chore(deps): refresh crates and adopt released aws-sigv4 (#9047)

* chore(deps): refresh crates and adopt released aws-sigv4

Run two update/upgrade rounds for the main and standalone fuzz workspaces.
Replace the temporary SigV4a Git override with aws-sigv4 1.6.4 from the
registry and remove its obsolete source permission.

Pin OpenID Connect to e20ddbb01ca68356f2979bc125f5613ab45b4e7e. Upgrade
async-nats to 0.51.0, hotpath to 0.30.0, libunftp to 0.24.0, and the TOML
test dependency to 1.1.8. Retain Arrow/Parquet 59.3.2 for DataFusion 55.2.0
and keep the fuzz PKCS#1 version compatible with pinned RSA rc.18.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(deps): retain safe openidconnect pin against HMAC panic

The e20ddbb pin verifies HMAC signatures with mac.verify(signature.into()),
which panics in generic-array on malformed length instead of returning
InvalidIdentityToken. Retain the previous reviewed pin that uses
verify_slice and add a malformed-signature regression covering HS256,
HS384, and HS512.

* chore(deps): upgrade sysinfo to 0.40.0

Update the main and standalone fuzz lockfiles. Adapt fallible system
queries, CPU usage accessors, and OS-native network interface names.
Preserve process sampling fallbacks and report unavailable diagnostic
sources without panicking. Cover missing host and process collectors.

Validation: 167 focused tests passed; production and all five fuzz targets
passed cargo check. Formatting, logging guardrails, and the existing
dependency policy passed. Linux and Windows runtime checks were not run.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): use the admin error facade for sysinfo failures

Keep the sysinfo initialization error behind the existing admin S3 facade.
The facade builds the same error code and message without expanding the
direct S3 error-macro surface guarded by the migration ratchet.

Validation: S3 footprint guard, formatting, whitespace, and production
cargo check with sftp, hotpath, and connect_top_net passed.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh hotpath and ctutils (#9055)

Run two cargo update/upgrade rounds in both the main and fuzz workspaces
on top of PR #9047. Upgrade the hotpath family to 0.30.1 and ctutils to
0.4.7. Retain fuzz pkcs1 0.8.0-rc.4 for the pinned RSA 0.10.0-rc.18;
rc.5 fails to compile against that RSA version.

Validation: 1096 library tests passed with 19 skipped; upstream ctutils
and hotpath-drain property/concurrency suites passed. Production and all
five fuzz targets passed cargo check. Formatting, S3 footprint, and the
existing dependency policy passed.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=a67a347f26be186c598b5beaf68e6bcbe1855350a35e1423dd5b8a81bcba06a5
-sha256-linux=ad4fabe811c6cca1aba89fad34ccd404b5832a6a955875c2826f6e0fa1ea4f1f
+sha256-darwin=67251cb7d9978d38e899268cac4d7898297ff7e5b5c48c9c1546dc760b93568d
+sha256-linux=673da76c2e5710c4622fe2fa32d2b297730895769487fcf1152ee1c2366f32f4

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=f065e7071aac9d48a7c1b301f328fa7cc14f11c4e7dc46da3e55057c4ea29bd5
+sha256=e9dc896774122a304e2b8b90f28c82d4d4e0af8ded36a82b147b30945ec994e3

.config/make/tests.mak

@@ -64,6 +64,8 @@ script-tests: ## Run shell script tests
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py

.config/nextest.toml

@@ -57,6 +57,11 @@ command = ['sh', '-c', 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"']
 [scripts.setup.ecstore-large-stack]
 command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
 
+# The receipt corruption oracle does not simulate a crash. Prefer tmpfs only
+# when its full fixture fits; retain the ordinary temporary directory otherwise.
+[scripts.setup.ecstore-receipt-tmpdir]
+command = ['sh', 'scripts/ci/receipt-fixture-tmpdir.sh']
+
 # The serial ILM selection builds the same deep storage futures in both the
 # lifecycle transition module and scanner integration binary. Different tests
 # in each have overflowed first across otherwise unrelated CI runs.
@@ -91,7 +96,7 @@ threads-required = "num-test-threads"
 # acquisition budget. Reserve capacity from sibling test processes while
 # preserving the probes' internal concurrency, deadlines, and assertions.
 [[profile.default.overrides]]
-filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
+filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
 threads-required = "num-test-threads"
 
 # These eight-disk decommission fixtures exercise concurrent retries,
@@ -128,7 +133,7 @@ setup = 'ecstore-large-stack'
 # path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need
 # 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI.
 [[profile.default.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
 setup = 'ecstore-base-stack'
 
 [[profile.default.scripts]]
@@ -145,10 +150,16 @@ setup = 'ecstore-base-stack'
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
 
-# The pool metadata placement regression composes the full erasure-heal
-# startup future and overflows the default Linux test-thread stack.
+# This conditional GET fixture composes real-store futures that overflowed
+# the default Linux test-thread stack.
 [[profile.default.scripts]]
-filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
+filter = 'package(rustfs) & test(=app::object::get::tests::conditional_get_304_uses_metadata_without_decoding_damaged_data)'
+setup = 'ecstore-large-stack'
+
+# These heal fixtures compose deep real-store recovery futures that can
+# overflow the default Linux test-thread stack.
+[[profile.default.scripts]]
+filter = 'package(rustfs-heal) & (test(=replacement_pool_metadata_follows_real_two_set_placement) | (binary(heal_integration_test) & test(=bucket_removal_recovers_delete_proofs_before_removing_empty_volumes)))'
 setup = 'ecstore-large-stack'
 
 # These real-disk scanner fixtures assert quiet-workload publication and
@@ -385,7 +396,7 @@ threads-required = "num-test-threads"
 # acquisition budget. Reserve capacity from sibling test processes while
 # preserving the probes' internal concurrency, deadlines, and assertions.
 [[profile.ci.overrides]]
-filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
+filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
 threads-required = "num-test-threads"
 
 # These eight-disk decommission fixtures exercise concurrent retries,
@@ -413,13 +424,22 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
 threads-required = "num-test-threads"
 
+# Emit this expensive fixture's stage timings in CI and use tmpfs when safe.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
+success-output = 'immediate'
+
+[[profile.ci.scripts]]
+filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
+setup = 'ecstore-receipt-tmpdir'
+
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
 # These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
 [[profile.ci.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
 setup = 'ecstore-base-stack'
 
 [[profile.ci.scripts]]
@@ -434,10 +454,16 @@ setup = 'ecstore-base-stack'
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
 
-# The pool metadata placement regression composes the full erasure-heal
-# startup future and overflows the default Linux test-thread stack.
+# This conditional GET fixture composes real-store futures that overflowed
+# the default Linux test-thread stack.
 [[profile.ci.scripts]]
-filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
+filter = 'package(rustfs) & test(=app::object::get::tests::conditional_get_304_uses_metadata_without_decoding_damaged_data)'
+setup = 'ecstore-large-stack'
+
+# These heal fixtures compose deep real-store recovery futures that can
+# overflow the default Linux test-thread stack.
+[[profile.ci.scripts]]
+filter = 'package(rustfs-heal) & (test(=replacement_pool_metadata_follows_real_two_set_placement) | (binary(heal_integration_test) & test(=bucket_removal_recovers_delete_proofs_before_removing_empty_volumes)))'
 setup = 'ecstore-large-stack'
 
 # These real-disk scanner fixtures assert quiet-workload publication and

.github/workflows/README.md

@@ -16,7 +16,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
 | `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
 | `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
 | `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
-| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
+| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | Docker workloads with a measured resource or environment requirement beyond the hosted runner |
 | `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
 
 The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; `ci.yml` builds the debug binary on `sm-standard-4` and runs it on both pools.
@@ -27,11 +27,15 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
 
 1. **Disk.** It restores or produces the workspace `target/`: `cargo build` or `cargo test` with `--all-targets`, `cargo nextest run --all`, or any reader of the `ci-dev` / `ci-feat-*` caches. The workspace test build leaves about 64 GB in `target/`; a hosted runner has about 14 GB.
 2. **Critical-path CPU.** It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (`build-rustfs-debug-binary` gates the e2e and s3-tests lanes).
-3. **Hardware.** Native aarch64, Docker-in-Docker, or the lab machines.
+3. **Hardware or environment.** Native aarch64 or access to the lab machines. Docker alone is not a reason: hosted Ubuntu ships Docker and supports the QEMU and Buildx setup actions.
 4. **Cache ownership.** It reads a `rust-cache` key that a self-hosted job writes (see the cache rule below).
 
 Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
 
+The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection stays on the operator-managed `smoke-testing` runner as part of the automation environment. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
+
+Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
+
 A hosted runner is also the only choice for a few things the pods cannot do: `mount -o loop` and `mount -t tmpfs` fail inside the ARC pods, and a container runtime cannot guarantee that `io_uring_setup` is allowed. `uring-integration` and `e2e-distributed` are on `ubuntu-latest` for that reason, not for cost.
 
 ## Cache rule

.github/workflows/docker.yml

@@ -337,7 +337,7 @@ jobs:
     name: Build Docker Images
     needs: build-check
     if: needs.build-check.outputs.should_build == 'true'
-    runs-on: dind-sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 60
     strategy:
       fail-fast: false

.github/workflows/runner-hygiene.yml

@@ -44,7 +44,7 @@ jobs:
   check-ephemerality:
     if: github.repository == 'rustfs/rustfs'
     name: Check runner ephemerality
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
       - name: Checkout repository

.github/workflows/rustfs-functional-chain.yml

@@ -20,13 +20,91 @@ on:
   workflow_dispatch:
     inputs:
       build_run_id:
-        description: Successful nightly build run on main
+        description: Successful nightly or validation package run; empty for package-only build
         type: string
-        required: true
+        required: false
+        default: ''
       build_run_attempt:
         description: Exact successful build attempt
         type: string
-        required: true
+        required: false
+        default: ''
+      validation_package_sha:
+        description: Exact source SHA for an isolated package-only build
+        type: string
+        required: false
+        default: ''
+      validation_case:
+        description: Canonical case ID, environment.inspect for read-only resources, or empty for functional chain
+        type: string
+        required: false
+        default: ''
+      i05_diagnostic_branch:
+        description: I05 source-plan runs both H1/H2 controllers with exact native evidence; subsets remain diagnostic
+        type: choice
+        options: [source-plan, timeout, queued-successor]
+        required: false
+        default: source-plan
+      auto_testing_sha:
+        description: Exact auto-testing commit for a validation case
+        type: string
+        required: false
+        default: ''
+      validation_retain_evidence_run_id:
+        description: Upload existing persistent evidence from one interrupted validation run without fleet access
+        type: string
+        required: false
+        default: ''
+      validation_retain_evidence_run_attempt:
+        description: Exact interrupted run attempt to retain
+        type: string
+        required: false
+        default: '1'
+      validation_runner_inspect_only:
+        description: Inspect GitHub runner registration and queued jobs on a hosted runner without fleet access
+        type: boolean
+        required: false
+        default: false
+      validation_tools_only:
+        description: Build candidate-bound validation tools without running a fleet case
+        type: boolean
+        required: false
+        default: false
+      validation_native_tools:
+        description: Include exact-source native regressions in a tools-only build
+        type: boolean
+        required: false
+        default: false
+      validation_shared_tools:
+        description: Include the controlled C10 shared-reference encoder in a tools-only build
+        type: boolean
+        required: false
+        default: false
+      validation_manager_tools:
+        description: Include exact checkpoint/MRF/incarnation regressions in a tools-only build
+        type: boolean
+        required: false
+        default: false
+      validation_cleanup_run_id:
+        description: Failed validation run whose supported retained environment needs evidence-based finalization
+        type: string
+        required: false
+        default: ''
+      validation_cleanup_retained_evidence:
+        description: Recover from verified persistent runner-local evidence when the failed run uploaded no artifact
+        type: boolean
+        required: false
+        default: false
+      validation_tools_run_id:
+        description: Successful tool build run; required by Heal physical metadata checks
+        type: string
+        required: false
+        default: ''
+      validation_tools_run_attempt:
+        description: Exact tool build attempt
+        type: string
+        required: false
+        default: '1'
   workflow_run:
     workflows: ["Nightly GNU Build"]
     types: [completed]
@@ -36,13 +114,104 @@ permissions:
   actions: read
 
 concurrency:
-  group: rustfs-functional-chain-runs
+  group: ${{ github.event_name == 'workflow_dispatch' && inputs.validation_runner_inspect_only && 'rustfs-functional-validation-runner-inspection' || github.event_name == 'workflow_dispatch' && inputs.validation_package_sha != '' && format('rustfs-functional-validation-package-{0}', inputs.validation_package_sha) || github.event_name == 'workflow_dispatch' && inputs.validation_tools_only && 'rustfs-functional-validation-tools' || 'rustfs-functional-chain-runs' }}
   cancel-in-progress: false
 
 jobs:
-  prepare:
-    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
+  validation-runner-inspection:
+    if: ${{ github.event_name == 'workflow_dispatch' && inputs.validation_runner_inspect_only }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+      - name: Inspect shared runner registration and queue without fleet access
+        env:
+          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          RUNNER_INSPECTION_INPUTS: ${{ toJSON(inputs) }}
+        run: >-
+          python3 -B scripts/inspect_validation_runner.py
+          --output "${RUNNER_TEMP}/validation-runner-inspection/report.json"
+      - name: Upload read-only runner inspection including incomplete reports
+        if: ${{ always() }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: validation-runner-inspection-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ runner.temp }}/validation-runner-inspection/report.json
+          if-no-files-found: error
+  validation-evidence-retention:
+    if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id != '' }}
+    # The original interrupted evidence exists only on this long-lived runner.
     runs-on: smoke-testing
+    timeout-minutes: 15
+    concurrency:
+      group: rustfs-shared-functional-tests-v2
+      cancel-in-progress: false
+    steps:
+      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+      - name: Verify existing interrupted evidence without changing it
+        env:
+          GH_TOKEN: ${{ github.token }}
+          EVIDENCE_RETAIN_INPUTS: ${{ toJSON(inputs) }}
+        run: >-
+          python3 -B scripts/retain_validation_evidence.py
+          --report "${RUNNER_TEMP}/validation-evidence-retention/report.json"
+      - name: Upload admitted original evidence under collection provenance
+        if: ${{ env.VALIDATION_RETAIN_ROOT != '' }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rustfs-validation-retained-${{ env.VALIDATION_RETAIN_SOURCE_RUN }}-${{ env.VALIDATION_RETAIN_SOURCE_ATTEMPT }}-by-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ env.VALIDATION_RETAIN_ROOT }}
+          if-no-files-found: error
+          include-hidden-files: true
+      - name: Upload retention receipt including incomplete reports
+        if: ${{ always() }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: validation-evidence-retention-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ runner.temp }}/validation-evidence-retention/report.json
+          if-no-files-found: error
+  validation-package-request:
+    if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha != '' }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Validate exclusive package-only request
+        env:
+          PACKAGE_INPUTS: ${{ toJSON(inputs) }}
+        run: |
+          python3 - <<'PY'
+          import json
+          import os
+          import re
+          value = json.loads(os.environ["PACKAGE_INPUTS"])
+          if re.fullmatch(r"[0-9a-f]{40}", value["validation_package_sha"]) is None:
+              raise ValueError("package build requires an exact source SHA")
+          exclusive = ("validation_retain_evidence_run_id", "validation_runner_inspect_only", "build_run_id", "build_run_attempt", "validation_case", "auto_testing_sha",
+                       "validation_tools_only", "validation_native_tools", "validation_shared_tools",
+                       "validation_manager_tools", "validation_cleanup_run_id", "validation_cleanup_retained_evidence", "validation_tools_run_id")
+          if any(value.get(name) for name in exclusive):
+              raise ValueError("package-only build cannot select existing candidate, tools, fleet or recovery execution")
+          PY
+
+  validation-package:
+    needs: validation-package-request
+    if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha != '' }}
+    uses: ./.github/workflows/rustfs-validation-package.yml
+    with:
+      source_sha: ${{ inputs.validation_package_sha }}
+    secrets:
+      R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+      R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+      R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
+      R2_BUCKET: ${{ secrets.R2_BUCKET }}
+
+  prepare:
+    if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha == '' && inputs.validation_case == '' && !inputs.validation_tools_only && !inputs.validation_native_tools && !inputs.validation_shared_tools && !inputs.validation_manager_tools && !inputs.validation_cleanup_retained_evidence && inputs.validation_cleanup_run_id == '') || github.event.workflow_run.event == 'schedule' }}
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     outputs:
       manifest: ${{ steps.candidate.outputs.manifest }}
@@ -89,6 +258,411 @@ jobs:
             echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
           fi
 
+  validation:
+    if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha == '' && (inputs.validation_case != '' || inputs.validation_tools_only || inputs.validation_native_tools || inputs.validation_shared_tools || inputs.validation_manager_tools || inputs.validation_cleanup_run_id != '' || inputs.validation_cleanup_retained_evidence) }}
+    runs-on: ${{ inputs.validation_tools_only && 'ubuntu-latest' || 'smoke-testing' }}
+    timeout-minutes: 90
+    outputs:
+      source_sha: ${{ steps.candidate.outputs.source_sha }}
+    concurrency:
+      group: ${{ inputs.validation_tools_only && 'rustfs-validation-tools-resolver' || 'rustfs-shared-functional-tests-v2' }}
+      cancel-in-progress: false
+    env:
+      RUSTFS_ACCESS_KEY: ${{ !inputs.validation_tools_only && secrets.RUSTFS_ACCESS_KEY || '' }}
+      RUSTFS_SECRET_KEY: ${{ !inputs.validation_tools_only && secrets.RUSTFS_SECRET_KEY || '' }}
+      RUSTFS_API_ENDPOINT: ${{ !inputs.validation_tools_only && (secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT) || '' }}
+      RUSTFS_NODES: ${{ !inputs.validation_tools_only && (secrets.RUSTFS_NODES || vars.RUSTFS_NODES) || '' }}
+      RUSTFS_SSH_USER: ${{ !inputs.validation_tools_only && (secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER) || '' }}
+    steps:
+      - name: Checkout candidate resolver
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Initialize validation evidence
+        env:
+          VALIDATION_TOOLS_ONLY: ${{ inputs.validation_tools_only }}
+        run: |
+          set -euo pipefail
+          umask 077
+          python3 -B scripts/initialize_validation_evidence.py
+
+      - name: Resolve immutable candidate
+        id: candidate
+        env:
+          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          BUILD_RUN_ID: ${{ inputs.build_run_id }}
+          BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
+          AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+          CLEANUP_RUN_ID: ${{ inputs.validation_cleanup_run_id }}
+          CLEANUP_RETAINED_EVIDENCE: ${{ inputs.validation_cleanup_retained_evidence }}
+          SELECTED_CASE: ${{ inputs.validation_case }}
+          TOOLS_ONLY: ${{ inputs.validation_tools_only }}
+          NATIVE_TOOLS: ${{ inputs.validation_native_tools }}
+          SHARED_TOOLS: ${{ inputs.validation_shared_tools }}
+          MANAGER_TOOLS: ${{ inputs.validation_manager_tools }}
+          I05_DIAGNOSTIC_BRANCH: ${{ inputs.i05_diagnostic_branch }}
+        run: |
+          set -euo pipefail
+          python3 - <<'PY'
+          import json
+          import os
+          from pathlib import Path
+          import re
+          from scripts.resolve_functional_candidate import resolve
+
+          revision = os.environ["AUTO_TESTING_SHA"]
+          cleanup = os.environ["CLEANUP_RUN_ID"]
+          if os.environ["CLEANUP_RETAINED_EVIDENCE"] == "true" and not cleanup:
+              raise ValueError("persistent recovery input requires one exact cleanup run")
+          if os.environ["TOOLS_ONLY"] == "true" and (cleanup or os.environ["SELECTED_CASE"]):
+              raise ValueError("tools-only execution cannot select a fleet case or cleanup")
+          if os.environ["NATIVE_TOOLS"] == "true" and (os.environ["TOOLS_ONLY"] != "true" or cleanup or os.environ["SELECTED_CASE"]):
+              raise ValueError("native regressions require tools-only execution, without a fleet case or cleanup")
+          if os.environ["SHARED_TOOLS"] == "true" and (os.environ["TOOLS_ONLY"] != "true" or cleanup or os.environ["SELECTED_CASE"]):
+              raise ValueError("shared fixture builds require tools-only execution, without a fleet case or cleanup")
+          if os.environ["MANAGER_TOOLS"] == "true" and (os.environ["TOOLS_ONLY"] != "true" or cleanup or os.environ["SELECTED_CASE"]):
+              raise ValueError("manager regressions require tools-only execution, without a fleet case or cleanup")
+          if cleanup and (not cleanup.isdigit() or int(cleanup) <= 0 or os.environ["SELECTED_CASE"] or os.environ["TOOLS_ONLY"] == "true"):
+              raise ValueError("cleanup requires one exact failed run and is exclusive with case/tool execution")
+          diagnostic_branch = os.environ["I05_DIAGNOSTIC_BRANCH"] or "source-plan"
+          if diagnostic_branch not in {"source-plan", "timeout", "queued-successor"}:
+              raise ValueError("I05 diagnostic branch is not registered")
+          if diagnostic_branch == "queued-successor" and os.environ["SELECTED_CASE"] != "heal.lock-timeout-and-caller-abort":
+              raise ValueError("queued-successor is available only for the I05 diagnostic case")
+          if re.fullmatch(r"[0-9a-f]{40}", revision) is None:
+              raise ValueError("validation requires an exact auto-testing commit SHA")
+          candidate = resolve(int(os.environ["BUILD_RUN_ID"]), int(os.environ["BUILD_RUN_ATTEMPT"]), allow_validation=True)
+          manifest = candidate["manifest"]
+          root = Path(os.environ["VALIDATION_ROOT"])
+          (root / "candidate-source.json").write_text(json.dumps(candidate, sort_keys=True) + "\n")
+          (root / "candidate.json").write_text(json.dumps({
+              "package_url": manifest["package_url"],
+              "package_sha256": manifest["package_sha256"],
+              "source_sha": manifest["source_sha"],
+          }, sort_keys=True) + "\n")
+          with open(os.environ["GITHUB_OUTPUT"], "a") as output:
+              output.write("source_sha=" + manifest["source_sha"] + "\n")
+          PY
+
+      - name: Checkout exact auto-testing revision
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          repository: rustfs/auto-testing
+          ref: ${{ inputs.auto_testing_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          path: auto-testing
+          persist-credentials: false
+
+      - name: Prepare validation Python runtime
+        if: ${{ !inputs.validation_tools_only && (startsWith(inputs.validation_case, 'ilm.') || startsWith(inputs.validation_case, 'pool.') || startsWith(inputs.validation_case, 'bucket-replication.') || startsWith(inputs.validation_case, 'site-replication.') || startsWith(inputs.validation_case, 'heal.')) }}
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+
+      - name: Install validation Python client
+        if: ${{ !inputs.validation_tools_only && (startsWith(inputs.validation_case, 'ilm.') || startsWith(inputs.validation_case, 'pool.') || startsWith(inputs.validation_case, 'bucket-replication.') || startsWith(inputs.validation_case, 'site-replication.') || startsWith(inputs.validation_case, 'heal.')) }}
+        run: |
+          set -euo pipefail
+          validation_python="${RUNNER_TEMP}/rustfs-validation-python-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          test ! -e "${validation_python}"
+          python3 -m venv "${validation_python}"
+          "${validation_python}/bin/python" -m pip install \
+            -r auto-testing/scripts/validation_ilm/requirements.txt
+          printf '%s\n' "${validation_python}/bin" >> "${GITHUB_PATH}"
+
+      - name: Validate Heal tool build identity
+        if: ${{ !inputs.validation_tools_only && startsWith(inputs.validation_case, 'heal.') }}
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TOOL_RUN_ID: ${{ inputs.validation_tools_run_id }}
+          TOOL_RUN_ATTEMPT: ${{ inputs.validation_tools_run_attempt }}
+        run: |
+          set -euo pipefail
+          [[ "${TOOL_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
+          [[ "${TOOL_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
+          gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${TOOL_RUN_ID}/attempts/${TOOL_RUN_ATTEMPT}" \
+            > "${VALIDATION_ROOT}/tool-run.json"
+          python3 - <<'PY'
+          import json
+          import os
+          from pathlib import Path
+          value = json.loads((Path(os.environ["VALIDATION_ROOT"]) / "tool-run.json").read_text())
+          if (value.get("conclusion") != "success" or value.get("status") != "completed"
+                  or value.get("run_attempt") != int(os.environ["TOOL_RUN_ATTEMPT"])
+                  or value.get("path") not in {".github/workflows/rustfs-functional-chain.yml", ".github/workflows/rustfs-validation-tools.yml"}):
+              raise ValueError("Heal tool run is not an exact successful validation build attempt")
+          PY
+
+      - name: Download pinned Heal metadata inspector
+        if: ${{ !inputs.validation_tools_only && startsWith(inputs.validation_case, 'heal.') }}
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          github-token: ${{ github.token }}
+          run-id: ${{ inputs.validation_tools_run_id }}
+          name: rustfs-validation-tools-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
+          path: ${{ runner.temp }}/rustfs-validation-tools-${{ github.run_id }}-${{ github.run_attempt }}
+
+      - name: Verify Heal metadata inspector
+        if: ${{ !inputs.validation_tools_only && startsWith(inputs.validation_case, 'heal.') }}
+        run: |
+          set -euo pipefail
+          TOOL_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-tools-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          python3 -B auto-testing/scripts/validation_heal/inspector.py \
+            --directory "${TOOL_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+            > "${VALIDATION_ROOT}/tool-manifest.json"
+          printf 'VALIDATION_HEAL_TOOL_DIRECTORY=%s\n' "${TOOL_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Select exact-source race native evidence
+        id: race_native
+        if: ${{ inputs.validation_case == 'heal.race-unversioned-overwrite' || inputs.validation_case == 'heal.race-versioned-put' || inputs.validation_case == 'heal.race-version-delete' || inputs.validation_case == 'heal.race-markers-and-batch-delete' || inputs.validation_case == 'heal.race-multipart-complete-abort' || inputs.validation_case == 'heal.admin-nolock-safety' || inputs.validation_case == 'heal.race-copy-and-metadata-update' }}
+        env:
+          CANDIDATE_SHA: ${{ steps.candidate.outputs.source_sha }}
+          VALIDATION_CASE_ID: ${{ inputs.validation_case }}
+        run: |
+          set -euo pipefail
+          required=false
+          if [ -f auto-testing/scripts/validation_heal/race_native.py ]; then
+            status=0
+            python3 -B auto-testing/scripts/validation_heal/race_native.py supports \
+              --commit "${CANDIDATE_SHA}" --case-id "${VALIDATION_CASE_ID}" || status=$?
+            if [ "${status}" -eq 0 ]; then
+              required=true
+            elif [ "${status}" -ne 2 ]; then
+              exit "${status}"
+            fi
+          fi
+          reviewed=false
+          case "${CANDIDATE_SHA}:${VALIDATION_CASE_ID}" in
+            6afdec9fe1b0b88007f2e5b445c19bf73d51d3ca:heal.race-unversioned-overwrite | \
+            bd0671ba781d8ebf509f4f03533228dfb44496f1:heal.race-unversioned-overwrite | \
+            bd0671ba781d8ebf509f4f03533228dfb44496f1:heal.race-versioned-put | \
+            bd0671ba781d8ebf509f4f03533228dfb44496f1:heal.race-version-delete | \
+            8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-unversioned-overwrite | \
+            8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-versioned-put | \
+            8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-version-delete | \
+            8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-markers-and-batch-delete | \
+            757b751482bb970bc321e92059690a7698554358:heal.admin-nolock-safety | \
+            757b751482bb970bc321e92059690a7698554358:heal.race-multipart-complete-abort | \
+            757b751482bb970bc321e92059690a7698554358:heal.race-copy-and-metadata-update) reviewed=true ;;
+          esac
+          if [ "${reviewed}" = true ] && [ "${required}" != true ]; then
+            printf 'Race candidate requires its reviewed native collector before fleet execution\n' >&2
+            exit 1
+          fi
+          printf 'required=%s\n' "${required}" >> "${GITHUB_OUTPUT}"
+
+      - name: Download pinned native storage evidence
+        if: ${{ inputs.validation_case == 'heal.conflicting-subquorum-authority' || inputs.validation_case == 'heal.bucket-incarnation-recreation' || steps.race_native.outputs.required == 'true' || (inputs.validation_case == 'heal.lock-timeout-and-caller-abort' && inputs.i05_diagnostic_branch != 'timeout') || (inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' && steps.candidate.outputs.source_sha == '1aa9be87d18cf941e5020f19e8622c26ba4b10ac') }}
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          github-token: ${{ github.token }}
+          run-id: ${{ inputs.validation_tools_run_id }}
+          name: rustfs-validation-native-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
+          path: ${{ runner.temp }}/rustfs-validation-native-${{ github.run_id }}-${{ github.run_attempt }}
+
+      - name: Verify C07 native source and exact test evidence
+        if: ${{ inputs.validation_case == 'heal.conflicting-subquorum-authority' }}
+        run: |
+          set -euo pipefail
+          NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
+          python3 -B auto-testing/scripts/validation_heal/authority.py \
+            --directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+            > "${VALIDATION_ROOT}/native-authority-manifest.json"
+          printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Verify C11 physical storage and exact final-rename barrier
+        if: ${{ inputs.validation_case == 'heal.bucket-incarnation-recreation' }}
+        run: |
+          set -euo pipefail
+          NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
+          python3 -B auto-testing/scripts/validation_heal/incarnation_native.py \
+            --directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+            --testing-commit "${{ inputs.auto_testing_sha }}" \
+            > "${VALIDATION_ROOT}/native-incarnation-manifest.json"
+          printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Verify exact-source race native and runtime evidence
+        if: ${{ steps.race_native.outputs.required == 'true' }}
+        env:
+          AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+          VALIDATION_CASE_ID: ${{ inputs.validation_case }}
+        run: |
+          set -euo pipefail
+          NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
+          python3 -B auto-testing/scripts/validation_heal/race_native_evidence.py \
+            --directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+            --testing-commit "${AUTO_TESTING_SHA}" --case-id "${VALIDATION_CASE_ID}" \
+            > "${VALIDATION_ROOT}/native-race-manifest.json"
+          runtime_driver=race.py
+          if [ "${VALIDATION_CASE_ID}" = 'heal.admin-nolock-safety' ]; then
+            runtime_driver=nolock.py
+          fi
+          VALIDATION_HEAL_NATIVE_DIRECTORY="${NATIVE_DIRECTORY}" \
+            python3 -B "auto-testing/scripts/validation_heal/${runtime_driver}" preflight \
+              --candidate "${VALIDATION_ROOT}/candidate.json" --testing-commit "${AUTO_TESTING_SHA}" \
+              --case-id "${VALIDATION_CASE_ID}" \
+              > "${VALIDATION_ROOT}/native-race-runtime-preflight.json"
+          printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Verify I05 source-plan or queued-successor native prerequisites
+        if: ${{ inputs.validation_case == 'heal.lock-timeout-and-caller-abort' && inputs.i05_diagnostic_branch != 'timeout' }}
+        env:
+          AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+          I05_DIAGNOSTIC_BRANCH: ${{ inputs.i05_diagnostic_branch }}
+        run: |
+          set -euo pipefail
+          NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
+          if [ "${I05_DIAGNOSTIC_BRANCH}" = 'queued-successor' ]; then
+            python3 -B auto-testing/scripts/validation_heal/race_native_evidence.py \
+              --directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+              --testing-commit "${AUTO_TESTING_SHA}" --case-id heal.lock-timeout-and-caller-abort \
+              > "${VALIDATION_ROOT}/native-i05-prerequisite.json"
+          else
+            python3 -B auto-testing/scripts/validation_heal/lock_source.py \
+              --directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+              --testing-commit "${AUTO_TESTING_SHA}" \
+              > "${VALIDATION_ROOT}/native-i05-source-prerequisites.json"
+          fi
+          printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Bind F07 exact-candidate directory fsync evidence
+        if: ${{ inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' && steps.candidate.outputs.source_sha == '1aa9be87d18cf941e5020f19e8622c26ba4b10ac' }}
+        run: |
+          set -euo pipefail
+          NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
+          printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Download pinned manager persistence and incarnation evidence
+        if: ${{ inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' || inputs.validation_case == 'heal.bucket-incarnation-recreation' || inputs.validation_case == 'heal.outcome-counters-and-error-truth' }}
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          github-token: ${{ github.token }}
+          run-id: ${{ inputs.validation_tools_run_id }}
+          name: rustfs-validation-manager-native-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
+          path: ${{ runner.temp }}/rustfs-validation-manager-native-${{ github.run_id }}-${{ github.run_attempt }}
+
+      - name: Verify exact manager native prerequisites before fleet mutation
+        if: ${{ inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' || inputs.validation_case == 'heal.bucket-incarnation-recreation' || inputs.validation_case == 'heal.outcome-counters-and-error-truth' }}
+        env:
+          VALIDATION_CASE: ${{ inputs.validation_case }}
+        run: |
+          set -euo pipefail
+          MANAGER_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-manager-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${MANAGER_DIRECTORY}/tool-run.json"
+          python3 -B auto-testing/scripts/validation_heal/manager_fixture.py \
+            --directory "${MANAGER_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+            --case-id "${VALIDATION_CASE}" > "${VALIDATION_ROOT}/manager-prerequisite-manifest.json"
+          printf 'VALIDATION_HEAL_MANAGER_DIRECTORY=%s\n' "${MANAGER_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Download pinned C10 shared-reference encoder
+        if: ${{ inputs.validation_case == 'heal.dangling-and-orphan-safety' }}
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          github-token: ${{ github.token }}
+          run-id: ${{ inputs.validation_tools_run_id }}
+          name: rustfs-validation-shared-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
+          path: ${{ runner.temp }}/rustfs-validation-shared-${{ github.run_id }}-${{ github.run_attempt }}
+
+      - name: Verify C10 exact-source codec and original tests
+        if: ${{ inputs.validation_case == 'heal.dangling-and-orphan-safety' }}
+        run: |
+          set -euo pipefail
+          SHARED_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-shared-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          cp "${VALIDATION_ROOT}/tool-run.json" "${SHARED_DIRECTORY}/tool-run.json"
+          python3 -B auto-testing/scripts/validation_heal/shared_fixture.py \
+            --directory "${SHARED_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
+            > "${VALIDATION_ROOT}/shared-fixture-manifest.json"
+          printf 'VALIDATION_HEAL_SHARED_DIRECTORY=%s\n' "${SHARED_DIRECTORY}" >> "${GITHUB_ENV}"
+
+      - name: Run selected canonical validation case
+        if: ${{ !inputs.validation_tools_only && inputs.validation_cleanup_run_id == '' && inputs.validation_case != 'environment.inspect' }}
+        env:
+          VALIDATION_CASE: ${{ inputs.validation_case }}
+          AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+          I05_DIAGNOSTIC_BRANCH: ${{ inputs.i05_diagnostic_branch }}
+        run: |
+          set -euo pipefail
+          test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
+          ./auto-testing/scripts/run_validation_case.sh \
+            "${VALIDATION_CASE}" "${VALIDATION_ROOT}/candidate.json" \
+            "${AUTO_TESTING_SHA}" "${VALIDATION_ROOT}/run"
+
+      - name: Inspect current Heal fleet resources without mutation
+        if: ${{ !inputs.validation_tools_only && inputs.validation_case == 'environment.inspect' }}
+        env:
+          AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+        run: |
+          set -euo pipefail
+          test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
+          python3 -B auto-testing/scripts/validation_heal/environment.py \
+            --candidate "${VALIDATION_ROOT}/candidate.json" \
+            --testing-commit "${AUTO_TESTING_SHA}" --evidence "${VALIDATION_ROOT}/environment"
+
+      - name: Audit and finalize retained validation environment
+        if: ${{ inputs.validation_cleanup_run_id != '' }}
+        env:
+          GH_TOKEN: ${{ github.token }}
+          FAILED_RUN_ID: ${{ inputs.validation_cleanup_run_id }}
+          AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+          CLEANUP_RETAINED_EVIDENCE: ${{ inputs.validation_cleanup_retained_evidence }}
+        run: |
+          set -euo pipefail
+          test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
+          gh run view "${FAILED_RUN_ID}" --repo "${GITHUB_REPOSITORY}" \
+            --json status,conclusion,headBranch,workflowName > "${VALIDATION_ROOT}/failed-run.json"
+          python3 - <<'PY'
+          import json
+          import os
+          from pathlib import Path
+          run = json.loads((Path(os.environ["VALIDATION_ROOT"]) / "failed-run.json").read_text())
+          if (run["status"] != "completed" or run["conclusion"] not in {"failure", "cancelled"}
+              or run["workflowName"] != "RustFS Functional Chain" or run["headBranch"] != os.environ["GITHUB_REF_NAME"]):
+              raise ValueError("cleanup source must be a completed failed or cancelled Functional Chain run")
+          PY
+          if [ "${CLEANUP_RETAINED_EVIDENCE}" = true ]; then
+            python3 -B scripts/prepare_retained_validation_recovery.py \
+              --report "${VALIDATION_ROOT}/retained-recovery-input.json"
+          else
+            gh run download "${FAILED_RUN_ID}" --repo "${GITHUB_REPOSITORY}" \
+              --pattern "rustfs-validation-${FAILED_RUN_ID}-*" --dir "${VALIDATION_ROOT}/failed-artifact"
+          fi
+          python3 -B auto-testing/scripts/validation_heal/recover_environment.py \
+            --failed-run-id "${FAILED_RUN_ID}" --artifact "${VALIDATION_ROOT}/failed-artifact" \
+            --candidate "${VALIDATION_ROOT}/candidate.json" --evidence "${VALIDATION_ROOT}/recovery"
+
+      - name: Show validation report
+        if: ${{ always() }}
+        run: |
+          if [ -f "${VALIDATION_ROOT}/run/report.md" ]; then
+            cat "${VALIDATION_ROOT}/run/report.md" >> "${GITHUB_STEP_SUMMARY}"
+          fi
+
+      - name: Upload validation evidence
+        if: ${{ always() && env.VALIDATION_ROOT != '' }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rustfs-validation-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ env.VALIDATION_ROOT }}
+          if-no-files-found: error
+
+  validation-tools:
+    needs: [validation]
+    if: ${{ github.event_name == 'workflow_dispatch' && inputs.validation_package_sha == '' && inputs.validation_tools_only }}
+    uses: ./.github/workflows/rustfs-validation-tools.yml
+    with:
+      candidate_sha: ${{ needs.validation.outputs.source_sha }}
+      auto_testing_sha: ${{ inputs.auto_testing_sha }}
+      build_native_heal: ${{ inputs.validation_native_tools }}
+      build_shared_heal: ${{ inputs.validation_shared_tools }}
+      build_manager_native: ${{ inputs.validation_manager_tools }}
+    secrets: inherit
+
   upgrade:
     needs: [prepare]
     if: ${{ always() && needs.prepare.result == 'success' }}
@@ -188,7 +762,7 @@ jobs:
   complete-chain:
     needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
     if: ${{ always() && needs.prepare.result != 'skipped' }}
-    runs-on: smoke-testing
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/rustfs-performance-test.yml

@@ -90,7 +90,7 @@ env:
 jobs:
   runner-check:
     if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/rustfs-validation-package.yml

@@ -0,0 +1,244 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+
+name: RustFS Validation Package
+
+on:
+  workflow_call:
+    inputs:
+      source_sha:
+        description: Exact RustFS commit SHA to build
+        required: true
+        type: string
+    secrets:
+      R2_ACCESS_KEY_ID:
+        required: true
+      R2_SECRET_ACCESS_KEY:
+        required: true
+      R2_ENDPOINT:
+        required: true
+      R2_BUCKET:
+        required: true
+  workflow_dispatch:
+    inputs:
+      source_sha:
+        description: Exact RustFS commit SHA to build
+        required: true
+        type: string
+
+permissions:
+  contents: read
+
+concurrency:
+  group: rustfs-validation-package-${{ inputs.source_sha }}
+  cancel-in-progress: false
+
+jobs:
+  resolve-source:
+    if: github.repository == 'rustfs/rustfs'
+    name: Resolve exact validation source
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    outputs:
+      source_sha: ${{ steps.source.outputs.sha }}
+      source_ref: ${{ steps.source.outputs.ref }}
+    steps:
+      - name: Validate requested source
+        id: source
+        env:
+          REQUESTED_SHA: ${{ inputs.source_sha }}
+        run: |
+          set -euo pipefail
+          [[ "${REQUESTED_SHA}" =~ ^[0-9a-f]{40}$ ]]
+          source_directory="$(mktemp -d "${RUNNER_TEMP}/validation-source.XXXXXX")"
+          trap 'rm -rf "${source_directory}"' EXIT
+          git init -q "${source_directory}"
+          git -C "${source_directory}" fetch --no-tags --depth=1 https://github.com/rustfs/rustfs.git "${REQUESTED_SHA}"
+          test "$(git -C "${source_directory}" rev-parse FETCH_HEAD)" = "${REQUESTED_SHA}"
+          echo "sha=${REQUESTED_SHA}" >> "${GITHUB_OUTPUT}"
+          echo "ref=${REQUESTED_SHA}" >> "${GITHUB_OUTPUT}"
+
+  build:
+    needs: resolve-source
+    name: Build immutable validation DEB
+    runs-on: ubuntu-latest
+    timeout-minutes: 150
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      VALIDATION_SOURCE_SHA: ${{ needs.resolve-source.outputs.source_sha }}
+    steps:
+      - name: Checkout exact source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+          ref: ${{ needs.resolve-source.outputs.source_sha }}
+
+      - name: Confirm checked out source
+        env:
+          EXPECTED_SHA: ${{ needs.resolve-source.outputs.source_sha }}
+        run: test "$(git rev-parse HEAD)" = "${EXPECTED_SHA}"
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          cache-shared-key: validation-package-hosted-x86_64-unknown-linux-gnu
+          cache-save-if: "false"
+          install-build-packaging-tools: "false"
+          install-test-tools: "false"
+
+      - name: Build RustFS
+        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
+
+      - name: Build validation DEB
+        shell: bash
+        run: |
+          set -euo pipefail
+          command -v fakeroot >/dev/null 2>&1 || sudo apt-get install -y -qq fakeroot
+          package_dir="rustfs-validation-${VALIDATION_SOURCE_SHA}"
+          package_file="${RUNNER_TEMP}/validation-package/rustfs.deb"
+          mkdir -p "${RUNNER_TEMP}/validation-package"
+          binary="target/x86_64-unknown-linux-gnu/release/rustfs"
+          test -x "${binary}"
+          mkdir -p "${package_dir}/DEBIAN" "${package_dir}/usr/bin" "${package_dir}/etc/default" \
+            "${package_dir}/lib/systemd/system" "${package_dir}/usr/share/doc/rustfs"
+          cp "${binary}" "${package_dir}/usr/bin/rustfs"
+          chmod 755 "${package_dir}/usr/bin/rustfs"
+          cp deploy/build/rustfs.service "${package_dir}/lib/systemd/system/"
+          cat > "${package_dir}/etc/default/rustfs" <<'ENVEOF'
+          # RustFS Environment Configuration
+          # RUSTFS_ACCESS_KEY=<changeme>
+          # RUSTFS_SECRET_KEY=<changeme>
+          # RUSTFS_VOLUMES=""
+          # RUSTFS_ADDRESS=":9000"
+          ENVEOF
+          cat > "${package_dir}/DEBIAN/control" <<'CONTROLEOF'
+          Package: rustfs
+          Version: 0.0.0~validation
+          Section: utils
+          Priority: optional
+          Architecture: amd64
+          Depends: libc6 (>= 2.31)
+          Maintainer: RustFS Team <support@rustfs.com>
+          Description: RustFS immutable validation package
+           Candidate package for RustFS validation only.
+          CONTROLEOF
+          printf '%s\n' /etc/default/rustfs > "${package_dir}/DEBIAN/conffiles"
+          ./scripts/release/package_service_scripts.sh after-install > "${package_dir}/DEBIAN/postinst"
+          ./scripts/release/package_service_scripts.sh before-remove > "${package_dir}/DEBIAN/prerm"
+          ./scripts/release/package_service_scripts.sh after-remove > "${package_dir}/DEBIAN/postrm"
+          chmod 755 "${package_dir}/DEBIAN/postinst" "${package_dir}/DEBIAN/prerm" "${package_dir}/DEBIAN/postrm"
+          cp LICENSE README.md "${package_dir}/usr/share/doc/rustfs/"
+          fakeroot dpkg-deb --build "${package_dir}" "${package_file}"
+          test -s "${package_file}"
+
+      - name: Upload opaque validation package
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: validation-package-bytes-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ runner.temp }}/validation-package/rustfs.deb
+          if-no-files-found: error
+
+  publish:
+    needs: [resolve-source, build]
+    name: Publish immutable validation DEB
+    # Candidate-controlled build scripts never run on this fresh hosted runner.
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    env:
+      VALIDATION_SOURCE_SHA: ${{ needs.resolve-source.outputs.source_sha }}
+      VALIDATION_SOURCE_REF: ${{ needs.resolve-source.outputs.source_ref }}
+    steps:
+      - name: Download opaque validation package
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: validation-package-bytes-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ runner.temp }}/validation-package-input
+
+      - name: Validate and stage opaque package bytes
+        shell: bash
+        run: |
+          set -euo pipefail
+          python3 -I - <<'PYTHON'
+          import os
+          from pathlib import Path
+          import shutil
+          import stat
+
+          root = Path(os.environ["RUNNER_TEMP"])
+          incoming = root / "validation-package-input"
+          if incoming.is_symlink() or not incoming.is_dir() or sorted(p.name for p in incoming.iterdir()) != ["rustfs.deb"]:
+              raise SystemExit("Validation artifact must contain exactly one rustfs.deb file")
+          fd = os.open(incoming / "rustfs.deb", os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
+          with os.fdopen(fd, "rb") as source:
+              info = os.fstat(source.fileno())
+              if not stat.S_ISREG(info.st_mode) or info.st_size == 0:
+                  raise SystemExit("Validation package must be a nonempty regular file")
+              staged = root / "validation-publication"
+              staged.mkdir(mode=0o700)
+              with (staged / "rustfs.deb").open("xb") as target:
+                  shutil.copyfileobj(source, target)
+          PYTHON
+
+      - name: Prepare trusted publication tools
+        id: tools
+        env:
+          AWS_EC2_METADATA_DISABLED: true
+        shell: bash
+        run: |
+          set -euo pipefail
+          AWS_CLI=aws
+          if ! "${AWS_CLI}" s3api put-object --generate-cli-skeleton input | jq -e 'has("IfNoneMatch")' >/dev/null; then
+            sudo apt-get update
+            sudo apt-get install -y -qq python3-venv
+            AWS_CLI_DIR="$(mktemp -d "${RUNNER_TEMP}/validation-awscli.XXXXXX")"
+            python3 -m venv "${AWS_CLI_DIR}"
+            "${AWS_CLI_DIR}/bin/python" -m pip install --disable-pip-version-check 'awscli==1.44.79'
+            AWS_CLI="${AWS_CLI_DIR}/bin/aws"
+          fi
+          "${AWS_CLI}" s3api put-object --generate-cli-skeleton input | jq -e 'has("IfNoneMatch")' >/dev/null
+          echo "aws_cli=${AWS_CLI}" >> "${GITHUB_OUTPUT}"
+
+      - name: Upload validation package to immutable R2 path
+        id: publish
+        env:
+          PACKAGE_FILE: ${{ runner.temp }}/validation-publication/rustfs.deb
+          AWS_CLI: ${{ steps.tools.outputs.aws_cli }}
+          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
+          R2_BUCKET: ${{ secrets.R2_BUCKET }}
+          AWS_EC2_METADATA_DISABLED: true
+        shell: bash
+        run: |
+          set -euo pipefail
+          if [[ -z "${R2_ACCESS_KEY_ID}" || -z "${R2_SECRET_ACCESS_KEY}" || -z "${R2_ENDPOINT}" || -z "${R2_BUCKET}" ]]; then
+            echo "Validation package publication requires complete R2 configuration" >&2
+            exit 1
+          fi
+          [[ "${VALIDATION_SOURCE_SHA}" =~ ^[0-9a-f]{40}$ ]]
+          test "${VALIDATION_SOURCE_REF}" = "${VALIDATION_SOURCE_SHA}"
+          test -s "${PACKAGE_FILE}" && test ! -L "${PACKAGE_FILE}"
+          export AWS_ACCESS_KEY_ID="${R2_ACCESS_KEY_ID}" AWS_SECRET_ACCESS_KEY="${R2_SECRET_ACCESS_KEY}" AWS_DEFAULT_REGION=auto
+          package_sha256="$(sha256sum "${PACKAGE_FILE}" | cut -d ' ' -f 1)"
+          key="artifacts/rustfs/packages/validation/runs/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/${package_sha256}/rustfs.deb"
+          url="https://dl.rustfs.com/${key}"
+          "${AWS_CLI}" s3api put-object --bucket "${R2_BUCKET}" --key "${key}" --body "${PACKAGE_FILE}" \
+            --if-none-match '*' --endpoint-url "${R2_ENDPOINT}"
+          downloaded_sha256="$(curl -fsSL --retry 3 --connect-timeout 15 --max-time 300 "${url}" | sha256sum | cut -d ' ' -f 1)"
+          test "${downloaded_sha256}" = "${package_sha256}"
+          manifest="${RUNNER_TEMP}/validation-candidate-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
+          jq -n --arg workflow_sha "${GITHUB_SHA}" --arg source_ref "${VALIDATION_SOURCE_REF}" \
+            --arg source_sha "${VALIDATION_SOURCE_SHA}" --arg channel validation \
+            --argjson build_run_id "${GITHUB_RUN_ID}" --argjson build_run_attempt "${GITHUB_RUN_ATTEMPT}" \
+            --arg package_url "${url}" --arg package_sha256 "${package_sha256}" \
+            '{schema: 3, workflow_sha: $workflow_sha, source_ref: $source_ref, source_sha: $source_sha, channel: $channel, build_run_id: $build_run_id, build_run_attempt: $build_run_attempt, package_url: $package_url, package_sha256: $package_sha256}' > "${manifest}"
+          echo "candidate_file=${manifest}" >> "${GITHUB_OUTPUT}"
+
+      - name: Upload validation candidate manifest
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: validation-candidate-${{ github.run_id }}-${{ github.run_attempt }}
+          path: ${{ steps.publish.outputs.candidate_file }}
+          if-no-files-found: error

.github/workflows/rustfs-validation-tools.yml

@@ -0,0 +1,343 @@
+# Build source-bound helpers separately so live validation needs only artifacts.
+name: RustFS Validation Tools
+
+on:
+  workflow_call:
+    inputs:
+      candidate_sha:
+        type: string
+        required: true
+      auto_testing_sha:
+        type: string
+        required: true
+      build_native_heal:
+        type: boolean
+        required: false
+        default: false
+      build_shared_heal:
+        type: boolean
+        required: false
+        default: false
+      build_manager_native:
+        type: boolean
+        required: false
+        default: false
+    secrets:
+      PF_TESTING_GH_TOKEN:
+        required: true
+  workflow_dispatch:
+    inputs:
+      candidate_sha:
+        description: Exact RustFS source SHA from the validation candidate manifest
+        type: string
+        required: true
+      auto_testing_sha:
+        description: Exact auto-testing SHA containing the Heal metadata inspector
+        type: string
+        required: true
+      build_native_heal:
+        description: Also compile and run separate exact-source native Heal regressions
+        type: boolean
+        required: false
+        default: false
+      build_shared_heal:
+        description: Build the controlled C10 shared-reference fixture encoder
+        type: boolean
+        required: false
+        default: false
+      build_manager_native:
+        description: Build and run exact candidate checkpoint/MRF/incarnation regressions
+        type: boolean
+        required: false
+        default: false
+
+permissions:
+  contents: read
+
+jobs:
+  manager-native-regressions:
+    if: ${{ inputs.build_manager_native }}
+    runs-on: ubuntu-22.04
+    timeout-minutes: 180
+    env:
+      CANDIDATE_SHA: ${{ inputs.candidate_sha }}
+      AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+    steps:
+      - name: Validate immutable manager native inputs
+        run: |
+          set -euo pipefail
+          [[ "${CANDIDATE_SHA}" =~ ^[0-9a-f]{40}$ ]]
+          [[ "${AUTO_TESTING_SHA}" =~ ^[0-9a-f]{40}$ ]]
+
+      - name: Checkout exact manager candidate source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          ref: ${{ inputs.candidate_sha }}
+          path: candidate
+          persist-credentials: false
+
+      - name: Checkout exact manager collector source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          repository: rustfs/auto-testing
+          ref: ${{ inputs.auto_testing_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          path: auto-testing
+          persist-credentials: false
+
+      - name: Install manager native build dependencies
+        run: |
+          set -euo pipefail
+          sudo apt-get update
+          sudo apt-get install -y clang lld cmake pkg-config libssl-dev protobuf-compiler
+          df -h .
+
+      - name: Build exact unmodified candidate manager test executable
+        run: |
+          set -euo pipefail
+          test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
+          python3 -B auto-testing/scripts/validation_heal/manager_native.py build \
+            --source candidate --commit "${CANDIDATE_SHA}" --out manager-tools
+
+      - name: Run exact manager regressions independently
+        run: |
+          set -euo pipefail
+          python3 -B auto-testing/scripts/validation_heal/manager_native.py run \
+            --directory manager-tools --out manager-evidence
+
+      - name: Upload manager executable identities and original diagnostics
+        if: ${{ always() }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rustfs-validation-manager-native-${{ github.run_id }}-${{ github.run_attempt }}
+          path: |
+            manager-tools/heal-manager-native-tests
+            manager-tools/*.json
+            manager-tools/*.jsonl
+            manager-tools/*.stderr
+            manager-tools/*.log
+            manager-evidence/
+          if-no-files-found: warn
+
+  native-heal-regressions:
+    if: ${{ inputs.build_native_heal }}
+    runs-on: ubuntu-22.04
+    timeout-minutes: 180
+    env:
+      CANDIDATE_SHA: ${{ inputs.candidate_sha }}
+      AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+    steps:
+      - name: Validate immutable native source inputs
+        run: |
+          set -euo pipefail
+          [[ "${CANDIDATE_SHA}" =~ ^[0-9a-f]{40}$ ]]
+          [[ "${AUTO_TESTING_SHA}" =~ ^[0-9a-f]{40}$ ]]
+
+      - name: Checkout exact native candidate source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          ref: ${{ inputs.candidate_sha }}
+          path: candidate
+          persist-credentials: false
+
+      - name: Checkout exact native collector source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          repository: rustfs/auto-testing
+          ref: ${{ inputs.auto_testing_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          path: auto-testing
+          persist-credentials: false
+
+      - name: Install native build dependencies
+        run: |
+          set -euo pipefail
+          sudo apt-get update
+          sudo apt-get install -y clang lld cmake pkg-config libssl-dev protobuf-compiler
+          df -h .
+
+      - name: Build and enumerate the unmodified candidate test executable
+        run: |
+          set -euo pipefail
+          test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
+          use_race_native=false
+          if [[ -f auto-testing/scripts/validation_heal/race_native.py ]]; then
+            support_rc=0
+            python3 -B auto-testing/scripts/validation_heal/race_native.py supports \
+              --commit "${CANDIDATE_SHA}" || support_rc=$?
+            case "${support_rc}" in
+              0) use_race_native=true ;;
+              2) ;; # The original exact-source collector handles legacy candidates.
+              *) exit "${support_rc}" ;;
+            esac
+          fi
+          if [[ "${use_race_native}" == true ]]; then
+            python3 -B auto-testing/scripts/validation_heal/race_native.py build \
+              --source candidate --commit "${CANDIDATE_SHA}" --testing-commit "${AUTO_TESTING_SHA}" --out native-tools
+          else
+            python3 -B auto-testing/scripts/validation_heal/native_regressions.py build \
+              --source candidate --commit "${CANDIDATE_SHA}" --out native-tools
+          fi
+
+      - name: Run exact native regressions separately from fleet validation
+        run: |
+          set -euo pipefail
+          if [[ -f native-tools/race-native.json ]]; then
+            test ! -e native-tools/native-regressions.json
+            python3 -B auto-testing/scripts/validation_heal/race_native.py run \
+              --directory native-tools --out native-evidence
+          else
+            python3 -B auto-testing/scripts/validation_heal/native_regressions.py run \
+              --directory native-tools --out native-evidence
+          fi
+
+      - name: Run existing F07 directory fsync failure regressions
+        if: ${{ inputs.candidate_sha == '1aa9be87d18cf941e5020f19e8622c26ba4b10ac' }}
+        run: |
+          set -euo pipefail
+          # Older immutable harness pins retain their original native suite;
+          # absence of this additive collector must never fabricate its proof.
+          if [[ ! -f auto-testing/scripts/validation_heal/persistence_native.py ]]; then
+            mkdir native-persistence-evidence
+            echo '{"complete":false,"reason":"pinned harness predates F07 directory-fsync collector","requiredNativeCoverageComplete":false,"canonicalCasePassClaimed":false}' \
+              > native-persistence-evidence/not-available.json
+            exit 0
+          fi
+          python3 -B auto-testing/scripts/validation_heal/persistence_native.py run \
+            --directory native-tools --native-evidence native-evidence --out native-persistence-evidence \
+            --testing-commit "${AUTO_TESTING_SHA}"
+          python3 -B auto-testing/scripts/validation_heal/persistence_native.py verify \
+            --directory native-tools --native-evidence native-evidence --out native-persistence-evidence \
+            --testing-commit "${AUTO_TESTING_SHA}"
+
+      - name: Upload native executable identities and original diagnostics
+        if: ${{ always() }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rustfs-validation-native-${{ github.run_id }}-${{ github.run_attempt }}
+          path: |
+            native-tools/ecstore-native-tests
+            native-tools/ecstore-race-native-tests
+            native-tools/*.json
+            native-tools/*.jsonl
+            native-tools/*.stderr
+            native-tools/*.log
+            native-evidence/
+            native-persistence-evidence/
+          if-no-files-found: warn
+
+  heal-metadata-inspector:
+    runs-on: ubuntu-22.04
+    timeout-minutes: 30
+    env:
+      CANDIDATE_SHA: ${{ inputs.candidate_sha }}
+      AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
+    steps:
+      - name: Validate immutable source inputs
+        run: |
+          set -euo pipefail
+          [[ "${CANDIDATE_SHA}" =~ ^[0-9a-f]{40}$ ]]
+          [[ "${AUTO_TESTING_SHA}" =~ ^[0-9a-f]{40}$ ]]
+
+      - name: Checkout exact candidate source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          ref: ${{ inputs.candidate_sha }}
+          path: candidate
+          persist-credentials: false
+
+      - name: Checkout exact inspector source
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          repository: rustfs/auto-testing
+          ref: ${{ inputs.auto_testing_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          path: auto-testing
+          persist-credentials: false
+
+      - name: Build the read-only candidate metadata decoder
+        run: |
+          set -euo pipefail
+          test "$(git -C candidate rev-parse HEAD)" = "${CANDIDATE_SHA}"
+          test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
+          df -h .
+          python3 auto-testing/scripts/validation_heal/prepare_heal_meta_inspector.py \
+            --source candidate --commit "${CANDIDATE_SHA}" --out inspector-build
+          # Seed the helper's separate graph from the candidate's frozen graph.
+          # Cargo adds the helper package; both lock identities are retained.
+          cp candidate/Cargo.lock inspector-build/Cargo.lock
+          cargo build --release --manifest-path inspector-build/Cargo.toml
+          python3 auto-testing/scripts/validation_heal/check_bitrot_tool.py \
+            --binary inspector-build/target/release/heal-meta-inspect --source candidate
+          mkdir validation-tools
+          cp inspector-build/target/release/heal-meta-inspect validation-tools/
+          cp inspector-build/Cargo.lock validation-tools/inspector-Cargo.lock
+          python3 - <<'PY'
+          import hashlib
+          import json
+          import os
+          from pathlib import Path
+          import subprocess
+
+          source = json.loads(Path("inspector-build/source.json").read_text())
+          binary = Path("validation-tools/heal-meta-inspect")
+          source.update(built=True, testingCommit=os.environ["AUTO_TESTING_SHA"],
+                        binarySha256=hashlib.sha256(binary.read_bytes()).hexdigest(),
+                        inspectorLockSha256=hashlib.sha256(Path("validation-tools/inspector-Cargo.lock").read_bytes()).hexdigest(),
+                        toolchain=subprocess.check_output(["rustc", "-vV"], text=True),
+                        target="x86_64-unknown-linux-gnu", runId=os.environ["GITHUB_RUN_ID"],
+                        runAttempt=os.environ["GITHUB_RUN_ATTEMPT"])
+          Path("validation-tools/heal-meta-inspect.json").write_text(json.dumps(source, indent=2) + "\n")
+          PY
+          git -C candidate diff --exit-code
+
+      - name: Build and test the controlled C10 metadata encoder
+        if: ${{ inputs.build_shared_heal }}
+        run: |
+          set -euo pipefail
+          python3 -B auto-testing/scripts/validation_heal/prepare_heal_shared_fixture.py \
+            --source candidate --commit "${CANDIDATE_SHA}" --out shared-build
+          cp candidate/Cargo.lock shared-build/Cargo.lock
+          export CARGO_TARGET_DIR="${GITHUB_WORKSPACE}/inspector-build/target"
+          cargo build --release --manifest-path shared-build/Cargo.toml
+          mkdir shared-tools
+          if ! cargo test --release --manifest-path shared-build/Cargo.toml -- --test-threads=1 \
+            > shared-tools/shared-fixture-tests.log 2>&1; then
+            cat shared-tools/shared-fixture-tests.log
+            exit 1
+          fi
+          cp "${CARGO_TARGET_DIR}/release/heal-shared-fixture" shared-tools/
+          cp shared-build/Cargo.lock shared-tools/shared-Cargo.lock
+          python3 - <<'PY'
+          import hashlib
+          import json
+          import os
+          from pathlib import Path
+          import subprocess
+
+          source = json.loads(Path("shared-build/source.json").read_text())
+          source.update(schemaVersion=1, built=True, sourceUnmodified=True,
+              candidateSourceTree=subprocess.check_output(["git", "-C", "candidate", "rev-parse", "HEAD^{tree}"], text=True).strip(),
+              testingCommit=os.environ["AUTO_TESTING_SHA"],
+              binarySha256=hashlib.sha256(Path("shared-tools/heal-shared-fixture").read_bytes()).hexdigest(),
+              sharedLockSha256=hashlib.sha256(Path("shared-tools/shared-Cargo.lock").read_bytes()).hexdigest(),
+              toolchain=subprocess.check_output(["rustc", "-vV"], text=True), target="x86_64-unknown-linux-gnu",
+              runId=os.environ["GITHUB_RUN_ID"], runAttempt=os.environ["GITHUB_RUN_ATTEMPT"])
+          Path("shared-tools/heal-shared-fixture.json").write_text(json.dumps(source, indent=2) + "\n")
+          PY
+          git -C candidate diff --exit-code
+
+      - name: Upload controlled C10 fixture encoder evidence
+        if: ${{ always() && inputs.build_shared_heal }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rustfs-validation-shared-${{ github.run_id }}-${{ github.run_attempt }}
+          path: shared-tools/
+          if-no-files-found: warn
+
+      - name: Upload immutable validation tools
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: rustfs-validation-tools-${{ github.run_id }}-${{ github.run_attempt }}
+          path: validation-tools/
+          if-no-files-found: error

CHANGELOG.md

@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Deployment
+
+- Existing volumes created by older root-running container images must be writable by the non-root runtime identity before upgrading. The default container UID/GID is `10001:10001`; stop the affected instances, verify host mount paths, and for ordinary bind mounts run `sudo chown -R 10001:10001 /actual/data /actual/logs` on each RustFS data/log mount. Account for ACLs, NFS, Kubernetes, and rootless UID mappings. Image-build ownership changes do not migrate mounted data. See the Docker Quick Start in [README.md](README.md).
+- Local disk initialization now checks file creation, writing, and deletion in the data root and fixed internal metadata directories. Unwritable disks are rejected using existing startup/quorum rules, with filesystem errors and Unix runtime/owner IDs logged for permission failures, instead of admitting those disks and discovering the mismatch on a later write. Existing object trees are not scanned and volume ownership is never changed automatically.
+
 ### Security
 
 - **Header-signed SigV4 requests honour only signed headers** (GHSA-xm99-m3gq-83g8): a request authenticated with a SigV4 `Authorization` header that carries an `x-amz-*` request header not listed in its `SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3 and the presigned rule from GHSA-g8w9-qw9q-fghr. Previously anyone holding one header-signed `PutObject` request could add an unsigned `x-amz-copy-source` and turn it into a `CopyObject` that ran with the signer's permissions, copying any object the signer could read into the target. An `Authorization` header whose algorithm token is not `AWS4-HMAC-SHA256` is now rejected instead of being verified as SigV4. The request-envelope headers `x-amz-content-sha256`, `x-amz-decoded-content-length`, `x-amz-trailer` and `x-amz-checksum-algorithm` (the same set the upstream `s3s` fix exempts) and `x-amz-cf-id` (CloudFront) remain tolerated unsigned; AWS SDKs and RustFS's own signers already sign every other `x-amz-*` header. SigV2, JWT and anonymous requests are unchanged.

Cargo.lock

@@ -4,11 +4,11 @@ version = 4
 
 [[package]]
 name = "addr2line"
-version = "0.25.1"
+version = "0.27.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1b5d307320b3181d6d7954e663bd7c774a838b8220fe0593c86d9fb09f498b4b"
+checksum = "e567177890eb1617b1f774005b66b26b2377afd138a2ca37aae7d8f0c81429d4"
 dependencies = [
- "gimli 0.32.3",
+ "gimli 0.34.0",
 ]
 
 [[package]]
@@ -286,7 +286,7 @@ dependencies = [
  "crc32fast",
  "digest 0.11.3",
  "log",
- "miniz_oxide 0.9.1",
+ "miniz_oxide",
  "num-bigint 0.4.8",
  "ouroboros",
  "quad-rand",
@@ -721,9 +721,9 @@ dependencies = [
 
 [[package]]
 name = "async-nats"
-version = "0.50.0"
+version = "0.51.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d83a251fa1a4c9d0fe6e816b7acd60549e473e08d14f27a1d992c2675abff05f"
+checksum = "a4f92759f418774b78c36ef07885849150ce950c67d9bcdeff8a9c253f57d9a1"
 dependencies = [
  "aws-lc-rs",
  "base64 0.22.1",
@@ -910,9 +910,9 @@ dependencies = [
 
 [[package]]
 name = "aws-lc-rs"
-version = "1.18.2"
+version = "1.19.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "faac5829c2b74c28f830747e7818ccfb684261b5f48a1118b1e2a13d36dfab13"
+checksum = "048006790404103c488e6c5616b3e7e702a71292aadda08e8f7698a9e0d2eb53"
 dependencies = [
  "aws-lc-sys",
  "untrusted 0.7.1",
@@ -921,9 +921,9 @@ dependencies = [
 
 [[package]]
 name = "aws-lc-sys"
-version = "0.46.0"
+version = "0.47.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f1622d8446a2d4b2ce0c7eefc73dd43a99779028d5ee5c2dd8073a658ba8a2bc"
+checksum = "8bfbe32ac36b3c428696ce315464181bc4029d689fd58e3f2a4e58c252242f52"
 dependencies = [
  "cc",
  "cmake",
@@ -960,9 +960,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-kms"
-version = "1.126.0"
+version = "1.128.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6c17ea00d100b3a51e9d7b7922baf2912977d66545038f2fa78553461f75a450"
+checksum = "ef519fda62fd0fdc066948f5ef33ecb8ae1bf0187bf1a12e8a775c93c4c47e7e"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -985,9 +985,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-s3"
-version = "1.155.0"
+version = "1.156.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5a542f0e9261b9ce56708d591a62062079fa3c58a362ffbf49a2e48a4f6ced16"
+checksum = "1072c9aa20159b9b417115342d38c4c53108fc98e5359fcb86b7d4575747eb59"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1021,9 +1021,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-sso"
-version = "1.117.0"
+version = "1.118.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5316813498fa9c403877007934ae448aa44b18136f58051f8fa3c5b79b44bdc6"
+checksum = "b4a89786db8ee6dff2c9302a354ef081354fcb0f527e16ee3cb249a6a5cb6df7"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1046,9 +1046,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-ssooidc"
-version = "1.119.0"
+version = "1.120.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "31db35e5662787b786daba15f431e4393c7ea512b70fa112bd048d1f3ecf877a"
+checksum = "521c6a11a25f1244efddf947523f277a51a63f0a7ec228f0f62d826e88106366"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1071,9 +1071,9 @@ dependencies = [
 
 [[package]]
 name = "aws-sdk-sts"
-version = "1.122.0"
+version = "1.123.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a10491638d9e560db90c484dbc77d73b8ad202780cdc9b9e92180603acb07ada"
+checksum = "8754d38d29448bd1d4aa4a6312ed010dfa296b3436eb2c551cc6e42fc6a66a38"
 dependencies = [
  "arc-swap",
  "aws-credential-types",
@@ -1098,7 +1098,8 @@ dependencies = [
 [[package]]
 name = "aws-sigv4"
 version = "1.6.4"
-source = "git+https://github.com/smithy-lang/smithy-rs?rev=5c1c4bca2ea7af97496e077c9f1a718eaac64bf8#5c1c4bca2ea7af97496e077c9f1a718eaac64bf8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f385da9b7757ed4c8214797ad663ff534fb622006143530aa80e21f8e06ff85c"
 dependencies = [
  "aws-credential-types",
  "aws-smithy-eventstream",
@@ -1166,9 +1167,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-http"
-version = "0.64.2"
+version = "0.64.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba8960956e0ee0ca3f3161e62913a61bc65a7ef638ca3ef1d4c7159f8443e398"
+checksum = "19af609dd5edf8b056cd0bb001c706c8b3b1458055328b334ec13d7f46f6aaa8"
 dependencies = [
  "aws-smithy-eventstream",
  "aws-smithy-runtime-api",
@@ -1187,9 +1188,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-http-client"
-version = "1.5.1"
+version = "1.5.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aeca69f65362f7375891008ea525fd440ad6db1836458ac188c87b126804114e"
+checksum = "859a81364f8185349c584fb41cd8765a400cf982f447adada590cf008a9f3b85"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-protocol-test",
@@ -1217,9 +1218,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-json"
-version = "0.65.0"
+version = "0.65.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9f6835fb38bc6fc8c91e584b0366eb4ff595e5c485134dbf4f0bdeaa18856666"
+checksum = "b45f9fc77ffe92c92c341165a01a565bd30c39ecf5aa8c0719b41719009604f9"
 dependencies = [
  "aws-smithy-runtime-api",
  "aws-smithy-schema",
@@ -1269,9 +1270,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-runtime"
-version = "1.16.1"
+version = "1.16.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "79cef0c41e8ca40f74f68ed8ee4d1ddbde12bb847b95ef9b3c025681ddfae58a"
+checksum = "4cb80fb19c084da5fca9d797845735e184445e5b958599440d67529bf6ef1b64"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-http",
@@ -1292,9 +1293,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-runtime-api"
-version = "1.19.1"
+version = "1.19.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "06788004dea79018413f381891da6dcca23196acd79c4d51b767fbdacc76eae4"
+checksum = "8c4e85cffe9944b10d576b075432bd8f98dbc081e8681cb37d5b2d87dc27bc84"
 dependencies = [
  "aws-smithy-async",
  "aws-smithy-runtime-api-macros",
@@ -1309,9 +1310,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-runtime-api-macros"
-version = "1.1.0"
+version = "1.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "221eaa237ddf1ca79b60d1372aad77e47f9c0ea5b3ce5099da8c61d027dc77b3"
+checksum = "ce237f890a2283c9b328955c3b0918d2d5bf66a1cb6ceccd36a0047246fbc6e6"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -1331,9 +1332,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-types"
-version = "1.9.0"
+version = "1.9.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "55752025c39c0e094962fd73f1c57014b191f678612e517f219468e575af9822"
+checksum = "cf183d70774ea2cfcc101ca74a8884e4c76be8b9df65cd303f6a1175c88e8f80"
 dependencies = [
  "base64-simd",
  "bytes",
@@ -1354,9 +1355,9 @@ dependencies = [
 
 [[package]]
 name = "aws-smithy-xml"
-version = "0.64.0"
+version = "0.64.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a6581618a2fd006159016d1c36881d94d61b4a017f4a8d631310875e18252413"
+checksum = "590fdc1a70546b2f7f2061c42eaf173fb6f26d935330d069edc04dbd8f01d924"
 dependencies = [
  "aws-smithy-runtime-api",
  "aws-smithy-schema",
@@ -1442,15 +1443,15 @@ dependencies = [
 
 [[package]]
 name = "backtrace"
-version = "0.3.76"
+version = "0.3.77"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bb531853791a215d7c62a30daf0dde835f381ab5de4589cfe7c649d2cbe92bd6"
+checksum = "42f4220217f3288537b83684130f42fad43320b1c3b805f1910d683d52ded24f"
 dependencies = [
  "addr2line",
  "cfg-if",
  "libc",
- "miniz_oxide 0.8.9",
- "object 0.37.3",
+ "miniz_oxide",
+ "object 0.40.0",
  "rustc-demangle",
  "windows-link",
 ]
@@ -1609,14 +1610,14 @@ dependencies = [
 
 [[package]]
 name = "blazesym"
-version = "0.2.6"
+version = "0.2.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "847a0a95b041ad5aae1bdc44f2bd54743f76eb0065c4f86b139f81343c287eaf"
+checksum = "3b7b394f5f7049b06702df3207b82932e03e8de884b048670ee2516273a10880"
 dependencies = [
  "cpp_demangle",
  "crc32fast",
  "flate2",
- "gimli 0.33.0",
+ "gimli 0.34.0",
  "libc",
  "memmap2",
  "rustc-demangle",
@@ -1743,9 +1744,9 @@ dependencies = [
 
 [[package]]
 name = "brotli"
-version = "9.1.0"
+version = "9.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6737ce8b81460bf9407909f73ff50bf13f7cb8577e1e76715f57ed0eba58ba44"
+checksum = "627b7cfb0578f0eafce087a90fcd40add1231900b69e4d4de2d456260a4f5d81"
 dependencies = [
  "alloc-no-stdlib 3.0.0",
  "alloc-stdlib 0.3.0",
@@ -2236,7 +2237,7 @@ version = "0.4.46"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cae36683101b2899f48bd31f8db9839437356188e3b1ff23096e73dab09ba8fe"
 dependencies = [
- "brotli 9.1.0",
+ "brotli 9.1.1",
  "bzip2",
  "compression-core",
  "flate2",
@@ -2657,9 +2658,9 @@ dependencies = [
 
 [[package]]
 name = "ctutils"
-version = "0.4.6"
+version = "0.4.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fe3e9d21d39e6f39ff46e03d5b621424f88ee23d8a1672c6bfc9ccc7f2da4f6a"
+checksum = "a2376ffd6f96d8a9db59875b4b6b0237d6900b5baf64279714e05493ce64828a"
 dependencies = [
  "cmov",
  "subtle",
@@ -3797,9 +3798,9 @@ dependencies = [
 
 [[package]]
 name = "dial9"
-version = "0.5.5"
+version = "0.5.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "634f30ebbf229703503cd7c24c106bb832c26c80e4a4de0dff9d9e81bbd10c70"
+checksum = "11185c62245c5ff61c753ec8ec3d56341a13fe4195344ecc100ce94f780533cc"
 dependencies = [
  "bon",
  "dial9-core",
@@ -3814,9 +3815,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-core"
-version = "0.5.5"
+version = "0.5.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "944efbaf9995121a93b92a9dacdbb6225332914215af1f08758ce3b9ec3a6693"
+checksum = "54f1dcb5d20bdc85b9beb7c2864540526c1394c00783e9f3544feed1cfe529e8"
 dependencies = [
  "ahash",
  "arc-swap",
@@ -3837,9 +3838,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-macro"
-version = "0.5.0"
+version = "0.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4177b8f8bf95d8e3769c1aad596f5d6fb231953a45c4c3d2a5946fe8e4ec327b"
+checksum = "bc225a6797592a51b6d68611927296ea582cf2317f62d6751d697ed2a2783f52"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -3848,9 +3849,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-metrique"
-version = "0.5.5"
+version = "0.5.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1aefb67b126274a6e0c31a82f624b347be1d531554d35766a2518b5eb04f1f40"
+checksum = "56645773b4185f51a9528625a5aedcd4bbc5b5919ba68e7a6108cf076f635efc"
 dependencies = [
  "dial9-core",
  "dial9-trace-format",
@@ -3862,9 +3863,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-perf-self-profile"
-version = "0.5.5"
+version = "0.5.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3ad6d9ce2232b6ba154d8fa19416594f65ce283233953eb716d0184e0d5eb3e1"
+checksum = "00c8c96d18b28ec76adde2107f7d2d5f8bb53b0670fba31bc012841703bc49ca"
 dependencies = [
  "blazesym",
  "bon",
@@ -3881,9 +3882,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-tokio-telemetry"
-version = "0.5.5"
+version = "0.5.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e30fba16e95037d93825bad06f087e99f60babce447fd5f344d039540cd39a39"
+checksum = "7ccc79f3683219471a8758b8ce459589167c55d72a649d27b89068a1a4cfffe9"
 dependencies = [
  "bon",
  "bytes",
@@ -3906,9 +3907,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-trace-format"
-version = "0.5.2"
+version = "0.5.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "258116805212db80e607656c81d2bf9830e66d5747e6ed6a29d17eeb2e673ff2"
+checksum = "44026339a4c6cbd4914c3eb85e653e741d5739f1e401695a039127ba49c19731"
 dependencies = [
  "ahash",
  "dial9-trace-format-derive",
@@ -3918,9 +3919,9 @@ dependencies = [
 
 [[package]]
 name = "dial9-trace-format-derive"
-version = "0.5.2"
+version = "0.5.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "23b6700840c894de4a66d698850647560a1ae889183d419211877d049a063aad"
+checksum = "bbdbf5c76eb6824b0576e6891811caae93fc0bc96c23003556e3f752d069a84d"
 dependencies = [
  "proc-macro-crate",
  "proc-macro2",
@@ -4420,7 +4421,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
 dependencies = [
  "crc32fast",
- "miniz_oxide 0.9.1",
+ "miniz_oxide",
  "zlib-rs",
 ]
 
@@ -4686,20 +4687,23 @@ dependencies = [
  "zeroize",
 ]
 
-[[package]]
-name = "gimli"
-version = "0.32.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7"
-
 [[package]]
 name = "gimli"
 version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0bf7f043f89559805f8c7cacc432749b2fa0d0a0a9ee46ce47164ed5ba7f126c"
+dependencies = [
+ "stable_deref_trait",
+]
+
+[[package]]
+name = "gimli"
+version = "0.34.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1033caf0b349c518623b5396bfb2cf0bddf44f0306d543a250e5743297aafd10"
 dependencies = [
  "fnv",
- "hashbrown 0.16.1",
+ "hashbrown 0.17.1",
  "indexmap 2.14.2",
  "stable_deref_trait",
 ]
@@ -5273,9 +5277,9 @@ dependencies = [
 
 [[package]]
 name = "hotpath"
-version = "0.29.2"
+version = "0.30.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5b4af2b5ecf2650278fe33b0e49b68c8c484d2306475c127628316dea740cf5c"
+checksum = "502dc37a7c218b9b1d3a0ccfb904da81d10d6dda0e899c722c503092d55576ea"
 dependencies = [
  "arc-swap",
  "async-channel",
@@ -5308,15 +5312,15 @@ dependencies = [
 
 [[package]]
 name = "hotpath-drain"
-version = "0.29.2"
+version = "0.30.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a5cceb6c15fbaaeb00b51acc7be7fad533d0b56cb422b1cfc16cf9ca1624d33a"
+checksum = "0ea17d04c231b90b1f68a91f6f80fb0762c5dea07c611dce9ae157b56a50ff4c"
 
 [[package]]
 name = "hotpath-macros"
-version = "0.29.2"
+version = "0.30.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08c0a42eb299ebb7ec6b9161489e5bf132603a34ca12cecadb8490937aa0492a"
+checksum = "a76e0af911930fc97abc766b393b48cdca77fea9f5f6b388cf2123a6d9c30deb"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -6165,9 +6169,9 @@ dependencies = [
 
 [[package]]
 name = "libunftp"
-version = "0.23.1"
+version = "0.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3dc54288ecbf8beefa5cab3a9921ab03bf1f18a3a0872cbcdb5313fc0baebaeb"
+checksum = "7d94587cfb90dd87b7cced3bff07d32cb56ad1192baa5d688dfaa8728f22d48d"
 dependencies = [
  "async-trait",
  "bitflags 2.13.2",
@@ -6179,10 +6183,11 @@ dependencies = [
  "getrandom 0.4.3",
  "lazy_static",
  "moka",
- "nix 0.30.1",
+ "nix 0.31.3",
  "prometheus",
  "proxy-protocol",
  "rustls",
+ "rustls-webpki",
  "slog",
  "slog-stdlog",
  "thiserror 2.0.21",
@@ -6572,15 +6577,6 @@ version = "0.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
 
-[[package]]
-name = "miniz_oxide"
-version = "0.8.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316"
-dependencies = [
- "adler2",
-]
-
 [[package]]
 name = "miniz_oxide"
 version = "0.9.1"
@@ -6732,7 +6728,7 @@ dependencies = [
  "tracing",
  "twox-hash",
  "url",
- "webpki-roots 1.0.11",
+ "webpki-roots",
 ]
 
 [[package]]
@@ -6830,18 +6826,6 @@ dependencies = [
  "memoffset",
 ]
 
-[[package]]
-name = "nix"
-version = "0.30.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6"
-dependencies = [
- "bitflags 2.13.2",
- "cfg-if",
- "cfg_aliases",
- "libc",
-]
-
 [[package]]
 name = "nix"
 version = "0.31.3"
@@ -7124,15 +7108,6 @@ dependencies = [
  "memchr",
 ]
 
-[[package]]
-name = "object"
-version = "0.37.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe"
-dependencies = [
- "memchr",
-]
-
 [[package]]
 name = "object"
 version = "0.39.1"
@@ -7245,9 +7220,9 @@ dependencies = [
 
 [[package]]
 name = "openssl-probe"
-version = "0.2.1"
+version = "0.2.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
+checksum = "7e036b411754132fd6b74e94b74e9a13c5cf606fa93d1885c35860d3060820b1"
 
 [[package]]
 name = "opentelemetry"
@@ -8377,7 +8352,7 @@ dependencies = [
  "tracing",
  "url",
  "uuid",
- "webpki-roots 1.0.11",
+ "webpki-roots",
 ]
 
 [[package]]
@@ -9858,7 +9833,6 @@ version = "1.0.2"
 dependencies = [
  "serde",
  "serde_json",
- "thiserror 2.0.21",
  "tokio",
  "uuid",
 ]
@@ -9963,7 +9937,7 @@ dependencies = [
  "tracing",
  "twox-hash",
  "uuid",
- "webpki-roots 1.0.11",
+ "webpki-roots",
 ]
 
 [[package]]
@@ -9986,7 +9960,7 @@ dependencies = [
  "tokio-rustls",
  "tracing",
  "uuid",
- "webpki-roots 1.0.11",
+ "webpki-roots",
 ]
 
 [[package]]
@@ -10885,7 +10859,7 @@ version = "1.0.2"
 dependencies = [
  "base64-simd",
  "blake2",
- "brotli 9.1.0",
+ "brotli 9.1.1",
  "bytes",
  "convert_case 0.12.0",
  "crc-fast",
@@ -10985,9 +10959,9 @@ dependencies = [
 
 [[package]]
 name = "rustix"
-version = "1.1.5"
+version = "1.1.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
+checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa"
 dependencies = [
  "bitflags 2.13.2",
  "errno",
@@ -12240,9 +12214,9 @@ dependencies = [
 
 [[package]]
 name = "sysinfo"
-version = "0.39.6"
+version = "0.40.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d2071df9448915b71c4fe6d25deaf1c22f12bd234f01540b77312bb8e41361e6"
+checksum = "110e37d98880fd804cd47509b90a8444f7aaaab655ae625e83cd442a8bc2ebf2"
 dependencies = [
  "libc",
  "memchr",
@@ -12661,47 +12635,38 @@ dependencies = [
 
 [[package]]
 name = "tokio-websockets"
-version = "0.10.1"
+version = "0.13.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f591660438b3038dd04d16c938271c79e7e06260ad2ea2885a4861bfb238605d"
+checksum = "d52efb639344a7c6adb8e62c6f3d2c19c001ff1b79a5041ba1c6ed42e19c6aa5"
 dependencies = [
- "aws-lc-rs",
  "base64 0.22.1",
  "bytes",
  "futures-core",
  "futures-sink",
  "http 1.5.0",
  "httparse",
- "rand 0.8.8",
+ "rand 0.10.3",
  "rustls-pki-types",
+ "sha1_smol",
+ "simdutf8",
  "tokio",
  "tokio-rustls",
  "tokio-util",
- "webpki-roots 0.26.11",
 ]
 
 [[package]]
 name = "toml"
-version = "0.9.12+spec-1.1.0"
+version = "1.1.8+spec-1.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cf92845e79fc2e2def6a5d828f0801e29a2f8acc037becc5ab08595c7d5e9863"
+checksum = "20489e00e4d8741d6be680764cc12e270655e375a20d1011e844a9c3379e678d"
 dependencies = [
  "indexmap 2.14.2",
  "serde_core",
  "serde_spanned",
- "toml_datetime 0.7.5+spec-1.1.0",
+ "toml_datetime",
  "toml_parser",
  "toml_writer",
- "winnow 0.7.15",
-]
-
-[[package]]
-name = "toml_datetime"
-version = "0.7.5+spec-1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347"
-dependencies = [
- "serde_core",
+ "winnow",
 ]
 
 [[package]]
@@ -12720,9 +12685,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c"
 dependencies = [
  "indexmap 2.14.2",
- "toml_datetime 1.1.2+spec-1.1.0",
+ "toml_datetime",
  "toml_parser",
- "winnow 1.0.4",
+ "winnow",
 ]
 
 [[package]]
@@ -12731,7 +12696,7 @@ version = "1.1.5+spec-1.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c"
 dependencies = [
- "winnow 1.0.4",
+ "winnow",
 ]
 
 [[package]]
@@ -12742,9 +12707,9 @@ checksum = "06bdbd8cfc056b8d2e2e85f29b56a3bdbecb527cef81eb39e3e7b98af4652770"
 
 [[package]]
 name = "tonic"
-version = "0.14.6"
+version = "0.14.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef"
+checksum = "cd1fa9b1a51188f3266a823208078315811269d06d272c3a61be0b9a194194ff"
 dependencies = [
  "async-trait",
  "axum",
@@ -12766,6 +12731,7 @@ dependencies = [
  "tokio",
  "tokio-rustls",
  "tokio-stream",
+ "tokio-util",
  "tower",
  "tower-layer",
  "tower-service",
@@ -12774,9 +12740,9 @@ dependencies = [
 
 [[package]]
 name = "tonic-build"
-version = "0.14.6"
+version = "0.14.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c68f61875ac5293cf72e6c8cf0158086428c82c37229e98c840878f1706b0322"
+checksum = "2c07bd4e9f99e2a3ed8bebad596ae02a94e53745b68acc6900e22c3475c8bc44"
 dependencies = [
  "prettyplease 0.2.37",
  "proc-macro2",
@@ -12786,9 +12752,9 @@ dependencies = [
 
 [[package]]
 name = "tonic-prost"
-version = "0.14.6"
+version = "0.14.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "50849f68853be452acf590cde0b146665b8d507b3b8af17261df47e02c209ea0"
+checksum = "f63c616ffaa648f7c5dd5657e97d9a285fcb44ff3b70da761d436f5a13ff7be6"
 dependencies = [
  "bytes",
  "prost 0.14.4",
@@ -12797,9 +12763,9 @@ dependencies = [
 
 [[package]]
 name = "tonic-prost-build"
-version = "0.14.6"
+version = "0.14.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "654e5643eff75d7f8c99197ce1440ed19a3474eada74c12bbac488b2cafdae27"
+checksum = "4a05d8957fac6b3542999796cb4449d45da790751962f974a457e5e00c43d78c"
 dependencies = [
  "prettyplease 0.2.37",
  "proc-macro2",
@@ -13139,9 +13105,9 @@ checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8"
 
 [[package]]
 name = "unicode-width"
-version = "0.2.2"
+version = "0.2.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
+checksum = "ca718471876992059f251e33aafc334346ada08042b7661dd1fe313a9bced557"
 
 [[package]]
 name = "unicode-xid"
@@ -13412,15 +13378,6 @@ dependencies = [
  "rustls-pki-types",
 ]
 
-[[package]]
-name = "webpki-roots"
-version = "0.26.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
-dependencies = [
- "webpki-roots 1.0.11",
-]
-
 [[package]]
 name = "webpki-roots"
 version = "1.0.11"
@@ -13766,12 +13723,6 @@ version = "0.53.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
 
-[[package]]
-name = "winnow"
-version = "0.7.15"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945"
-
 [[package]]
 name = "winnow"
 version = "1.0.4"
@@ -13859,9 +13810,9 @@ dependencies = [
 
 [[package]]
 name = "xattr"
-version = "1.6.1"
+version = "1.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156"
+checksum = "963d6fab92d5504b971dc8bd90ca743c2575e590bc8071ea5bc571598667ae08"
 dependencies = [
  "libc",
  "rustix",
@@ -14038,9 +13989,9 @@ dependencies = [
 
 [[package]]
 name = "zip"
-version = "9.1.2"
+version = "9.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ec2e3161de27091f18a831d9494e9c53378a34833af5c1dda35ad85849da18d6"
+checksum = "1156b39d142a34b999c2310a50455e0cb3760bf7efad0122f796d813bee26ccb"
 dependencies = [
  "aes 0.9.3",
  "bzip2",

Cargo.toml

@@ -149,7 +149,7 @@ async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
 async-compression = { version = "0.4.51" }
 async-trait = "0.1.92"
-async-nats = { version = "0.50.0", default-features = false }
+async-nats = { version = "0.51.0", default-features = false }
 axum = "0.8.9"
 futures = "0.3.34"
 futures-core = "0.3.34"
@@ -175,9 +175,9 @@ tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
 tokio-util = { version = "0.7.20" }
-tonic = { version = "0.14.6" }
-tonic-prost = { version = "0.14.6" }
-tonic-prost-build = { version = "0.14.6" }
+tonic = { version = "0.14.8" }
+tonic-prost = { version = "0.14.8" }
+tonic-prost-build = { version = "0.14.8" }
 tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
 h3 = "0.0.8"
@@ -253,15 +253,15 @@ atoi = "3.1.1"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.13.0" }
 aws-credential-types = { version = "1.3.1" }
-aws-sdk-kms = { default-features = false, version = "1.126.0" }
-aws-sdk-s3 = { default-features = false, version = "1.155.0" }
-aws-sdk-sts = { default-features = false, version = "1.122.0" }
+aws-sdk-kms = { default-features = false, version = "1.128.0" }
+aws-sdk-s3 = { default-features = false, version = "1.156.0" }
+aws-sdk-sts = { default-features = false, version = "1.123.0" }
 aws-smithy-async = { version = "1.3.1" }
-aws-smithy-http-client = { default-features = false, version = "1.5.1" }
-aws-smithy-runtime-api = { version = "1.19.1" }
-aws-smithy-types = { version = "1.9.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.2" }
+aws-smithy-runtime-api = { version = "1.19.2" }
+aws-smithy-types = { version = "1.9.1" }
 base64-simd = "0.8.0"
-brotli = "9.1.0"
+brotli = "9.1.1"
 clap = { version = "4.6.7" }
 const-str = { version = "1.1.0" }
 convert_case = "0.12.0"
@@ -321,7 +321,7 @@ regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
 redis = { version = "1.8.0" }
 rustify = { version = "0.7", default-features = false }
-rustix = { version = "1.1.5" }
+rustix = { version = "1.1.6" }
 rust-embed = { version = "8.13.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
@@ -334,7 +334,7 @@ compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
-sysinfo = "0.39.6"
+sysinfo = "0.40.0"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.4.0"
@@ -356,13 +356,13 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "9.1.2"
+zip = "9.1.3"
 zstd = "0.14.3"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.5", default-features = false }
+dial9 = { version = "0.5.6", default-features = false }
 opentelemetry = { version = "0.33.1" }
 opentelemetry-appender-tracing = { version = "0.33.1" }
 opentelemetry-otlp = { version = "0.33.1" }
@@ -373,7 +373,7 @@ opentelemetry-stdout = { version = "0.33.1" }
 pyroscope = { version = "2.1.5" }
 
 # FTP and SFTP
-libunftp = { version = "0.23.1" }
+libunftp = { version = "0.24.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "12.2.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
@@ -386,7 +386,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.7.2" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.29.2", default-features = false }
+hotpath = { version = "0.30.1", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.49" }
 
@@ -431,12 +431,3 @@ zstd = { path = "crates/zstd-compat" }
 # Upstream: openidconnect-rs#244 and oauth2-rs#352.
 openidconnect = { git = "https://github.com/ramosbugs/openidconnect-rs", rev = "2f77ee73bc4554e1cb6279adb386005429fb4e41" }
 oauth2 = { git = "https://github.com/ramosbugs/oauth2-rs", rev = "89585f4a280afd7b285eaf1c9be0068874e03d22" }
-aws-sigv4 = { git = "https://github.com/smithy-lang/smithy-rs", rev = "5c1c4bca2ea7af97496e077c9f1a718eaac64bf8" }
-
-# Keep the signer's path dependencies on the SDK's existing registry types.
-[patch."https://github.com/smithy-lang/smithy-rs"]
-aws-credential-types = { version = "=1.3.1" }
-aws-smithy-eventstream = { version = "=0.61.6" }
-aws-smithy-http = { version = "=0.64.2" }
-aws-smithy-runtime-api = { version = "=1.19.1" }
-aws-smithy-types = { version = "=1.9.0" }

README.md

@@ -130,6 +130,10 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 The RustFS container runs as a non-root user `rustfs` (UID/GID `10001:10001`). If you bind-mount host directories with Docker or Compose, every mounted path must be writable by that user, otherwise startup may fail with permission denied errors. This applies to data directories, log directories, and TLS certificate directories when `RUSTFS_TLS_PATH` is enabled.
 
+**Upgrading an existing data volume:** some older images ran as root. Replacing the image does not change ownership of mounted files. Before upgrading to a non-root image, stop the instances using the volume, confirm the actual host mount paths and runtime UID/GID, and grant that identity access to all existing data and log directories. For ordinary Docker bind mounts using the default identity, run `sudo chown -R 10001:10001 /actual/data /actual/logs` on the host. Repeat for each data mount; do not apply this recursively to unrelated shared directories. Kubernetes, NFS, ACLs, and rootless UID mappings may require a different ownership or access-control adjustment.
+
+RustFS checks creation, writing, and deletion of a small temporary file in each data root and fixed internal metadata directories when opening a local disk. A failed check rejects that disk through the existing startup/quorum rules and logs the filesystem error; permission diagnostics include the runtime UID/GID and path owner on Unix. This is not a recursive scan of existing buckets or objects: permissions must also allow access to their contents. RustFS does not change ownership automatically. After migration, verify upload, download, and deletion rather than relying only on a green health endpoint.
+
 ```bash
 # Create data and logs directories
 mkdir -p data logs

crates/concurrency/Cargo.toml

@@ -26,7 +26,7 @@ rustfs-io-core = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 
 # Async runtime
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time"] }
 
 # Logging
 tracing = { workspace = true }

crates/config/README.md

@@ -100,6 +100,7 @@ when restoring `/rustfs/console` for a custom OEM build.
 ## S3 API environment variables
 
 - `RUSTFS_API_OBJECT_MAX_VERSIONS` caps the number of retained versions for a single object. It defaults to `9223372036854775807`, matching MinIO's practical-unlimited default. Set a positive integer to enforce a lower per-object metadata bound.
+- `RUSTFS_VEEAM_SOSAPI_ENABLE` serves Veeam SOSAPI `system.xml` and `capacity.xml` for an authenticated GET or HEAD of those exact bucket-root keys. It defaults to `true`. Explicit version reads use the normal object path. Capacity requests return `ServiceUnavailable` until the scanner publishes a complete snapshot certified for the current bucket incarnation and authoritative quota metadata can be read. Legacy or mixed-version snapshots without an incarnation proof cannot serve capacity until a compatible scanner publishes a certified snapshot. Capacity also returns `ServiceUnavailable` when cluster or disk observations are missing, stale, or degraded. Retry after scanner convergence and storage or metadata recovery. Capacity assumes shard paths have independent backing space; shared physical storage can make the estimate optimistic. Set the flag to `false` to restore the normal object path, including deletion of a real object previously stored at that key.
 - `MINIO_API_OBJECT_MAX_VERSIONS` is accepted as a compatibility alias when the canonical RustFS variable is not set.
 
 ## Distributed endpoint locality

crates/config/src/constants/app.rs

@@ -222,6 +222,16 @@ pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
 /// Environment variable for server KMS enable.
 pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
 
+/// Environment variable for Veeam SOSAPI virtual system objects.
+///
+/// When true (the default), authenticated GET and HEAD of the two Veeam
+/// SOSAPI keys return generated XML. When false, those keys use the normal
+/// object path so an operator can remove a real object stored there earlier.
+pub const ENV_VEEAM_SOSAPI_ENABLE: &str = "RUSTFS_VEEAM_SOSAPI_ENABLE";
+
+/// Default Veeam SOSAPI enablement.
+pub const DEFAULT_VEEAM_SOSAPI_ENABLE: bool = true;
+
 /// Default KMS enable for server-side encryption
 /// This is the default value for enabling KMS encryption for server-side encryption.
 /// Default value: false
@@ -374,6 +384,9 @@ pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
 /// Environment variable: RUSTFS_OBS_LOG_STDOUT_ENABLED
 pub const DEFAULT_OBS_LOG_STDOUT_ENABLED: bool = false;
 
+/// Fixed target for sampled local-batch diagnostics with isolated JSON output.
+pub const LOCAL_BATCH_DIAGNOSTIC_LOG_TARGET: &str = "rustfs_local_batch_diagnostic";
+
 /// Constant representing 1 Kibibyte (1024 bytes)
 /// Default value: 1024
 pub const KI_B: usize = 1024;

crates/data-usage/src/data_usage.rs

@@ -466,6 +466,9 @@ pub struct BucketTargetUsageInfo {
 #[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
 pub struct BucketUsageInfo {
     pub size: u64,
+    /// Bucket identity certified by every set contributing to this snapshot.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub bucket_incarnation: Option<String>,
     // Following five fields suffixed with V1 are here for backward compatibility
     // Total Size for objects that have not yet been replicated
     pub replication_pending_size_v1: u64,
@@ -2316,6 +2319,9 @@ impl BucketUsageInfo {
     /// Add size summary to this bucket usage
     /// Merge another BucketUsageInfo into this one
     pub fn merge(&mut self, other: &BucketUsageInfo) {
+        if self.bucket_incarnation != other.bucket_incarnation {
+            self.bucket_incarnation = None;
+        }
         self.size += other.size;
         self.objects_count += other.objects_count;
         self.versions_count += other.versions_count;
@@ -3299,6 +3305,31 @@ mod tests {
         assert_eq!(usage1.versions_count, 15);
     }
 
+    #[test]
+    fn bucket_usage_merge_discards_incompatible_incarnation_proofs() {
+        let id = "00000000-0000-4000-8000-000000000001";
+        let mut same = BucketUsageInfo {
+            bucket_incarnation: Some(id.into()),
+            ..Default::default()
+        };
+        same.merge(&BucketUsageInfo {
+            bucket_incarnation: Some(id.into()),
+            ..Default::default()
+        });
+        assert_eq!(same.bucket_incarnation.as_deref(), Some(id));
+        same.merge(&BucketUsageInfo::default());
+        assert!(same.bucket_incarnation.is_none());
+        let mut different = BucketUsageInfo {
+            bucket_incarnation: Some(id.into()),
+            ..Default::default()
+        };
+        different.merge(&BucketUsageInfo {
+            bucket_incarnation: Some("00000000-0000-4000-8000-000000000002".into()),
+            ..Default::default()
+        });
+        assert!(different.bucket_incarnation.is_none());
+    }
+
     #[test]
     fn size_summary_add_saturates_instead_of_overflowing() {
         // The scanner folds one summary per object into a per-prefix total, so a

crates/e2e_test/src/distributed/multipart_fence_acceptance_test.rs

@@ -1129,9 +1129,198 @@ async fn phase_quorum_matrix(
     Ok(())
 }
 
+async fn list_parts_disk_loss_after_metadata_read(
+    cluster: &RustFSTestClusterEnvironment,
+    proxies: &[RpcFaultProxy],
+    clients: &[Client],
+    bucket: &str,
+) -> TestResult {
+    let key = "list-parts-after-metadata-loss";
+    let body = payload_for("list-parts-acknowledged", 4096);
+    let upload = clients[0].create_multipart_upload().bucket(bucket).key(key).send().await?;
+    let upload_id = upload.upload_id().ok_or("missing upload ID")?.to_owned();
+    let part = clients[0]
+        .upload_part()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .part_number(1)
+        .body(ByteStream::from(body.clone()))
+        .send()
+        .await?;
+    let etag = part.e_tag().ok_or("missing part ETag")?.to_owned();
+    let path = upload_path_scope(bucket, key, &upload_id)?;
+    let root = Path::new(&cluster.nodes[0].data_dirs[0])
+        .join(RUSTFS_META_MULTIPART_BUCKET)
+        .join(&path);
+    let data_dirs = std::fs::read_dir(&root)?
+        .filter_map(Result::ok)
+        .map(|entry| entry.path())
+        .filter(|entry| entry.join("part.1").is_file() && entry.join("part.1.meta").is_file())
+        .collect::<Vec<_>>();
+    assert_eq!(data_dirs.len(), 1, "the coordinator's actual part directory must be identified");
+    let data_dir = &data_dirs[0];
+    let dir_name = data_dir
+        .file_name()
+        .and_then(|name| name.to_str())
+        .ok_or("non-UTF8 data directory")?;
+    let scoped_path = format!("{path}/{dir_name}");
+    let held_dir = data_dir.with_extension("e2e-held");
+    std::fs::rename(data_dir, &held_dir)?;
+    let pauses = proxies[1..]
+        .iter()
+        .map(|proxy| proxy.pause(&["ListDir"], usize::MAX, Some(&scoped_path)))
+        .collect::<Result<Vec<_>, _>>()?;
+    let request = clients[0].list_parts().bucket(bucket).key(key).upload_id(&upload_id);
+    let task = tokio::spawn(async move { request.send().await });
+    // ListDir is dispatched only after the upload metadata read succeeded.
+    // Require all three scoped remote requests; unrelated failures do not count.
+    wait_for_faults(&pauses, &["ListDir"]).await?;
+    let result = tokio::time::timeout(DEADLINE, task).await??;
+    for pause in &pauses {
+        pause.discard_pending();
+    }
+    drop(pauses);
+    std::fs::rename(&held_dir, data_dir)?;
+    let error = result.expect_err("one missing directory and three unavailable participants cannot prove an empty upload");
+    let response = error.raw_response().expect("ListParts must return an S3 service response");
+    assert_eq!(response.status().as_u16(), 503);
+    assert_eq!(error.as_service_error().and_then(|err| err.meta().code()), Some("SlowDownRead"));
+    assert!(response.headers().get("x-rustfs-error-cause").is_none());
+    assert!(response.headers().get("retry-after").is_none());
+    assert!(
+        response
+            .headers()
+            .get("content-type")
+            .is_some_and(|value| value.starts_with("application/xml"))
+    );
+    assert!(
+        response
+            .headers()
+            .get("x-amz-request-id")
+            .is_some_and(|value| !value.is_empty())
+    );
+    let listed = recover_after_fault("ListParts after directory restoration", || {
+        clients[0].list_parts().bucket(bucket).key(key).upload_id(&upload_id).send()
+    })
+    .await??;
+    assert_eq!(listed.parts().len(), 1);
+    assert_eq!(listed.parts()[0].e_tag(), Some(etag.as_str()));
+    clients[0]
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .parts(CompletedPart::builder().part_number(1).e_tag(&etag).build())
+                .build(),
+        )
+        .send()
+        .await?;
+    for client in clients {
+        assert_object_bytes_after_fault(client, bucket, key, &body).await?;
+    }
+    eprintln!(
+        "MPU list proof: metadata read completed, three scoped ListDir RPCs held, missing local directory rejected; restored part and all readers verified"
+    );
+    Ok(())
+}
+
+async fn concurrent_capped_parts_respect_limit_on_distinct_nodes(clients: &[Client], bucket: &str) -> TestResult {
+    let key = "concurrent-capped-parts";
+    let upload = clients[0]
+        .create_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .customize()
+        .mutate_request(|request| {
+            let separator = if request.uri().contains('?') { "&" } else { "?" };
+            let uri = format!("{}{separator}x-rustfs-max-total-object-size=10", request.uri());
+            request
+                .set_uri(uri)
+                .expect("the signed multipart size query must be a valid URI");
+        })
+        .send()
+        .await?;
+    let upload_id = upload.upload_id().ok_or("missing capped upload ID")?.to_owned();
+    let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(3));
+    let mut tasks = Vec::new();
+    for (index, number, byte) in [(0_usize, 1_i32, 0x41_u8), (1, 2, 0x42)] {
+        let client = clients[index].clone();
+        let bucket = bucket.to_owned();
+        let upload_id = upload_id.clone();
+        let barrier = std::sync::Arc::clone(&barrier);
+        tasks.push(tokio::spawn(async move {
+            barrier.wait().await;
+            let result = client
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(&upload_id)
+                .part_number(number)
+                .body(ByteStream::from(vec![byte; 6]))
+                .send()
+                .await;
+            (number, byte, result)
+        }));
+    }
+    barrier.wait().await;
+    let mut winner = None;
+    let mut rejected = 0;
+    for task in tasks {
+        let (number, byte, result) = tokio::time::timeout(DEADLINE, task).await??;
+        match result {
+            Ok(part) => {
+                assert!(winner.is_none(), "both six-byte parts cannot fit the ten-byte limit");
+                winner = Some((number, byte, part.e_tag().ok_or("missing accepted ETag")?.to_owned()));
+            }
+            Err(error) => {
+                assert_eq!(
+                    error.as_service_error().and_then(|err| err.meta().code()),
+                    Some("EntityTooLarge"),
+                    "{error:?}"
+                );
+                rejected += 1;
+            }
+        }
+    }
+    let (number, byte, etag) = winner.ok_or("no part was admitted")?;
+    assert_eq!(rejected, 1);
+    let listed = clients[0]
+        .list_parts()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .send()
+        .await?;
+    assert_eq!(listed.parts().len(), 1);
+    assert_eq!(listed.parts()[0].part_number(), Some(number));
+    assert_eq!(listed.parts()[0].e_tag(), Some(etag.as_str()));
+    clients[0]
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .parts(CompletedPart::builder().part_number(number).e_tag(&etag).build())
+                .build(),
+        )
+        .send()
+        .await?;
+    for client in clients {
+        assert_object_bytes_after_fault(client, bucket, key, &[byte; 6]).await?;
+    }
+    eprintln!("MPU size limit: two coordinators raced; exactly one six-byte part admitted under ten-byte limit");
+    Ok(())
+}
+
 #[tokio::test]
 async fn metadata_delay_preserves_write_quorum_and_rejects_quorum_minus_one() -> TestResult {
     let (cluster, proxies, clients, bucket) = fixture(1, "EC:2").await?;
+    list_parts_disk_loss_after_metadata_read(&cluster, &proxies, &clients, &bucket).await?;
+    concurrent_capped_parts_respect_limit_on_distinct_nodes(&clients, &bucket).await?;
     phase_quorum_matrix(&cluster, &proxies, &clients, &bucket, false, 1, 3).await
 }
 
@@ -1214,6 +1403,22 @@ async fn mixed_version_failed_complete_requires_current_coordinator_for_abort()
     cluster.start_node(0).await?;
     crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness(&clients, &bucket, "mixed-mpu-current-writer", DEADLINE)
         .await?;
+    // Read-only ListParts remains available during the documented mutation pause.
+    // Compare the actual old and current coordinators on the same legacy upload.
+    for (version, client) in [("current", &clients[0]), ("previous", &clients[1])] {
+        let listed = client
+            .list_parts()
+            .bucket(&bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .send()
+            .await?;
+        assert_eq!(listed.parts().len(), 1, "{version} coordinator must list the legacy part");
+        assert_eq!(listed.parts()[0].part_number(), Some(1));
+        assert_eq!(listed.parts()[0].size(), Some(9 * 1024 * 1024));
+        assert_eq!(listed.parts()[0].e_tag(), part.e_tag());
+    }
+    eprintln!("MPU mixed version: old and current coordinators agree on legacy ListParts identity and size");
     let scope = upload_scope(upload_id)?;
     let observers = proxies[1..]
         .iter()
@@ -1344,7 +1549,7 @@ async fn native_certified_partial_abort_retry_reclaims_survivors_below_metadata_
         capability_sha256: Option<[u8; 32]>,
         abort: Option<AbortRef>,
     }
-    let (cluster, proxies, clients, bucket) = fixture(4, "EC:4").await?;
+    let (mut cluster, proxies, clients, bucket) = fixture(4, "EC:4").await?;
     let key = "partial-abort";
     let committed = payload_for("committed-before-abort", 4096);
     clients[0]
@@ -1475,6 +1680,48 @@ async fn native_certified_partial_abort_retry_reclaims_survivors_below_metadata_
             .raw_response()
             .and_then(|response| response.headers().get("x-amz-request-id"))
     );
+    assert!(
+        retained.values().all(|source| source.participant.0 != 3),
+        "the selected offline node must have no surviving root"
+    );
+    cluster.stop_node(3)?;
+    let offline = tokio::time::timeout(
+        DEADLINE,
+        clients[0]
+            .abort_multipart_upload()
+            .bucket(&bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .send(),
+    )
+    .await?
+    .expect_err("an unavailable original participant cannot be replaced by the certificate");
+    assert_unavailable(&offline, "storage_write_quorum", None);
+    assert!(
+        multipart_sources(&cluster, Some(&hash))? == retained,
+        "an unavailable participant must not authorize deletion or change retained evidence"
+    );
+    cluster.start_node(3).await?;
+    wait_for_storage_peer_recovery(&proxies, &clients[0], &bucket, upload_id, 4).await?;
+    eprintln!("MPU certified residual: unavailable original returned storage_write_quorum; all surviving bytes retained");
+
+    // Kill the coordinator and a node owning a real surviving source. The
+    // retained bytes and full-N certificate must survive both process restarts
+    // before a fresh coordinator can authorize the retry.
+    let survivor_node = retained.values().next().ok_or("missing surviving source")?.participant.0;
+    let mut restarted = vec![0, survivor_node];
+    restarted.sort_unstable();
+    restarted.dedup();
+    for node in restarted {
+        cluster.stop_node(node)?;
+        cluster.start_node(node).await?;
+        wait_for_storage_peer_recovery(&proxies, &clients[0], &bucket, upload_id, 4).await?;
+        assert!(
+            multipart_sources(&cluster, Some(&hash))? == retained,
+            "restart must preserve every surviving source and certificate byte"
+        );
+    }
+    eprintln!("MPU certified residual: coordinator and surviving owner restarted; all retained bytes unchanged");
     let delete_observers = proxies
         .iter()
         .map(|proxy| proxy.pause(&["RecoverMultipartSource"], usize::MAX, Some(&path)))

crates/e2e_test/src/distributed/upgrade_test.rs

@@ -47,6 +47,9 @@ struct UpgradeSeed {
     history_bucket: String,
     history_key: &'static str,
     history_body: Vec<u8>,
+    multipart_upload: String,
+    multipart_etag: String,
+    multipart_body: Vec<u8>,
     versioned_bucket: String,
     versioned_key: &'static str,
     version1: String,
@@ -241,6 +244,25 @@ async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
     let history_body = b"written by the previous 4-node release".to_vec();
     put_object(&root, &history_bucket, history_key, history_body.clone()).await?;
 
+    let multipart_body = vec![0x5a; 4096];
+    let multipart = root
+        .create_multipart_upload()
+        .bucket(&history_bucket)
+        .key("legacy-multipart.bin")
+        .send()
+        .await?;
+    let multipart_upload = multipart.upload_id().ok_or("legacy multipart upload ID missing")?.to_owned();
+    let part = root
+        .upload_part()
+        .bucket(&history_bucket)
+        .key("legacy-multipart.bin")
+        .upload_id(&multipart_upload)
+        .part_number(1)
+        .body(aws_sdk_s3::primitives::ByteStream::from(multipart_body.clone()))
+        .send()
+        .await?;
+    let multipart_etag = part.e_tag().ok_or("legacy multipart part ETag missing")?.to_owned();
+
     let versioned_key = "versioned-history.txt";
     let version1_body = b"version-one-before-upgrade".to_vec();
     let version1 = root
@@ -278,6 +300,9 @@ async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
         history_bucket,
         history_key,
         history_body,
+        multipart_upload,
+        multipart_etag,
+        multipart_body,
         versioned_bucket,
         versioned_key,
         version1,
@@ -304,6 +329,21 @@ async fn assert_history_and_iam(dist: &DistCluster, seed: &UpgradeSeed, context:
     )
     .await?;
     assert_object_bytes(&root_a, &seed.history_bucket, seed.history_key, &seed.history_body).await?;
+    // During mixed-version operation this is read-only; mutation resumes only
+    // after the whole fleet is upgraded.
+    for client in [&root_a, &root_b] {
+        let listed = client
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send()
+            .await?;
+        assert_eq!(listed.parts().len(), 1, "{context}: legacy part listing");
+        assert_eq!(listed.parts()[0].part_number(), Some(1));
+        assert_eq!(listed.parts()[0].size(), Some(4096));
+        assert_eq!(listed.parts()[0].e_tag(), Some(seed.multipart_etag.as_str()));
+    }
 
     let v1 = root_b
         .get_object()
@@ -387,17 +427,146 @@ async fn assert_history_and_iam(dist: &DistCluster, seed: &UpgradeSeed, context:
     Ok(())
 }
 
+async fn compare_degraded_legacy_list_parts(dist: &mut DistCluster, seed: &UpgradeSeed, previous: &Path) -> TestResult {
+    let decoded = String::from_utf8(base64_simd::URL_SAFE_NO_PAD.decode_to_vec(seed.multipart_upload.as_bytes())?)?;
+    let (_, upload_dir) = decoded.split_once('.').ok_or("legacy upload ID lacks deployment delimiter")?;
+    let hash = sha256_hex(format!("{}/legacy-multipart.bin", seed.history_bucket).as_bytes());
+    let root = format!("{hash}/{upload_dir}");
+    let mut held = Vec::new();
+    let mut original_files = Vec::new();
+    for (index, node) in dist.cluster.nodes.iter().enumerate() {
+        let path = Path::new(&node.data_dirs[0])
+            .join(crate::storage_api::multipart_fence_acceptance::RUSTFS_META_MULTIPART_BUCKET)
+            .join(&root);
+        let entries = std::fs::read_dir(path)?.collect::<Result<Vec<_>, _>>()?;
+        let dirs = entries
+            .into_iter()
+            .map(|entry| entry.path())
+            .filter(|entry| entry.join("part.1").is_file() && entry.join("part.1.meta").is_file())
+            .collect::<Vec<_>>();
+        assert_eq!(dirs.len(), 1, "each configured disk must hold the original part before injection");
+        let source_root = dirs[0].parent().ok_or("part directory has no upload root")?;
+        for path in [
+            source_root.join("xl.meta"),
+            dirs[0].join("part.1"),
+            dirs[0].join("part.1.meta"),
+        ] {
+            original_files.push((path.clone(), std::fs::read(path)?));
+        }
+        if index < 3 {
+            held.push((dirs[0].clone(), dirs[0].with_extension("held-for-mixed-test")));
+        }
+    }
+    dist.cluster.stop_node(3)?;
+    for (original, temporary) in &held {
+        std::fs::rename(original, temporary)?;
+    }
+
+    let clients = dist.clients()?;
+    let current = tokio::time::timeout(
+        CREDENTIAL_TIMEOUT,
+        clients[0]
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send(),
+    )
+    .await?;
+    let legacy = tokio::time::timeout(
+        CREDENTIAL_TIMEOUT,
+        clients[1]
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send(),
+    )
+    .await?;
+    for (original, temporary) in &held {
+        std::fs::rename(temporary, original)?;
+    }
+    // Three online disks retain metadata quorum but have no part directory;
+    // the fourth disk is offline. A legacy empty response is not a proof that
+    // the unavailable participant has no part.
+    let legacy = legacy.expect("previous coordinator should expose its legacy empty-list behavior");
+    assert!(legacy.parts().is_empty());
+    let current = current.expect_err("current coordinator must not infer empty from missing directories and offline disks");
+    assert_eq!(current.raw_response().expect("current S3 response").status().as_u16(), 503);
+    assert_eq!(current.as_service_error().and_then(|error| error.meta().code()), Some("SlowDownRead"));
+    dist.cluster.start_node_from_binary(3, previous).await?;
+    wait_for_upgrade_write_readiness(&clients, &seed.history_bucket, "mixed-list-restored", CREDENTIAL_TIMEOUT).await?;
+    for (path, before) in original_files {
+        assert_eq!(
+            std::fs::read(&path)?,
+            before,
+            "the fault must preserve every original source file: {path:?}"
+        );
+    }
+    for client in [&clients[0], &clients[1]] {
+        let listed = client
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send()
+            .await?;
+        assert_eq!(listed.parts().len(), 1);
+        assert_eq!(listed.parts()[0].e_tag(), Some(seed.multipart_etag.as_str()));
+    }
+    eprintln!(
+        "MPU mixed degraded: one offline disk, three missing part directories; previous returned empty, current returned 503 SlowDownRead; original part restored on both"
+    );
+    Ok(())
+}
+
+async fn complete_legacy_multipart(dist: &DistCluster, seed: &UpgradeSeed) -> TestResult {
+    // Native legacy-source upgrade needs every original participant, not just
+    // the ordinary three-of-four write quorum proven by the earlier probes.
+    let clients = dist.clients()?;
+    crate::upgrade_compatibility_test::wait_for_drive_readmission(
+        &clients,
+        &seed.history_bucket,
+        "legacy-multipart-all-drives",
+        CREDENTIAL_TIMEOUT,
+    )
+    .await?;
+    let root = &clients[0];
+    root.complete_multipart_upload()
+        .bucket(&seed.history_bucket)
+        .key("legacy-multipart.bin")
+        .upload_id(&seed.multipart_upload)
+        .multipart_upload(
+            aws_sdk_s3::types::CompletedMultipartUpload::builder()
+                .parts(
+                    aws_sdk_s3::types::CompletedPart::builder()
+                        .part_number(1)
+                        .e_tag(&seed.multipart_etag)
+                        .build(),
+                )
+                .build(),
+        )
+        .send()
+        .await?;
+    for client in &clients {
+        assert_object_bytes(client, &seed.history_bucket, "legacy-multipart.bin", &seed.multipart_body).await?;
+    }
+    Ok(())
+}
+
 #[tokio::test]
 async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> TestResult {
     init_logging();
     let previous = source_binary()?;
     let mut dist = DistCluster::new_stopped(DistLayout::FourNodeFourDisk).await?;
+    dist.cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
     capture_upgrade_logs(&mut dist, "direct-upgrade")?;
     dist.start_from_binary(&previous).await?;
 
     let seed = seed_history_and_iam(&dist).await?;
     dist.restart_with_current_binary().await?;
     assert_history_and_iam(&dist, &seed, "direct").await?;
+    complete_legacy_multipart(&dist, &seed).await?;
     Ok(())
 }
 
@@ -419,6 +588,7 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
     init_logging();
     let previous = source_binary()?;
     let mut dist = DistCluster::new_stopped(DistLayout::FourNodeFourDisk).await?;
+    dist.cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
     capture_upgrade_logs(&mut dist, "rolling-upgrade")?;
     dist.start_from_binary(&previous).await?;
 
@@ -426,6 +596,7 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
     assert_history_and_iam(&dist, &seed, "one-current-node").await?;
+    compare_degraded_legacy_list_parts(&mut dist, &seed, &previous).await?;
 
     for node_idx in [1, 2] {
         replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
@@ -434,5 +605,6 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
     assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
+    complete_legacy_multipart(&dist, &seed).await?;
     Ok(())
 }

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1716,6 +1716,15 @@ mod tests {
         cluster.set_env("RUSTFS_HEAL_AUTO_HEAL_ENABLE", background_enabled.to_string());
         cluster.set_env("RUSTFS_HEAL_MRF_ENABLE", background_enabled.to_string());
         cluster.set_env("RUSTFS_SCANNER_ENABLED", background_enabled.to_string());
+        let bucket = "heal-restart-during-rebuild";
+        let outage_key_prefix = "cluster/written-while-node-down-";
+        if !background_enabled {
+            // With MRF disabled, an outage PUT still hands its missing shard to a
+            // best-effort repair on the PUT coordinator. That repair retries until
+            // the target rejoins, so suppress it to keep the explicit root heal
+            // the only owner of the outage object.
+            cluster.set_env("RUSTFS_E2E_SUPPRESS_RENAME_TAIL_HEAL_PREFIX", format!("{bucket}/{outage_key_prefix}"));
+        }
         if background_enabled {
             // Only the scanner cadence is accelerated. Keep normal Heal
             // concurrency and every automatic recovery owner enabled.
@@ -1748,7 +1757,6 @@ mod tests {
         cluster.start_with_binary(&server_binary).await?;
         let clients = cluster.create_all_clients()?;
 
-        let bucket = "heal-restart-during-rebuild";
         clients[0].create_bucket().bucket(bucket).send().await?;
 
         let ReplacementDriveSelection {
@@ -1876,7 +1884,7 @@ mod tests {
         let mut replacement_outage_erasure_index = None;
         let mut rejected_outage_keys = Vec::new();
         for attempt in 0..max_outage_write_attempts {
-            let candidate_key = format!("cluster/written-while-node-down-{attempt:04}.bin");
+            let candidate_key = format!("{outage_key_prefix}{attempt:04}.bin");
             let put_result = timeout(
                 Duration::from_secs(30),
                 clients[2]
@@ -1986,25 +1994,6 @@ mod tests {
         }
 
         let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
-        if !background_enabled {
-            // The PUT path may have admitted a direct Internal object repair while
-            // node 1 was offline. Cancel the isolated bucket path before the target
-            // returns; otherwise it could rebuild the outage object and invalidate
-            // the explicit-root ownership assertion below.
-            let cancel_outage_heal_path = format!("/rustfs/admin/v3/heal/{bucket}?forceStop=true");
-            let (cancel_status, cancel_body) = admin_request(
-                &cluster.nodes[0].url,
-                Method::POST,
-                &cancel_outage_heal_path,
-                Some(heal_body.to_string()),
-                &cluster.access_key,
-                &cluster.secret_key,
-            )
-            .await?;
-            if !cancel_status.is_success() {
-                return Err(format!("cancel outage heal failed: {cancel_status} {cancel_body}").into());
-            }
-        }
 
         // Keep the partial-repair checkpoint stable across readiness and admin
         // requests. Endpoint-blackhole tests must prove their own network stall.

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -38,6 +38,8 @@ mod tests {
     use tracing::info;
 
     const PAGINATION_TIMEOUT: Duration = Duration::from_secs(120);
+    // Bound queued durable commits on the single fixture disk under CI I/O pressure.
+    const FIXTURE_PUT_CONCURRENCY: usize = 4;
 
     // Fixture population can require thousands of durable PUTs. Bound the
     // listing traversal itself with one deadline shared by every page.
@@ -564,7 +566,7 @@ mod tests {
         let object_count = 1002;
         // Finish every durable PUT before listing, with the same bounded fanout as the delimiter fixtures.
         stream::iter(0..object_count)
-            .for_each_concurrent(16, |i| {
+            .for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |i| {
                 let client = &client;
                 async move {
                     let key = format!("object{i:04}.txt");
@@ -575,7 +577,7 @@ mod tests {
                         .body(ByteStream::from_static(b"test content"))
                         .send()
                         .await
-                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
                 }
             })
             .await;
@@ -822,7 +824,7 @@ mod tests {
             .flat_map(|dir| (0..10).map(move |file| format!("dir-{dir:03}/file{file:02}.txt")))
             .collect();
         stream::iter(&all_keys)
-            .for_each_concurrent(16, |key| {
+            .for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |key| {
                 let client = &client;
                 async move {
                     client
@@ -832,7 +834,7 @@ mod tests {
                         .body(ByteStream::from_static(b"x"))
                         .send()
                         .await
-                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
                 }
             })
             .await;
@@ -1055,7 +1057,7 @@ mod tests {
         let files_per_dir = 100;
         let fixture_started = std::time::Instant::now();
         stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
-            .for_each_concurrent(16, |(d, f)| {
+            .for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |(d, f)| {
                 let client = &client;
                 async move {
                     let key = format!("dir-{d:02}/file{f:03}.txt");
@@ -1066,7 +1068,7 @@ mod tests {
                         .body(ByteStream::from_static(b"x"))
                         .send()
                         .await
-                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
                 }
             })
             .await;

crates/e2e_test/src/snowball_auto_extract_test.rs

@@ -261,6 +261,61 @@ mod tests {
         archive
     }
 
+    #[tokio::test]
+    async fn snowball_auto_extract_rejects_veeam_virtual_members() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_VEEAM_SOSAPI_ENABLE", "true")])
+            .await?;
+        let client = env.create_s3_client();
+        let bucket = "snowball-veeam-reserved";
+        client.create_bucket().bucket(bucket).send().await?;
+        let prefix = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c";
+        for (member, destination, use_prefix) in [
+            (format!("{prefix}/system.xml"), format!("{prefix}/system.xml"), false),
+            ("capacity.xml".to_string(), format!("{prefix}/capacity.xml"), true),
+        ] {
+            let mut builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
+            append_typed_entry(&mut builder, &member, tokio_tar::EntryType::Regular, b"blocked").await?;
+            let archive = builder.into_inner().await?.into_inner();
+            let mut request = client
+                .put_object()
+                .bucket(bucket)
+                .key("fixture.tar")
+                .metadata("Snowball-Auto-Extract", "true")
+                .body(ByteStream::from(archive));
+            if use_prefix {
+                request = request
+                    .metadata("Minio-Snowball-Prefix", prefix)
+                    .metadata("Minio-Snowball-Ignore-Errors", "true");
+            }
+            let error = request.send().await.expect_err("virtual archive members must be rejected");
+            assert_eq!(error.code(), Some("MethodNotAllowed"));
+            let listed = client.list_objects_v2().bucket(bucket).prefix(&destination).send().await?;
+            assert!(listed.contents().is_empty(), "rejected virtual member must not be stored");
+        }
+        let mut builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
+        append_typed_entry(&mut builder, "ordinary.xml", tokio_tar::EntryType::Regular, b"ordinary").await?;
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("fixture.tar")
+            .metadata("Snowball-Auto-Extract", "true")
+            .metadata("Minio-Snowball-Prefix", prefix)
+            .body(ByteStream::from(builder.into_inner().await?.into_inner()))
+            .send()
+            .await?;
+        let ordinary = client
+            .get_object()
+            .bucket(bucket)
+            .key(format!("{prefix}/ordinary.xml"))
+            .send()
+            .await?;
+        assert_eq!(ordinary.body.collect().await?.into_bytes().as_ref(), b"ordinary");
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     async fn snowball_auto_extract_supports_minio_prefix_and_directory_markers() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();

crates/e2e_test/src/stale_multipart_cleanup_cluster_test.rs

@@ -13,9 +13,13 @@
 // limitations under the License.
 
 use crate::common::{RustFSTestClusterEnvironment, init_logging};
+use crate::storage_api::multipart_fence_acceptance::RUSTFS_META_MULTIPART_BUCKET;
 use aws_sdk_s3::error::SdkError;
 use aws_sdk_s3::primitives::ByteStream;
-use tokio::time::{Duration, sleep};
+use sha2::{Digest, Sha256};
+use std::collections::BTreeMap;
+use std::path::{Path, PathBuf};
+use tokio::time::{Duration, Instant, sleep};
 use tracing::info;
 use uuid::Uuid;
 
@@ -84,6 +88,126 @@ async fn wait_for_cleanup_on_all_nodes(
     Err("stale multipart upload was not cleaned up on all nodes within timeout".into())
 }
 
+type PhysicalTree = BTreeMap<PathBuf, Option<Vec<u8>>>;
+
+fn physical_tree(root: &Path) -> Result<PhysicalTree, Box<dyn std::error::Error + Send + Sync>> {
+    let mut result = BTreeMap::new();
+    let mut pending = vec![root.to_path_buf()];
+    while let Some(directory) = pending.pop() {
+        for entry in std::fs::read_dir(directory)? {
+            let entry = entry?;
+            let path = entry.path();
+            let relative = path.strip_prefix(root)?.to_path_buf();
+            let kind = entry.file_type()?;
+            if kind.is_dir() {
+                result.insert(relative, None);
+                pending.push(path);
+            } else if kind.is_file() {
+                result.insert(relative, Some(std::fs::read(path)?));
+            } else {
+                return Err(format!("unexpected staging entry type: {relative:?}").into());
+            }
+        }
+    }
+    Ok(result)
+}
+
+async fn assert_low_quorum_residuals_are_diagnosed_and_preserved(
+    cluster: &RustFSTestClusterEnvironment,
+    clients: &[aws_sdk_s3::Client],
+    log_paths: &[PathBuf],
+) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    let mut residuals = Vec::new();
+    for copies in [1_usize, 2] {
+        let key = format!("multipart/partial-{copies}-{}", Uuid::new_v4().simple());
+        let upload = clients[0]
+            .create_multipart_upload()
+            .bucket(CLEANUP_BUCKET)
+            .key(&key)
+            .send()
+            .await?;
+        let upload_id = upload.upload_id().ok_or("create response missing upload ID")?;
+        clients[0]
+            .upload_part()
+            .bucket(CLEANUP_BUCKET)
+            .key(&key)
+            .upload_id(upload_id)
+            .part_number(1)
+            .body(ByteStream::from(vec![0x41; 4096]))
+            .send()
+            .await?;
+        let decoded = String::from_utf8(base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes())?)?;
+        let (_, upload_dir) = decoded.split_once('.').ok_or("upload ID lacks deployment delimiter")?;
+        let digest = Sha256::digest(format!("{CLEANUP_BUCKET}/{key}").as_bytes());
+        let hash = hex_simd::encode_to_string(digest, hex_simd::AsciiCase::Lower);
+        let relative = format!("{hash}/{upload_dir}");
+        let roots = cluster
+            .nodes
+            .iter()
+            .map(|node| {
+                Path::new(&node.data_dirs[0])
+                    .join(RUSTFS_META_MULTIPART_BUCKET)
+                    .join(&relative)
+            })
+            .collect::<Vec<_>>();
+        assert_eq!(roots.len(), 4);
+        let mut preserved = Vec::new();
+        for root in &roots {
+            assert!(
+                root.join("xl.meta").is_file(),
+                "all four original metadata files must exist before injection"
+            );
+            preserved.push(physical_tree(root)?);
+        }
+        for root in roots.iter().skip(copies) {
+            std::fs::remove_dir_all(root)?;
+        }
+        residuals.push((relative, roots, preserved, copies));
+    }
+
+    let deadline = Instant::now() + Duration::from_secs(45);
+    loop {
+        let logs = log_paths
+            .iter()
+            .filter_map(|path| std::fs::read_to_string(path).ok())
+            .collect::<Vec<_>>();
+        let diagnosed = residuals.iter().all(|(path, _, _, _)| {
+            logs.iter().any(|log| {
+                log.lines()
+                    .any(|line| line.contains(path) && line.contains("multipart_cleanup_lock_or_recheck_failed"))
+            })
+        });
+        if diagnosed {
+            break;
+        }
+        if Instant::now() >= deadline {
+            return Err("stale low-quorum residuals were not diagnosed before the deadline".into());
+        }
+        sleep(Duration::from_millis(200)).await;
+    }
+    for (path, roots, preserved, copies) in &residuals {
+        for slot in 0..*copies {
+            assert_eq!(
+                physical_tree(&roots[slot])?,
+                preserved[slot],
+                "retained source evidence changed: {path}, slot {slot}"
+            );
+        }
+        for root in roots.iter().skip(*copies) {
+            assert!(!root.exists(), "cleanup must not recreate missing source roots: {path}");
+        }
+    }
+    for path in log_paths {
+        let log = std::fs::read_to_string(path)?;
+        assert!(
+            !log.contains("op=\"upload_part\"") && !log.contains("\"op\":\"upload_part\""),
+            "background cleanup must not be logged as a client UploadPart"
+        );
+    }
+    eprintln!("Stale MPU low-quorum: EC:2 one/two-copy roots diagnosed; full surviving file inventories preserved");
+    Ok(())
+}
+
 #[tokio::test]
 async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -92,6 +216,22 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
     let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
     cluster.set_env("RUSTFS_API_STALE_UPLOADS_EXPIRY", "5s");
     cluster.set_env("RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL", "1s");
+    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
+    cluster.set_env(
+        "RUST_LOG",
+        concat!("warn,rustfs_ecstore", "::bucket::lifecycle::bucket_lifecycle_ops=debug"),
+    );
+    let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR")
+        .map(PathBuf::from)
+        .map(|root| root.join(format!("stale-multipart-{}", Uuid::new_v4().simple())))
+        .unwrap_or_else(|| PathBuf::from(&cluster.temp_dir));
+    std::fs::create_dir_all(&log_dir)?;
+    let log_paths = (0..cluster.nodes.len())
+        .map(|index| log_dir.join(format!("stale-node-{index}.log")))
+        .collect::<Vec<_>>();
+    for (index, path) in log_paths.iter().enumerate() {
+        cluster.set_node_capture_log_path(index, path.to_string_lossy().into_owned())?;
+    }
     cluster.start().await?;
     cluster.create_test_bucket(CLEANUP_BUCKET).await?;
 
@@ -137,6 +277,7 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
     );
 
     wait_for_cleanup_on_all_nodes(&clients, CLEANUP_BUCKET, &key, &upload_id).await?;
+    assert_low_quorum_residuals_are_diagnosed_and_preserved(&cluster, &clients, &log_paths).await?;
 
     Ok(())
 }

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -1066,7 +1066,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness(
 /// reduced-redundancy class then fails with 503 `ServiceUnavailable` until the
 /// serving node writes through every drive, on the previous release and on
 /// the current build alike.
-async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult {
+pub(crate) async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult {
     wait_for_upgrade_writes(
         clients,
         bucket,

crates/e2e_test/src/version_id_regression_test.rs

@@ -600,6 +600,154 @@ mod tests {
         info!("✅ PASSED: Veeam backup workflow simulation completed successfully");
     }
 
+    /// Veeam B&R discovers SOSAPI by GET/HEAD of two well-known keys.
+    #[tokio::test]
+    async fn test_veeam_sosapi_system_and_capacity() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        use crate::common::local_http_client;
+
+        const SYSTEM_KEY: &str = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml";
+        const CAPACITY_KEY: &str = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/capacity.xml";
+
+        init_logging();
+        info!("🧪 TEST: Veeam SOSAPI system.xml and capacity.xml");
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server_with_env(vec![], crate::common::FAST_DATA_USAGE_SCANNER_ENV)
+            .await?;
+        let client = create_s3_client(&env);
+        let bucket = "veeam-sosapi";
+        create_bucket(&client, bucket).await?;
+
+        let system = client.get_object().bucket(bucket).key(SYSTEM_KEY).send().await?;
+        let system_etag = system.e_tag().map(str::to_string);
+        let system_len = system.content_length();
+        assert_eq!(system.content_type(), Some("application/xml"));
+        let system_body = system.body.collect().await?.into_bytes();
+        let system_xml = std::str::from_utf8(&system_body)?;
+        assert!(system_xml.contains("<ProtocolVersion>\"1.0\"</ProtocolVersion>"), "{system_xml}");
+        assert!(
+            system_xml.contains("<ModelName>\"RustFS ") && system_xml.contains("\"</ModelName>"),
+            "{system_xml}"
+        );
+        assert!(system_xml.contains("<CapacityInfo>true</CapacityInfo>"));
+        assert!(system_xml.contains("<UploadSessions>false</UploadSessions>"));
+        assert!(system_xml.contains("<IAMSTS>false</IAMSTS>"));
+        assert!(!system_xml.contains("SystemRecommendations"));
+        assert!(!system_xml.contains("APIEndpoints"));
+        assert!(!system_xml.contains("&quot;"));
+
+        let head = client.head_object().bucket(bucket).key(SYSTEM_KEY).send().await?;
+        assert_eq!(head.content_type(), Some("application/xml"));
+        assert_eq!(head.content_length(), system_len);
+        assert_eq!(head.e_tag().map(str::to_string), system_etag);
+        assert!(head.last_modified().is_some());
+
+        let ranged = client
+            .get_object()
+            .bucket(bucket)
+            .key(SYSTEM_KEY)
+            .range("bytes=0-3")
+            .send()
+            .await?;
+        assert_eq!(ranged.content_length(), Some(4));
+        let ranged_header = ranged.content_range().unwrap_or("");
+        assert!(ranged_header.starts_with("bytes 0-3/"), "{ranged_header}");
+        let ranged_body = ranged.body.collect().await?.into_bytes();
+        assert_eq!(&ranged_body[..], &system_body[..4]);
+
+        let capacity = tokio::time::timeout(std::time::Duration::from_secs(180), async {
+            loop {
+                match client.get_object().bucket(bucket).key(CAPACITY_KEY).send().await {
+                    Ok(capacity) => break Ok(capacity),
+                    Err(err) if err.code() == Some("ServiceUnavailable") => {
+                        tokio::time::sleep(std::time::Duration::from_secs(2)).await;
+                    }
+                    Err(err) => break Err(err),
+                }
+            }
+        })
+        .await??;
+        assert_eq!(capacity.content_type(), Some("application/xml"));
+        let capacity_body = capacity.body.collect().await?.into_bytes();
+        let capacity_xml = std::str::from_utf8(&capacity_body)?;
+        let capacity_value = xml_u64(capacity_xml, "Capacity")?;
+        let available_value = xml_u64(capacity_xml, "Available")?;
+        let used_value = xml_u64(capacity_xml, "Used")?;
+        assert!(used_value <= capacity_value, "{capacity_xml}");
+        assert!(available_value <= capacity_value.saturating_sub(used_value), "{capacity_xml}");
+
+        let put_err = client
+            .put_object()
+            .bucket(bucket)
+            .key(SYSTEM_KEY)
+            .body(ByteStream::from_static(b"overwrite"))
+            .send()
+            .await
+            .expect_err("SOSAPI put must fail");
+        assert_eq!(put_err.code(), Some("MethodNotAllowed"));
+        let delete_err = client
+            .delete_object()
+            .bucket(bucket)
+            .key(CAPACITY_KEY)
+            .send()
+            .await
+            .expect_err("SOSAPI delete must fail");
+        assert_eq!(delete_err.code(), Some("MethodNotAllowed"));
+
+        let normal_key = "Veeam/Backup/job.dat";
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(normal_key)
+            .body(ByteStream::from_static(b"backup-bytes"))
+            .send()
+            .await?;
+        let normal = client.get_object().bucket(bucket).key(normal_key).send().await?;
+        let normal_body = normal.body.collect().await?.into_bytes();
+        assert_eq!(&normal_body[..], b"backup-bytes");
+
+        let anon = local_http_client()
+            .get(format!("{}/{bucket}/{SYSTEM_KEY}", env.url))
+            .send()
+            .await?;
+        assert_eq!(anon.status(), reqwest::StatusCode::FORBIDDEN);
+        let anon_body = anon.text().await?;
+        assert!(!anon_body.contains("ProtocolVersion"), "{anon_body}");
+
+        let now = time::OffsetDateTime::now_utc();
+        let scope_date = format!("{:04}{:02}{:02}", now.year(), u8::from(now.month()), now.day());
+        let amz_date = format!("{scope_date}T{:02}{:02}{:02}Z", now.hour(), now.minute(), now.second());
+        let bad = local_http_client()
+            .get(format!("{}/{bucket}/{SYSTEM_KEY}", env.url))
+            .header("x-amz-date", &amz_date)
+            .header("x-amz-content-sha256", "UNSIGNED-PAYLOAD")
+            .header(
+                "authorization",
+                format!(
+                    "AWS4-HMAC-SHA256 Credential=bad/{scope_date}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000"
+                ),
+            )
+            .send()
+            .await?;
+        assert_eq!(bad.status(), reqwest::StatusCode::FORBIDDEN);
+        let bad_body = bad.text().await?;
+        assert!(!bad_body.contains("ProtocolVersion"), "{bad_body}");
+
+        info!("✅ PASSED: Veeam SOSAPI system.xml and capacity.xml");
+        Ok(())
+    }
+
+    fn xml_u64(xml: &str, element: &str) -> Result<u64, Box<dyn std::error::Error + Send + Sync>> {
+        let start = format!("<{element}>");
+        let end = format!("</{element}>");
+        let start_idx = xml.find(&start).ok_or_else(|| format!("missing {element} in {xml}"))? + start.len();
+        let end_idx = xml[start_idx..]
+            .find(&end)
+            .ok_or_else(|| format!("unclosed {element} in {xml}"))?
+            + start_idx;
+        Ok(xml[start_idx..end_idx].parse()?)
+    }
+
     #[tokio::test]
     async fn test_terraform_put_after_delete() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         init_logging();

crates/ecstore/Cargo.toml

@@ -32,6 +32,8 @@ workspace = true
 [features]
 default = []
 diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
+# Sampled wall/CPU diagnostics only; does not change the read backend.
+diagnostic-local-batch = ["rustix/time"]
 gcs = ["dep:google-cloud-storage", "dep:google-cloud-auth"]
 rio-v2 = ["dep:rustfs-rio-v2"]
 hotpath = [

crates/ecstore/src/api/mod.rs

@@ -14,12 +14,16 @@
 
 //! Explicit ECStore public facades for outer crate compatibility boundaries.
 
-#[cfg(feature = "diagnostic-read-version-caller")]
+#[cfg(any(feature = "diagnostic-read-version-caller", feature = "diagnostic-local-batch"))]
 pub mod diagnostic {
+    #[cfg(feature = "diagnostic-read-version-caller")]
     pub use crate::cluster::rpc::runtime_sources::{
         current_read_version_entry_source, observe_read_version_context_boundary, with_captured_read_version_context,
         with_read_version_entry, with_read_version_spawn_context,
     };
+    #[cfg(feature = "diagnostic-local-batch")]
+    pub use crate::diagnostics::batch_read_diagnostics::{current_request_id, in_request_context, request_context};
+    #[cfg(feature = "diagnostic-read-version-caller")]
     pub use rustfs_io_metrics::internode_metrics::{ReadVersionContextBoundary, ReadVersionEntrySource};
 }
 
@@ -141,6 +145,12 @@ pub mod bucket {
             pub use crate::bucket::lifecycle::rule::{Filter, NoncurrentVersionTransitionOps, TransitionOps};
         }
 
+        pub mod tier_free_version_recovery {
+            pub use crate::bucket::lifecycle::tier_free_version_recovery::{
+                TierCleanupRetryReport, TierCleanupRetryStatus, retry_tier_free_version_cleanup,
+            };
+        }
+
         pub mod tier_delete_journal {
             #[cfg(feature = "test-util")]
             pub use crate::bucket::lifecycle::tier_delete_journal::recover_tier_delete_journal_entries;
@@ -252,7 +262,7 @@ pub mod bucket {
     pub mod quota {
         pub use crate::bucket::quota::{
             BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
-            QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration,
+            QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration, QuotaType,
         };
 
         pub mod checker {
@@ -371,15 +381,24 @@ pub mod compression {
 
 pub mod config {
     pub mod com {
+        /// Preserve empty existing objects so callers can distinguish corruption from absence.
+        pub async fn read_config_preserve_empty_with_metadata_opts(
+            api: std::sync::Arc<crate::store::ECStore>,
+            file: &str,
+            opts: &crate::object_api::ObjectOptions,
+        ) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
+            crate::config::com::read_config_preserve_empty_with_metadata_opts(api, file, opts).await
+        }
+
         pub use crate::config::com::{
             COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
             ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
             is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
             read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
-            read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_config_with_opts_quiet,
-            save_server_config, save_server_config_no_lock, save_server_config_snapshot,
-            save_server_config_snapshot_with_generation, server_config_path, try_migrate_server_config,
-            with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
+            read_server_config_snapshot, read_server_config_with_generation, save_config, save_config_no_lock,
+            save_config_with_opts, save_config_with_opts_quiet, save_server_config, save_server_config_no_lock,
+            save_server_config_snapshot, save_server_config_snapshot_with_generation, server_config_path,
+            try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
             with_server_config_write_lock,
         };
     }
@@ -416,9 +435,10 @@ pub mod data_usage {
         DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
         init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
         invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
-        load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size,
-        record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
-        record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
+        load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
+        load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
+        record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
+        record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
         refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
         remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
         store_data_usage_in_backend,

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -3472,23 +3472,47 @@ mod tests {
         (port, handle)
     }
 
-    fn spawn_http_status_server(status: u16) -> (u16, std::thread::JoinHandle<()>) {
-        use std::io::{Read, Write};
+    fn spawn_http_status_server(status: u16) -> (u16, std::sync::mpsc::Sender<()>, std::thread::JoinHandle<usize>) {
+        use std::io::{ErrorKind, Read, Write};
 
         let listener = std::net::TcpListener::bind(("127.0.0.1", 0)).expect("test HTTP listener should bind");
+        listener
+            .set_nonblocking(true)
+            .expect("test HTTP listener should be nonblocking");
         let port = listener
             .local_addr()
             .expect("test HTTP listener should have an address")
             .port();
+        let (stop, stopped) = std::sync::mpsc::channel();
         let handle = std::thread::spawn(move || {
-            let (mut stream, _) = listener.accept().expect("test HTTP client should connect");
-            let mut request = [0_u8; 8192];
-            let bytes_read = stream.read(&mut request).expect("test HTTP request should be read");
-            assert!(bytes_read > 0, "test HTTP request should not be empty");
-            write!(stream, "HTTP/1.1 {status} Test\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
-                .expect("test HTTP response should be written");
+            let mut requests = 0;
+            loop {
+                match stopped.try_recv() {
+                    Ok(()) | Err(std::sync::mpsc::TryRecvError::Disconnected) => break,
+                    Err(std::sync::mpsc::TryRecvError::Empty) => {}
+                }
+                let (mut stream, _) = match listener.accept() {
+                    Ok(connection) => connection,
+                    Err(err) if err.kind() == ErrorKind::WouldBlock => {
+                        std::thread::sleep(Duration::from_millis(5));
+                        continue;
+                    }
+                    Err(err) => panic!("test HTTP listener failed: {err}"),
+                };
+                stream.set_nonblocking(false).expect("test HTTP stream should be blocking");
+                stream
+                    .set_read_timeout(Some(Duration::from_secs(3)))
+                    .expect("test HTTP read timeout should configure");
+                let mut request = [0_u8; 8192];
+                let bytes_read = stream.read(&mut request).expect("test HTTP request should be read");
+                assert!(bytes_read > 0, "test HTTP request should not be empty");
+                requests += 1;
+                write!(stream, "HTTP/1.1 {status} Test\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
+                    .expect("test HTTP response should be written");
+            }
+            requests
         });
-        (port, handle)
+        (port, stop, handle)
     }
 
     fn spawn_delayed_http_server() -> (
@@ -4492,16 +4516,26 @@ mod tests {
     #[tokio::test]
     async fn target_health_check_treats_client_errors_as_online_and_server_errors_as_offline() {
         for (status, expected_online) in [(403, true), (500, false)] {
-            let (port, server) = spawn_http_status_server(status);
+            let (port, stop, server) = spawn_http_status_server(status);
             let endpoint = format!("http://127.0.0.1:{port}");
-            let target = target_client_for_test(
-                &format!("arn:http-{status}"),
-                endpoint.clone(),
-                s3_client_for_endpoint_test(endpoint, None),
-            );
+            let client = s3_client_for_endpoint_test(endpoint.clone(), None);
+            let config = client
+                .config()
+                .to_builder()
+                .retry_config(
+                    aws_sdk_s3::config::retry::RetryConfig::standard()
+                        .with_max_attempts(2)
+                        .with_initial_backoff(Duration::from_millis(1))
+                        .with_max_backoff(Duration::from_millis(1)),
+                )
+                .build();
+            let target = target_client_for_test(&format!("arn:http-{status}"), endpoint, S3Client::from_conf(config));
 
-            assert_eq!(BucketTargetSys::check_endpoint_health(&target).await, expected_online);
-            server.join().expect("test HTTP server should stop");
+            let online = BucketTargetSys::check_endpoint_health(&target).await;
+            stop.send(()).expect("test HTTP server should be running");
+            let requests = server.join().expect("test HTTP server should stop");
+            assert_eq!(requests, if status == 500 { 2 } else { 1 }, "unexpected probe count for HTTP {status}");
+            assert_eq!(online, expected_online, "unexpected health for HTTP {status}");
         }
     }
 

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -63,6 +63,7 @@ use crate::services::tier::{
 };
 use crate::set_disk::{
     MAX_PARTS_COUNT, RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY, SetDisks, get_lock_acquire_timeout,
+    multipart_part_logical_size, should_persist_encryption_original_size,
 };
 use crate::storage_api_contracts::{
     lifecycle::ExpirationOptions,
@@ -119,6 +120,7 @@ const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
 const EVENT_LIFECYCLE_WORKER_STATE: &str = "lifecycle_worker_state";
 const EVENT_LIFECYCLE_TRANSITION_COMPENSATION: &str = "lifecycle_transition_compensation";
 const EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP: &str = "lifecycle_stale_multipart_cleanup";
+const METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL: &str = "rustfs_stale_multipart_cleanup_blocked_total";
 const EVENT_LIFECYCLE_SCAN_SKIPPED: &str = "lifecycle_scan_skipped";
 const EVENT_LIFECYCLE_EVALUATION_FAILED: &str = "lifecycle_evaluation_failed";
 const EVENT_LIFECYCLE_EXPIRED_DETECTED: &str = "lifecycle_expired_detected";
@@ -1086,8 +1088,9 @@ impl ExpiryState {
         }))
     }
 
-    #[cfg(test)]
-    fn new_with_unconsumed_worker_channel(capacity: usize) -> Arc<RwLock<Self>> {
+    #[cfg(any(test, feature = "test-util"))]
+    #[doc(hidden)]
+    pub fn new_with_unconsumed_worker_channel(capacity: usize) -> Arc<RwLock<Self>> {
         let (tx, rx) = mpsc::channel(capacity);
         Arc::new(RwLock::new(Self {
             tasks_tx: vec![tx],
@@ -3510,19 +3513,29 @@ async fn stale_upload_current_size_with_opts(
             &ObjectOptions {
                 data_movement,
                 no_lock,
+                strict_multipart_part_sizes: true,
                 ..Default::default()
             },
         )
         .await
         .ok()?;
 
-    Some(
-        parts
-            .parts
-            .iter()
-            .map(|part| part.actual_size.max(part.size as i64).max(0) as usize)
-            .sum(),
-    )
+    stale_upload_size_from_parts(&parts)
+}
+
+fn stale_upload_size_from_parts(parts: &crate::storage_api_contracts::multipart::ListPartsInfo) -> Option<usize> {
+    if parts.is_truncated {
+        return None;
+    }
+    let transformed = rustfs_utils::http::contains_key_str(&parts.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION)
+        || should_persist_encryption_original_size(&parts.user_defined);
+    parts.parts.iter().try_fold(0usize, |total, part| {
+        // ListParts represents a quorum-confirmed missing part with an empty ETag.
+        if part.etag.as_deref().is_none_or(str::is_empty) {
+            return None;
+        }
+        total.checked_add(multipart_part_logical_size(part.size, part.actual_size, transformed)?)
+    })
 }
 
 async fn stale_upload_lifecycle_due(
@@ -3539,18 +3552,44 @@ async fn stale_upload_lifecycle_due(
     let bucket = metadata.get(RUSTFS_MULTIPART_BUCKET_KEY)?;
     let object = metadata.get(RUSTFS_MULTIPART_OBJECT_KEY)?;
 
-    let lifecycle = match metadata_boundary::get_lifecycle_config(bucket).await {
+    let mut lifecycle = match metadata_boundary::get_lifecycle_config(bucket).await {
         Ok((lifecycle, _)) => lifecycle,
         Err(_) => return None,
     };
 
+    let has_size_filter = |filter: &s3s::dto::LifecycleRuleFilter| {
+        filter.object_size_greater_than.is_some()
+            || filter.object_size_less_than.is_some()
+            || filter
+                .and
+                .as_ref()
+                .is_some_and(|and| and.object_size_greater_than.is_some() || and.object_size_less_than.is_some())
+    };
+    let needs_size = lifecycle.rules.iter().any(|rule| {
+        rule.status.as_str() != ExpirationStatus::DISABLED
+            && rule.abort_incomplete_multipart_upload.is_some()
+            && rule.filter.as_ref().is_some_and(has_size_filter)
+    });
+    let size = if needs_size {
+        match stale_upload_current_size_with_opts(set, metadata, upload_dir, no_lock).await {
+            Some(size) => size,
+            None => {
+                // An unknown size cannot authorize a size-filtered abort.
+                // Unrelated rules can still be evaluated independently.
+                lifecycle
+                    .rules
+                    .retain(|rule| !rule.filter.as_ref().is_some_and(has_size_filter));
+                0
+            }
+        }
+    } else {
+        0
+    };
     let object_opts = ObjectOpts {
         name: object.clone(),
         user_tags: metadata.get(AMZ_OBJECT_TAGGING).cloned().unwrap_or_default(),
         mod_time: Some(initiated),
-        size: stale_upload_current_size_with_opts(set, metadata, upload_dir, no_lock)
-            .await
-            .unwrap_or_default(),
+        size,
         is_latest: true,
         delete_marker: false,
         user_defined: metadata.clone(),
@@ -3707,6 +3746,23 @@ async fn cleanup_empty_multipart_sha_dirs_on_local_disks(set: &Arc<SetDisks>) {
     }
 }
 
+fn record_stale_multipart_cleanup_blocked(stage: &'static str) {
+    metrics::counter!(METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL, "stage" => stage).increment(1);
+}
+
+fn record_stale_multipart_cleanup_lock_lost(path: &str) {
+    record_stale_multipart_cleanup_blocked("lock_lost");
+    debug!(
+        event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
+        component = LOG_COMPONENT_ECSTORE,
+        subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+        state = "skipped",
+        path = %path,
+        reason = "multipart_cleanup_lock_lost",
+        "Skipped stale multipart cleanup"
+    );
+}
+
 async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: OffsetDateTime, default_expiry: StdDuration) -> usize {
     let mut deleted = 0usize;
     let mut candidates = HashMap::new();
@@ -3803,7 +3859,8 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
     for candidate in candidates.into_values() {
         let upload_dir = candidate.path.rsplit('/').next().unwrap_or_default().to_string();
         let mut due = stale_upload_default_due(candidate.initiated, default_expiry);
-        if let Some(metadata) = candidate.metadata.as_ref()
+        if now < due
+            && let Some(metadata) = candidate.metadata.as_ref()
             && let Some(lifecycle_due) = stale_upload_lifecycle_due(set, metadata, candidate.initiated, &upload_dir, false).await
             && lifecycle_due < due
         {
@@ -3817,6 +3874,7 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
         let cleanup_guard = match set.lock_stale_multipart_cleanup(&candidate.path).await {
             Ok(guard) => guard,
             Err(err) => {
+                record_stale_multipart_cleanup_blocked("recheck");
                 debug!(
                     event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
                     component = LOG_COMPONENT_ECSTORE,
@@ -3832,13 +3890,18 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
         let current_metadata = cleanup_guard.file_info().metadata.clone();
         let current_initiated = initiated_from_upload_dir(&upload_dir, cleanup_guard.file_info().mod_time);
         let mut current_due = stale_upload_default_due(current_initiated, default_expiry);
-        if let Some(lifecycle_due) =
-            stale_upload_lifecycle_due(set, &current_metadata, current_initiated, &upload_dir, true).await
+        if now < current_due
+            && let Some(lifecycle_due) =
+                stale_upload_lifecycle_due(set, &current_metadata, current_initiated, &upload_dir, true).await
             && lifecycle_due < current_due
         {
             current_due = lifecycle_due;
         }
-        if now < current_due || cleanup_guard.is_lock_lost() {
+        if now < current_due {
+            continue;
+        }
+        if cleanup_guard.is_lock_lost() {
+            record_stale_multipart_cleanup_lock_lost(&candidate.path);
             continue;
         }
 
@@ -3858,15 +3921,18 @@ async fn cleanup_stale_multipart_uploads_in_set(set: &Arc<SetDisks>, now: Offset
                     "Removed stale multipart upload"
                 );
             }
-            Err(err) => debug!(
-                event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                path = %candidate.path,
-                error = ?err,
-                reason = "multipart_remove_failed",
-                "Failed to remove stale multipart upload"
-            ),
+            Err(err) => {
+                record_stale_multipart_cleanup_blocked("delete");
+                debug!(
+                    event = EVENT_LIFECYCLE_STALE_MULTIPART_CLEANUP,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                    path = %candidate.path,
+                    error = ?err,
+                    reason = "multipart_remove_failed",
+                    "Failed to remove stale multipart upload"
+                );
+            }
         }
     }
 
@@ -14973,6 +15039,226 @@ mod tests {
         reset_runtime_expiry_state(&ecstore).await;
     }
 
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial]
+    async fn tier_cleanup_retry_enqueues_only_target_and_retains_durable_marker() {
+        use crate::bucket::lifecycle::tier_free_version_recovery::{TierCleanupRetryStatus, retry_tier_free_version_cleanup};
+        let (disks, api) = setup_test_env().await;
+        let (_backend, identity) = register_recovery_mock_tier(&api).await;
+        let state = install_unconsumed_runtime_expiry_worker(&api, 4).await;
+        let bucket = format!("tier-retry-{}", Uuid::new_v4());
+        create_test_bucket(&api, &bucket).await;
+        seed_recoverable_free_version(&disks, &bucket, "owned", None, Some(identity)).await;
+        let check = TierConfigMgr::deletion_check(&api.tier_config_mgr(), api.clone(), "WARM")
+            .await
+            .expect("physical free-version must be diagnosed through ECStore");
+        assert!(!check.proof_complete);
+        assert_eq!(
+            check.scan_status,
+            crate::services::tier::tier::TierDeletionScanStatus::StoppedAfterBlocker
+        );
+        assert_eq!(check.reasons.len(), 1);
+        assert_eq!(check.reasons[0].kind, crate::services::tier::tier::TierDeletionReasonKind::CleanupPending);
+        let report = retry_tier_free_version_cleanup(api.clone(), "WARM")
+            .await
+            .expect("retry should enqueue durable cleanup");
+        assert_eq!(report.status, TierCleanupRetryStatus::Accepted);
+        assert_eq!(report.enqueued, 1);
+        assert_eq!(report.failed, 0);
+        let rx = Arc::clone(&state.read().await.tasks_rx[0]);
+        let task = rx.lock().await.try_recv().expect("cleanup was queued").expect("queue item");
+        let task = task
+            .as_any()
+            .downcast_ref::<FreeVersionTask>()
+            .expect("existing free-version worker task");
+        assert_eq!(
+            (&task.0.bucket, task.0.name.as_str(), task.0.transitioned_object.tier.as_str()),
+            (&bucket, "owned", "WARM")
+        );
+        assert_eq!(
+            crate::services::tier::test_util::free_version_count(&disks[0], &bucket, "owned").await,
+            1,
+            "enqueue acceptance must retain ownership until remote cleanup commits"
+        );
+        crate::services::tier::test_util::register_mock_tier(&api.tier_config_mgr(), "OTHER").await;
+        let other = retry_tier_free_version_cleanup(api.clone(), "OTHER")
+            .await
+            .expect("other tier retry should be bounded");
+        assert_eq!(other.enqueued, 0, "retry must not enqueue a different tier's marker");
+        assert!(rx.lock().await.try_recv().is_err());
+        api.tier_config_mgr()
+            .write()
+            .await
+            .remove("OTHER", true)
+            .await
+            .expect("remove other mock tier");
+        reset_runtime_expiry_state(&api).await;
+        let failed = retry_tier_free_version_cleanup(api.clone(), "WARM")
+            .await
+            .expect("queue failure remains retryable");
+        assert_eq!(failed.failed, 1);
+        assert_eq!(failed.enqueued, 0);
+        assert_eq!(crate::services::tier::test_util::free_version_count(&disks[0], &bucket, "owned").await, 1);
+        remove_seeded_free_version(&disks, &bucket, "owned").await;
+        api.tier_config_mgr()
+            .write()
+            .await
+            .remove("WARM", true)
+            .await
+            .expect("remove mock tier");
+        api.delete_bucket(&bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("remove test bucket");
+        reset_runtime_expiry_state(&api).await;
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial]
+    async fn tier_cleanup_retry_strictly_bounds_versions_within_one_key() {
+        use crate::bucket::lifecycle::tier_free_version_recovery::{
+            DEFAULT_FREE_VERSION_RECOVERY_LIMIT, RecoveryWalkTestAction, retry_tier_free_version_cleanup,
+            set_recovery_walk_test_hook,
+        };
+        let (_disks, api) = setup_test_env().await;
+        let (_backend, _) = register_recovery_mock_tier(&api).await;
+        let state = install_unconsumed_runtime_expiry_worker(&api, DEFAULT_FREE_VERSION_RECOVERY_LIMIT + 2).await;
+        let bucket = format!("retry-version-limit-{}", Uuid::new_v4());
+        create_test_bucket(&api, &bucket).await;
+        let selected_bucket = bucket.clone();
+        let hook = set_recovery_walk_test_hook(move |name| {
+            (name == selected_bucket).then(|| {
+                RecoveryWalkTestAction::SendItems(
+                    (0..DEFAULT_FREE_VERSION_RECOVERY_LIMIT + 1)
+                        .map(|_| {
+                            let mut object = ObjectInfo {
+                                bucket: selected_bucket.clone(),
+                                name: "same-key".to_string(),
+                                version_id: Some(Uuid::new_v4()),
+                                ..Default::default()
+                            };
+                            object.transitioned_object.free_version = true;
+                            object.transitioned_object.name = "remote/object".to_string();
+                            object.transitioned_object.tier = "WARM".to_string();
+                            object
+                        })
+                        .collect(),
+                )
+            })
+        });
+        let report = retry_tier_free_version_cleanup(api.clone(), "WARM")
+            .await
+            .expect("bounded retry succeeds");
+        assert_eq!(report.enqueued, DEFAULT_FREE_VERSION_RECOVERY_LIMIT);
+        assert_eq!(report.failed, 0);
+        assert!(report.scan_truncated, "a partial key must remain visibly truncated");
+        let receiver = Arc::clone(&state.read().await.tasks_rx[0]);
+        let mut queued = 0;
+        while receiver.lock().await.try_recv().is_ok() {
+            queued += 1;
+        }
+        assert_eq!(queued, DEFAULT_FREE_VERSION_RECOVERY_LIMIT);
+        drop(hook);
+        reset_runtime_expiry_state(&api).await;
+        api.tier_config_mgr()
+            .write()
+            .await
+            .remove("WARM", true)
+            .await
+            .expect("remove mock tier");
+        api.delete_bucket(&bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("remove empty test bucket");
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial]
+    async fn tier_cleanup_retry_coalesces_by_backend_identity() {
+        use crate::bucket::lifecycle::tier_free_version_recovery::{TierCleanupRetryStatus, retry_tier_free_version_cleanup};
+        let (_disks, api) = setup_test_env().await;
+        let (_backend, _) = register_recovery_mock_tier(&api).await;
+        let state = install_unconsumed_runtime_expiry_worker(&api, 1).await;
+        let lease = TierConfigMgr::acquire_operation_lease(&api.tier_config_mgr(), "WARM")
+            .await
+            .expect("test lease");
+        let name = format!(
+            "tier-cleanup-retry/{}",
+            hex_simd::encode_to_string(lease.backend_identity(), hex_simd::AsciiCase::Lower)
+        );
+        let lock = api
+            .new_ns_lock(crate::disk::RUSTFS_META_BUCKET, &name)
+            .await
+            .expect("retry lock");
+        let mut guard = lock
+            .get_write_lock_quiet(StdDuration::from_secs(1))
+            .await
+            .expect("hold concurrent retry");
+        let report = retry_tier_free_version_cleanup(api.clone(), "WARM")
+            .await
+            .expect("busy retry is observable");
+        assert_eq!(report.status, TierCleanupRetryStatus::Busy);
+        assert_eq!(report.enqueued, 0);
+        assert_eq!(report.scanned, 0);
+        assert!(state.read().await.tasks_rx[0].lock().await.try_recv().is_err());
+        guard.release();
+        drop(lease);
+        api.tier_config_mgr()
+            .write()
+            .await
+            .remove("WARM", true)
+            .await
+            .expect("remove mock tier");
+        reset_runtime_expiry_state(&api).await;
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial]
+    async fn tier_cleanup_retry_cancels_walk_without_dropping_cleanup_ownership() {
+        use crate::bucket::lifecycle::tier_free_version_recovery::{
+            RecoveryWalkTestAction, retry_tier_free_version_cleanup_with_cancel, set_recovery_walk_test_hook,
+        };
+        let (disks, api) = setup_test_env().await;
+        let (_backend, identity) = register_recovery_mock_tier(&api).await;
+        let bucket = format!("retry-cancel-{}", Uuid::new_v4());
+        create_test_bucket(&api, &bucket).await;
+        seed_recoverable_free_version(&disks, &bucket, "owned", None, Some(identity)).await;
+        let started = Arc::new(tokio::sync::Notify::new());
+        let notify = started.clone();
+        let selected_bucket = bucket.clone();
+        let hook = set_recovery_walk_test_hook(move |name| {
+            (name == selected_bucket).then(|| RecoveryWalkTestAction::WaitForCancellation(notify.clone()))
+        });
+        let cancel = CancellationToken::new();
+        let task_api = api.clone();
+        let task_cancel = cancel.clone();
+        let task = tokio::spawn(async move { retry_tier_free_version_cleanup_with_cancel(task_api, "WARM", task_cancel).await });
+        tokio::time::timeout(StdDuration::from_secs(5), started.notified())
+            .await
+            .expect("retry starts a walk");
+        cancel.cancel();
+        let error = tokio::time::timeout(StdDuration::from_secs(2), task)
+            .await
+            .expect("cancelled retry must stop")
+            .expect("retry joins")
+            .expect_err("cancelled scan cannot report acceptance");
+        assert_eq!(error.status_code, http::StatusCode::SERVICE_UNAVAILABLE);
+        assert_eq!(crate::services::tier::test_util::free_version_count(&disks[0], &bucket, "owned").await, 1);
+        drop(hook);
+        remove_seeded_free_version(&disks, &bucket, "owned").await;
+        api.tier_config_mgr()
+            .write()
+            .await
+            .remove("WARM", true)
+            .await
+            .expect("remove mock tier");
+        api.delete_bucket(&bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("remove test bucket");
+    }
+
     #[tokio::test]
     #[serial]
     async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
@@ -15868,6 +16154,536 @@ mod tests {
         assert!(ecstore.is_ok(), "fresh local ECStore boot should succeed, got {ecstore:?}");
     }
 
+    #[tokio::test(flavor = "current_thread")]
+    #[serial]
+    async fn stale_multipart_cleanup_preserves_partial_metadata_without_upload_part_warning() {
+        use tracing::instrument::WithSubscriber as _;
+        #[derive(Clone, Default)]
+        struct CapturedLogs(Arc<StdMutex<Vec<u8>>>);
+        struct LogWriter(Arc<StdMutex<Vec<u8>>>);
+        impl std::io::Write for LogWriter {
+            fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
+                self.0.lock().unwrap().extend_from_slice(bytes);
+                Ok(bytes.len())
+            }
+            fn flush(&mut self) -> std::io::Result<()> {
+                Ok(())
+            }
+        }
+        impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for CapturedLogs {
+            type Writer = LogWriter;
+            fn make_writer(&'a self) -> Self::Writer {
+                LogWriter(self.0.clone())
+            }
+        }
+
+        let (paths, ecstore) = setup_test_env().await;
+        let bucket = format!("stale-partial-{}", Uuid::new_v4().simple());
+        create_test_bucket(&ecstore, &bucket).await;
+        for copies in [1, 2] {
+            let object = format!("partial-{copies}.bin");
+            let initiated = OffsetDateTime::now_utc() - time::Duration::hours(30);
+            let upload = ecstore
+                .new_multipart_upload(
+                    &bucket,
+                    &object,
+                    &ObjectOptions {
+                        mod_time: Some(initiated),
+                        max_parity: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("multipart upload should be created");
+            let decoded = base64_simd::URL_SAFE_NO_PAD
+                .decode_to_vec(upload.upload_id.as_bytes())
+                .unwrap();
+            let decoded = String::from_utf8(decoded).unwrap();
+            let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload id");
+            let upload_path = format!("{}/{upload_dir}", multipart_sha_dir(&bucket, &object));
+            let physical_paths: Vec<_> = paths
+                .iter()
+                .map(|path| path.join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path))
+                .collect();
+            let original: Vec<_> = physical_paths
+                .iter()
+                .take(copies)
+                .map(|path| std::fs::read(path.join(STORAGE_FORMAT_FILE)).unwrap())
+                .collect();
+            let info = super::get_file_info(
+                &original[0],
+                RUSTFS_META_MULTIPART_BUCKET,
+                &upload_path,
+                "",
+                super::FileInfoOpts {
+                    data: false,
+                    include_free_versions: false,
+                    include_part_checksums: false,
+                },
+            )
+            .unwrap();
+            assert_eq!((info.erasure.data_blocks, info.erasure.parity_blocks), (2, 2));
+            for path in physical_paths.iter().skip(copies) {
+                fs::remove_dir_all(path).await.expect("remove selected metadata copies");
+            }
+            if copies == 2 {
+                ecstore
+                    .get_multipart_info(&bucket, &object, &upload.upload_id, &ObjectOptions::default())
+                    .await
+                    .expect("two metadata copies must meet read quorum");
+            }
+
+            let logs = CapturedLogs::default();
+            let subscriber = tracing_subscriber::fmt()
+                .with_max_level(tracing::Level::DEBUG)
+                .with_writer(logs.clone())
+                .with_ansi(false)
+                .without_time()
+                .finish();
+            let recorder = crate::test_metrics::CapturingRecorder::default();
+            let recorder_guard = metrics::set_default_local_recorder(&recorder);
+            cleanup_stale_multipart_uploads_once_at(
+                ecstore.clone(),
+                OffsetDateTime::now_utc(),
+                StdDuration::from_secs(24 * 60 * 60),
+            )
+            .with_subscriber(subscriber)
+            .await;
+            drop(recorder_guard);
+            for (path, expected) in physical_paths.iter().zip(&original) {
+                assert_eq!(fs::read(path.join(STORAGE_FORMAT_FILE)).await.unwrap(), *expected);
+            }
+            let log_output = String::from_utf8(logs.0.lock().unwrap().clone()).unwrap();
+            let rechecks = log_output
+                .lines()
+                .filter(|line| line.contains("multipart_cleanup_lock_or_recheck_failed"));
+            let matching_path = rechecks.clone().any(|line| line.contains(&upload_path));
+            assert!(
+                matching_path,
+                "missing path-specific background diagnostic for {upload_path}: {log_output}"
+            );
+            assert_eq!(
+                recorder.counter_value(super::METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL, &[("stage", "recheck")]),
+                u64::try_from(rechecks.count()).expect("diagnostic count fits u64"),
+                "each observed recheck failure must increment the metric exactly once"
+            );
+            assert!(!log_output.contains("op=\"upload_part\""), "unexpected upload-part warning: {log_output}");
+            for path in physical_paths.iter().take(copies) {
+                fs::remove_dir_all(path).await.unwrap();
+            }
+        }
+
+        let logs = CapturedLogs::default();
+        let subscriber = tracing_subscriber::fmt()
+            .with_max_level(tracing::Level::DEBUG)
+            .with_writer(logs.clone())
+            .with_ansi(false)
+            .without_time()
+            .finish();
+        let recorder = crate::test_metrics::CapturingRecorder::default();
+        metrics::with_local_recorder(&recorder, || {
+            tracing::subscriber::with_default(subscriber, || {
+                super::record_stale_multipart_cleanup_lock_lost("test-hash/test-upload");
+            })
+        });
+        let output = String::from_utf8(logs.0.lock().unwrap().clone()).unwrap();
+        assert!(
+            output.contains("multipart_cleanup_lock_lost") && output.contains("test-hash/test-upload"),
+            "{output}"
+        );
+        assert!(!output.contains("op=\"upload_part\""), "{output}");
+        assert_eq!(
+            recorder.counter_value(super::METRIC_STALE_MULTIPART_CLEANUP_BLOCKED_TOTAL, &[("stage", "lock_lost")]),
+            1
+        );
+    }
+
+    #[tokio::test(flavor = "current_thread")]
+    #[serial]
+    async fn stale_multipart_cleanup_lists_parts_only_when_size_is_needed() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+        use tracing::instrument::WithSubscriber as _;
+        use tracing_subscriber::prelude::*;
+
+        struct PartListCounter {
+            calls: Arc<AtomicUsize>,
+            bucket: String,
+            object: &'static str,
+        }
+        struct PartListFields<'a> {
+            bucket: &'a str,
+            object: &'a str,
+            matching_bucket: bool,
+            matching_object: bool,
+        }
+        impl tracing::field::Visit for PartListFields<'_> {
+            fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
+                match field.name() {
+                    "bucket" => self.matching_bucket = value == self.bucket,
+                    "object" => self.matching_object = value == self.object,
+                    _ => {}
+                }
+            }
+            fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
+                let value = format!("{value:?}");
+                match field.name() {
+                    "bucket" => self.matching_bucket = value == format!("{:?}", self.bucket),
+                    "object" => self.matching_object = value == format!("{:?}", self.object),
+                    _ => {}
+                }
+            }
+        }
+        impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for PartListCounter {
+            fn on_new_span(
+                &self,
+                attrs: &tracing::span::Attributes<'_>,
+                _id: &tracing::span::Id,
+                _ctx: tracing_subscriber::layer::Context<'_, S>,
+            ) {
+                if attrs.metadata().name() != "list_object_parts" {
+                    return;
+                }
+                let mut fields = PartListFields {
+                    bucket: &self.bucket,
+                    object: self.object,
+                    matching_bucket: false,
+                    matching_object: false,
+                };
+                attrs.record(&mut fields);
+                if fields.matching_bucket && fields.matching_object {
+                    self.calls.fetch_add(1, Ordering::Relaxed);
+                }
+            }
+        }
+
+        let (_, ecstore) = setup_test_env().await;
+        let bucket = format!("stale-list-parts-{}", Uuid::new_v4().simple());
+        create_test_bucket(&ecstore, &bucket).await;
+        set_abort_incomplete_lifecycle(&bucket, "logs/", 0).await;
+        let now = OffsetDateTime::now_utc();
+        for (object, age, size_filter, should_list) in [
+            ("logs/default-expired", time::Duration::hours(30), false, false),
+            ("logs/lifecycle-only", time::Duration::minutes(5), false, false),
+            ("logs/size-filter", time::Duration::minutes(5), true, true),
+            ("logs/disabled-size-filter", time::Duration::minutes(5), false, false),
+        ] {
+            if size_filter {
+                set_abort_incomplete_lifecycle_with_size(&bucket, "logs/", 0, 0).await;
+            } else if object == "logs/disabled-size-filter" {
+                let rules = r#"<LifecycleConfiguration><Rule><ID>enabled</ID><Status>Enabled</Status><Filter><Prefix>logs/</Prefix></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule><Rule><ID>disabled-size</ID><Status>Disabled</Status><Filter><ObjectSizeGreaterThan>0</ObjectSizeGreaterThan></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule></LifecycleConfiguration>"#;
+                metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rules.as_bytes().to_vec())
+                    .await
+                    .expect("install enabled rule and disabled size rule");
+            }
+            let upload = ecstore
+                .new_multipart_upload(
+                    &bucket,
+                    object,
+                    &ObjectOptions {
+                        mod_time: Some(now - age),
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("create multipart upload");
+            if size_filter {
+                let mut data = PutObjReader::from_vec(vec![1, 2, 3, 4]);
+                ecstore
+                    .put_object_part(&bucket, object, &upload.upload_id, 1, &mut data, &ObjectOptions::default())
+                    .await
+                    .expect("upload a part for size filtering");
+            }
+            let calls = Arc::new(AtomicUsize::new(0));
+            let subscriber = tracing_subscriber::registry().with(PartListCounter {
+                calls: Arc::clone(&calls),
+                bucket: bucket.clone(),
+                object,
+            });
+            cleanup_stale_multipart_uploads_once_at(ecstore.clone(), now, StdDuration::from_secs(24 * 60 * 60))
+                .with_subscriber(subscriber)
+                .await;
+            let listed = calls.load(Ordering::Relaxed);
+            assert_eq!(listed > 0, should_list, "unexpected ListParts calls for {object}: {listed}");
+            let result = ecstore
+                .get_multipart_info(&bucket, object, &upload.upload_id, &ObjectOptions::default())
+                .await;
+            assert!(
+                matches!(result, Err(ref err) if is_err_invalid_upload_id(err)),
+                "upload must be removed: {result:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn stale_multipart_cleanup_rejects_incomplete_or_overflowing_part_sizes() {
+        use crate::storage_api_contracts::multipart::{ListPartsInfo, PartInfo};
+
+        let part = |size, actual_size, etag: &str| PartInfo {
+            size,
+            actual_size,
+            etag: Some(etag.to_owned()),
+            ..Default::default()
+        };
+        let mut parts = ListPartsInfo {
+            parts: vec![part(6, 6, "valid")],
+            ..Default::default()
+        };
+        assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
+        parts.is_truncated = true;
+        assert_eq!(super::stale_upload_size_from_parts(&parts), None);
+        parts.is_truncated = false;
+        parts.parts = vec![part(0, 0, "")];
+        assert_eq!(super::stale_upload_size_from_parts(&parts), None);
+        parts.parts = vec![part(1, -1, "valid")];
+        assert_eq!(super::stale_upload_size_from_parts(&parts), None);
+        parts.parts = vec![part(8, 6, "valid")];
+        assert_eq!(super::stale_upload_size_from_parts(&parts), None);
+        parts.parts = vec![part(4, 6, "valid")];
+        assert_eq!(super::stale_upload_size_from_parts(&parts), None);
+        rustfs_utils::http::insert_str(&mut parts.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION, "test".to_owned());
+        assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
+        parts.parts = vec![part(8, 6, "valid")];
+        assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
+        rustfs_utils::http::remove_str(&mut parts.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION);
+        parts
+            .user_defined
+            .insert("x-amz-server-side-encryption".to_owned(), "AES256".to_owned());
+        assert_eq!(super::stale_upload_size_from_parts(&parts), Some(6));
+        parts.user_defined.clear();
+        parts.parts = vec![part(usize::MAX, 0, "valid"), part(1, 1, "valid")];
+        assert_eq!(super::stale_upload_size_from_parts(&parts), None);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn stale_multipart_cleanup_handles_unknown_size_without_false_abort() {
+        let (_, ecstore) = setup_test_env().await;
+        let bucket = format!("stale-unknown-size-{}", Uuid::new_v4().simple());
+        let object = "logs/object";
+        create_test_bucket(&ecstore, &bucket).await;
+        let size_rule = r#"<Rule><ID>size</ID><Status>Enabled</Status><Filter><ObjectSizeLessThan>5</ObjectSizeLessThan></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule>"#;
+        let independent_rule = r#"<Rule><ID>prefix</ID><Status>Enabled</Status><Filter><Prefix>logs/</Prefix></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule>"#;
+        let lifecycle = |rules: &str| format!("<LifecycleConfiguration>{rules}</LifecycleConfiguration>").into_bytes();
+        metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle(size_rule))
+            .await
+            .expect("install upper-size lifecycle rule");
+        let metadata = HashMap::from([
+            (super::RUSTFS_MULTIPART_BUCKET_KEY.to_owned(), bucket.clone()),
+            (super::RUSTFS_MULTIPART_OBJECT_KEY.to_owned(), object.to_owned()),
+        ]);
+        let initiated = OffsetDateTime::now_utc() - time::Duration::hours(1);
+        let set = &ecstore.pools[0].disk_set[0];
+        let upload = ecstore
+            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("create empty multipart upload");
+        let decoded = base64_simd::URL_SAFE_NO_PAD
+            .decode_to_vec(upload.upload_id.as_bytes())
+            .expect("decode upload ID");
+        let decoded = String::from_utf8(decoded).expect("upload ID must be UTF-8");
+        let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload ID");
+        assert!(
+            super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
+                .await
+                .is_some(),
+            "a known empty upload must match the upper-size rule"
+        );
+        let missing_upload = format!("{}x{}", Uuid::new_v4(), initiated.unix_timestamp_nanos());
+        assert!(
+            super::stale_upload_lifecycle_due(set, &metadata, initiated, &missing_upload, false)
+                .await
+                .is_none(),
+            "an unreadable size must not match the upper-size rule"
+        );
+        metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle(&format!("{size_rule}{independent_rule}")))
+            .await
+            .expect("install size-independent lifecycle rule");
+        assert!(
+            super::stale_upload_lifecycle_due(set, &metadata, initiated, &missing_upload, false)
+                .await
+                .is_some(),
+            "a size-independent rule remains eligible when size is unknown"
+        );
+        ecstore
+            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
+            .await
+            .expect("remove this test's remaining upload from the shared fixture");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn strict_multipart_lifecycle_size_skips_unavailable_participant_and_recovers() {
+        let (_, ecstore) = setup_test_env().await;
+        let bucket = format!("stale-strict-size-{}", Uuid::new_v4().simple());
+        let object = "logs/object";
+        create_test_bucket(&ecstore, &bucket).await;
+        let rule = br#"<LifecycleConfiguration><Rule><ID>size</ID><Status>Enabled</Status><Filter><ObjectSizeLessThan>5</ObjectSizeLessThan></Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule></LifecycleConfiguration>"#;
+        metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule.to_vec())
+            .await
+            .expect("install upper-size rule");
+        let upload = ecstore
+            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("create empty upload");
+        let decoded = base64_simd::URL_SAFE_NO_PAD
+            .decode_to_vec(upload.upload_id.as_bytes())
+            .expect("decode upload ID");
+        let decoded = String::from_utf8(decoded).expect("upload ID is UTF-8");
+        let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload ID");
+        let metadata = HashMap::from([
+            (super::RUSTFS_MULTIPART_BUCKET_KEY.to_owned(), bucket.clone()),
+            (super::RUSTFS_MULTIPART_OBJECT_KEY.to_owned(), object.to_owned()),
+        ]);
+        let initiated = OffsetDateTime::now_utc() - time::Duration::hours(1);
+        let set = &ecstore.pools[0].disk_set[0];
+        assert!(
+            super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
+                .await
+                .is_some()
+        );
+        let disk = set.disks.write().await[3].take();
+        assert!(disk.is_some());
+        let due = super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false).await;
+        set.disks.write().await[3] = disk;
+        assert!(due.is_none(), "unknown size must not match an upper-size rule");
+        assert!(
+            super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
+                .await
+                .is_some()
+        );
+        ecstore
+            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
+            .await
+            .expect("remove this test's remaining upload from the shared fixture");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn strict_multipart_lifecycle_does_not_treat_a_minority_part_as_empty() {
+        let (paths, ecstore) = setup_test_env().await;
+        let bucket = format!("stale-part-minority-{}", Uuid::new_v4().simple());
+        let object = "logs/object";
+        create_test_bucket(&ecstore, &bucket).await;
+        let initiated = OffsetDateTime::now_utc() - time::Duration::hours(1);
+        let upload = ecstore
+            .new_multipart_upload(
+                &bucket,
+                object,
+                &ObjectOptions {
+                    mod_time: Some(initiated),
+                    max_parity: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("create upload");
+        let mut body = PutObjReader::from_vec(vec![7; 6]);
+        ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut body, &ObjectOptions::default())
+            .await
+            .expect("write a known six-byte part");
+        let decoded = base64_simd::URL_SAFE_NO_PAD
+            .decode_to_vec(upload.upload_id.as_bytes())
+            .expect("decode upload ID");
+        let decoded = String::from_utf8(decoded).expect("upload ID is UTF-8");
+        let (_, upload_dir) = decoded.split_once('.').expect("deployment-prefixed upload ID");
+        let upload_path = format!("{}/{upload_dir}", multipart_sha_dir(&bucket, object));
+        let root = paths[0].join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path);
+        let bytes = fs::read(root.join(STORAGE_FORMAT_FILE)).await.expect("read source metadata");
+        let info = super::get_file_info(
+            &bytes,
+            RUSTFS_META_MULTIPART_BUCKET,
+            &upload_path,
+            "",
+            super::FileInfoOpts {
+                data: false,
+                include_free_versions: false,
+                include_part_checksums: false,
+            },
+        )
+        .expect("decode source metadata");
+        let data_dir = info.data_dir.expect("upload data directory").to_string();
+        let metadata = HashMap::from([
+            (super::RUSTFS_MULTIPART_BUCKET_KEY.to_owned(), bucket.clone()),
+            (super::RUSTFS_MULTIPART_OBJECT_KEY.to_owned(), object.to_owned()),
+        ]);
+        let set = &ecstore.pools[0].disk_set[0];
+        let rule = |filter: &str| {
+            format!("<LifecycleConfiguration><Rule><ID>size</ID><Status>Enabled</Status><Filter>{filter}</Filter><AbortIncompleteMultipartUpload><DaysAfterInitiation>0</DaysAfterInitiation></AbortIncompleteMultipartUpload></Rule></LifecycleConfiguration>").into_bytes()
+        };
+        let filters = [
+            "<ObjectSizeLessThan>10</ObjectSizeLessThan>",
+            "<ObjectSizeGreaterThan>5</ObjectSizeGreaterThan>",
+        ];
+        for filter in filters {
+            metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule(filter))
+                .await
+                .expect("install baseline size filter");
+            assert!(
+                super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
+                    .await
+                    .is_some(),
+                "known six-byte part must satisfy {filter} before the fault"
+            );
+        }
+        let mut saved = Vec::new();
+        for path in paths.iter().take(3) {
+            for file in ["part.1", "part.1.meta"] {
+                let path = path
+                    .join(RUSTFS_META_MULTIPART_BUCKET)
+                    .join(&upload_path)
+                    .join(&data_dir)
+                    .join(file);
+                let bytes = fs::read(&path).await.expect("snapshot original part");
+                fs::remove_file(&path).await.expect("simulate a missing part on this disk");
+                saved.push((path, bytes));
+            }
+        }
+        for filter in filters {
+            metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule(filter))
+                .await
+                .expect("install size filter");
+            super::metadata_boundary::get_lifecycle_config(&bucket)
+                .await
+                .expect("size filter must be readable during the fault");
+            assert!(
+                super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false)
+                    .await
+                    .is_none(),
+                "a minority part must not satisfy {filter}"
+            );
+        }
+        cleanup_stale_multipart_uploads_once_at(
+            ecstore.clone(),
+            OffsetDateTime::now_utc(),
+            StdDuration::from_secs(7 * 24 * 60 * 60),
+        )
+        .await;
+        ecstore
+            .get_multipart_info(&bucket, object, &upload.upload_id, &ObjectOptions::default())
+            .await
+            .expect("unknown size must preserve the upload during cleanup");
+        for (path, bytes) in saved {
+            fs::write(path, bytes).await.expect("restore original part");
+        }
+        let size = super::stale_upload_current_size_with_opts(set, &metadata, upload_dir, false).await;
+        assert_eq!(size, Some(6), "restored part must have its original logical size");
+        for filter in filters {
+            metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, rule(filter))
+                .await
+                .expect("restore size filter");
+            let config = super::metadata_boundary::get_lifecycle_config(&bucket)
+                .await
+                .expect("read lifecycle");
+            let due = super::stale_upload_lifecycle_due(set, &metadata, initiated, upload_dir, false).await;
+            assert!(due.is_some(), "restored part must satisfy {filter}; size={size:?}, config={config:?}");
+        }
+        ecstore
+            .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default())
+            .await
+            .expect("remove the restored test upload from the shared fixture");
+    }
+
     #[tokio::test]
     #[serial]
     async fn stale_multipart_cleanup_uses_default_expiry_without_lifecycle() {

crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs

@@ -63,14 +63,7 @@ pub(crate) async fn get_expiry_configs_for_options(
     bucket: &str,
     opts: &crate::object_api::ObjectOptions,
 ) -> Result<LifecycleExpiryConfigs> {
-    let bucket_incarnation_id = match opts
-        .object_lock_config_snapshot
-        .as_ref()
-        .and_then(|snapshot| snapshot.validated_bucket_incarnation_for(api.id, bucket, opts.expected_bucket_incarnation_id))
-    {
-        Some(incarnation_id) => incarnation_id,
-        None => metadata_sys::get_bucket_incarnation_id_for_options_in(&api.ctx, bucket, opts).await?,
-    };
+    let bucket_incarnation_id = metadata_sys::get_bucket_incarnation_id_for_options_in(&api.ctx, bucket, opts).await?;
     let metadata = get_authoritative_metadata(api, bucket, bucket_incarnation_id).await?;
     let table_bucket_enabled = metadata.table_bucket_enabled();
 

crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs

@@ -24,12 +24,16 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_versi
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
 use crate::object_api::ObjectInfo;
+use crate::services::tier::tier::{ERR_TIER_INVALID_CONFIG, TierConfigMgr};
+use crate::storage_api_contracts::namespace::NamespaceLocking as _;
 use crate::storage_api_contracts::{
     bucket::{BucketOperations, BucketOptions},
     list::{ListOperations as _, StorageObjectInfoOrErr, StorageWalkOptions},
 };
 use crate::store::ECStore;
 use rustfs_filemeta::FileInfo;
+use rustfs_lock::error::LockError;
+use rustfs_s3_client::admin_handler_utils::AdminError;
 
 pub const DEFAULT_FREE_VERSION_RECOVERY_LIMIT: usize = 1_000;
 // These are page-wide repair budgets. Applying them per bucket would still let
@@ -130,6 +134,153 @@ pub(super) fn set_recovery_bucket_list_wait_hook(started: Arc<tokio::sync::Notif
     RecoveryBucketListWaitHookGuard
 }
 
+/// A bounded retry queues existing cleanup ownership; it never deletes a tier.
+#[derive(Debug, serde::Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct TierCleanupRetryReport {
+    pub tier_name: String,
+    pub status: TierCleanupRetryStatus,
+    pub scanned: usize,
+    pub enqueued: usize,
+    pub failed: usize,
+    pub scan_truncated: bool,
+}
+
+#[derive(Debug, serde::Serialize, PartialEq, Eq)]
+#[serde(rename_all = "snake_case")]
+pub enum TierCleanupRetryStatus {
+    Accepted,
+    Busy,
+}
+
+impl TierCleanupRetryReport {
+    fn new(tier_name: &str, status: TierCleanupRetryStatus) -> Self {
+        Self {
+            tier_name: tier_name.to_string(),
+            status,
+            scanned: 0,
+            enqueued: 0,
+            failed: 0,
+            scan_truncated: false,
+        }
+    }
+}
+
+fn tier_cleanup_retry_error() -> AdminError {
+    let mut error = ERR_TIER_INVALID_CONFIG.clone();
+    error.code = "XRustFSAdminTierCleanupRetryFailed".to_string();
+    error.status_code = http::StatusCode::SERVICE_UNAVAILABLE;
+    error.message = "Tier cleanup retry could not finish; existing cleanup records are retained".to_string();
+    error
+}
+
+/// Reuses the recovery reader and expiry queue on whichever node receives the
+/// request. This does not depend on a node-local wakeup reaching the recovery
+/// leader. The identity lease and distributed retry lock are acquired in that
+/// order; no manager lock is held during namespace I/O.
+pub async fn retry_tier_free_version_cleanup(
+    api: Arc<ECStore>,
+    tier_name: &str,
+) -> std::result::Result<TierCleanupRetryReport, AdminError> {
+    let cancel = api.ctx.background_cancel_token().unwrap_or_default().child_token();
+    let _cancel_on_drop = cancel.clone().drop_guard();
+    let deadline = tokio::time::sleep(Duration::from_secs(10));
+    tokio::pin!(deadline);
+    let retry = retry_tier_free_version_cleanup_with_cancel(api, tier_name, cancel.clone());
+    tokio::pin!(retry);
+    tokio::select! {
+        biased;
+        _ = cancel.cancelled() => {
+            let _ = retry.await;
+            Err(tier_cleanup_retry_error())
+        }
+        _ = &mut deadline => {
+            cancel.cancel();
+            // Poll cancellation through the recovery reader's walk shutdown.
+            let _ = retry.await;
+            Err(tier_cleanup_retry_error())
+        }
+        result = &mut retry => result,
+    }
+}
+
+pub(super) async fn retry_tier_free_version_cleanup_with_cancel(
+    api: Arc<ECStore>,
+    tier_name: &str,
+    cancel: CancellationToken,
+) -> std::result::Result<TierCleanupRetryReport, AdminError> {
+    let manager = api.tier_config_mgr();
+    let lease = tokio::select! {
+        biased;
+        _ = cancel.cancelled() => return Err(tier_cleanup_retry_error()),
+        result = TierConfigMgr::acquire_operation_lease(&manager, tier_name) => result?,
+    };
+    let lock_name = format!(
+        "tier-cleanup-retry/{}",
+        hex_simd::encode_to_string(lease.backend_identity(), hex_simd::AsciiCase::Lower)
+    );
+    let lock = tokio::select! {
+        biased;
+        _ = cancel.cancelled() => return Err(tier_cleanup_retry_error()),
+        result = api.new_ns_lock(RUSTFS_META_BUCKET, &lock_name) => result.map_err(|_| tier_cleanup_retry_error())?,
+    };
+    let mut guard = tokio::select! {
+        biased;
+        _ = cancel.cancelled() => return Err(tier_cleanup_retry_error()),
+        result = lock.get_write_lock_quiet(Duration::from_millis(100)) => match result {
+            Ok(guard) => guard,
+            Err(LockError::Timeout { .. }) => return Ok(TierCleanupRetryReport::new(tier_name, TierCleanupRetryStatus::Busy)),
+            Err(_) => return Err(tier_cleanup_retry_error()),
+        },
+    };
+    let recovery = retry_tier_free_version_cleanup_page(api, tier_name, cancel.clone());
+    tokio::pin!(recovery);
+    let result = tokio::select! {
+        biased;
+        _ = guard.lock_lost_notified() => {
+            cancel.cancel();
+            let _ = recovery.await;
+            Err(tier_cleanup_retry_error())
+        }
+        result = &mut recovery => result,
+    };
+    guard.release();
+    drop(lease);
+    result
+}
+
+async fn retry_tier_free_version_cleanup_page(
+    api: Arc<ECStore>,
+    tier_name: &str,
+    cancel: CancellationToken,
+) -> std::result::Result<TierCleanupRetryReport, AdminError> {
+    let page = list_tier_free_versions_for_tier(
+        api.clone(),
+        DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
+        None,
+        None,
+        cancel.clone(),
+        recovery_work_budget(DEFAULT_FREE_VERSION_RECOVERY_LIMIT),
+        Some(tier_name),
+    )
+    .await
+    .map_err(|_| tier_cleanup_retry_error())?;
+    let mut report = TierCleanupRetryReport::new(tier_name, TierCleanupRetryStatus::Accepted);
+    report.scanned = page.scanned_entries;
+    report.scan_truncated = page.truncated;
+    for object in page.items {
+        if cancel.is_cancelled() {
+            return Err(tier_cleanup_retry_error());
+        }
+        if enqueue_recovered_free_version_with_cancel(&api, object, &cancel).await {
+            report.enqueued += 1;
+        } else {
+            report.failed += 1;
+        }
+    }
+    Ok(report)
+}
+
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct FreeVersionRecoveryStats {
     pub scanned: usize,
@@ -302,6 +453,18 @@ pub(super) async fn list_tier_free_versions_with_budget(
     object_marker: Option<String>,
     cancel_token: CancellationToken,
     work_budget: RecoveryWorkBudget,
+) -> Result<FreeVersionRecoveryPage> {
+    list_tier_free_versions_for_tier(api, limit, bucket_marker, object_marker, cancel_token, work_budget, None).await
+}
+
+async fn list_tier_free_versions_for_tier(
+    api: Arc<ECStore>,
+    limit: usize,
+    bucket_marker: Option<String>,
+    object_marker: Option<String>,
+    cancel_token: CancellationToken,
+    work_budget: RecoveryWorkBudget,
+    tier_name: Option<&str>,
 ) -> Result<FreeVersionRecoveryPage> {
     let mut page = FreeVersionRecoveryPage {
         items: Vec::new(),
@@ -539,7 +702,7 @@ pub(super) async fn list_tier_free_versions_with_budget(
                 drain_deadline = Some(tokio::time::Instant::now() + BACKGROUND_WALK_SHUTDOWN_TIMEOUT);
                 continue;
             }
-            if is_recoverable_tier_free_version(&oi) {
+            if is_recoverable_tier_free_version_for_tier(&oi, tier_name) {
                 let current_cursor = RecoveryCursor {
                     bucket: bucket.name.clone(),
                     object: oi.name.clone(),
@@ -547,7 +710,15 @@ pub(super) async fn list_tier_free_versions_with_budget(
                 page.items.push(oi);
                 page.next_bucket_marker = Some(current_cursor.bucket.clone());
                 page.next_object_marker = Some(current_cursor.object.clone());
-                if page.items.len() >= limit && truncated_after.is_none() {
+                if page.items.len() >= limit && tier_name.is_some() {
+                    // Manual passes do not expose a continuation cursor, so they can
+                    // stop within one key. Background recovery keeps complete key
+                    // groups to avoid skipping versions when it resumes by key.
+                    page.truncated = true;
+                    cancel.cancel();
+                    draining_after_truncation = true;
+                    drain_deadline = Some(tokio::time::Instant::now() + BACKGROUND_WALK_SHUTDOWN_TIMEOUT);
+                } else if page.items.len() >= limit && truncated_after.is_none() {
                     truncated_after = Some(current_cursor);
                 }
             }
@@ -624,6 +795,10 @@ fn mark_scan_truncated_if_needed(
     }
 }
 
+fn is_recoverable_tier_free_version_for_tier(oi: &ObjectInfo, tier_name: Option<&str>) -> bool {
+    is_recoverable_tier_free_version(oi) && tier_name.is_none_or(|tier| oi.transitioned_object.tier == tier)
+}
+
 fn is_recoverable_tier_free_version(oi: &ObjectInfo) -> bool {
     // A free version transitioned to an unversioned remote tier legally carries an
     // empty `version_id` (see AGENTS.md: a tier version of `None`/`""` means the

crates/ecstore/src/bucket/metadata_sys.rs

@@ -564,12 +564,39 @@ pub(crate) async fn validate_bucket_reconstruction_in(
     Ok(())
 }
 
-/// Validate against disk while retaining an already-held Object Lock fence.
+/// Resolve the authoritative bucket incarnation id for `opts`:
+/// 1. When namespace locking is enabled and `opts.object_lock_config_snapshot` is
+///    present, validate and return the snapshot's incarnation under its held
+///    lifecycle and metadata-transaction fences.
+/// 2. When namespace locking is enabled and `opts.bucket_lifecycle_lock_fence` is
+///    present and not lost, reuse the store's `BucketFenceRegistry` memo if a
+///    registered lifecycle read guard is currently held without lock loss.
+/// 3. Otherwise, acquire the bucket metadata transaction read lock (unless
+///    already held on `opts`) and read the incarnation from disk.
 pub(crate) async fn get_bucket_incarnation_id_for_options_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
     opts: &ObjectOptions,
 ) -> Result<Uuid> {
+    if !ctx.lock_manager().is_disabled() {
+        if let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() {
+            let store = object_store_in(ctx).await?;
+            return snapshot
+                .validated_bucket_incarnation_for(store.id, bucket, opts.expected_bucket_incarnation_id)
+                .ok_or_else(|| {
+                    Error::other("Object Lock snapshot does not hold a valid metadata transaction fence for this bucket")
+                });
+        }
+        if opts
+            .bucket_lifecycle_lock_fence
+            .as_ref()
+            .is_some_and(|fence| !fence.is_lock_lost())
+            && let Some(store) = object_store_if_initialized_in(ctx).await
+            && let Some(incarnation) = store.bucket_fence_registry.validated_incarnation(bucket)
+        {
+            return Ok(incarnation);
+        }
+    }
     let sys = bucket_metadata_sys_of(ctx)?;
     let sys = sys.read().await.clone();
     let guard = acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
@@ -3186,9 +3213,17 @@ pub mod test_support {
     use crate::store::init_local_disks_with_instance_ctx;
 
     pub async fn isolated_store_over_temp_disks() -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
-        let mut dirs = Vec::with_capacity(4);
-        let mut endpoints = Vec::with_capacity(4);
-        for disk_idx in 0..4 {
+        isolated_store_over_temp_disks_with_count(4).await
+    }
+
+    pub async fn isolated_store_over_temp_disks_with_count(disk_count: usize) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
+        assert!(
+            matches!(disk_count, 4 | 16),
+            "isolated store fixture requires a registered logical set geometry"
+        );
+        let mut dirs = Vec::with_capacity(disk_count);
+        let mut endpoints = Vec::with_capacity(disk_count);
+        for disk_idx in 0..disk_count {
             let dir = tempfile::tempdir().expect("tempdir should be created");
             let mut endpoint =
                 Endpoint::try_from(dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
@@ -3201,7 +3236,7 @@ pub mod test_support {
         let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
             legacy: false,
             set_count: 1,
-            drives_per_set: 4,
+            drives_per_set: disk_count,
             endpoints: Endpoints::from(endpoints),
             cmd_line: "metadata-sys-cache-test".to_string(),
             platform: "test".to_string(),

crates/ecstore/src/cluster/rpc/remote_locker.rs

@@ -1074,7 +1074,6 @@ impl RemoteClient {
 #[async_trait]
 impl LockClient for RemoteClient {
     async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
-        info!("remote acquire_exclusive for {}", request.resource);
         let admission = match self.admit_lock_request("lock") {
             Some(admission) => admission,
             None => return Ok(Self::rpc_backoff_failure_response(request)),

crates/ecstore/src/config/com.rs

@@ -488,6 +488,18 @@ where
     Ok(data)
 }
 
+/// Read config bytes and metadata while preserving an existing empty payload.
+pub(crate) async fn read_config_preserve_empty_with_metadata_opts<S>(
+    api: Arc<S>,
+    file: &str,
+    opts: &ObjectOptions,
+) -> Result<(Vec<u8>, ObjectInfo)>
+where
+    S: EcstoreObjectIO,
+{
+    read_config_with_metadata_inner(api, file, opts, true, None).await
+}
+
 pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
 where
     S: ObjectIO<
@@ -2605,6 +2617,15 @@ where
     save_server_config_snapshot(api, cfg, &snapshot).await.map(|_| ())
 }
 
+/// Read config and generation from the same object without taking the transaction write lock.
+pub async fn read_server_config_with_generation(api: Arc<ECStore>) -> Result<(Config, Option<Uuid>)> {
+    match read_config_with_metadata_inner(api.clone(), &server_config_path(), &ObjectOptions::default(), false, None).await {
+        Ok((raw, info)) => Ok((decode_persisted_server_config(&raw)?.merge(), info.data_dir.filter(|id| !id.is_nil()))),
+        Err(Error::ConfigNotFound) => read_config_without_migrate(api).await.map(|config| (config, None)),
+        Err(error) => Err(error),
+    }
+}
+
 /// One serialized read of the persisted server config used as the seed and
 /// compare-and-swap fence for an admin update.
 #[derive(Debug)]

crates/ecstore/src/data_usage/mod.rs

@@ -1493,6 +1493,48 @@ pub async fn load_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<
     project_data_usage_from_backend_cached(store, DataUsageInfo::clone).await
 }
 
+/// Read one bucket's persisted usage without cloning the complete snapshot.
+pub async fn load_bucket_usage_from_backend_cached(
+    store: Arc<ECStore>,
+    bucket: &str,
+    incarnation: uuid::Uuid,
+) -> Result<Option<u64>, Error> {
+    project_data_usage_from_backend_cached(store, |info| certified_bucket_usage(info, bucket, incarnation)).await
+}
+
+fn certified_bucket_usage(info: &DataUsageInfo, bucket: &str, incarnation: uuid::Uuid) -> Option<u64> {
+    if incarnation.is_nil()
+        || !info.is_complete_bucket_usage_snapshot()
+        || info.usage_snapshot_observational
+        || info.scanner_cycle.is_none()
+        || info.scanner_epoch.is_none()
+    {
+        return None;
+    }
+    let pending_count = u64::try_from(info.usage_snapshot_pending_buckets.len()).ok()?;
+    if pending_count != info.usage_snapshot_pending_bucket_count {
+        return None;
+    }
+    match info.usage_snapshot_converged {
+        Some(true) if pending_count == 0 => {}
+        Some(false)
+            if pending_count > 0
+                && info.usage_snapshot_pending_buckets.windows(2).all(|pair| pair[0] < pair[1])
+                && info
+                    .usage_snapshot_pending_buckets
+                    .binary_search_by(|pending| pending.as_str().cmp(bucket))
+                    .is_err() => {}
+        _ => return None,
+    }
+    let usage = info.buckets_usage.get(bucket)?;
+    let certified = usage
+        .bucket_incarnation
+        .as_deref()
+        .and_then(|id| uuid::Uuid::parse_str(id).ok())
+        .filter(|id| !id.is_nil())?;
+    (certified == incarnation).then_some(usage.size)
+}
+
 // Project under the cache guard so scalar readers never copy the snapshot's
 // bucket maps. The owned API shares the same TTL, refresh and invalidation path.
 async fn project_data_usage_from_backend_cached<T>(

crates/ecstore/src/data_usage/tests/snapshot_projection.rs

@@ -129,6 +129,41 @@ fn projected_publication_moves_bucket_maps_into_cache() {
     );
 }
 
+#[test]
+fn certified_bucket_usage_rejects_unproven_and_pending_data() {
+    let id = uuid::Uuid::from_u128(7);
+    let mut info = data_usage_info_for_test("known", 1, 42, SystemTime::UNIX_EPOCH);
+    info.usage_snapshot_converged = Some(true);
+    info.scanner_cycle = Some(1);
+    info.scanner_epoch = Some(1);
+    info.buckets_usage.get_mut("known").unwrap().bucket_incarnation = Some(id.to_string());
+    assert_eq!(certified_bucket_usage(&info, "known", id), Some(42));
+    assert_eq!(certified_bucket_usage(&info, "known", uuid::Uuid::from_u128(8)), None);
+    assert_eq!(certified_bucket_usage(&info, "known", uuid::Uuid::nil()), None);
+    let mut legacy = info.clone();
+    legacy.buckets_usage.get_mut("known").unwrap().bucket_incarnation = None;
+    assert_eq!(certified_bucket_usage(&legacy, "known", id), None);
+    legacy.buckets_usage.get_mut("known").unwrap().bucket_incarnation = Some("corrupt".into());
+    assert_eq!(certified_bucket_usage(&legacy, "known", id), None);
+    let mut pending = info.clone();
+    pending.usage_snapshot_converged = Some(false);
+    pending.usage_snapshot_pending_buckets = vec!["other".into()];
+    pending.usage_snapshot_pending_bucket_count = 1;
+    assert_eq!(certified_bucket_usage(&pending, "known", id), Some(42));
+    pending.usage_snapshot_pending_buckets = vec!["known".into()];
+    assert_eq!(certified_bucket_usage(&pending, "known", id), None);
+    pending.usage_snapshot_pending_bucket_count = 0;
+    assert_eq!(certified_bucket_usage(&pending, "known", id), None);
+    pending.usage_snapshot_pending_buckets = vec!["z".into(), "a".into()];
+    pending.usage_snapshot_pending_bucket_count = 2;
+    assert_eq!(certified_bucket_usage(&pending, "known", id), None);
+    let mut observational = info.clone();
+    observational.usage_snapshot_observational = true;
+    assert_eq!(certified_bucket_usage(&observational, "known", id), None);
+    info.usage_snapshot_converged = None;
+    assert_eq!(certified_bucket_usage(&info, "known", id), None);
+}
+
 #[tokio::test]
 #[serial]
 async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
@@ -140,6 +175,13 @@ async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
     info.buckets_usage.insert("empty".to_string(), BucketUsageInfo::default());
     info.bucket_sizes.insert("empty".to_string(), 0);
     info.buckets_count = 2;
+    info.usage_snapshot_converged = Some(true);
+    info.scanner_cycle = Some(1);
+    info.scanner_epoch = Some(1);
+    let incarnation = uuid::Uuid::from_u128(7);
+    for usage in info.buckets_usage.values_mut() {
+        usage.bucket_incarnation = Some(incarnation.to_string());
+    }
     save_config(
         store.clone(),
         DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -151,11 +193,29 @@ async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
 
     for (bucket, expected) in [("known", Some(42)), ("empty", Some(0)), ("missing", None)] {
         assert_eq!(persisted_bucket_usage(store.clone(), bucket).await, expected);
+        assert_eq!(
+            load_bucket_usage_from_backend_cached(store.clone(), bucket, incarnation)
+                .await
+                .expect("project bucket usage"),
+            expected
+        );
         let owned = load_data_usage_from_backend_cached(store.clone())
             .await
             .expect("owned cached snapshot");
         assert_eq!(expected, owned.buckets_usage.get(bucket).map(|usage| usage.size));
     }
+    assert_eq!(
+        load_bucket_usage_from_backend_cached(store.clone(), "known", uuid::Uuid::from_u128(8))
+            .await
+            .unwrap(),
+        None
+    );
+    assert_eq!(
+        load_bucket_usage_from_backend_cached(store.clone(), "known", uuid::Uuid::nil())
+            .await
+            .unwrap(),
+        None
+    );
     info.buckets_usage.get_mut("known").expect("known bucket").size = 84;
     info.bucket_sizes.insert("known".to_string(), 84);
     save_config(

crates/ecstore/src/diagnostics/batch_read_diagnostics.rs

@@ -0,0 +1,592 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Sampled local batch diagnostics for a separate, default-off build.
+//!
+//! Event fields contain server-owned correlation IDs, sizes and timing, never
+//! object/disk paths or I/O error strings. They are not performance acceptance.
+//! Linux supplies thread CPU time; other targets report it absent.
+
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, OnceLock};
+use std::time::{Instant, SystemTime, UNIX_EPOCH};
+use tracing::{Dispatch, Level};
+
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_LOCAL_BATCH: &str = "local_batch_diagnostic";
+const EVENT_LOCAL_BATCH_DIAGNOSTIC: &str = "local_batch_diagnostic";
+const TARGET: &str = rustfs_config::LOCAL_BATCH_DIAGNOSTIC_LOG_TARGET;
+const ENV_SAMPLE_EVERY: &str = "RUSTFS_DIAGNOSTIC_LOCAL_BATCH_SAMPLE_EVERY";
+const MAX_SAMPLE_EVERY: u64 = 65_536;
+static SAMPLE_EVERY: OnceLock<u64> = OnceLock::new();
+static NEXT_BATCH: AtomicU64 = AtomicU64::new(0);
+
+tokio::task_local! {
+    static REQUEST_ID: Option<Arc<str>>;
+}
+
+fn parse_rate(value: Option<&str>) -> u64 {
+    value
+        .and_then(|value| value.parse::<u64>().ok())
+        .filter(|rate| (1..=MAX_SAMPLE_EVERY).contains(rate))
+        .unwrap_or(0)
+}
+
+fn enabled_rate() -> u64 {
+    *SAMPLE_EVERY.get_or_init(|| parse_rate(std::env::var(ENV_SAMPLE_EVERY).ok().as_deref()))
+}
+
+fn recording_enabled() -> bool {
+    enabled_rate() != 0 && tracing::enabled!(target: TARGET, Level::TRACE)
+}
+
+fn valid_request_id(id: &str) -> bool {
+    !id.is_empty()
+        && id.len() <= 128
+        && id
+            .bytes()
+            .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_'))
+}
+
+/// Retain a bounded server-owned canonical ID only while diagnostics are enabled.
+///
+/// The external S3 boundary must supply its canonical ID, not a client header.
+pub fn request_context(id: &str) -> Option<Arc<str>> {
+    (recording_enabled() && valid_request_id(id)).then(|| Arc::from(id))
+}
+
+/// Poll a request under its diagnostic ID without retaining request fields.
+pub fn in_request_context<T>(id: Option<&Arc<str>>, operation: impl FnOnce() -> T) -> T {
+    REQUEST_ID.sync_scope(id.cloned(), operation)
+}
+
+/// Return the current ID for explicit diagnostic propagation checks.
+pub fn current_request_id() -> Option<Arc<str>> {
+    REQUEST_ID.try_with(Clone::clone).ok().flatten()
+}
+
+/// Capture before spawning the legacy producer; task locals are not inherited.
+pub(crate) fn with_captured_request_context<F: std::future::Future>(future: F) -> impl std::future::Future<Output = F::Output> {
+    REQUEST_ID.scope(current_request_id(), future)
+}
+
+#[derive(Clone, Copy)]
+pub(crate) enum Stage {
+    Execution,
+    Metadata,
+    Open,
+    Allocation,
+    Read,
+}
+
+impl Stage {
+    fn as_str(self) -> &'static str {
+        match self {
+            Self::Execution => "execution",
+            Self::Metadata => "metadata",
+            Self::Open => "open",
+            Self::Allocation => "allocation",
+            Self::Read => "read",
+        }
+    }
+}
+
+fn unix_ns() -> Option<u64> {
+    SystemTime::now().duration_since(UNIX_EPOCH).ok()?.as_nanos().try_into().ok()
+}
+
+#[cfg(target_os = "linux")]
+fn cpu_ns() -> Option<u64> {
+    use rustix::time::{ClockId, DynamicClockId, clock_gettime_dynamic};
+    let time = clock_gettime_dynamic(DynamicClockId::Known(ClockId::ThreadCPUTime)).ok()?;
+    let seconds = u64::try_from(time.tv_sec).ok()?;
+    let nanos = u64::try_from(time.tv_nsec).ok().filter(|value| *value < 1_000_000_000)?;
+    seconds.checked_mul(1_000_000_000)?.checked_add(nanos)
+}
+
+#[cfg(not(target_os = "linux"))]
+fn cpu_ns() -> Option<u64> {
+    None
+}
+
+fn thread_id() -> Option<i32> {
+    #[cfg(target_os = "linux")]
+    {
+        thread_local! { static TID: i32 = rustix::thread::gettid().as_raw_nonzero().get(); }
+        Some(TID.with(|id| *id))
+    }
+    #[cfg(not(target_os = "linux"))]
+    {
+        None
+    }
+}
+
+pub(crate) struct Started {
+    wall: Instant,
+    unix_ns: Option<u64>,
+    cpu_ns: Option<u64>,
+}
+struct Timing {
+    unix_ns: Option<u64>,
+    wall_ns: Option<u64>,
+    cpu_ns: Option<u64>,
+    cpu_status: &'static str,
+}
+
+pub(crate) struct Sample {
+    batch_id: u64,
+    request_id: Option<Arc<str>>,
+    submitted: Instant,
+    submitted_unix_ns: Option<u64>,
+    dispatch: Dispatch,
+}
+
+impl Sample {
+    pub(crate) fn capture() -> Option<Self> {
+        Self::capture_with(enabled_rate(), tracing::enabled!(target: TARGET, Level::TRACE), &NEXT_BATCH)
+    }
+
+    fn capture_with(rate: u64, enabled: bool, sequence: &AtomicU64) -> Option<Self> {
+        if rate == 0 || !enabled {
+            return None;
+        }
+        let batch_id = sequence.fetch_add(1, Ordering::Relaxed);
+        if !batch_id.is_multiple_of(rate) {
+            return None;
+        }
+        Some(Self {
+            batch_id,
+            request_id: current_request_id(),
+            submitted: Instant::now(),
+            submitted_unix_ns: unix_ns(),
+            dispatch: tracing::dispatcher::get_default(Clone::clone),
+        })
+    }
+
+    pub(crate) fn dispatch(&self) -> &Dispatch {
+        &self.dispatch
+    }
+
+    pub(crate) fn start(&self) -> Started {
+        Started {
+            wall: Instant::now(),
+            unix_ns: unix_ns(),
+            cpu_ns: cpu_ns(),
+        }
+    }
+
+    pub(crate) fn worker_started(&self) {
+        self.emit(
+            "queue_wait",
+            None,
+            true,
+            Timing {
+                unix_ns: self.submitted_unix_ns,
+                wall_ns: self.submitted.elapsed().as_nanos().try_into().ok(),
+                cpu_ns: None,
+                cpu_status: "not_applicable",
+            },
+        );
+    }
+
+    pub(crate) fn finish(&self, stage: Stage, shard: Option<usize>, start: Started, success: bool) {
+        let cpu = start
+            .cpu_ns
+            .zip(cpu_ns())
+            .and_then(|(before, after)| after.checked_sub(before));
+        self.emit(
+            stage.as_str(),
+            shard,
+            success,
+            Timing {
+                unix_ns: start.unix_ns,
+                wall_ns: start.wall.elapsed().as_nanos().try_into().ok(),
+                cpu_ns: cpu,
+                cpu_status: if cpu.is_some() { "available" } else { "unavailable" },
+            },
+        );
+    }
+
+    fn emit(&self, stage: &'static str, shard: Option<usize>, success: bool, timing: Timing) {
+        tracing::trace!(target: TARGET, parent: None,
+            event = EVENT_LOCAL_BATCH_DIAGNOSTIC, component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_LOCAL_BATCH, result = if success { "success" } else { "error" },
+            stage, batch_id = self.batch_id, request_id = self.request_id.as_deref(),
+            request_context = if self.request_id.is_some() { "present" } else { "missing" },
+            pid = std::process::id(), tid = thread_id(), shard_index = shard,
+            submitted_unix_ns = self.submitted_unix_ns, start_unix_ns = timing.unix_ns,
+            wall_ns = timing.wall_ns, cpu_ns = timing.cpu_ns, cpu_clock_status = timing.cpu_status,
+            "Sampled local batch stage");
+    }
+
+    pub(crate) fn shard_result(&self, shard: usize, requested: usize, read: usize, success: bool) {
+        tracing::trace!(target: TARGET, parent: None,
+            event = EVENT_LOCAL_BATCH_DIAGNOSTIC, component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_LOCAL_BATCH, result = if success { "success" } else { "error" },
+            stage = "shard_result", batch_id = self.batch_id, request_id = self.request_id.as_deref(),
+            request_context = if self.request_id.is_some() { "present" } else { "missing" },
+            pid = std::process::id(), tid = thread_id(), submitted_unix_ns = self.submitted_unix_ns,
+            shard_index = shard, requested_bytes = requested, read_bytes = read,
+            "Sampled local batch result");
+    }
+}
+
+pub(crate) fn measure<T, E>(
+    sample: Option<&Sample>,
+    stage: Stage,
+    shard: usize,
+    operation: impl FnOnce() -> Result<T, E>,
+) -> Result<T, E> {
+    let Some(sample) = sample else { return operation() };
+    let start = sample.start();
+    let result = operation();
+    sample.finish(stage, Some(shard), start, result.is_ok());
+    result
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use serde_json::{Value, json};
+    use std::collections::BTreeMap;
+    use std::sync::{Barrier, Mutex, mpsc};
+    use tracing::field::{Field, Visit};
+    use tracing_subscriber::{Layer, layer::Context, prelude::*};
+
+    #[derive(Default)]
+    struct Values(BTreeMap<String, Value>);
+    impl Visit for Values {
+        fn record_u64(&mut self, field: &Field, value: u64) {
+            self.0.insert(field.name().into(), json!(value));
+        }
+        fn record_i64(&mut self, field: &Field, value: i64) {
+            self.0.insert(field.name().into(), json!(value));
+        }
+        fn record_bool(&mut self, field: &Field, value: bool) {
+            self.0.insert(field.name().into(), json!(value));
+        }
+        fn record_str(&mut self, field: &Field, value: &str) {
+            self.0.insert(field.name().into(), json!(value));
+        }
+        fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
+            self.0.insert(field.name().into(), json!(format!("{value:?}")));
+        }
+    }
+
+    struct Records {
+        events: Mutex<Vec<BTreeMap<String, Value>>>,
+        block_queue: std::sync::atomic::AtomicBool,
+        queue_started: Mutex<Option<mpsc::Sender<()>>>,
+        release: Barrier,
+    }
+
+    struct Recorder(Arc<Records>);
+    impl<S: tracing::Subscriber> Layer<S> for Recorder {
+        fn on_event(&self, event: &tracing::Event<'_>, _: Context<'_, S>) {
+            if event.metadata().target() != TARGET {
+                return;
+            }
+            assert!(event.is_root(), "diagnostic events must not inherit URI-bearing spans");
+            let mut fields = Values::default();
+            event.record(&mut fields);
+            let queue = fields.0.get("stage") == Some(&json!("queue_wait"));
+            self.0.events.lock().expect("record diagnostic event").push(fields.0);
+            if queue && self.0.block_queue.swap(false, Ordering::SeqCst) {
+                if let Some(sender) = self.0.queue_started.lock().expect("queue sender").take() {
+                    sender.send(()).expect("signal worker start");
+                }
+                self.0.release.wait();
+            }
+        }
+    }
+
+    fn records() -> Arc<Records> {
+        Arc::new(Records {
+            events: Mutex::new(Vec::new()),
+            block_queue: false.into(),
+            queue_started: Mutex::new(None),
+            release: Barrier::new(2),
+        })
+    }
+
+    #[test]
+    fn rate_is_default_off_and_bounded() {
+        for value in [None, Some("0"), Some("invalid"), Some("-1"), Some("65537")] {
+            assert_eq!(parse_rate(value), 0);
+        }
+        assert_eq!(parse_rate(Some("1")), 1);
+        assert_eq!(parse_rate(Some("64")), 64);
+        assert_eq!(parse_rate(Some("65536")), MAX_SAMPLE_EVERY);
+    }
+
+    #[test]
+    fn disabled_or_filtered_capture_does_not_advance_sampling() {
+        let sequence = AtomicU64::new(0);
+        assert!(Sample::capture_with(0, true, &sequence).is_none());
+        assert!(Sample::capture_with(64, false, &sequence).is_none());
+        assert_eq!(sequence.load(Ordering::Relaxed), 0);
+        let result = measure::<(), _>(None, Stage::Read, 0, || Err("original error"));
+        assert_eq!(result, Err("original error"));
+    }
+
+    #[test]
+    fn sampling_keeps_the_registered_interval() {
+        let sequence = AtomicU64::new(0);
+        let selected: Vec<_> = (0..12)
+            .filter_map(|_| Sample::capture_with(4, true, &sequence).map(|sample| sample.batch_id))
+            .collect();
+        assert_eq!(selected, [0, 4, 8]);
+    }
+
+    #[test]
+    fn request_context_is_bounded_and_restored_on_unwind() {
+        assert!(valid_request_id("server-owned-uuid_123"));
+        assert!(valid_request_id(&"a".repeat(128)));
+        for value in ["", "secret/path", "line\nbreak", &"a".repeat(129)] {
+            assert!(!valid_request_id(value));
+        }
+        let outer: Arc<str> = Arc::from("server-outer");
+        in_request_context(Some(&outer), || {
+            in_request_context(None, || assert!(current_request_id().is_none()));
+            assert_eq!(current_request_id().as_deref(), Some("server-outer"));
+            let nested: Arc<str> = Arc::from("server-inner");
+            let result = std::panic::catch_unwind(|| in_request_context(Some(&nested), || panic!("test unwind")));
+            assert!(result.is_err());
+            assert_eq!(current_request_id().as_deref(), Some("server-outer"));
+        });
+        assert!(current_request_id().is_none());
+    }
+
+    #[tokio::test]
+    async fn captured_request_context_crosses_spawn_without_leaking_to_caller() {
+        let original: Arc<str> = Arc::from("server-detached-producer");
+        let task = in_request_context(Some(&original), || {
+            tokio::spawn(with_captured_request_context(async {
+                tokio::task::yield_now().await;
+                current_request_id()
+            }))
+        });
+        assert!(current_request_id().is_none());
+        assert_eq!(task.await.expect("detached diagnostic context").as_deref(), Some(original.as_ref()));
+        assert!(current_request_id().is_none());
+    }
+
+    #[test]
+    fn failed_partial_operation_preserves_error_and_initialized_byte_count() {
+        let _interest = crate::test_tracing::pin_callsite_interest_for_test();
+        let state = records();
+        let subscriber = tracing_subscriber::registry().with(Recorder(Arc::clone(&state)));
+        tracing::subscriber::with_default(subscriber, || {
+            let parent = tracing::info_span!("request", uri = "/must-not-copy-parent-path");
+            let _entered = parent.enter();
+            let sample = Sample::capture_with(1, true, &AtomicU64::new(0)).expect("sampled diagnostic");
+            let mut read = 0;
+            let result = measure::<(), _>(Some(&sample), Stage::Read, 0, || {
+                read = 2;
+                Err("partial read failure")
+            });
+            assert_eq!(result, Err("partial read failure"));
+            sample.shard_result(0, 4, read, false);
+        });
+        let events = state.events.lock().expect("read events");
+        assert!(
+            events
+                .iter()
+                .any(|e| e.get("stage") == Some(&json!("read")) && e.get("result") == Some(&json!("error")))
+        );
+        assert!(
+            events
+                .iter()
+                .any(|e| e.get("read_bytes") == Some(&json!(2)) && e.get("requested_bytes") == Some(&json!(4)))
+        );
+    }
+
+    #[test]
+    fn unavailable_cpu_is_absent_and_not_a_zero_measurement() {
+        let _interest = crate::test_tracing::pin_callsite_interest_for_test();
+        let state = records();
+        let subscriber = tracing_subscriber::registry().with(Recorder(Arc::clone(&state)));
+        tracing::subscriber::with_default(subscriber, || {
+            let sample = Sample::capture_with(1, true, &AtomicU64::new(0)).expect("diagnostic sample");
+            sample.emit(
+                "metadata",
+                Some(0),
+                true,
+                Timing {
+                    unix_ns: None,
+                    wall_ns: Some(1),
+                    cpu_ns: None,
+                    cpu_status: "unavailable",
+                },
+            );
+            sample.emit(
+                "metadata",
+                Some(0),
+                true,
+                Timing {
+                    unix_ns: None,
+                    wall_ns: Some(1),
+                    cpu_ns: Some(0),
+                    cpu_status: "available",
+                },
+            );
+        });
+        let events = state.events.lock().expect("CPU availability events");
+        assert!(!events[0].contains_key("cpu_ns"));
+        assert_eq!(events[0]["cpu_clock_status"], "unavailable");
+        assert_eq!(events[1]["cpu_ns"], 0);
+        assert_eq!(events[1]["cpu_clock_status"], "available");
+    }
+
+    #[cfg(target_os = "linux")]
+    #[test]
+    fn thread_cpu_clock_is_available_without_zero_substitution() {
+        let before = cpu_ns().expect("Linux thread CPU clock available");
+        for value in 0..10_000 {
+            std::hint::black_box(value);
+        }
+        let after = cpu_ns().expect("Linux thread CPU clock remains available");
+        assert!(after >= before);
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn production_batch_diagnostic_survives_caller_cancellation() {
+        const CHILD: &str = "RUSTFS_TEST_BATCH_DIAGNOSTIC_CHILD";
+        if std::env::var_os(CHILD).is_none() {
+            let status = std::process::Command::new(std::env::current_exe().expect("current test executable"))
+                .args([
+                    "--exact",
+                    "diagnostics::batch_read_diagnostics::tests::production_batch_diagnostic_survives_caller_cancellation",
+                    "--nocapture",
+                ])
+                .env(CHILD, "1")
+                .env(ENV_SAMPLE_EVERY, "1")
+                .status()
+                .expect("run isolated diagnostic test");
+            assert!(status.success(), "isolated production diagnostic failed");
+            return;
+        }
+        let _interest = crate::test_tracing::pin_callsite_interest_for_test();
+        let state = records();
+        let recorder = crate::test_metrics::CapturingRecorder::default();
+        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
+        let subscriber = tracing_subscriber::registry().with(Recorder(Arc::clone(&state)));
+        metrics::with_local_recorder(&recorder, || {
+            tracing::subscriber::with_default(subscriber, || {
+                let runtime = tokio::runtime::Builder::new_current_thread()
+                    .enable_all()
+                    .build()
+                    .expect("test runtime");
+                runtime.block_on(async {
+                    let directory = tempfile::tempdir().expect("diagnostic test directory");
+                    let path = directory.path().join("must-not-appear-in-trace");
+                    std::fs::write(&path, b"abcdef").expect("write payload");
+                    let results = REQUEST_ID
+                        .scope(
+                            Some(Arc::from("server-original")),
+                            crate::disk::local::batch_shard_pread(vec![
+                                (path.clone(), 1, 3),
+                                (directory.path().join("missing-secret-path"), 0, 3),
+                                (path.clone(), 0, 0),
+                                (path.clone(), 7, 1),
+                            ]),
+                        )
+                        .await;
+                    assert_eq!(results[0].as_ref().expect("positioned read").as_ref(), b"bcd");
+                    assert!(results[1].is_err());
+                    assert!(results[2].as_ref().expect("zero-length read").is_empty());
+                    assert!(results[3].is_err());
+                    let counter = "rustfs_io_get_object_local_batch_shards_total";
+                    assert_eq!(recorder.counter_value(counter, &[("outcome", "success")]), 2);
+                    assert_eq!(recorder.counter_value(counter, &[("outcome", "error")]), 2);
+                    assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 3);
+                    {
+                        let events = state.events.lock().expect("normal events");
+                        let indices: Vec<_> = events
+                            .iter()
+                            .filter(|event| event.get("stage") == Some(&json!("shard_result")))
+                            .map(|event| event["shard_index"].as_u64().expect("shard index"))
+                            .collect();
+                        assert_eq!(indices, [0, 1, 2, 3], "errors and empty reads retain their input index");
+                        for (index, result) in [(0, "success"), (1, "error"), (2, "success"), (3, "error")] {
+                            assert!(
+                                events.iter().any(|event| {
+                                    event.get("stage") == Some(&json!("shard_result"))
+                                        && event.get("shard_index") == Some(&json!(index))
+                                        && event.get("result") == Some(&json!(result))
+                                }),
+                                "missing diagnostic result for shard {index}"
+                            );
+                        }
+                        for stage in ["open", "metadata", "allocation", "read"] {
+                            assert!(
+                                events.iter().any(|event| {
+                                    event.get("stage") == Some(&json!(stage))
+                                        && event.get("shard_index") == Some(&json!(0))
+                                        && event.get("result") == Some(&json!("success"))
+                                }),
+                                "missing successful {stage} diagnostic"
+                            );
+                        }
+                        assert!(
+                            events.iter().any(|event| {
+                                event.get("stage") == Some(&json!("open"))
+                                    && event.get("shard_index") == Some(&json!(1))
+                                    && event.get("result") == Some(&json!("error"))
+                            }),
+                            "missing shard must emit an open failure"
+                        );
+                        assert!(
+                            !events.iter().any(|event| {
+                                event.get("stage") == Some(&json!("metadata")) && event.get("shard_index") == Some(&json!(1))
+                            }),
+                            "metadata must not be measured when open fails"
+                        );
+                        assert!(
+                            !events
+                                .iter()
+                                .any(|e| e.get("stage") == Some(&json!("read")) && e.get("shard_index") == Some(&json!(2)))
+                        );
+                        assert!(!format!("{events:?}").contains("must-not-appear"));
+                        assert!(!format!("{events:?}").contains("missing-secret"));
+                    }
+                    let (sender, receiver) = mpsc::channel();
+                    *state.queue_started.lock().expect("install queue signal") = Some(sender);
+                    state.block_queue.store(true, Ordering::SeqCst);
+                    let mut future = Box::pin(REQUEST_ID.scope(
+                        Some(Arc::from("server-cancelled")),
+                        crate::disk::local::batch_shard_pread(vec![(path, 0, 6)]),
+                    ));
+                    assert!(futures::poll!(future.as_mut()).is_pending());
+                    receiver
+                        .recv_timeout(std::time::Duration::from_secs(5))
+                        .expect("blocking worker started");
+                    drop(future);
+                    state.release.wait();
+                    let deadline = Instant::now() + std::time::Duration::from_secs(5);
+                    loop {
+                        if state.events.lock().expect("completion events").iter().any(|e| {
+                            e.get("stage") == Some(&json!("execution")) && e.get("request_id") == Some(&json!("server-cancelled"))
+                        }) {
+                            break;
+                        }
+                        assert!(Instant::now() < deadline, "detached worker must complete diagnostic");
+                        tokio::time::sleep(std::time::Duration::from_millis(1)).await;
+                    }
+                });
+            })
+        });
+    }
+}

crates/ecstore/src/diagnostics/mod.rs

@@ -15,5 +15,7 @@
 // #730: diagnostics constants are staged for request-path telemetry migration.
 
 pub(crate) mod admin_server_info;
+#[cfg(feature = "diagnostic-local-batch")]
+pub(crate) mod batch_read_diagnostics;
 pub(crate) mod get;
 pub(crate) mod object_lock;

crates/ecstore/src/disk/local.rs

@@ -3129,6 +3129,9 @@ fn should_fail_after_delete_commit(_root: &Path, _path: &str) -> bool {
 }
 
 fn log_startup_disk_io_error(stage: &str, path: &Path, err: &IoError) {
+    if err.kind() == ErrorKind::PermissionDenied {
+        log_startup_disk_permission_error(stage, path);
+    }
     warn!(
         event = EVENT_DISK_LOCAL_STARTUP_CLEANUP,
         component = LOG_COMPONENT_ECSTORE,
@@ -3143,12 +3146,52 @@ fn log_startup_disk_io_error(stage: &str, path: &Path, err: &IoError) {
     );
 }
 
+fn log_startup_disk_permission_error(stage: &str, path: &Path) {
+    #[cfg(unix)]
+    {
+        use std::os::unix::fs::MetadataExt;
+        let metadata = std::fs::metadata(path).ok();
+        error!(
+            event = EVENT_DISK_LOCAL_STARTUP_CLEANUP,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
+            state = "permission_denied",
+            stage,
+            path = %path.display(),
+            runtime_uid = rustix::process::geteuid().as_raw(),
+            runtime_gid = rustix::process::getegid().as_raw(),
+            path_owner_uid = metadata.as_ref().map(MetadataExt::uid),
+            path_owner_gid = metadata.as_ref().map(MetadataExt::gid),
+            remediation = "Stop RustFS and grant the runtime UID/GID access to the data directory and existing contents. For container upgrades from root to the default user, review ownership and run chown -R 10001:10001 <data-directory> on the host; account for ACLs and user-namespace mappings",
+            "Disk local startup permission denied"
+        );
+    }
+    #[cfg(not(unix))]
+    error!(
+        event = EVENT_DISK_LOCAL_STARTUP_CLEANUP,
+        component = LOG_COMPONENT_ECSTORE,
+        subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
+        state = "permission_denied",
+        stage,
+        path = %path.display(),
+        remediation = "Grant the RustFS service account write and delete access to the data directory and existing contents",
+        "Disk local startup permission denied"
+    );
+}
+
 fn log_startup_disk_error(stage: &str, path: &Path, err: &DiskError) {
     if let DiskError::Io(io_err) = err {
         log_startup_disk_io_error(stage, path, io_err);
         return;
     }
 
+    if matches!(
+        err,
+        DiskError::DiskAccessDenied | DiskError::FileAccessDenied | DiskError::VolumeAccessDenied
+    ) {
+        log_startup_disk_permission_error(stage, path);
+    }
+
     warn!(
         event = EVENT_DISK_LOCAL_STARTUP_CLEANUP,
         component = LOG_COMPONENT_ECSTORE,
@@ -6114,6 +6157,8 @@ impl LocalDisk {
         #[cfg(not(target_os = "linux"))]
         let io_root = Self::open_mount_lease(&root)?;
 
+        Self::check_startup_writable(&publication_root, &root).await?;
+
         ensure_data_usage_layout(&io_root)
             .await
             .map_err(|e| e.narrow_to_disk().unwrap_or_else(DiskError::other))
@@ -6308,6 +6353,21 @@ impl LocalDisk {
             log_startup_disk_error("make_meta_volumes", &disk.root, err);
         })?;
 
+        // Existing directories may be readable but owned by the UID of an old
+        // container. mkdir_all alone does not establish that they are writable.
+        for directory in [
+            RUSTFS_META_BUCKET,
+            ".rustfs.sys/buckets",
+            RUSTFS_META_MULTIPART_BUCKET,
+            ".rustfs.sys/config",
+            RUSTFS_META_TMP_BUCKET,
+            RUSTFS_META_TMP_DELETED_BUCKET,
+            ".rustfs.sys/datausage",
+            ".rustfs.sys/datausage/state",
+        ] {
+            Self::check_startup_writable(&disk.publication_root, &disk.root.join(directory)).await?;
+        }
+
         let (exit_tx, exit_rx) = tokio::sync::broadcast::channel(1);
         disk.exit_signal = Some(exit_tx);
 
@@ -6666,6 +6726,28 @@ impl LocalDisk {
         let md = fs::metadata(&self.io_format_path).await.map_err(to_unformatted_disk_error)?;
         Ok(md)
     }
+    async fn check_startup_writable(publication_root: &os::PublicationRoot, path: &Path) -> Result<()> {
+        let root = publication_root.clone();
+        let probe_path = path.to_path_buf();
+        let probe = tokio::task::spawn_blocking(move || root.check_writable(&probe_path));
+        match timeout(crate::disk::disk_store::get_max_timeout_duration(), probe).await {
+            Ok(result) => result
+                .map_err(DiskError::other)?
+                .inspect_err(|err| log_startup_disk_io_error("check_writable", path, err))
+                .map_err(|err| {
+                    if err.kind() == ErrorKind::PermissionDenied {
+                        DiskError::FileAccessDenied
+                    } else {
+                        DiskError::from(err)
+                    }
+                }),
+            Err(_) => {
+                log_startup_disk_error("check_writable", path, &DiskError::Timeout);
+                Err(DiskError::Timeout)
+            }
+        }
+    }
+
     async fn make_meta_volumes(&self) -> Result<()> {
         let buckets = format!("{RUSTFS_META_BUCKET}/{BUCKET_META_PREFIX}");
         let multipart = format!("{}/{}", RUSTFS_META_BUCKET, "multipart");
@@ -8450,6 +8532,23 @@ impl LocalDisk {
                     return Err(std::io::Error::other("metadata read worker cancelled"));
                 }
                 let _io_permit = io_permit;
+                #[cfg(all(any(test, feature = "test-util"), not(windows)))]
+                {
+                    let opened = root.open_readonly(&path);
+                    if opened.is_ok() {
+                        os::prepared_publication_test_hooks::run(
+                            os::prepared_publication_test_hooks::Stage::MetadataReadAfterOpen,
+                            &path,
+                        );
+                    }
+                    let result = read_fn(opened);
+                    os::prepared_publication_test_hooks::run(
+                        os::prepared_publication_test_hooks::Stage::MetadataReadAfterRead,
+                        &path,
+                    );
+                    Ok(result)
+                }
+                #[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
                 Ok(read_fn(root.open_readonly(&path)))
             })
             .await
@@ -11311,10 +11410,39 @@ pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize,
     let n = requests.len();
     let metrics = rustfs_io_metrics::LocalBatchReadMetrics::capture();
     let submitted = metrics.as_ref().map(|_| std::time::Instant::now());
+    #[cfg(feature = "diagnostic-local-batch")]
+    let diagnostic = crate::diagnostics::batch_read_diagnostics::Sample::capture();
     tokio::task::spawn_blocking(move || {
         use rustfs_io_metrics::LocalBatchReadStage;
-        use std::os::unix::fs::FileExt;
+        use std::os::unix::fs::{FileExt, OpenOptionsExt};
 
+        // The disabled build expands directly to the original operation.
+        macro_rules! diagnostic_stage {
+            ($stage:ident, $index:expr, $operation:expr) => {{
+                #[cfg(feature = "diagnostic-local-batch")]
+                {
+                    crate::diagnostics::batch_read_diagnostics::measure(
+                        diagnostic.as_ref(),
+                        crate::diagnostics::batch_read_diagnostics::Stage::$stage,
+                        $index,
+                        $operation,
+                    )
+                }
+                #[cfg(not(feature = "diagnostic-local-batch"))]
+                {
+                    $operation()
+                }
+            }};
+        }
+        #[cfg(feature = "diagnostic-local-batch")]
+        let _dispatch = diagnostic
+            .as_ref()
+            .map(|sample| tracing::dispatcher::set_default(sample.dispatch()));
+        #[cfg(feature = "diagnostic-local-batch")]
+        let execution = diagnostic.as_ref().map(|sample| {
+            sample.worker_started();
+            sample.start()
+        });
         let execution_started = metrics.as_ref().map(|_| std::time::Instant::now());
         if let (Some(metrics), Some(submitted)) = (&metrics, submitted) {
             metrics.record_stage(LocalBatchReadStage::QueueWait, true, submitted.elapsed());
@@ -11322,41 +11450,71 @@ pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize,
         }
         let mut results = Vec::with_capacity(n);
         for (file_path, offset, length) in requests {
-            let mut bytes_read = 0;
+            #[cfg(feature = "diagnostic-local-batch")]
+            let _index = results.len();
+            let mut total = 0usize;
             let r = (|| -> Result<Bytes> {
-                let meta = measure_local_batch_stage(metrics.as_deref(), LocalBatchReadStage::Metadata, || {
-                    std::fs::metadata(&file_path).map_err(DiskError::from)
-                })?;
                 let end = offset.checked_add(length).ok_or(DiskError::FileCorrupt)?;
-                if meta.len() < u64::try_from(end).unwrap_or(u64::MAX) {
+                let end = u64::try_from(end).map_err(|_| DiskError::FileCorrupt)?;
+                // Bind validation and reads to one descriptor. Nonblocking open
+                // lets corrupt non-regular entries fail without waiting on a FIFO.
+                let file = diagnostic_stage!(Open, _index, || measure_local_batch_stage(
+                    metrics.as_deref(),
+                    LocalBatchReadStage::Open,
+                    || std::fs::OpenOptions::new()
+                        .read(true)
+                        .custom_flags(libc::O_NONBLOCK)
+                        .open(&file_path)
+                        .map_err(DiskError::from),
+                ))?;
+                let meta = diagnostic_stage!(Metadata, _index, || measure_local_batch_stage(
+                    metrics.as_deref(),
+                    LocalBatchReadStage::Metadata,
+                    || file.metadata().map_err(DiskError::from),
+                ))?;
+                if !meta.is_file() || meta.len() < end {
                     return Err(DiskError::FileCorrupt);
                 }
 
-                let file = measure_local_batch_stage(metrics.as_deref(), LocalBatchReadStage::Open, || {
-                    std::fs::File::open(&file_path).map_err(DiskError::from)
-                })?;
-                let mut buf = vec![0u8; length];
+                let mut buf = diagnostic_stage!(Allocation, _index, || Ok::<_, DiskError>(vec![0u8; length]))?;
                 if length > 0 {
-                    measure_local_batch_stage(metrics.as_deref(), LocalBatchReadStage::Read, || {
-                        while bytes_read < length {
-                            let nbytes = file
-                                .read_at(&mut buf[bytes_read..], u64::try_from(offset + bytes_read).unwrap_or(u64::MAX))
-                                .map_err(DiskError::from)?;
-                            if nbytes == 0 {
-                                return Err(DiskError::FileCorrupt);
+                    diagnostic_stage!(Read, _index, || measure_local_batch_stage(
+                        metrics.as_deref(),
+                        LocalBatchReadStage::Read,
+                        || {
+                            while total < length {
+                                let nbytes = file
+                                    .read_at(&mut buf[total..], u64::try_from(offset + total).unwrap_or(u64::MAX))
+                                    .map_err(DiskError::from)?;
+                                if nbytes == 0 {
+                                    return Err(DiskError::FileCorrupt);
+                                }
+                                total += nbytes;
                             }
-                            bytes_read += nbytes;
+                            Ok::<_, DiskError>(())
                         }
-                        Ok(())
-                    })?;
+                    ))?;
                 }
                 Ok(Bytes::from(buf))
             })();
+            #[cfg(feature = "diagnostic-local-batch")]
+            if let Some(sample) = &diagnostic {
+                sample.shard_result(_index, length, total, r.is_ok());
+            }
             if let Some(metrics) = &metrics {
-                metrics.record_shard(r.is_ok(), length, bytes_read);
+                metrics.record_shard(r.is_ok(), length, total);
             }
             results.push(r);
         }
+        #[cfg(feature = "diagnostic-local-batch")]
+        if let (Some(sample), Some(started)) = (&diagnostic, execution) {
+            sample.finish(
+                crate::diagnostics::batch_read_diagnostics::Stage::Execution,
+                None,
+                started,
+                results.iter().all(Result::is_ok),
+            );
+        }
         if let (Some(metrics), Some(started)) = (&metrics, execution_started) {
             metrics.record_stage(LocalBatchReadStage::Execution, results.iter().all(Result::is_ok), started.elapsed());
         }
@@ -13736,6 +13894,239 @@ mod test {
         }
     }
 
+    #[tokio::test]
+    async fn startup_writability_preserves_existing_data_and_removes_probes() {
+        let dir = tempfile::tempdir().expect("create disk directory");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse endpoint");
+        let disk = LocalDisk::new(&endpoint, false).await.expect("initialize writable disk");
+        let metadata = dir.path().join(".rustfs.sys/config/existing");
+        fs::write(&metadata, b"existing metadata")
+            .await
+            .expect("write existing metadata");
+        drop(disk);
+
+        LocalDisk::new(&endpoint, true).await.expect("reopen writable disk");
+        assert_eq!(fs::read(metadata).await.expect("read preserved metadata"), b"existing metadata");
+        for directory in [
+            "",
+            RUSTFS_META_BUCKET,
+            ".rustfs.sys/buckets",
+            RUSTFS_META_MULTIPART_BUCKET,
+            ".rustfs.sys/config",
+            RUSTFS_META_TMP_BUCKET,
+            RUSTFS_META_TMP_DELETED_BUCKET,
+            ".rustfs.sys/datausage",
+            ".rustfs.sys/datausage/state",
+        ] {
+            for entry in std::fs::read_dir(dir.path().join(directory)).expect("list probed directory") {
+                assert!(
+                    !entry
+                        .expect("read directory entry")
+                        .file_name()
+                        .to_string_lossy()
+                        .starts_with(".rustfs-write-check-"),
+                    "startup probes must be deleted"
+                );
+            }
+        }
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn startup_writability_rejects_unwritable_paths() {
+        use std::os::unix::fs::{MetadataExt, PermissionsExt};
+        use std::os::unix::process::CommandExt;
+
+        const CHILD_ROOT: &str = "RUSTFS_TEST_STARTUP_WRITABILITY_ROOT";
+        const CHILD_BLOCKED: &str = "RUSTFS_TEST_STARTUP_WRITABILITY_BLOCKED";
+        const CHILD_HEALTHY: &str = "RUSTFS_TEST_STARTUP_WRITABILITY_HEALTHY";
+        if let Some(root) = std::env::var_os(CHILD_ROOT) {
+            let root = PathBuf::from(root);
+            let logs = CapturedLogs::default();
+            let subscriber = tracing_subscriber::fmt()
+                .with_ansi(false)
+                .without_time()
+                .with_writer(logs.clone())
+                .finish();
+            let _guard = tracing::subscriber::set_default(subscriber);
+            assert_eq!(
+                std::fs::read(root.join(".rustfs.sys/config/existing")).expect("old metadata must remain readable"),
+                b"existing metadata"
+            );
+            let runtime = tokio::runtime::Builder::new_current_thread()
+                .enable_all()
+                .build()
+                .expect("create runtime");
+            let endpoint = Endpoint::try_from(root.to_str().expect("UTF-8 disk path")).expect("parse endpoint");
+            let result = runtime.block_on(LocalDisk::new(&endpoint, false));
+            assert!(
+                matches!(result, Err(DiskError::FileAccessDenied)),
+                "unwritable disk must not initialize: {result:?}"
+            );
+            crate::store::init_format::check_disk_fatal_errs(&[Some(DiskError::FileAccessDenied)])
+                .expect_err("a deployment with no writable disks must fail startup");
+            runtime.block_on(async {
+                use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
+
+                let healthy_root = std::env::var(CHILD_HEALTHY).expect("healthy disk directory");
+                let mut bad = endpoint.clone();
+                bad.set_pool_index(0);
+                bad.set_set_index(0);
+                bad.set_disk_index(0);
+                let mut healthy = Endpoint::try_from(healthy_root.as_str()).expect("parse healthy endpoint");
+                healthy.set_pool_index(0);
+                healthy.set_set_index(0);
+                healthy.set_disk_index(1);
+                let endpoints = Endpoints::from(vec![bad, healthy]);
+                let pools = EndpointServerPools(vec![PoolEndpoints {
+                    legacy: false,
+                    set_count: 1,
+                    drives_per_set: 2,
+                    endpoints: endpoints.clone(),
+                    cmd_line: "startup-writability".into(),
+                    platform: "test".into(),
+                }]);
+                let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
+                let options = crate::disk::DiskOption {
+                    cleanup: false,
+                    health_check: false,
+                };
+                runtime_sources::initialize_local_disk_maps(&ctx, pools, &options)
+                    .await
+                    .expect("an unwritable disk must not abort early registry setup");
+                let entries = ctx.local_disk_map();
+                let entries = entries.read().await;
+                assert!(
+                    entries
+                        .get(&endpoint.to_string())
+                        .expect("bad disk slot must remain registered")
+                        .is_none()
+                );
+                assert_eq!(entries.values().filter(|disk| disk.is_some()).count(), 1);
+                let slots = ctx.local_disk_set_drives();
+                let slots = slots.read().await;
+                assert!(slots[0][0][0].is_none());
+                assert!(slots[0][0][1].is_some());
+                drop(slots);
+                drop(entries);
+                let (disks, errors) = crate::store::init_format::init_disks(&endpoints, &options).await;
+                assert!(disks[0].is_none());
+                assert!(disks[1].is_some());
+                assert_eq!(errors[0], Some(DiskError::FileAccessDenied));
+                assert!(errors[1].is_none());
+                crate::store::init_format::check_disk_fatal_errs(&errors)
+                    .expect("mixed disk availability must reach the existing quorum decision");
+            });
+            let output = logs.contents();
+            for expected in [
+                "permission_denied",
+                "PermissionDenied",
+                "raw_os_error",
+                "runtime_uid",
+                "runtime_gid",
+                "path_owner_uid",
+                "path_owner_gid",
+                "chown -R 10001:10001",
+                &std::env::var(CHILD_BLOCKED).expect("blocked directory"),
+            ] {
+                assert!(output.contains(expected), "missing diagnostic {expected:?}: {output}");
+            }
+            let blocked = std::env::var(CHILD_BLOCKED).expect("blocked directory");
+            let owner = std::fs::metadata(blocked).expect("read blocked directory ownership");
+            for expected in [
+                format!("runtime_uid={}", rustix::process::geteuid().as_raw()),
+                format!("runtime_gid={}", rustix::process::getegid().as_raw()),
+                format!("path_owner_uid={}", owner.uid()),
+                format!("path_owner_gid={}", owner.gid()),
+            ] {
+                assert!(output.contains(&expected), "incorrect ownership diagnostic {expected:?}: {output}");
+            }
+            return;
+        }
+
+        let runtime = tokio::runtime::Runtime::new().expect("create fixture runtime");
+        for blocked in ["", ".rustfs.sys/config", ".rustfs.sys/datausage/state"] {
+            let dir = tempfile::tempdir().expect("create disk directory");
+            let healthy_dir = tempfile::tempdir().expect("create healthy disk directory");
+            let root = dir.path().canonicalize().expect("resolve fixture directory");
+            let endpoint = Endpoint::try_from(root.to_str().expect("UTF-8 disk path")).expect("parse endpoint");
+            runtime
+                .block_on(LocalDisk::new(&endpoint, false))
+                .expect("initialize old disk layout");
+            std::fs::write(root.join(".rustfs.sys/config/existing"), b"existing metadata").expect("write readable old metadata");
+            std::fs::set_permissions(root.join(".rustfs.sys/config/existing"), std::fs::Permissions::from_mode(0o644))
+                .expect("allow old metadata reads by the upgraded identity");
+            // Root CI still exercises real EACCES by running the child as an
+            // unprivileged user against root-owned directories.
+            let root_test = rustix::process::geteuid().is_root();
+            if root_test {
+                std::fs::set_permissions(healthy_dir.path(), std::fs::Permissions::from_mode(0o777))
+                    .expect("allow the upgraded identity to initialize the healthy disk");
+                for directory in [
+                    "",
+                    RUSTFS_META_BUCKET,
+                    ".rustfs.sys/buckets",
+                    RUSTFS_META_MULTIPART_BUCKET,
+                    ".rustfs.sys/config",
+                    RUSTFS_META_TMP_BUCKET,
+                    RUSTFS_META_TMP_DELETED_BUCKET,
+                    ".rustfs.sys/datausage",
+                    ".rustfs.sys/datausage/state",
+                ] {
+                    std::fs::set_permissions(root.join(directory), std::fs::Permissions::from_mode(0o777))
+                        .expect("make unblocked fixture paths accessible to child");
+                }
+            }
+            let blocked_path = if blocked.is_empty() {
+                root.clone()
+            } else {
+                root.join(blocked)
+            };
+            std::fs::set_permissions(&blocked_path, std::fs::Permissions::from_mode(0o555)).expect("deny directory writes");
+            let mut child = std::process::Command::new(std::env::current_exe().expect("find test executable"));
+            child
+                .args([
+                    "--exact",
+                    "disk::local::test::startup_writability_rejects_unwritable_paths",
+                    "--nocapture",
+                ])
+                .env(CHILD_ROOT, &root)
+                .env(CHILD_HEALTHY, healthy_dir.path())
+                .env(CHILD_BLOCKED, &blocked_path);
+            if root_test {
+                child.gid(65534).uid(65534);
+            }
+            let output = child.output().expect("run unprivileged startup regression");
+            std::fs::set_permissions(&blocked_path, std::fs::Permissions::from_mode(0o755)).expect("restore fixture permissions");
+            assert!(
+                output.status.success(),
+                "blocked path {blocked_path:?}: stdout={} stderr={}",
+                String::from_utf8_lossy(&output.stdout),
+                String::from_utf8_lossy(&output.stderr)
+            );
+            runtime
+                .block_on(LocalDisk::new(&endpoint, false))
+                .expect("restoring permissions must allow startup again");
+        }
+    }
+
+    #[cfg(target_os = "linux")]
+    #[tokio::test]
+    async fn startup_writability_rejects_symlinked_internal_directory() {
+        let dir = tempfile::tempdir().expect("create disk directory");
+        let outside = tempfile::tempdir().expect("create outside directory");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse endpoint");
+        LocalDisk::new(&endpoint, false).await.expect("initialize disk");
+        let config = dir.path().join(".rustfs.sys/config");
+        std::fs::remove_dir(&config).expect("remove empty config directory");
+        std::os::unix::fs::symlink(outside.path(), &config).expect("replace config directory with symlink");
+        assert!(
+            LocalDisk::new(&endpoint, false).await.is_err(),
+            "probe must not follow an internal symlink"
+        );
+        assert_eq!(std::fs::read_dir(outside.path()).expect("list outside directory").count(), 0);
+    }
+
     pub(super) fn test_file_info(name: &str, version_id: Uuid, data_dir: Option<Uuid>, data: Option<Bytes>) -> FileInfo {
         let size = data
             .as_ref()
@@ -31028,7 +31419,7 @@ mod test {
         assert_eq!(recorder.counter_value(counter, &[("outcome", "error")]), 2);
         assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 9);
         let stages = "rustfs_io_get_object_local_batch_stage_seconds";
-        for (stage, success, errors) in [("metadata", 3, 1), ("open", 2, 0), ("read", 1, 0), ("execution", 0, 1)] {
+        for (stage, success, errors) in [("open", 3, 1), ("metadata", 3, 0), ("read", 1, 0), ("execution", 0, 1)] {
             assert_eq!(
                 recorder
                     .histogram_values(stages, &[("stage", stage), ("outcome", "success")])
@@ -31097,6 +31488,70 @@ mod test {
         assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 4);
     }
 
+    #[cfg(unix)]
+    #[tokio::test]
+    async fn test_batch_shard_pread_rejects_nonregular_files_without_blocking() {
+        use std::os::unix::fs::OpenOptionsExt;
+
+        let dir = tempfile::tempdir().expect("nonregular shard fixture");
+        let fifo = dir.path().join("fifo");
+        assert!(
+            std::process::Command::new("mkfifo")
+                .arg(&fifo)
+                .status()
+                .expect("run POSIX mkfifo for the shard fixture")
+                .success()
+        );
+        let good = dir.path().join("good");
+        std::fs::write(&good, b"data").expect("write regular shard");
+        let mut batch = Box::pin(batch_shard_pread(vec![
+            (dir.path().to_owned(), 0, 0),
+            (fifo.clone(), 0, 0),
+            (fifo.clone(), 0, 4),
+            (good, 0, 4),
+        ]));
+        let results = match tokio::time::timeout(Duration::from_secs(2), &mut batch).await {
+            Ok(results) => results,
+            Err(_) => {
+                // Release a regressed blocking open before failing, so runtime
+                // shutdown does not leave the test waiting on its worker.
+                let _release = std::fs::OpenOptions::new()
+                    .read(true)
+                    .write(true)
+                    .custom_flags(libc::O_NONBLOCK)
+                    .open(&fifo)
+                    .expect("release blocked FIFO reader");
+                batch.await;
+                panic!("a nonregular shard must not block its batch");
+            }
+        };
+        for result in &results[..3] {
+            assert!(matches!(result, Err(DiskError::FileCorrupt)), "nonregular shard: {result:?}");
+        }
+        assert_eq!(results[3].as_ref().expect("later regular shard remains readable").as_ref(), b"data");
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    async fn test_batch_shard_pread_validates_descriptor_ranges() {
+        let dir = tempfile::tempdir().expect("range validation fixture");
+        let path = dir.path().join("shard");
+        std::fs::write(&path, b"01234567").expect("write range fixture");
+        let results = batch_shard_pread(vec![
+            (path.clone(), 3, 5),
+            (path.clone(), 8, 0),
+            (path.clone(), 8, 1),
+            (path.clone(), 9, 0),
+            (path, usize::MAX, 1),
+        ])
+        .await;
+        assert_eq!(results[0].as_ref().expect("range ending at EOF").as_ref(), b"34567");
+        assert!(results[1].as_ref().expect("empty range at EOF").is_empty());
+        for result in &results[2..] {
+            assert!(matches!(result, Err(DiskError::FileCorrupt)), "invalid range: {result:?}");
+        }
+    }
+
     #[cfg(unix)]
     #[tokio::test]
     async fn test_batch_shard_pread_basic() {

crates/ecstore/src/disk/os.rs

@@ -333,6 +333,8 @@ pub(crate) mod prepared_publication_test_hooks {
         PreparedRename,
         Rename,
         Remove,
+        MetadataReadAfterOpen,
+        MetadataReadAfterRead,
         #[cfg(test)]
         Rollback,
         #[cfg(test)]
@@ -457,6 +459,8 @@ pub enum LocalPublicationStage {
     PreparedRename,
     Rename,
     Remove,
+    MetadataReadAfterOpen,
+    MetadataReadAfterRead,
 }
 
 #[cfg(all(feature = "test-util", not(windows)))]
@@ -469,6 +473,8 @@ impl LocalPublicationPause {
             LocalPublicationStage::PreparedRename => prepared_publication_test_hooks::Stage::PreparedRename,
             LocalPublicationStage::Rename => prepared_publication_test_hooks::Stage::Rename,
             LocalPublicationStage::Remove => prepared_publication_test_hooks::Stage::Remove,
+            LocalPublicationStage::MetadataReadAfterOpen => prepared_publication_test_hooks::Stage::MetadataReadAfterOpen,
+            LocalPublicationStage::MetadataReadAfterRead => prepared_publication_test_hooks::Stage::MetadataReadAfterRead,
         };
         let (entered_tx, entered) = oneshot::channel();
         let (release, release_rx) = std::sync::mpsc::channel::<()>();
@@ -4705,6 +4711,27 @@ impl PublicationRoot {
         std::fs::File::open(path)
     }
 
+    pub(crate) fn check_writable(&self, path: &Path) -> io::Result<()> {
+        #[cfg(target_os = "linux")]
+        let directory = self.open_directory(path)?;
+        #[cfg(target_os = "linux")]
+        let probe_directory = {
+            use std::os::fd::AsRawFd;
+            PathBuf::from(format!("/proc/self/fd/{}/.", directory.as_raw_fd()))
+        };
+        #[cfg(not(target_os = "linux"))]
+        let probe_directory = path;
+
+        // Keep the Linux parent handle pinned through deletion, including when
+        // a startup timeout detaches the blocking task.
+        let mut probe = tempfile::Builder::new()
+            .prefix(".rustfs-write-check-")
+            .tempfile_in(probe_directory)?;
+        let write_result = std::io::Write::write_all(probe.as_file_mut(), b"rustfs");
+        let delete_result = probe.close();
+        write_result.and(delete_result)
+    }
+
     pub(crate) fn create_truncate(&self, path: &Path) -> io::Result<std::fs::File> {
         #[cfg(target_os = "linux")]
         {

crates/ecstore/src/object_api/types.rs

@@ -959,6 +959,10 @@ pub struct ObjectOptions {
     /// Persisted bucket incarnation observed before authorization.
     pub expected_bucket_incarnation_id: Option<Uuid>,
     pub no_lock: bool,
+    /// Require complete participant observations for internal multipart size decisions.
+    /// This transient flag is never derived from client input or stored on disk.
+    #[doc(hidden)]
+    pub strict_multipart_part_sizes: bool,
     /// A genuine target write guard; loss fences alone do not authorize a borrowed write.
     #[doc(hidden)]
     pub write_lock_context: Option<WriteLockContext>,

crates/ecstore/src/runtime/sources.rs

@@ -599,7 +599,17 @@ pub(crate) async fn initialize_local_disk_maps(
                 continue;
             }
 
-            let disk = new_disk(ep, opt).await?;
+            let disk = match new_disk(ep, opt).await {
+                Ok(disk) => disk,
+                Err(crate::disk::error::DiskError::FileAccessDenied) => {
+                    // LocalDisk already logged the filesystem cause and repair
+                    // hint. Keep this slot unavailable so ECStore initialization
+                    // can apply its existing fatal-error and quorum rules.
+                    global_local_disk_map.insert(ep.to_string(), None);
+                    continue;
+                }
+                Err(err) => return Err(err.into()),
+            };
             if let crate::disk::Disk::Local(wrapper) = disk.as_ref() {
                 wrapper.get_disk().bind_multipart_proof_authority(&multipart_authority)?;
             }

crates/ecstore/src/set_disk/core/io_primitives.rs

@@ -280,7 +280,7 @@ pub(in crate::set_disk) fn resolve_read_part_from_responses(
             continue;
         }
 
-        if !parts[part_idx].etag.is_empty() {
+        if !parts[part_idx].etag.is_empty() && parts[part_idx].error.is_none() {
             present_count += 1;
             let part = &parts[part_idx];
             let key = (part.etag.as_str(), part.number, part.size, part.actual_size, part.integrity.as_ref());
@@ -603,6 +603,10 @@ pub(in crate::set_disk) async fn submit_read_repair_heal_with_submitter(
     request.object_version_id = version_id.filter(|value| !value.is_empty()).map(str::to_string);
     // A degraded read's surviving minority is not a recreate instruction; the
     // locked storage read decides whether the target is absent (backlog#2900).
+    // Reader failures include checksum damage that metadata-only healing cannot detect.
+    if matches!(reason, "missing_shards" | "decode_error") {
+        request.scan_mode = Some(rustfs_heal_contracts::heal_channel::HealScanMode::Deep);
+    }
 
     let request_id = request.id.clone();
     let bucket = bucket.to_string();
@@ -1118,9 +1122,9 @@ pub(in crate::set_disk) fn fill_deferred_bitrot_readers(
     // offset can be advanced to the failing stripe. A parity reader that was
     // opened eagerly during setup is pinned at the stripe-0 stream position
     // and could never be engaged mid-object, so swap it for an unopened
-    // deferred reader carrying a stripe handle. The eager open already proved
-    // the shard is reachable; no shard bytes were read from it, and the
-    // ready/error bookkeeping that quorum decisions rely on is left untouched.
+    // deferred reader carrying a stripe handle. The eager setup already proved
+    // the shard is reachable; the ready/error bookkeeping that quorum decisions
+    // rely on is left untouched.
     // Gate off (default): keep the eagerly opened parity readers exactly as
     // before — the lockstep path reads them on every stripe.
     if !demand_bound_lockstep {
@@ -1217,8 +1221,8 @@ pub(in crate::set_disk) fn record_bitrot_reader_setup_strategy(
 }
 
 /// When all online shards are local and mmap-read is enabled, materialize
-/// shard bytes with one `batch_shard_pread` instead of per-shard
-/// `spawn_blocking` via `open_disk_reader`.
+/// shard bytes in batches instead of per-shard `spawn_blocking` via
+/// `open_disk_reader`. Demand-bound reads add parity batches only as needed.
 #[cfg(unix)]
 #[allow(clippy::too_many_arguments)]
 async fn try_create_bitrot_readers_via_batch_pread(
@@ -1232,6 +1236,10 @@ async fn try_create_bitrot_readers_via_batch_pread(
     shard_size: usize,
     checksum_algo: HashAlgorithm,
     skip_verify_bitrot: bool,
+    data_shards: usize,
+    parity_shards: usize,
+    mode: BitrotReaderSetupMode,
+    strategy: BitrotReaderSetupStrategy,
 ) -> Option<BitrotReaderSetup> {
     use crate::disk::local::batch_shard_pread;
     use rustfs_io_metrics::{LocalBatchReadDecision, record_local_batch_read_decision};
@@ -1275,42 +1283,62 @@ async fn try_create_bitrot_readers_via_batch_pread(
         return None;
     }
 
-    let requests: Vec<_> = batch_items.iter().map(|(_, p, off, len)| (p.clone(), *off, *len)).collect();
-    #[cfg(test)]
-    for (disk_index, _, _, _) in &batch_items {
-        disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, *disk_index);
-    }
-    record_decision(LocalBatchReadDecision::Selected);
-    let batch_results = batch_shard_pread(requests).await;
-
+    // Legacy lockstep consumes every shard, so keep its coalesced batch even
+    // when setup prefers data blocks. Only demand-bound decode can leave the
+    // parity readers unopened; preading them here would discard their bytes
+    // when fill_deferred_bitrot_readers installs stripe-aligned reserves.
+    let data_first = strategy != BitrotReaderSetupStrategy::AllShards
+        && crate::erasure::coding::decode::get_lockstep_data_shards_only_enabled();
     let mut setup = BitrotReaderSetup::new(disks.len());
-    for (i, (idx, _, _, _)) in batch_items.iter().enumerate() {
-        setup.mark_scheduled(*idx);
-        match &batch_results[i] {
-            Ok(bytes) => {
-                let reader = BitrotReader::new(
-                    ShardReader::InMemory(Cursor::new(bytes.clone())),
-                    shard_size,
-                    checksum_algo.clone(),
-                    skip_verify_bitrot,
-                );
-                setup.apply_reader_result(*idx, Ok(Some(reader)));
-            }
-            Err(e) => {
-                setup.apply_reader_result(*idx, Err(e.clone()));
-            }
-        }
-    }
-
-    for (idx, disk_op) in disks.iter().enumerate() {
-        if setup.scheduled[idx] {
-            continue;
-        }
-        setup.mark_scheduled(idx);
-        if disk_op.is_none() {
+    for (idx, disk) in disks.iter().enumerate() {
+        if disk.is_none() {
+            setup.mark_scheduled(idx);
             setup.apply_reader_result(idx, Ok(None));
         }
     }
+    let mut batch_size = if data_first {
+        batch_items.iter().take_while(|(idx, _, _, _)| *idx < data_shards).count()
+    } else {
+        batch_items.len()
+    };
+    let mut pending = batch_items.into_iter();
+    record_decision(LocalBatchReadDecision::Selected);
+    loop {
+        let mut indices = Vec::with_capacity(batch_size.min(pending.len()));
+        let requests: Vec<_> = pending
+            .by_ref()
+            .take(batch_size)
+            .map(|(idx, path, offset, length)| {
+                setup.mark_scheduled(idx);
+                indices.push(idx);
+                #[cfg(test)]
+                disk_call_counters::record(object, disk_call_counters::KIND_DISK_SHARD_OPEN, idx);
+                (path, offset, length)
+            })
+            .collect();
+        if !requests.is_empty() {
+            for (idx, result) in indices.into_iter().zip(batch_shard_pread(requests).await) {
+                let result = result.map(|bytes| {
+                    Some(BitrotReader::new(
+                        ShardReader::InMemory(Cursor::new(bytes)),
+                        shard_size,
+                        checksum_algo.clone(),
+                        skip_verify_bitrot,
+                    ))
+                });
+                setup.apply_reader_result(idx, result);
+            }
+        }
+        if !data_first || setup.has_setup_quorum(data_shards, parity_shards, mode) || pending.len() == 0 {
+            break;
+        }
+        // A missing/corrupt data file needs replacement parity. Reuse the
+        // canonical target, including the extra verification source required
+        // for reconstruction, and retry only the outstanding deficit.
+        batch_size = setup
+            .scheduling_target(data_shards, parity_shards, mode)
+            .saturating_sub(setup.available_shards());
+    }
 
     Some(setup)
 }
@@ -1328,6 +1356,10 @@ async fn try_create_bitrot_readers_via_batch_pread(
     _shard_size: usize,
     _checksum_algo: HashAlgorithm,
     _skip_verify_bitrot: bool,
+    _data_shards: usize,
+    _parity_shards: usize,
+    _mode: BitrotReaderSetupMode,
+    _strategy: BitrotReaderSetupStrategy,
 ) -> Option<BitrotReaderSetup> {
     rustfs_io_metrics::record_local_batch_read_decision(rustfs_io_metrics::LocalBatchReadDecision::UnsupportedPlatform, None);
     None
@@ -1517,6 +1549,10 @@ pub(in crate::set_disk) async fn create_bitrot_readers_until_quorum_with_prefere
             shard_size,
             checksum_algo.clone(),
             skip_verify_bitrot,
+            data_shards,
+            parity_shards,
+            mode,
+            strategy,
         )
         .await
     {
@@ -7889,6 +7925,329 @@ mod tests {
         (dirs, disks)
     }
 
+    #[cfg(unix)]
+    async fn local_batch_quorum_fixture(
+        bucket: &str,
+        object: &str,
+    ) -> (Vec<TempDir>, Vec<Option<DiskStore>>, Vec<FileInfo>, Vec<u8>) {
+        let payload: Vec<_> = (0..1024 * 1024)
+            .map(|index| u8::try_from(index % 251).expect("bounded byte"))
+            .collect();
+        let (dirs, disks) = call_counter_local_disks(bucket, 8).await;
+        let mut files = inline_metadata_fanout_fileinfos_with_geometry(bucket, object, &payload, false, 4, 4).await;
+        for (file, disk) in files.iter_mut().zip(&disks) {
+            let path = format!("{object}/{}/part.1", file.data_dir.expect("fixture data directory"));
+            disk.as_ref()
+                .expect("fixture disk")
+                .write_all(bucket, &path, file.data.take().expect("encoded fixture shard"))
+                .await
+                .expect("external shard fixture should write");
+        }
+        (dirs, disks, files, payload)
+    }
+
+    #[cfg(unix)]
+    async fn local_batch_quorum_setup(
+        files: &[FileInfo],
+        disks: &[Option<DiskStore>],
+        bucket: &str,
+        object: &str,
+        mode: BitrotReaderSetupMode,
+    ) -> BitrotReaderSetup {
+        let erasure = coding::Erasure::new_with_options(4, 4, files[0].erasure.block_size, false);
+        create_bitrot_readers_until_quorum_with_preference(
+            files,
+            disks,
+            bucket,
+            object,
+            1,
+            0,
+            erasure.shard_file_offset(0, 1024 * 1024, 1024 * 1024),
+            erasure.shard_size(),
+            HashAlgorithm::HighwayHash256S,
+            false,
+            true,
+            4,
+            4,
+            mode,
+            false,
+            None,
+            None,
+        )
+        .await
+    }
+
+    #[cfg(unix)]
+    async fn local_batch_quorum_decode(setup: BitrotReaderSetup, block_size: usize) -> (Vec<u8>, Option<std::io::Error>) {
+        let erasure = coding::Erasure::new_with_options(4, 4, block_size, false);
+        let mut output = Vec::new();
+        let (_, error) = erasure
+            .decode_with_stripe_handles_and_reopeners(
+                &mut output,
+                setup.readers,
+                0,
+                1024 * 1024,
+                1024 * 1024,
+                None,
+                setup.deferred_stripe_handles,
+                setup.deferred_reopeners,
+            )
+            .await;
+        (output, error)
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_batch_quorum_healthy_reads_only_data_when_demand_bound() {
+        temp_env::async_with_vars(
+            [
+                (ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
+                ("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("false")),
+            ],
+            crate::erasure::coding::decode::with_decode_read_policy(
+                crate::erasure::coding::decode::DecodeReadPolicy::DemandBound,
+                async {
+                    let bucket = "batch-data-only";
+                    let object = "healthy-one-mib";
+                    let (_dirs, disks, files, payload) = local_batch_quorum_fixture(bucket, object).await;
+                    let calls = disk_call_counters::observe(object);
+                    let setup = local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::ReadQuorum).await;
+                    assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
+                    assert_eq!(setup.available_shards(), 4);
+                    assert_eq!(setup.deferred_shards(), 4);
+                    for index in 4..8 {
+                        assert_eq!(calls.for_disk(disk_call_counters::KIND_DISK_SHARD_OPEN, index), 0);
+                        assert!(!setup.attempted[index], "parity must stay unopened until needed");
+                    }
+                    let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
+                    assert!(error.is_none(), "healthy decode failed: {error:?}");
+                    assert_eq!(output, payload);
+                    assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
+                },
+            ),
+        )
+        .await;
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_batch_quorum_preserves_legacy_and_all_shards_override() {
+        for (data_first, demand_bound) in [("true", "false"), ("false", "true")] {
+            temp_env::async_with_vars(
+                [
+                    (ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some(data_first)),
+                    ("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some(demand_bound)),
+                ],
+                async {
+                    let bucket = "batch-legacy";
+                    let object = "all-shards";
+                    let (_dirs, disks, files, payload) = local_batch_quorum_fixture(bucket, object).await;
+                    let calls = disk_call_counters::observe(object);
+                    let setup = local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::ReadQuorum).await;
+                    assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 8);
+                    assert_eq!(setup.available_shards(), 8);
+                    let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
+                    assert!(error.is_none(), "legacy decode failed: {error:?}");
+                    assert_eq!(output, payload);
+                },
+            )
+            .await;
+        }
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_batch_quorum_recovers_missing_truncated_and_bitrot_data() {
+        temp_env::async_with_vars(
+            [
+                (ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
+                ("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
+            ],
+            async {
+                for fault in ["missing", "truncated", "bitrot"] {
+                    let bucket = "batch-degraded";
+                    let object = format!("degraded-{fault}");
+                    let (dirs, disks, files, payload) = local_batch_quorum_fixture(bucket, &object).await;
+                    let path = dirs[0]
+                        .path()
+                        .join(bucket)
+                        .join(&object)
+                        .join(files[0].data_dir.expect("fixture directory").to_string())
+                        .join("part.1");
+                    match fault {
+                        "missing" => tokio::fs::remove_file(&path).await.expect("remove data shard"),
+                        "truncated" => tokio::fs::write(&path, b"short").await.expect("truncate data shard"),
+                        _ => {
+                            let mut bytes = tokio::fs::read(&path).await.expect("read bitrot fixture");
+                            bytes[32] ^= 0xff;
+                            tokio::fs::write(&path, bytes)
+                                .await
+                                .expect("corrupt data without changing length");
+                        }
+                    }
+                    let setup =
+                        local_batch_quorum_setup(&files, &disks, bucket, &object, BitrotReaderSetupMode::ReadQuorum).await;
+                    assert_eq!(setup.available_shards(), 4);
+                    if fault == "bitrot" {
+                        assert_eq!(setup.attempted_shards(), 4, "bitrot is detected on decode, not pread");
+                    } else {
+                        assert_eq!(setup.attempted_shards(), 5, "setup reads only one replacement parity shard");
+                    }
+                    let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
+                    assert!(error.is_none(), "{fault} should recover: {error:?}");
+                    assert_eq!(output, payload, "{fault} must reconstruct the exact body");
+                }
+            },
+        )
+        .await;
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_batch_quorum_keeps_extra_reconstruction_verification_source() {
+        temp_env::async_with_vars(
+            [
+                (ENV_RUSTFS_GET_CODEC_STREAMING_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
+                ("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
+            ],
+            async {
+                let bucket = "batch-verification";
+                let object = "missing-data";
+                let (_dirs, mut disks, files, payload) = local_batch_quorum_fixture(bucket, object).await;
+                disks[0] = None;
+                let calls = disk_call_counters::observe(object);
+                let setup =
+                    local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::VerifyReconstruction).await;
+                assert_eq!(setup.available_shards(), 5, "degraded reconstruction needs quorum plus verification");
+                assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 5);
+                assert!(setup.ready[4] && setup.ready[5], "two parity sources must be available");
+                assert!(!setup.attempted[6] && !setup.attempted[7], "unneeded parity stays deferred");
+                let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
+                assert!(error.is_none(), "verified reconstruction failed: {error:?}");
+                assert_eq!(output, payload);
+            },
+        )
+        .await;
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_batch_quorum_fails_without_read_quorum() {
+        temp_env::async_with_vars(
+            [
+                (ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
+                ("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
+            ],
+            async {
+                let bucket = "batch-no-quorum";
+                let object = "all-data-offline";
+                let (_dirs, mut disks, files, _payload) = local_batch_quorum_fixture(bucket, object).await;
+                disks[..5].fill(None);
+                let setup = local_batch_quorum_setup(&files, &disks, bucket, object, BitrotReaderSetupMode::ReadQuorum).await;
+                assert_eq!(setup.available_shards(), 3);
+                assert!(!setup.has_setup_quorum(4, 4, BitrotReaderSetupMode::ReadQuorum));
+                assert_eq!(setup.deferred_shards(), 0);
+                let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
+                assert!(error.is_some(), "insufficient shards must fail closed");
+                assert!(output.is_empty(), "no unverified bytes may be emitted");
+            },
+        )
+        .await;
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_batch_quorum_recovers_late_bitrot_after_range_offset() {
+        temp_env::async_with_vars(
+            [
+                (ENV_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP, Some("true")),
+                ("RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE", Some("true")),
+            ],
+            async {
+                let bucket = "batch-ranged";
+                let object = "late-bitrot";
+                let (_dirs, disks) = call_counter_local_disks(bucket, 8).await;
+                let erasure = coding::Erasure::new_with_options(4, 4, 64, false);
+                let payload: Vec<u8> = (0..192).collect();
+                let mut shards = vec![Vec::new(); 8];
+                for block in payload.chunks(64) {
+                    for (index, shard) in erasure
+                        .encode_data(block)
+                        .expect("encode range stripe")
+                        .into_iter()
+                        .enumerate()
+                    {
+                        let mut writer = coding::BitrotWriterWrapper::new(
+                            coding::CustomWriter::new_inline_buffer(),
+                            erasure.shard_size(),
+                            HashAlgorithm::HighwayHash256S,
+                        );
+                        writer.write(&shard).await.expect("write range stripe");
+                        writer.shutdown().await.expect("finish range stripe");
+                        shards[index].extend(writer.into_inline_data().expect("range stripe bytes"));
+                    }
+                }
+                // Corrupt the third stripe; the first requested stripe is healthy.
+                shards[0][2 * (32 + erasure.shard_size()) + 32] ^= 0xff;
+                let mut files = vec![FileInfo::new(object, 4, 4); 8];
+                let data_dir = Uuid::new_v4();
+                for ((file, disk), bytes) in files.iter_mut().zip(&disks).zip(shards) {
+                    file.data_dir = Some(data_dir);
+                    disk.as_ref()
+                        .expect("fixture disk")
+                        .write_all(bucket, &format!("{object}/{data_dir}/part.1"), Bytes::from(bytes))
+                        .await
+                        .expect("write multi-stripe shard");
+                }
+                let calls = disk_call_counters::observe(object);
+                let setup = create_bitrot_readers_until_quorum_with_preference(
+                    &files,
+                    &disks,
+                    bucket,
+                    object,
+                    1,
+                    erasure.shard_size(),
+                    2 * erasure.shard_size(),
+                    erasure.shard_size(),
+                    HashAlgorithm::HighwayHash256S,
+                    false,
+                    true,
+                    4,
+                    4,
+                    BitrotReaderSetupMode::ReadQuorum,
+                    false,
+                    None,
+                    None,
+                )
+                .await;
+                assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
+                let mut output = Vec::new();
+                let (written, error) = erasure
+                    .decode_with_stripe_handles_and_reopeners(
+                        &mut output,
+                        setup.readers,
+                        64,
+                        128,
+                        payload.len(),
+                        None,
+                        setup.deferred_stripe_handles,
+                        setup.deferred_reopeners,
+                    )
+                    .await;
+                assert!(error.is_none(), "late bitrot must recover from stripe-aligned parity: {error:?}");
+                assert_eq!(written, 128);
+                assert_eq!(output, payload[64..]);
+            },
+        )
+        .await;
+    }
+
     #[cfg(unix)]
     #[test]
     fn local_batch_read_decisions_follow_the_actual_reader_setup_entry() {
@@ -7957,6 +8316,10 @@ mod tests {
                             4,
                             HashAlgorithm::None,
                             false,
+                            1,
+                            1,
+                            BitrotReaderSetupMode::ReadQuorum,
+                            BitrotReaderSetupStrategy::AllShards,
                         )
                         .await
                         .is_none(),
@@ -7974,6 +8337,10 @@ mod tests {
                             4,
                             HashAlgorithm::HighwayHash256S,
                             false,
+                            1,
+                            1,
+                            BitrotReaderSetupMode::ReadQuorum,
+                            BitrotReaderSetupStrategy::AllShards,
                         )
                         .await
                         .is_none(),
@@ -7993,6 +8360,10 @@ mod tests {
                             4,
                             HashAlgorithm::None,
                             false,
+                            1,
+                            1,
+                            BitrotReaderSetupMode::ReadQuorum,
+                            BitrotReaderSetupStrategy::AllShards,
                         )
                         .await
                         .is_none()
@@ -8020,6 +8391,10 @@ mod tests {
                             4,
                             HashAlgorithm::None,
                             false,
+                            1,
+                            1,
+                            BitrotReaderSetupMode::ReadQuorum,
+                            BitrotReaderSetupStrategy::AllShards,
                         )
                         .await
                         .is_none(),
@@ -8037,6 +8412,10 @@ mod tests {
                             4,
                             HashAlgorithm::None,
                             false,
+                            1,
+                            1,
+                            BitrotReaderSetupMode::ReadQuorum,
+                            BitrotReaderSetupStrategy::AllShards,
                         )
                         .await
                         .is_none()
@@ -8054,6 +8433,10 @@ mod tests {
                             4,
                             HashAlgorithm::None,
                             false,
+                            1,
+                            1,
+                            BitrotReaderSetupMode::ReadQuorum,
+                            BitrotReaderSetupStrategy::AllShards,
                         )
                         .await
                         .is_none()
@@ -12823,6 +13206,20 @@ mod tests {
         assert_eq!(err, DiskError::ErasureReadQuorum);
     }
 
+    #[test]
+    fn resolve_read_part_does_not_count_error_response_with_etag_as_a_vote() {
+        let healthy = read_part_test_part(1, "same-etag");
+        let mut failed = healthy.clone();
+        failed.error = Some("metadata read failed".to_owned());
+        let responses = vec![Some(vec![healthy.clone()]), Some(vec![healthy]), Some(vec![failed])];
+        let err = resolve_read_part_from_responses("bucket", "upload/part.1.meta", 1, 0, 1, &responses, 3)
+            .expect_err("an error-bearing response must not supply the third vote");
+        assert_eq!(err, DiskError::ErasureReadQuorum);
+        let part = resolve_read_part_from_responses("bucket", "upload/part.1.meta", 1, 0, 1, &responses, 2)
+            .expect("the two healthy matching responses remain a valid quorum");
+        assert!(part.error.is_none());
+    }
+
     #[test]
     fn resolve_read_part_accepts_alternate_missing_error_markers() {
         let responses = vec![
@@ -13827,6 +14224,61 @@ mod tests {
         release_read_repair_heal_reservation(&key).await;
     }
 
+    #[tokio::test]
+    async fn read_repair_reader_failure_requests_deep_scan_for_the_exact_version() {
+        fn captured() -> &'static std::sync::Mutex<Option<rustfs_heal_contracts::heal_channel::HealChannelRequest>> {
+            static REQUEST: OnceLock<std::sync::Mutex<Option<rustfs_heal_contracts::heal_channel::HealChannelRequest>>> =
+                OnceLock::new();
+            REQUEST.get_or_init(|| std::sync::Mutex::new(None))
+        }
+        fn capture(request: rustfs_heal_contracts::heal_channel::HealChannelRequest) -> ReadRepairAdmissionFuture {
+            *captured().lock().expect("read-repair request capture must be available") = Some(request);
+            Box::pin(async { ReadRepairAdmissionOutcome::Response(HealAdmissionResult::Accepted) })
+        }
+
+        for (reason, expected_scan) in [
+            ("missing_shards", Some(rustfs_heal_contracts::heal_channel::HealScanMode::Deep)),
+            ("decode_error", Some(rustfs_heal_contracts::heal_channel::HealScanMode::Deep)),
+            ("metadata_read_error", None),
+        ] {
+            let object = format!("reader-failure-{}", Uuid::new_v4());
+            let version = Uuid::new_v4().to_string();
+            submit_read_repair_heal_with_submitter(
+                ReadRepairHealSubmission {
+                    bucket: "reader-failure",
+                    object: &object,
+                    version_id: Some(&version),
+                    pool_index: 1,
+                    set_index: 2,
+                    part_number: Some(1),
+                    reason,
+                    mrf_intent: None,
+                },
+                capture,
+            )
+            .await;
+            let request = tokio::time::timeout(Duration::from_secs(5), async {
+                loop {
+                    if let Some(request) = captured().lock().expect("captured request must be readable").take() {
+                        break request;
+                    }
+                    tokio::task::yield_now().await;
+                }
+            })
+            .await
+            .expect("the actual asynchronous admission must be captured");
+            assert_eq!(request.scan_mode, expected_scan, "reader damage needs payload verification");
+            assert_eq!(request.source, HealRequestSource::ReadRepair);
+            assert_eq!(request.bucket, "reader-failure");
+            assert_eq!(request.object_prefix.as_deref(), Some(object.as_str()));
+            assert_eq!(request.object_version_id.as_deref(), Some(version.as_str()));
+            assert_eq!((request.pool_index, request.set_index), (Some(1), Some(2)));
+            assert_eq!(request.recreate_missing, None, "storage evidence decides absence");
+            let key = ReadRepairHealCacheKey::new("reader-failure", &object, Some(&version), 1, 2);
+            release_read_repair_heal_reservation(&key).await;
+        }
+    }
+
     // ========================================================================
     // backlog#898 — groups A/C: pure old-data-dir cleanup classification.
     // ========================================================================

crates/ecstore/src/set_disk/core/multipart_source.rs

@@ -338,17 +338,26 @@ fn abort_slot(
     Ok(())
 }
 
-fn abort_certificate_cohort(
+pub(in crate::set_disk) fn abort_certificate_cohort(
     certificate: &MultipartAbortCertificate,
     ids: &[Option<Uuid>],
     authority: &ProofVerificationContext,
 ) -> Result<()> {
     certificate.verify_with_authority(authority)?;
-    if ids.len() != certificate.disk_ids.len()
-        || ids.iter().copied().collect::<Option<Vec<_>>>().as_ref() != Some(&certificate.disk_ids)
-    {
+    if ids.len() != certificate.disk_ids.len() {
         return Err(DiskError::FileCorrupt);
     }
+    let mut missing = false;
+    for (actual, expected) in ids.iter().zip(&certificate.disk_ids) {
+        match actual {
+            Some(actual) if actual != expected => return Err(DiskError::FileCorrupt),
+            Some(_) => {}
+            None => missing = true,
+        }
+    }
+    if missing {
+        return Err(DiskError::ErasureWriteQuorum);
+    }
     Ok(())
 }
 

crates/ecstore/src/set_disk/mod.rs

@@ -283,6 +283,23 @@ fn ensure_decommission_tier_free_version_commit_fence(bucket: &str, object: &str
     Ok(())
 }
 
+/// Names a `bucket/object` key prefix whose best-effort rename-tail repairs
+/// are not submitted. A root-heal fixture arms it so the explicit admin heal
+/// remains the only owner of objects written while their target was offline.
+#[cfg(feature = "e2e-test-hooks")]
+const RENAME_TAIL_HEAL_SUPPRESS_PREFIX_ENV: &str = "RUSTFS_E2E_SUPPRESS_RENAME_TAIL_HEAL_PREFIX";
+
+#[cfg(feature = "e2e-test-hooks")]
+fn rename_tail_heal_suppressed_for_e2e(bucket: &str, object: Option<&str>) -> bool {
+    let Some(prefix) = std::env::var(RENAME_TAIL_HEAL_SUPPRESS_PREFIX_ENV)
+        .ok()
+        .filter(|prefix| !prefix.is_empty())
+    else {
+        return false;
+    };
+    object.is_some_and(|object| format!("{bucket}/{object}").starts_with(&prefix))
+}
+
 impl SetDisks {
     pub(super) async fn require_current_restore_operation_id(
         &self,
@@ -486,10 +503,16 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
     rustfs_utils::http::metadata_compat::remove_str(metadata, SUFFIX_BUCKET_INCARNATION_ID);
 }
 
-fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
+pub(crate) fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
     metadata.keys().any(|key| is_object_encryption_marker(key))
 }
 
+/// Reject ambiguous part sizes unless the upload records a transformation.
+pub(crate) fn multipart_part_logical_size(stored_size: usize, actual_size: i64, transformed: bool) -> Option<usize> {
+    let actual_size = usize::try_from(actual_size).ok()?;
+    (transformed || actual_size == stored_size).then_some(actual_size)
+}
+
 /// Per-set memoized capacity dirty scope.
 ///
 /// The disk endpoint/path identity of each slot in a set is immutable for the
@@ -889,6 +912,8 @@ pub(crate) use ops::heal::AbsenceProofRequest;
 #[cfg(test)]
 pub(crate) use ops::heal::DanglingDeleteFailure;
 pub(crate) use ops::heal::HealedObjectAbsence;
+#[cfg(test)]
+pub(crate) use ops::heal::ReadRepairPauseScope;
 
 #[cfg(test)]
 pub(crate) use ops::hermetic_set_disks_isolated;
@@ -4245,14 +4270,23 @@ pub(super) struct MultipartWriteQuorumContext<'a> {
     part_number: Option<usize>,
 }
 
+fn record_multipart_write_quorum_failure(
+    stage: &'static str,
+    errs: &[Option<DiskError>],
+    write_quorum: usize,
+) -> crate::disk::error_reduce::WriteQuorumFailureSummary {
+    let summary = build_write_quorum_failure_summary(errs, OBJECT_OP_IGNORED_ERRS, write_quorum);
+    runtime_sources::record_erasure_write_quorum_failure(stage, summary.dominant_error_label);
+    summary
+}
+
 fn log_multipart_write_quorum_failure(
     context: MultipartWriteQuorumContext<'_>,
     errs: &[Option<DiskError>],
     write_quorum: usize,
     returned_error: &DiskError,
 ) {
-    let summary = build_write_quorum_failure_summary(errs, OBJECT_OP_IGNORED_ERRS, write_quorum);
-    runtime_sources::record_erasure_write_quorum_failure(context.stage, summary.dominant_error_label);
+    let summary = record_multipart_write_quorum_failure(context.stage, errs, write_quorum);
     warn!(
         target: "rustfs_ecstore::set_disk",
         event = EVENT_SET_DISK_MULTIPART,
@@ -5134,6 +5168,11 @@ impl SetDisks {
             }
         }
 
+        #[cfg(feature = "e2e-test-hooks")]
+        if rename_tail_heal_suppressed_for_e2e(&request.bucket, request.object_prefix.as_deref()) {
+            return Ok(());
+        }
+
         let _ = rustfs_heal_contracts::heal_channel::send_heal_request(request).await;
         Ok(())
     }

crates/ecstore/src/set_disk/ops/delete_recovery.rs

@@ -40,9 +40,35 @@ pub(crate) mod test_hooks {
     use std::sync::{Arc, Mutex};
 
     type Hook = Arc<dyn Fn() -> BoxFuture<'static, ()> + Send + Sync>;
+    type TransactionLockHook = Arc<dyn Fn(&rustfs_lock::NamespaceLockGuard) + Send + Sync>;
 
     static BEFORE_GC_FINALIZE: Mutex<Vec<(String, String, Hook)>> = Mutex::new(Vec::new());
     static CLEANUP_ATTEMPTS: Mutex<Vec<(String, String)>> = Mutex::new(Vec::new());
+    static AFTER_TRANSACTION_LOCK: Mutex<Vec<(String, String, TransactionLockHook)>> = Mutex::new(Vec::new());
+
+    pub(crate) fn after_transaction_lock_acquired(bucket: &str, object: &str, hook: TransactionLockHook) {
+        AFTER_TRANSACTION_LOCK
+            .lock()
+            .expect("transaction lock hooks")
+            .push((bucket.to_owned(), object.to_owned(), hook));
+    }
+
+    pub(in crate::set_disk) fn run_after_transaction_lock_acquired(
+        bucket: &str,
+        object: &str,
+        guard: &rustfs_lock::NamespaceLockGuard,
+    ) {
+        let hook = {
+            let mut hooks = AFTER_TRANSACTION_LOCK.lock().expect("transaction lock hooks");
+            hooks
+                .iter()
+                .position(|(hook_bucket, hook_object, _)| hook_bucket == bucket && hook_object == object)
+                .map(|index| hooks.remove(index).2)
+        };
+        if let Some(hook) = hook {
+            hook(guard);
+        }
+    }
 
     pub(super) fn record_cleanup_attempt(bucket: &str, object: &str) {
         CLEANUP_ATTEMPTS
@@ -112,6 +138,7 @@ pub(crate) struct DeleteRecoveryContext<'a> {
     /// Only the locked heal path may repair a live marker-creation source.
     pub repair_marker_source: bool,
     pub namespace_owner: Option<Arc<dyn Send + Sync>>,
+    pub transaction_guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
     pub recovery_ticket: Option<crate::store::delete_recovery_runtime::DeleteRecoveryTicket>,
     pub published_hint: Option<(Uuid, &'a [Result<DeleteRecoveryInspection>])>,
     pub can_commit: &'a (dyn Fn() -> bool + Sync),
@@ -125,6 +152,7 @@ impl<'a> DeleteRecoveryContext<'a> {
             explicit_delete: false,
             repair_marker_source: false,
             namespace_owner: None,
+            transaction_guard: None,
             recovery_ticket: None,
             published_hint: None,
             can_commit,
@@ -373,9 +401,22 @@ impl SetDisks {
         write_quorum: usize,
         mut context: DeleteRecoveryContext<'_>,
     ) -> Result<()> {
-        if context.namespace_owner.is_none() {
-            context.namespace_owner = Some(self.ctx.begin_namespace_commit());
-        }
+        let transaction_guard = match context.transaction_guard.take() {
+            Some(guard) => guard,
+            None => self
+                .acquire_delete_transaction_lock(bucket, object)
+                .await
+                .map_err(DiskError::other)?,
+        };
+        let outer_can_commit = context.can_commit;
+        let can_commit = || outer_can_commit() && !transaction_guard.is_lock_lost();
+        context.can_commit = &can_commit;
+        context.transaction_guard = Some(Arc::clone(&transaction_guard));
+        let owner = context
+            .namespace_owner
+            .take()
+            .unwrap_or_else(|| self.ctx.begin_namespace_commit());
+        context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
         let can_commit = context.can_commit;
         let states = inspect(disks, bucket, object, transaction).await;
         let untouched_absence = |state: &Result<DeleteRecoveryInspection>| matches!(state, Ok(state) if state.state == DeleteRecoveryState::Absent && state.current_digest.is_none());
@@ -547,8 +588,25 @@ impl SetDisks {
         &self,
         bucket: &str,
         object: &str,
-        context: DeleteRecoveryContext<'_>,
+        mut context: DeleteRecoveryContext<'_>,
     ) -> Result<DeleteRecoverySummary> {
+        let transaction_guard = match context.transaction_guard.take() {
+            Some(guard) => guard,
+            None => self
+                .acquire_delete_transaction_lock(bucket, object)
+                .await
+                .map_err(DiskError::other)?,
+        };
+        let outer_can_commit = context.can_commit;
+        let can_commit = || outer_can_commit() && !transaction_guard.is_lock_lost();
+        context.can_commit = &can_commit;
+        if !context.dry_run {
+            let owner = context
+                .namespace_owner
+                .take()
+                .unwrap_or_else(|| self.ctx.begin_namespace_commit());
+            context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
+        }
         let DeleteRecoveryContext {
             incarnation,
             dry_run,
@@ -558,6 +616,7 @@ impl SetDisks {
             published_hint,
             can_commit,
             recovery_ticket: _,
+            transaction_guard: _,
         } = context;
         let disks = self.disk_inventory().await;
         let complete_inventory = disks.len() == self.set_drive_count && disks.iter().all(Option::is_some);
@@ -1257,6 +1316,135 @@ mod tests {
         assert_eq!(body(&store, bucket, None).await, original);
     }
 
+    #[tokio::test]
+    async fn delete_recovery_deferred_batch_discards_released_alias_fence_and_preserves_caller_fence() {
+        use crate::bucket::metadata_sys::ObjectLockConfigState;
+        use crate::object_api::{NamespaceLockFence, NamespaceLockSignalTestFence, ObjectLockConfigSnapshot};
+        use crate::storage_api_contracts::object::ObjectToDelete;
+        use std::sync::Mutex;
+        use std::sync::atomic::{AtomicBool, Ordering};
+
+        for lose_caller in [false, true] {
+            let bucket = if lose_caller {
+                "deferred-batch-lost-caller"
+            } else {
+                "deferred-batch-released-alias"
+            };
+            let (_dirs, store, disks) = fixture(bucket).await;
+            put(&store, bucket, &[27; 4096], false).await;
+            let transaction = applied(&store, &disks, bucket, None, disks.len()).await;
+            let before = join_all(
+                disks
+                    .iter()
+                    .map(|disk| disk.inspect_delete_transaction(bucket, "object", transaction)),
+            )
+            .await
+            .into_iter()
+            .collect::<Result<Vec<_>>>()
+            .expect("capture pending recovery evidence");
+            let mut set = (*store.pools[0].disk_set[0]).clone();
+            set.lockers = vec![Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(
+                rustfs_lock::GlobalLockManager::Enabled(Arc::new(rustfs_lock::FastObjectLockManager::new())),
+            )))];
+            set.ctx
+                .update_erasure_type(crate::layout::endpoints::SetupType::DistErasure)
+                .await;
+
+            let (caller_fence, caller_lost) = NamespaceLockFence::loss_handle_for_test();
+            let released_alias_lost = Arc::new(AtomicBool::new(false));
+            let signal_fence = Arc::new(Mutex::new(None));
+            let captured_signal_fence = Arc::clone(&signal_fence);
+            let captured_alias_lost = Arc::clone(&released_alias_lost);
+            test_hooks::after_transaction_lock_acquired(
+                bucket,
+                "object",
+                Arc::new(move |guard| {
+                    let signal = guard.lock_lost_signal().expect("distributed alias signal");
+                    let fence = NamespaceLockSignalTestFence::install_with_loss_handle(&signal, Arc::clone(&captured_alias_lost));
+                    // Retain the signal so a later allocation cannot reuse its
+                    // registry identity after the original guard is dropped.
+                    *captured_signal_fence.lock().expect("captured alias fence") = Some((fence, signal));
+                }),
+            );
+            let replay_started = Arc::new(AtomicBool::new(false));
+            let captured_replay_started = Arc::clone(&replay_started);
+            let expired_alias = Arc::clone(&released_alias_lost);
+            test_hooks::after_transaction_lock_acquired(
+                bucket,
+                "object",
+                Arc::new(move |_| {
+                    // Acquiring the same transaction key again proves the first
+                    // lease was released. Only its copied fence becomes stale.
+                    expired_alias.store(true, Ordering::Release);
+                    caller_lost.store(lose_caller, Ordering::Release);
+                    captured_replay_started.store(true, Ordering::Release);
+                }),
+            );
+
+            let (deleted, errors) = set
+                .delete_objects(
+                    bucket,
+                    vec![ObjectToDelete {
+                        object_name: "object".into(),
+                        ..Default::default()
+                    }],
+                    ObjectOptions {
+                        namespace_lock_fence: Some(caller_fence),
+                        object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new(
+                            ObjectLockConfigState::ConfirmedAbsent,
+                        ))),
+                        ..Default::default()
+                    },
+                )
+                .await;
+            assert!(
+                replay_started.load(Ordering::Acquire),
+                "missing metadata must take the deferred replay path"
+            );
+            assert!(
+                released_alias_lost.load(Ordering::Acquire),
+                "the completed batch alias must be marked lost"
+            );
+            if lose_caller {
+                assert!(errors[0].is_some(), "losing the original caller fence must reject deferred replay");
+                for (disk, original) in disks.iter().zip(before) {
+                    let after = disk
+                        .inspect_delete_transaction(bucket, "object", transaction)
+                        .await
+                        .expect("retained proof");
+                    assert_eq!(
+                        after.fingerprint, original.fingerprint,
+                        "lost caller authority must preserve every prepared journal"
+                    );
+                }
+            } else {
+                assert!(
+                    errors.iter().all(Option::is_none),
+                    "released alias loss must not reject fresh recovery: {errors:?}"
+                );
+                assert!(
+                    !deleted[0].found,
+                    "fresh unversioned recovery commits absence before the idempotent batch retry"
+                );
+                for disk in &disks {
+                    assert!(
+                        discover(disk, bucket, Some("object"))
+                            .await
+                            .expect("final recovery evidence")
+                            .is_empty()
+                    );
+                    assert_eq!(
+                        disk.read_version("", bucket, "object", "", &ReadOptions::default())
+                            .await
+                            .expect_err("replayed delete must remove the source"),
+                        DiskError::FileNotFound
+                    );
+                }
+            }
+            drop(signal_fence);
+        }
+    }
+
     #[tokio::test]
     async fn delete_recovery_corrupt_bucket_does_not_prevent_other_bucket_rollback() {
         let bad_bucket = "recover-corrupt-bucket";

crates/ecstore/src/set_disk/ops/locking.rs

@@ -44,6 +44,21 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks {
 
     #[tracing::instrument(level = "trace", skip(self))]
     async fn new_ns_lock(&self, bucket: &str, object: &str) -> Result<NamespaceLockWrapper> {
+        // Lock the file the disk resolves, not the spelling: on Windows `a\b`
+        // and `a/b` share one xl.meta, so they must share one lock. The key is
+        // folded by the requesting node only; lock servers compare it verbatim.
+        let resource = ObjectKey {
+            bucket: Arc::from(bucket),
+            object: Arc::from(object_name_disk_identity(object).as_ref()),
+            version: None,
+        };
+
+        Ok(self.namespace_lock_for_key(resource).await)
+    }
+}
+
+impl SetDisks {
+    async fn namespace_lock_for_key(&self, resource: ObjectKey) -> NamespaceLockWrapper {
         // Resolved from this set's own instance context (backlog#1052), not the
         // ambient facade: the facade tracks whichever context is currently
         // published process-wide, so a second instance (or, in tests, another
@@ -69,20 +84,33 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks {
             NamespaceLock::with_local_manager_shared(self.set_lock_namespace.clone(), self.local_lock_manager.clone())
         };
 
-        // Lock the file the disk resolves, not the spelling: on Windows `a\b`
-        // and `a/b` share one xl.meta, so they must share one lock. The key is
-        // folded by the requesting node only; lock servers compare it verbatim.
-        let resource = ObjectKey {
-            bucket: Arc::from(bucket),
-            object: Arc::from(object_name_disk_identity(object).as_ref()),
-            version: None,
-        };
-
-        Ok(NamespaceLockWrapper::new(set_lock, resource, self.locker_owner.clone()))
+        NamespaceLockWrapper::new(set_lock, resource, self.locker_owner.clone())
+    }
+
+    /// Raw object locks precede this set-local transaction lock. Its versioned
+    /// resource cannot collide with normal namespace keys, which use None.
+    /// Keeping the target set's clients avoids nesting overlapping raw quorums.
+    pub(in crate::set_disk) async fn acquire_delete_transaction_lock(
+        &self,
+        bucket: &str,
+        object: &str,
+    ) -> Result<Arc<rustfs_lock::NamespaceLockGuard>> {
+        let resource = ObjectKey::with_version(
+            bucket,
+            object_name_disk_identity(object).as_ref(),
+            format!("rustfs:delete-transaction:{}:{}", self.pool_index, self.set_index),
+        );
+        let guard = self
+            .namespace_lock_for_key(resource)
+            .await
+            .get_write_lock(super::super::get_lock_acquire_timeout())
+            .await
+            .map_err(|error| self.map_namespace_lock_error(bucket, object, "delete_transaction", error))?;
+        #[cfg(test)]
+        super::delete_recovery::test_hooks::run_after_transaction_lock_acquired(bucket, object, &guard);
+        Ok(Arc::new(guard))
     }
-}
 
-impl SetDisks {
     #[allow(dead_code, reason = "lock diagnostics formatter with no caller in this port (backlog#1823)")]
     pub(in crate::set_disk) fn format_lock_error(&self, bucket: &str, object: &str, mode: &str, err: &LockResult) -> String {
         match err {

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -43,8 +43,9 @@ use super::super::{
     complete_multipart_part_error, complete_multipart_part_error_result, complete_part_checksum, contains_key_str,
     create_bitrot_writer, debug, disk, error, file_info_is_valid_for_metadata, get_complete_multipart_md5, get_header_map,
     get_str, insert_str, is_err_object_not_found, is_err_version_not_found, is_min_allowed_part_size,
-    log_multipart_write_quorum_failure, parts_after_marker, path_join_buf, prepare_complete_multipart_parts,
-    record_compression_total_memory, reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
+    log_multipart_write_quorum_failure, multipart_part_logical_size, parts_after_marker, path_join_buf,
+    prepare_complete_multipart_parts, record_compression_total_memory, record_multipart_write_quorum_failure,
+    reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
     restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, strip_internal_multipart_metadata,
     to_object_err, warn,
 };
@@ -233,6 +234,19 @@ fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u
     Ok(total)
 }
 
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum MultipartMetadataQuorum {
+    Read,
+    Write,
+    Cleanup,
+}
+
+impl From<bool> for MultipartMetadataQuorum {
+    fn from(write: bool) -> Self {
+        if write { Self::Write } else { Self::Read }
+    }
+}
+
 pub(crate) struct StaleMultipartCleanupGuard {
     file_info: FileInfo,
     upload_path: String,
@@ -410,6 +424,8 @@ impl StaleMultipartCleanupGuard {
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
 pub enum MultipartCommitPause {
     NewUploadBeforeLockLost,
+    #[cfg(test)]
+    ListPartsAfterMetadata,
     PutPartBeforeLockAcquire,
     PutPartBeforeLockLost,
     PutPartAfterCapacityAdmission,
@@ -824,7 +840,8 @@ where
     }
 
     if successful_responses < read_quorum {
-        if empty_upload_fallback_possible(successful_responses, &errs) {
+        // A single missing directory cannot speak for unavailable disks.
+        if !errs.is_empty() && errs.iter().all(|err| matches!(err, Some(DiskError::FileNotFound))) {
             return Err(DiskError::FileNotFound);
         }
 
@@ -834,6 +851,51 @@ where
     Ok((errs, object_parts))
 }
 
+// Size-dependent decisions must not discard a part merely because only a
+// minority of its files was observed. All participants must answer, and each
+// observed part must have a complete file/metadata pair on a read quorum.
+fn validate_part_size_observations(
+    errs: &[Option<DiskError>],
+    object_parts: &[Vec<String>],
+    read_quorum: usize,
+) -> disk::error::Result<Vec<usize>> {
+    if errs.iter().any(Option::is_some) {
+        return Err(DiskError::ErasureReadQuorum);
+    }
+    let mut counts: HashMap<usize, usize> = HashMap::new();
+    for entries in object_parts {
+        let mut observed: HashMap<usize, u8> = HashMap::new();
+        for entry in entries {
+            let Some(name) = entry.strip_prefix("part.") else {
+                continue;
+            };
+            let (number, flag) = match name.strip_suffix(".meta") {
+                Some(number) => (number, 2),
+                None => (name, 1),
+            };
+            let parsed = number.parse::<usize>().map_err(|_| DiskError::FileCorrupt)?;
+            if !(1..=MAX_PARTS_COUNT).contains(&parsed)
+                || !number.as_bytes().first().is_some_and(|digit| (b'1'..=b'9').contains(digit))
+            {
+                return Err(DiskError::FileCorrupt);
+            }
+            *observed.entry(parsed).or_default() |= flag;
+        }
+        for (number, flags) in observed {
+            if flags != 3 {
+                return Err(DiskError::FileCorrupt);
+            }
+            *counts.entry(number).or_default() += 1;
+        }
+    }
+    if counts.values().any(|&count| count < read_quorum) {
+        return Err(DiskError::ErasureReadQuorum);
+    }
+    let mut part_numbers: Vec<_> = counts.into_keys().collect();
+    part_numbers.sort_unstable();
+    Ok(part_numbers)
+}
+
 fn reduce_quorum_part_numbers(object_parts: Vec<Vec<String>>, read_quorum: usize) -> Vec<usize> {
     let mut part_quorum_map: HashMap<usize, usize> = HashMap::new();
 
@@ -1004,7 +1066,7 @@ impl SetDisks {
             ]),
             SLASH_SEPARATOR
         );
-        let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
+        let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum, true).await {
             Ok(parts) => parts,
             Err(DiskError::FileNotFound) => return Ok(0),
             Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
@@ -1016,28 +1078,45 @@ impl SetDisks {
             .iter()
             .map(|number| format!("{part_path}part.{number}.meta"))
             .collect::<Vec<_>>();
-        let existing_parts =
-            Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
-                .await
-                .map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
+        let existing_parts = Self::read_parts(
+            &online_disks,
+            RUSTFS_META_MULTIPART_BUCKET,
+            &part_meta_paths,
+            &part_numbers,
+            fi.write_quorum(self.default_write_quorum()),
+        )
+        .await
+        .map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
 
-        existing_parts.into_iter().try_fold(0_u64, |total, part| {
-            if part.error.is_some() || part.number == replacing_part {
-                return if part.error.is_some() {
-                    Err(Error::PartMissingOrCorrupt)
-                } else {
-                    Ok(total)
-                };
-            }
-            let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
-            total.checked_add(part_size).ok_or_else(|| {
-                Error::InvalidArgument(
-                    "multipart upload".to_string(),
-                    SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
-                    "logical size overflow".to_string(),
-                )
+        if existing_parts.len() != part_numbers.len() {
+            return Err(Error::PartMissingOrCorrupt);
+        }
+        let transformed = fi.is_compressed() || should_persist_encryption_original_size(&fi.metadata);
+        existing_parts
+            .into_iter()
+            .zip(part_numbers)
+            .try_fold(0_u64, |total, (part, number)| {
+                if part.number != number {
+                    return Err(Error::PartMissingOrCorrupt);
+                }
+                if part.error.is_some() || part.number == replacing_part {
+                    return if part.error.is_some() {
+                        Err(Error::PartMissingOrCorrupt)
+                    } else {
+                        Ok(total)
+                    };
+                }
+                let part_size = multipart_part_logical_size(part.size, part.actual_size, transformed)
+                    .and_then(|size| u64::try_from(size).ok())
+                    .ok_or(Error::PartMissingOrCorrupt)?;
+                total.checked_add(part_size).ok_or_else(|| {
+                    Error::InvalidArgument(
+                        "multipart upload".to_string(),
+                        SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+                        "logical size overflow".to_string(),
+                    )
+                })
             })
-        })
     }
 
     async fn discover_multipart_upload_paths(
@@ -1139,7 +1218,7 @@ impl SetDisks {
             }
             let upload_id = runtime_sources::deployment_upload_id(raw_upload_id);
             let file_info = match self
-                .check_multipart_upload_path_exists(bucket, object, &upload_id, &upload_path, false, true)
+                .check_multipart_upload_path_exists(bucket, object, &upload_id, &upload_path, MultipartMetadataQuorum::Read, true)
                 .await
             {
                 Ok((file_info, _)) => file_info,
@@ -1226,7 +1305,7 @@ impl SetDisks {
             .acquire_write_lock_diag("abort_data_movement_multipart", RUSTFS_META_MULTIPART_BUCKET, &upload_path)
             .await?;
         let file_info = match self
-            .check_multipart_upload_path_exists(bucket, object, upload_id, &upload_path, true, true)
+            .check_multipart_upload_path_exists(bucket, object, upload_id, &upload_path, MultipartMetadataQuorum::Write, true)
             .await
         {
             Ok((file_info, _)) => file_info,
@@ -1269,6 +1348,7 @@ impl SetDisks {
         disks: &[Option<DiskStore>],
         part_path: &str,
         read_quorum: usize,
+        strict_size: bool,
     ) -> disk::error::Result<Vec<usize>> {
         let mut futures = Vec::with_capacity(disks.len());
         let part_path = part_path.to_string();
@@ -1277,21 +1357,24 @@ impl SetDisks {
             let part_path = part_path.clone();
             futures.push(async move {
                 if let Some(disk) = disk {
-                    disk.list_dir(RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_MULTIPART_BUCKET, part_path.as_str(), -1)
+                    match disk
+                        .list_dir(RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_MULTIPART_BUCKET, part_path.as_str(), -1)
                         .await
+                    {
+                        Err(DiskError::FileNotFound) if strict_size => Ok(Vec::new()),
+                        result => result,
+                    }
                 } else {
                     Err(DiskError::DiskNotFound)
                 }
             });
         }
 
-        let mut errs = Vec::with_capacity(disks.len());
-        let mut object_parts = Vec::with_capacity(disks.len());
-
-        let (collected_errs, collected_parts) = collect_list_parts_results(futures, read_quorum).await?;
-        errs.extend(collected_errs);
-        object_parts.extend(collected_parts);
+        let (errs, object_parts) = collect_list_parts_results(futures, read_quorum).await?;
 
+        if strict_size {
+            return validate_part_size_observations(&errs, &object_parts, read_quorum);
+        }
         if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, read_quorum) {
             return Err(err);
         }
@@ -1333,7 +1416,7 @@ impl SetDisks {
         data_movement: bool,
     ) -> Result<(FileInfo, Vec<FileInfo>)> {
         let upload_id_path = Self::validated_multipart_upload_path(bucket, object, upload_id, data_movement)?;
-        self.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, write, false)
+        self.check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, write.into(), false)
             .await
     }
 
@@ -1373,7 +1456,7 @@ impl SetDisks {
         object: &str,
         upload_id: &str,
         upload_id_path: &str,
-        write: bool,
+        quorum_mode: MultipartMetadataQuorum,
         include_recovery: bool,
     ) -> Result<(FileInfo, Vec<FileInfo>)> {
         // Slow metadata I/O must not hold the topology registry's read lock.
@@ -1489,22 +1572,26 @@ impl SetDisks {
         }
 
         let mut quorum = read_quorum as usize;
-        if write {
+        if quorum_mode != MultipartMetadataQuorum::Read {
             quorum = write_quorum as usize;
 
             if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, quorum) {
-                log_multipart_write_quorum_failure(
-                    MultipartWriteQuorumContext {
-                        stage: MULTIPART_WRITE_QUORUM_UPLOAD_METADATA,
-                        bucket,
-                        object,
-                        upload_id,
-                        part_number: None,
-                    },
-                    &errs,
-                    quorum,
-                    &err,
-                );
+                if quorum_mode == MultipartMetadataQuorum::Write {
+                    log_multipart_write_quorum_failure(
+                        MultipartWriteQuorumContext {
+                            stage: MULTIPART_WRITE_QUORUM_UPLOAD_METADATA,
+                            bucket,
+                            object,
+                            upload_id,
+                            part_number: None,
+                        },
+                        &errs,
+                        quorum,
+                        &err,
+                    );
+                } else {
+                    record_multipart_write_quorum_failure(MULTIPART_WRITE_QUORUM_UPLOAD_METADATA, &errs, quorum);
+                }
                 return Err(map_error(err));
             }
         } else if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, quorum) {
@@ -1523,7 +1610,7 @@ impl SetDisks {
             .acquire_write_lock_diag("stale_multipart_cleanup", RUSTFS_META_MULTIPART_BUCKET, upload_path)
             .await?;
         let (file_info, _) = self
-            .check_multipart_upload_path_exists("", "", "", upload_path, true, true)
+            .check_multipart_upload_path_exists("", "", "", upload_path, MultipartMetadataQuorum::Cleanup, true)
             .await?;
         let write_quorum = file_info.write_quorum(self.default_write_quorum());
         Ok(StaleMultipartCleanupGuard {
@@ -2634,6 +2721,8 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
         ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
             .await?;
+        #[cfg(test)]
+        pause_multipart_commit(bucket, object, MultipartCommitPause::ListPartsAfterMetadata).await;
 
         let upload_id_path = Self::validated_multipart_upload_path(bucket, object, upload_id, opts.data_movement)?;
 
@@ -2682,16 +2771,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             SLASH_SEPARATOR
         );
 
-        let mut part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
-            Ok(parts) => parts,
-            Err(err) => {
-                if err == DiskError::FileNotFound {
-                    return Ok(ret);
-                }
+        let mut part_numbers =
+            match Self::list_parts(&online_disks, &part_path, read_quorum, opts.strict_multipart_part_sizes).await {
+                Ok(parts) => parts,
+                Err(err) => {
+                    if err == DiskError::FileNotFound {
+                        return Ok(ret);
+                    }
 
-                return Err(to_object_err(err.into(), vec![bucket, object]));
-            }
-        };
+                    return Err(to_object_err(err.into(), vec![bucket, object]));
+                }
+            };
 
         if part_numbers.is_empty() {
             return Ok(ret);
@@ -2708,11 +2798,29 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             .map(|v| format!("{part_path}part.{v}.meta"))
             .collect::<Vec<String>>();
 
-        let object_parts =
-            Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
-                .await
-                .map_err(|e| to_object_err(e.into(), vec![bucket, object, upload_id]))?;
+        let object_parts = Self::read_parts(
+            &online_disks,
+            RUSTFS_META_MULTIPART_BUCKET,
+            &part_meta_paths,
+            &part_numbers,
+            if opts.strict_multipart_part_sizes {
+                fi.write_quorum(self.default_write_quorum())
+            } else {
+                read_quorum
+            },
+        )
+        .await
+        .map_err(|e| to_object_err(e.into(), vec![bucket, object, upload_id]))?;
 
+        if opts.strict_multipart_part_sizes
+            && (object_parts.len() != part_numbers.len()
+                || object_parts
+                    .iter()
+                    .zip(&part_numbers)
+                    .any(|(part, number)| part.number != *number || part.error.is_some() || part.etag.is_empty()))
+        {
+            return Err(Error::PartMissingOrCorrupt);
+        }
         let mut count = max_parts;
 
         for part in object_parts.iter() {
@@ -3032,7 +3140,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         // its full write quorum and the upload's incarnation/lifecycle fences.
         let upload_id_path = Self::validated_multipart_upload_path(bucket, object, upload_id, opts.data_movement)?;
         let (fi, _) = self
-            .check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, false, true)
+            .check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, MultipartMetadataQuorum::Read, true)
             .await?;
         ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
         ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
@@ -3192,7 +3300,14 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                     }
                     SourceRecovery::Missing => {
                         return match self
-                            .check_multipart_upload_path_exists(bucket, object, upload_id, &upload_id_path, false, true)
+                            .check_multipart_upload_path_exists(
+                                bucket,
+                                object,
+                                upload_id,
+                                &upload_id_path,
+                                MultipartMetadataQuorum::Read,
+                                true,
+                            )
                             .await
                         {
                             Ok(_) => Err(Error::ErasureWriteQuorum),
@@ -7492,7 +7607,7 @@ mod tests {
     }
 
     #[tokio::test]
-    async fn collect_list_parts_results_returns_file_not_found_for_empty_upload_dirs() {
+    async fn collect_list_parts_results_rejects_unavailable_empty_upload_dirs() {
         let tasks: Vec<_> = vec![
             (5_u64, Err(DiskError::FileNotFound)),
             (10, Err(DiskError::DiskNotFound)),
@@ -7507,11 +7622,372 @@ mod tests {
 
         let err = collect_list_parts_results(tasks, 2)
             .await
-            .expect_err("missing multipart directories should be treated as empty uploads");
+            .expect_err("an unavailable disk cannot prove an empty upload");
 
+        assert_eq!(err, DiskError::ErasureReadQuorum);
+    }
+
+    #[tokio::test(flavor = "multi_thread")]
+    #[serial]
+    async fn list_parts_rejects_disk_loss_after_upload_metadata_was_read() {
+        let (dirs, disks, set) = hermetic_set_disks(4).await;
+        let bucket = "multipart-list-loss-after-metadata";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let opts = ObjectOptions {
+            max_parity: true,
+            ..Default::default()
+        };
+        let (upload_id, _) = stage_upload_with_create_opts(&set, bucket, object, &[1; 32], &opts).await;
+        let (fi, _) = set
+            .check_upload_id_exists(bucket, object, &upload_id, false)
+            .await
+            .expect("read initial metadata");
+        let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
+        let part_dir = dirs[0]
+            .path()
+            .join(RUSTFS_META_MULTIPART_BUCKET)
+            .join(upload_path)
+            .join(fi.data_dir.expect("upload data directory").to_string());
+        let held_dir = part_dir.with_extension("held-for-test");
+        let barrier = MultipartCommitBarrier::install(bucket, object, MultipartCommitPause::ListPartsAfterMetadata);
+        let query_set = Arc::clone(&set);
+        let query_id = upload_id.clone();
+        let query = tokio::spawn(async move {
+            query_set
+                .list_object_parts(bucket, object, &query_id, None, 100, &ObjectOptions::default())
+                .await
+        });
+        barrier.wait_until_paused().await;
+        tokio::fs::rename(&part_dir, &held_dir)
+            .await
+            .expect("simulate the surviving disk missing the part directory");
+        {
+            let mut live = set.disks.write().await;
+            for slot in 1..4 {
+                live[slot] = None;
+            }
+        }
+        barrier.release();
+        let result = query.await.expect("list task must finish");
+        drop(barrier);
+        tokio::fs::rename(&held_dir, &part_dir)
+            .await
+            .expect("restore the original directory");
+        {
+            let mut live = set.disks.write().await;
+            for slot in 1..4 {
+                live[slot] = Some(disks[slot].clone());
+            }
+        }
+        assert!(
+            matches!(result, Err(StorageError::InsufficientReadQuorum(..))),
+            "disk loss must fail rather than return an empty part list: {result:?}"
+        );
+        let recovered = set
+            .list_object_parts(bucket, object, &upload_id, None, 100, &ObjectOptions::default())
+            .await
+            .expect("listing resumes after the original disks return");
+        assert_eq!(recovered.parts.len(), 1);
+    }
+
+    #[tokio::test]
+    async fn collect_list_parts_results_proves_all_directories_missing() {
+        let tasks = (0..4)
+            .map(|_| async { Err::<Vec<String>, _>(DiskError::FileNotFound) })
+            .collect();
+        let err = collect_list_parts_results(tasks, 2)
+            .await
+            .expect_err("all missing directories are empty");
         assert_eq!(err, DiskError::FileNotFound);
     }
 
+    #[test]
+    fn strict_multipart_part_sizes_reject_incomplete_observations() {
+        let pair = || vec!["part.1".to_string(), "part.1.meta".to_string()];
+        let all_online = vec![None; 4];
+        assert!(validate_part_size_observations(&all_online, &[vec![], vec![], vec![], vec![]], 2).is_ok());
+        assert_eq!(
+            validate_part_size_observations(&all_online, &[pair(), pair(), vec![], vec![]], 2),
+            Ok(vec![1])
+        );
+        let unordered = vec!["part.9.meta", "part.1", "part.9", "part.1.meta", "part.1"]
+            .into_iter()
+            .map(str::to_owned)
+            .collect::<Vec<_>>();
+        assert_eq!(
+            validate_part_size_observations(&all_online, &[unordered.clone(), unordered, vec![], vec![]], 2),
+            Ok(vec![1, 9])
+        );
+        assert_eq!(
+            validate_part_size_observations(&all_online, &[pair(), vec![], vec![], vec![]], 2),
+            Err(DiskError::ErasureReadQuorum)
+        );
+        assert_eq!(
+            validate_part_size_observations(&all_online, &[vec!["part.1.meta".to_string()], pair(), pair(), vec![]], 2),
+            Err(DiskError::FileCorrupt)
+        );
+        for invalid in ["part.0", "part.10001", "part.01", "part.+1", "part.1x"] {
+            let entries = vec![invalid.to_string(), format!("{invalid}.meta")];
+            assert_eq!(
+                validate_part_size_observations(&all_online, &[entries.clone(), entries.clone(), entries.clone(), entries], 2),
+                Err(DiskError::FileCorrupt),
+                "noncanonical part names are not valid size evidence even on every disk"
+            );
+        }
+        assert_eq!(
+            validate_part_size_observations(
+                &[None, None, None, Some(DiskError::DiskNotFound)],
+                &[pair(), pair(), pair(), vec![]],
+                2
+            ),
+            Err(DiskError::ErasureReadQuorum)
+        );
+    }
+
+    #[tokio::test]
+    async fn strict_multipart_capped_upload_rejects_unavailable_participant_and_recovers() {
+        let (_dirs, disks, set) = hermetic_set_disks(4).await;
+        let bucket = "strict-multipart-size-bucket";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let mut metadata = HashMap::new();
+        insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "10".to_string());
+        let opts = ObjectOptions {
+            max_parity: true,
+            user_defined: metadata,
+            ..Default::default()
+        };
+        let (upload_id, _) = stage_upload_with_create_opts(&set, bucket, object, &[1; 6], &opts).await;
+
+        set.disks.write().await[3] = None;
+        let mut part = PutObjReader::from_vec(vec![2; 4]);
+        let err = set
+            .put_object_part(bucket, object, &upload_id, 2, &mut part, &ObjectOptions::default())
+            .await;
+        assert!(err.is_err(), "an unavailable disk must not produce a size admission: {err:?}");
+        set.disks.write().await[3] = Some(disks[3].clone());
+
+        // Two distinct part identities at read quorum cannot prove which size
+        // belongs to the acknowledged write; require the write quorum instead.
+        let (other_id, _) = stage_upload_with_create_opts(&set, bucket, "other", &[9; 2], &ObjectOptions::default()).await;
+        let (fi, _) = set
+            .check_upload_id_exists(bucket, object, &upload_id, false)
+            .await
+            .expect("read capped upload metadata");
+        let (other_fi, _) = set
+            .check_upload_id_exists(bucket, "other", &other_id, false)
+            .await
+            .expect("read other upload metadata");
+        let path = format!(
+            "{}/{}/part.1.meta",
+            SetDisks::get_upload_id_dir(bucket, object, &upload_id),
+            fi.data_dir.expect("data dir")
+        );
+        let other_path = format!(
+            "{}/{}/part.1.meta",
+            SetDisks::get_upload_id_dir(bucket, "other", &other_id),
+            other_fi.data_dir.expect("data dir")
+        );
+        let mut originals = Vec::new();
+        for disk in disks.iter().take(2) {
+            originals.push(
+                disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &path)
+                    .await
+                    .expect("save original part metadata"),
+            );
+            let conflicting = disk
+                .read_all(RUSTFS_META_MULTIPART_BUCKET, &other_path)
+                .await
+                .expect("read conflicting part metadata");
+            disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, conflicting)
+                .await
+                .expect("install split part metadata");
+        }
+        let mut split = PutObjReader::from_vec(vec![5; 4]);
+        let split_result = set
+            .put_object_part(bucket, object, &upload_id, 2, &mut split, &ObjectOptions::default())
+            .await;
+        for (disk, original) in disks.iter().zip(originals) {
+            disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, original)
+                .await
+                .expect("restore original part metadata");
+        }
+        assert!(
+            split_result.is_err(),
+            "split part identities must not authorize a size decision: {split_result:?}"
+        );
+
+        let mut originals = Vec::new();
+        for disk in disks.iter().take(3) {
+            let original = disk
+                .read_all(RUSTFS_META_MULTIPART_BUCKET, &path)
+                .await
+                .expect("save original part metadata");
+            let mut stale = ObjectPartInfo::unmarshal(&original).expect("decode original part metadata");
+            stale.actual_size = 1;
+            disk.write_all(
+                RUSTFS_META_MULTIPART_BUCKET,
+                &path,
+                Bytes::from(stale.marshal_msg().expect("encode stale part")),
+            )
+            .await
+            .expect("install stale size on write quorum");
+            originals.push(original);
+        }
+        let mut undercounted = PutObjReader::from_vec(vec![6; 4]);
+        let result = set
+            .put_object_part(bucket, object, &upload_id, 2, &mut undercounted, &ObjectOptions::default())
+            .await;
+        for (disk, original) in disks.iter().zip(originals) {
+            disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, original)
+                .await
+                .expect("restore original part metadata");
+        }
+        assert!(
+            matches!(result, Err(StorageError::PartMissingOrCorrupt)),
+            "smaller logical size must fail closed: {result:?}"
+        );
+
+        let mut originals = Vec::new();
+        for disk in disks.iter().take(3) {
+            let original = disk
+                .read_all(RUSTFS_META_MULTIPART_BUCKET, &path)
+                .await
+                .expect("save original part metadata");
+            let mut wrong_identity = ObjectPartInfo::unmarshal(&original).expect("decode original part metadata");
+            wrong_identity.number = 2;
+            disk.write_all(
+                RUSTFS_META_MULTIPART_BUCKET,
+                &path,
+                Bytes::from(wrong_identity.marshal_msg().expect("encode wrong identity")),
+            )
+            .await
+            .expect("install mismatched part identity on write quorum");
+            originals.push(original);
+        }
+        let list_result = set
+            .list_object_parts(
+                bucket,
+                object,
+                &upload_id,
+                None,
+                100,
+                &ObjectOptions {
+                    strict_multipart_part_sizes: true,
+                    ..Default::default()
+                },
+            )
+            .await;
+        let mut replacement = PutObjReader::from_vec(vec![7; 4]);
+        let result = set
+            .put_object_part(bucket, object, &upload_id, 2, &mut replacement, &ObjectOptions::default())
+            .await;
+        for (disk, original) in disks.iter().zip(originals) {
+            disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &path, original)
+                .await
+                .expect("restore original part metadata");
+        }
+        assert!(
+            matches!(list_result, Err(StorageError::PartMissingOrCorrupt)),
+            "strict listing must reject mismatched part identity: {list_result:?}"
+        );
+        assert!(
+            matches!(result, Err(StorageError::PartMissingOrCorrupt)),
+            "mismatched part identity must not bypass the cap: {result:?}"
+        );
+
+        let mut too_large = PutObjReader::from_vec(vec![3; 5]);
+        let err = set
+            .put_object_part(bucket, object, &upload_id, 2, &mut too_large, &ObjectOptions::default())
+            .await;
+        assert!(
+            matches!(err, Err(StorageError::EntityTooLarge(11, 10))),
+            "existing part must count: {err:?}"
+        );
+        let mut fits = PutObjReader::from_vec(vec![4; 4]);
+        set.put_object_part(bucket, object, &upload_id, 2, &mut fits, &ObjectOptions::default())
+            .await
+            .expect("the upload should resume at the exact limit after all disks recover");
+    }
+
+    #[tokio::test(flavor = "multi_thread")]
+    async fn capped_multipart_concurrent_parts_cannot_exceed_logical_limit() {
+        let (_dirs, disks, set) = hermetic_set_disks(4).await;
+        let bucket = "concurrent-capped-multipart";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let mut metadata = HashMap::new();
+        insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "10".to_owned());
+        let opts = ObjectOptions {
+            max_parity: true,
+            user_defined: metadata,
+            ..Default::default()
+        };
+        let upload = set
+            .new_multipart_upload(bucket, object, &opts)
+            .await
+            .expect("create capped upload");
+        let barrier = Arc::new(tokio::sync::Barrier::new(3));
+        let mut tasks = Vec::new();
+        for (number, byte) in [(1_usize, 0x41_u8), (2, 0x42)] {
+            let set = Arc::clone(&set);
+            let barrier = Arc::clone(&barrier);
+            let upload_id = upload.upload_id.clone();
+            tasks.push(tokio::spawn(async move {
+                let mut body = PutObjReader::from_vec(vec![byte; 6]);
+                barrier.wait().await;
+                set.put_object_part(bucket, object, &upload_id, number, &mut body, &ObjectOptions::default())
+                    .await
+            }));
+        }
+        barrier.wait().await;
+        let mut accepted = Vec::new();
+        let mut rejected = 0;
+        for task in tasks {
+            match tokio::time::timeout(Duration::from_secs(30), task)
+                .await
+                .expect("concurrent part deadline")
+                .expect("part task")
+            {
+                Ok(part) => accepted.push(part),
+                Err(StorageError::EntityTooLarge(12, 10)) => rejected += 1,
+                Err(other) => panic!("unexpected concurrent admission result: {other:?}"),
+            }
+        }
+        assert_eq!(accepted.len(), 1, "only one six-byte part fits the ten-byte limit");
+        assert_eq!(rejected, 1);
+        let listed = set
+            .list_object_parts(bucket, object, &upload.upload_id, None, 100, &ObjectOptions::default())
+            .await
+            .expect("read final part listing");
+        assert_eq!(listed.parts.len(), 1);
+        assert_eq!(listed.parts[0].part_num, accepted[0].part_num);
+        let winner = accepted.remove(0);
+        let part_number = winner.part_num;
+        set.clone()
+            .complete_multipart_upload(
+                bucket,
+                object,
+                &upload.upload_id,
+                vec![CompletePart {
+                    part_num: part_number,
+                    etag: winner.etag,
+                    ..Default::default()
+                }],
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("complete the admitted part");
+        let mut reader = set
+            .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+            .await
+            .expect("read completed object");
+        let mut actual = Vec::new();
+        reader.stream.read_to_end(&mut actual).await.expect("read whole object");
+        assert_eq!(actual, vec![if part_number == 1 { 0x41 } else { 0x42 }; 6]);
+    }
+
     #[tokio::test]
     async fn collect_list_parts_results_fails_early_when_file_not_found_fallback_is_impossible() {
         let started = Instant::now();
@@ -8131,9 +8607,7 @@ mod tests {
         assert!(listed.parts.is_empty(), "part lock loss before rename must not publish the part");
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn put_object_part_fences_upload_lock_loss_before_rename() {
+    async fn put_object_part_fences_upload_lock_loss_before_rename_case(capped: bool) {
         let target = Arc::new(std::sync::RwLock::new(None));
         let refresh_calls = Arc::new(AtomicUsize::new(0));
         let lockers: Vec<Arc<dyn LockClient>> = (0..4)
@@ -8145,10 +8619,12 @@ mod tests {
         let bucket = "multipart-put-part-lock-loss-bucket";
         let object = "object";
         make_bucket_on_all(&disk_stores, bucket).await;
+        let mut create_opts = ObjectOptions::default();
+        if capped {
+            insert_str(&mut create_opts.user_defined, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "8192".to_owned());
+        }
         let upload = temp_env::async_with_vars([(ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, Some("false"))], async {
-            set_disks
-                .new_multipart_upload(bucket, object, &ObjectOptions::default())
-                .await
+            set_disks.new_multipart_upload(bucket, object, &create_opts).await
         })
         .await
         .expect("multipart upload should be created");
@@ -8199,6 +8675,18 @@ mod tests {
         assert!(listed.parts.is_empty(), "lock loss before rename must not publish the part");
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn put_object_part_fences_upload_lock_loss_before_rename() {
+        put_object_part_fences_upload_lock_loss_before_rename_case(false).await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn capped_multipart_part_fences_upload_lock_loss_before_rename() {
+        put_object_part_fences_upload_lock_loss_before_rename_case(true).await;
+    }
+
     #[tokio::test]
     async fn complete_encrypted_multipart_marks_quorum_validated_layout() {
         temp_env::async_with_vars([(crate::object_api::ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some("true"))], async {
@@ -11133,7 +11621,7 @@ mod tests {
             *disk = None;
         }
         let (info, _) = set
-            .check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
+            .check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
             .await
             .expect("four primary plus four recovery copies reach metadata read quorum");
         assert_eq!(info.name, path);
@@ -11154,7 +11642,7 @@ mod tests {
             *disk = None;
         }
         assert!(
-            set.check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
+            set.check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
                 .await
                 .is_err(),
             "two files on each of four disks must not count as eight voters"
@@ -14471,7 +14959,7 @@ mod tests {
                     object,
                     &u_new,
                     &SetDisks::get_upload_id_dir(bucket, object, &u_new),
-                    true,
+                    MultipartMetadataQuorum::Write,
                     true,
                 )
                 .await
@@ -14516,7 +15004,7 @@ mod tests {
                 object,
                 &upload,
                 &SetDisks::get_upload_id_dir(bucket, object, &upload),
-                true,
+                MultipartMetadataQuorum::Write,
                 true,
             )
             .await
@@ -14599,7 +15087,7 @@ mod tests {
                     assert_completion_files_preserved(&disk_stores, RUSTFS_META_MULTIPART_BUCKET, &source_files).await;
                     set_disks
                         .check_multipart_upload_path_exists(
-                            bucket, object, &u_new, &SetDisks::get_upload_id_dir(bucket, object, &u_new), true, true,
+                            bucket, object, &u_new, &SetDisks::get_upload_id_dir(bucket, object, &u_new), MultipartMetadataQuorum::Write, true,
                         )
                         .await
                         .expect("upload authority must survive the post-commit crash");

crates/ecstore/src/set_disk/ops/multipart_abort_tests.rs

@@ -363,6 +363,202 @@ mod native_abort_closure_tests {
         }
     }
 
+    #[tokio::test]
+    async fn stale_cleanup_reclaims_certified_one_and_two_disk_residuals_after_reopen() {
+        for survivors in [1, 2] {
+            let (dirs, disks, set) = hermetic_set_disks(4).await;
+            let bucket = "native-stale-certified";
+            let object = format!("survivors-{survivors}");
+            make_bucket_on_all(&disks, bucket).await;
+            let opts = ObjectOptions {
+                max_parity: true,
+                ..Default::default()
+            };
+            let (upload, _) = stage_upload_with_create_opts(&set, bucket, &object, &[0x61; 4096], &opts).await;
+            let path = SetDisks::get_upload_id_dir(bucket, &object, &upload);
+            let before = originals(&disks, &path).await;
+            {
+                let physical = disks.iter().cloned().map(Some).collect::<Vec<_>>();
+                let SourceCleanupAuthority::Native { certificate, snapshots } =
+                    prepare_source_cleanup(&physical, bucket, &object, &path, true, &set.multipart_proof_verification_context())
+                        .await
+                        .expect("close every original and persist the certificate")
+                else {
+                    panic!("native upload must retain a native certificate")
+                };
+                for slot in 0..(4 - survivors) {
+                    let response = operation(
+                        &disks[slot],
+                        &path,
+                        MultipartSourceOperation::DeleteAborted {
+                            expected: snapshots[slot].clone().expect("actual certified source"),
+                            certificate: certificate.as_ref().clone(),
+                        },
+                    )
+                    .await;
+                    assert!(matches!(response, MultipartSourceResponse::AbortDeleted(_)));
+                }
+            }
+            let (reopened, cold) = cold_reopen(disks, set).await;
+            for slot in (4 - survivors)..4 {
+                assert_eq!(
+                    retained_native_multipart_original(&reopened[slot], slot, bucket, &object, &path).await,
+                    before[slot]
+                );
+            }
+            cold.lock_stale_multipart_cleanup(&path)
+                .await
+                .expect("complete certificate authorizes the background retry")
+                .delete(&cold)
+                .await
+                .expect("background retry removes certified survivors");
+            for dir in &dirs {
+                assert!(!dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&path).exists());
+            }
+            assert!(
+                cold.lock_stale_multipart_cleanup(&path).await.is_err(),
+                "a repeated pass must not recreate the retired upload"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn stale_certified_residual_survives_lock_loss_then_converges() {
+        let target = Arc::new(std::sync::RwLock::new(None));
+        let refresh_calls = Arc::new(AtomicUsize::new(0));
+        let lockers: Vec<Arc<dyn LockClient>> = (0..4)
+            .map(|_| {
+                Arc::new(SelectiveLockLossClient::new(Arc::clone(&target), Arc::clone(&refresh_calls))) as Arc<dyn LockClient>
+            })
+            .collect();
+        let (dirs, disks, set) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await;
+        let bucket = "stale-certified-lock-loss";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let opts = ObjectOptions {
+            max_parity: true,
+            ..Default::default()
+        };
+        let (upload, _) = stage_upload_with_create_opts(&set, bucket, object, &[0x63; 4096], &opts).await;
+        let path = SetDisks::get_upload_id_dir(bucket, object, &upload);
+        let (fi, _) = set
+            .check_upload_id_exists(bucket, object, &upload, false)
+            .await
+            .expect("source metadata before closure");
+        let part_path = format!("{path}/{}/part.1", fi.data_dir.expect("source data directory"));
+        let original = originals(&disks, &path).await;
+        let surviving_part = disks[3]
+            .read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path)
+            .await
+            .expect("snapshot surviving shard");
+        {
+            let physical = disks.iter().cloned().map(Some).collect::<Vec<_>>();
+            let SourceCleanupAuthority::Native { certificate, snapshots } =
+                prepare_source_cleanup(&physical, bucket, object, &path, true, &set.multipart_proof_verification_context())
+                    .await
+                    .expect("all original participants close and persist the certificate")
+            else {
+                panic!("native certificate required")
+            };
+            let authority = set.multipart_proof_verification_context();
+            let mut ids = certificate.disk_ids.iter().copied().map(Some).collect::<Vec<_>>();
+            ids[0] = None;
+            assert_eq!(
+                crate::set_disk::core::multipart_source::abort_certificate_cohort(&certificate, &ids, &authority),
+                Err(DiskError::ErasureWriteQuorum),
+                "an unavailable participant is a retryable proof gap"
+            );
+            ids[1] = Some(Uuid::new_v4());
+            assert_eq!(
+                crate::set_disk::core::multipart_source::abort_certificate_cohort(&certificate, &ids, &authority),
+                Err(DiskError::FileCorrupt),
+                "a missing participant cannot hide a known identity conflict"
+            );
+            ids[0] = Some(Uuid::new_v4());
+            ids[1] = Some(certificate.disk_ids[1]);
+            assert_eq!(
+                crate::set_disk::core::multipart_source::abort_certificate_cohort(&certificate, &ids, &authority),
+                Err(DiskError::FileCorrupt),
+                "a known different identity remains a certificate conflict"
+            );
+            for slot in 0..3 {
+                let response = operation(
+                    &disks[slot],
+                    &path,
+                    MultipartSourceOperation::DeleteAborted {
+                        expected: snapshots[slot].clone().expect("certified source"),
+                        certificate: certificate.as_ref().clone(),
+                    },
+                )
+                .await;
+                assert!(matches!(response, MultipartSourceResponse::AbortDeleted(_)));
+            }
+        }
+        assert!(dirs[3].path().join(RUSTFS_META_MULTIPART_BUCKET).join(&path).exists());
+        let offline = set.disks.write().await[0]
+            .take()
+            .expect("take an original participant offline");
+        let rejected = set.lock_stale_multipart_cleanup(&path).await;
+        set.disks.write().await[0] = Some(offline);
+        let err = rejected
+            .err()
+            .expect("a certificate must not substitute for an unavailable participant");
+        assert!(
+            matches!(err, StorageError::ErasureWriteQuorum | StorageError::InsufficientWriteQuorum(..)),
+            "the missing participant must be reported as a quorum failure: {err:?}"
+        );
+        assert_eq!(retained_native_multipart_original(&disks[3], 3, bucket, object, &path).await, original[3]);
+        assert_eq!(
+            disks[3]
+                .read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path)
+                .await
+                .expect("surviving shard while a participant was offline"),
+            surviving_part
+        );
+        tokio::time::pause();
+        *target.write().expect("lock-loss target") =
+            Some(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, path.clone()));
+        let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
+        let guard = set
+            .lock_stale_multipart_cleanup(&path)
+            .await
+            .expect("certificate admits the one-disk residual");
+        tokio::time::advance(Duration::from_secs(11)).await;
+        for _ in 0..20 {
+            if guard.is_lock_lost() {
+                break;
+            }
+            tokio::task::yield_now().await;
+        }
+        assert!(refresh_calls.load(Ordering::Acquire) > 0, "the real lock refresh must fail");
+        assert!(guard.is_lock_lost(), "the cleanup guard must observe lock loss");
+        // Real disk I/O must not run under Tokio's auto-advancing paused clock:
+        // a pending blocking operation could otherwise exhaust its timeout.
+        tokio::time::resume();
+        let err = guard.delete(&set).await.expect_err("lock loss must fence physical deletion");
+        assert!(matches!(err, StorageError::NamespaceLockQuorumUnavailable { .. }), "{err:?}");
+        assert_eq!(retained_native_multipart_original(&disks[3], 3, bucket, object, &path).await, original[3]);
+        assert_eq!(
+            disks[3]
+                .read_all(RUSTFS_META_MULTIPART_BUCKET, &part_path)
+                .await
+                .expect("surviving shard after loss"),
+            surviving_part
+        );
+
+        *target.write().expect("restore lock service") = None;
+        set.lock_stale_multipart_cleanup(&path)
+            .await
+            .expect("reacquire the cleanup lock")
+            .delete(&set)
+            .await
+            .expect("the certified residual converges after lock recovery");
+        for dir in &dirs {
+            assert!(!dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&path).exists());
+        }
+    }
+
     #[tokio::test]
     async fn native_abort_partial_delete_surviving_producer_confirms_current_and_historical_exact_target() {
         use crate::set_disk::core::io_primitives::{rename_fault_injection, rollback_fault_injection};
@@ -507,7 +703,7 @@ mod native_abort_closure_tests {
         let path = SetDisks::get_upload_id_dir(bucket, object, &upload);
         pin_historical_unmarked_upload(&disks, bucket, &path).await;
         let before = originals(&disks, &path).await;
-        set.check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
+        set.check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
             .await
             .expect("readonly legacy cleanup identity lookup");
         assert_eq!(
@@ -786,7 +982,7 @@ mod native_abort_closure_tests {
             "known orphan native evidence remains unknown: {inspected:?}"
         );
         let readonly = set
-            .check_multipart_upload_path_exists(bucket, object, &upload, &path, false, true)
+            .check_multipart_upload_path_exists(bucket, object, &upload, &path, MultipartMetadataQuorum::Read, true)
             .await;
         assert!(
             matches!(

crates/ecstore/src/set_disk/ops/object.rs

@@ -73,6 +73,7 @@ struct DeleteCommitContext<'a> {
     incarnation: Option<Uuid>,
     options: Option<&'a ObjectOptions>,
     guard: Option<Arc<super::super::ObjectLockDiagGuard>>,
+    transaction_guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
 }
 
 impl<'a> DeleteCommitContext<'a> {
@@ -81,17 +82,37 @@ impl<'a> DeleteCommitContext<'a> {
             incarnation: options.expected_bucket_incarnation_id,
             options: Some(options),
             guard: guard.cloned(),
+            transaction_guard: None,
         }
     }
 
+    fn with_transaction_guard(mut self, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) -> Self {
+        self.transaction_guard = guard.cloned();
+        self
+    }
+
+    fn ensure_transaction_lock_held(&self, bucket: &str, object: &str) -> Result<()> {
+        if self.transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
+            return Err(StorageError::NamespaceLockQuorumUnavailable {
+                mode: "delete_transaction",
+                bucket: bucket.to_owned(),
+                object: object.to_owned(),
+                required: 1,
+                achieved: 0,
+            });
+        }
+        Ok(())
+    }
+
     fn can_commit(&self, bucket: &str, object: &str) -> bool {
-        self.options.is_none_or(|options| {
-            ensure_delete_commit_locks_held(self.guard.as_deref(), bucket, object, options).is_ok()
-                && options
-                    .scanner_publication_commit_scope
-                    .as_ref()
-                    .is_none_or(|scope| scope.can_commit())
-        })
+        !self.transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
+            && self.options.is_none_or(|options| {
+                ensure_delete_commit_locks_held(self.guard.as_deref(), bucket, object, options).is_ok()
+                    && options
+                        .scanner_publication_commit_scope
+                        .as_ref()
+                        .is_none_or(|scope| scope.can_commit())
+            })
     }
 }
 
@@ -3604,7 +3625,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
         let read_policy = crate::set_disk::get_object_read_policy();
         let erasure_cache = Arc::clone(&self.erasure_cache);
         let (fi, files, disks) = snapshot.into_owned();
-        tokio::spawn(async move {
+        let producer = async move {
             let _guard = read_lock_guard;
             let mut writer = GetObjectDownstreamWriter::new(wd);
             // Do not wrap the entire read+write pipeline in `disk_read_timeout`.
@@ -3687,7 +3708,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                 }
             };
             let _ = producer_terminal_tx.send(producer_result.map(|_| ()));
-        });
+        };
+        #[cfg(feature = "diagnostic-local-batch")]
+        let producer = crate::diagnostics::batch_read_diagnostics::with_captured_request_context(producer);
+        tokio::spawn(producer);
 
         Ok(reader)
     }
@@ -3852,6 +3876,7 @@ impl SetDisks {
         object: &str,
         opts: &ObjectOptions,
         lock_guard: Option<&Arc<super::super::ObjectLockDiagGuard>>,
+        transaction_guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>,
     ) -> Result<bool> {
         if is_meta_bucketname(bucket)
             || opts.version_id.is_some()
@@ -3888,6 +3913,7 @@ impl SetDisks {
                 object,
                 DeleteRecoveryContext {
                     explicit_delete: true,
+                    transaction_guard: transaction_guard.cloned(),
                     namespace_owner: Some(Arc::new((
                         self.ctx.begin_namespace_commit(),
                         lock_guard.cloned(),
@@ -8177,13 +8203,23 @@ fn notify_put_object_commit_namespace_acquired(bucket: &str, object: &str) {
     }
 }
 
+#[cfg(test)]
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum DeleteObjectCommitPause {
+    BeforeNamespace,
+    BeforeCommit,
+    AfterPublish,
+}
+
 #[cfg(test)]
 struct DeleteObjectCommitBarrierState {
     bucket: String,
     object: String,
-    pause_after_publish: bool,
+    pause: DeleteObjectCommitPause,
     arrived: tokio::sync::Notify,
     release: tokio::sync::Notify,
+    namespace_pending: tokio::sync::Notify,
+    namespace_acquired: std::sync::atomic::AtomicBool,
 }
 
 #[cfg(test)]
@@ -8198,21 +8234,27 @@ static DELETE_OBJECT_COMMIT_BARRIER: std::sync::OnceLock<std::sync::Mutex<Option
 #[cfg(test)]
 impl DeleteObjectCommitBarrier {
     pub(crate) fn install(bucket: &str, object: &str) -> Self {
-        Self::install_with_mode(bucket, object, false)
+        Self::install_with_mode(bucket, object, DeleteObjectCommitPause::BeforeCommit)
+    }
+
+    pub(crate) fn install_before_namespace(bucket: &str, object: &str) -> Self {
+        Self::install_with_mode(bucket, object, DeleteObjectCommitPause::BeforeNamespace)
     }
 
     #[cfg(feature = "test-util")]
     pub(crate) fn install_after_publish(bucket: &str, object: &str) -> Self {
-        Self::install_with_mode(bucket, object, true)
+        Self::install_with_mode(bucket, object, DeleteObjectCommitPause::AfterPublish)
     }
 
-    fn install_with_mode(bucket: &str, object: &str, pause_after_publish: bool) -> Self {
+    fn install_with_mode(bucket: &str, object: &str, pause: DeleteObjectCommitPause) -> Self {
         let state = Arc::new(DeleteObjectCommitBarrierState {
             bucket: bucket.to_string(),
             object: object.to_string(),
-            pause_after_publish,
+            pause,
             arrived: tokio::sync::Notify::new(),
             release: tokio::sync::Notify::new(),
+            namespace_pending: tokio::sync::Notify::new(),
+            namespace_acquired: std::sync::atomic::AtomicBool::new(false),
         });
         let mut slot = DELETE_OBJECT_COMMIT_BARRIER
             .get_or_init(|| std::sync::Mutex::new(None))
@@ -8232,6 +8274,19 @@ impl DeleteObjectCommitBarrier {
     pub(crate) fn release(&self) {
         self.state.release.notify_one();
     }
+
+    pub(crate) async fn release_and_wait_until_namespace_pending(&self) {
+        assert_eq!(self.state.pause, DeleteObjectCommitPause::BeforeNamespace);
+        let pending = self.state.namespace_pending.notified();
+        self.release();
+        tokio::time::timeout(Duration::from_secs(5), pending)
+            .await
+            .expect("delete object must poll its actual namespace acquisition to Pending");
+    }
+
+    pub(crate) fn namespace_acquired(&self) -> bool {
+        self.state.namespace_acquired.load(std::sync::atomic::Ordering::Acquire)
+    }
 }
 
 #[cfg(test)]
@@ -8248,6 +8303,43 @@ impl Drop for DeleteObjectCommitBarrier {
     }
 }
 
+#[cfg(test)]
+async fn pause_delete_object_before_namespace(bucket: &str, object: &str) {
+    let barrier = DELETE_OBJECT_COMMIT_BARRIER
+        .get_or_init(|| std::sync::Mutex::new(None))
+        .lock()
+        .expect("delete object commit barrier mutex should not poison")
+        .as_ref()
+        .filter(|barrier| {
+            barrier.bucket == bucket && barrier.object == object && barrier.pause == DeleteObjectCommitPause::BeforeNamespace
+        })
+        .cloned();
+    if let Some(barrier) = barrier {
+        barrier.arrived.notify_one();
+        barrier.release.notified().await;
+    }
+}
+
+#[cfg(test)]
+fn notify_delete_object_namespace(bucket: &str, object: &str, acquired: bool) {
+    let barrier = DELETE_OBJECT_COMMIT_BARRIER
+        .get_or_init(|| std::sync::Mutex::new(None))
+        .lock()
+        .expect("delete object commit barrier mutex should not poison")
+        .as_ref()
+        .filter(|barrier| {
+            barrier.bucket == bucket && barrier.object == object && barrier.pause == DeleteObjectCommitPause::BeforeNamespace
+        })
+        .cloned();
+    if let Some(barrier) = barrier {
+        if acquired {
+            barrier.namespace_acquired.store(true, std::sync::atomic::Ordering::Release);
+        } else {
+            barrier.namespace_pending.notify_one();
+        }
+    }
+}
+
 #[cfg(test)]
 async fn pause_delete_object_commit(bucket: &str, object: &str) {
     let barrier = DELETE_OBJECT_COMMIT_BARRIER
@@ -8255,7 +8347,9 @@ async fn pause_delete_object_commit(bucket: &str, object: &str) {
         .lock()
         .expect("delete object commit barrier mutex should not poison")
         .as_ref()
-        .filter(|barrier| barrier.bucket == bucket && barrier.object == object && !barrier.pause_after_publish)
+        .filter(|barrier| {
+            barrier.bucket == bucket && barrier.object == object && barrier.pause == DeleteObjectCommitPause::BeforeCommit
+        })
         .cloned();
     if let Some(barrier) = barrier {
         barrier.arrived.notify_one();
@@ -8270,7 +8364,9 @@ async fn pause_delete_object_commit_after_publish(bucket: &str, object: &str) {
         .lock()
         .expect("delete object commit barrier mutex should not poison")
         .as_ref()
-        .filter(|barrier| barrier.bucket == bucket && barrier.object == object && barrier.pause_after_publish)
+        .filter(|barrier| {
+            barrier.bucket == bucket && barrier.object == object && barrier.pause == DeleteObjectCommitPause::AfterPublish
+        })
         .cloned();
     if let Some(barrier) = barrier {
         barrier.arrived.notify_one();
@@ -8877,8 +8973,11 @@ impl SetDisks {
         fi: &FileInfo,
         force_del_marker: bool,
         delete_marker_purge: Option<MrfDeleteMarkerPurge>,
-        commit_context: DeleteCommitContext<'_>,
+        mut commit_context: DeleteCommitContext<'_>,
     ) -> Result<()> {
+        if !is_meta_bucketname(bucket) && commit_context.transaction_guard.is_none() {
+            commit_context.transaction_guard = Some(self.acquire_delete_transaction_lock(bucket, object).await?);
+        }
         let source_bucket_incarnation_id = commit_context.incarnation;
         let transported = delete_file_info_with_replication_transport_metadata(fi);
         let fi = &transported;
@@ -8887,6 +8986,7 @@ impl SetDisks {
             Arc::new((
                 self.ctx.begin_namespace_commit(),
                 commit_context.guard.clone(),
+                commit_context.transaction_guard.clone(),
                 commit_context
                     .options
                     .and_then(|options| options.object_lock_config_snapshot.clone()),
@@ -8926,7 +9026,11 @@ impl SetDisks {
 
         for disk in disks.iter() {
             let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>);
+            let transaction_guard = commit_context.transaction_guard.as_ref();
             futures.push(async move {
+                if transaction_guard.is_some_and(|guard| guard.is_lock_lost()) {
+                    return Err(DiskError::DiskOngoingReq);
+                }
                 if let Some(disk) = disk {
                     let options = DeleteOptions {
                         old_data_dir: Some(rollback_dir),
@@ -8976,6 +9080,7 @@ impl SetDisks {
             }
         }
 
+        commit_context.ensure_transaction_lock_held(bucket, object)?;
         let mut quorum_result = resolve_tiered_decommission_write_quorum_result(&errs, write_quorum, bucket, object);
         if !is_meta_bucketname(bucket) && errs.iter().flatten().any(super::delete_recovery::unsupported_recovery) {
             quorum_result = Err(DiskError::UnsupportedDisk.into());
@@ -9008,6 +9113,7 @@ impl SetDisks {
                 write_quorum,
                 DeleteRecoveryContext {
                     recovery_ticket,
+                    transaction_guard: commit_context.transaction_guard.clone(),
                     namespace_owner: namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>),
                     ..DeleteRecoveryContext::new(source_bucket_incarnation_id, &|| commit_context.can_commit(bucket, object))
                 },
@@ -9017,6 +9123,7 @@ impl SetDisks {
         }
         #[cfg(test)]
         pause_delete_before_cleanup(bucket).await;
+        commit_context.ensure_transaction_lock_held(bucket, object)?;
         let mut rollback_futures = Vec::new();
         for (index, err) in errs.iter().enumerate() {
             if structured_cleanup {
@@ -9042,7 +9149,11 @@ impl SetDisks {
             let object = object.to_string();
             let fi = fi.clone();
             let disk_namespace_owner = namespace_owner.clone().map(|owner| owner as Arc<dyn Send + Sync>);
+            let transaction_guard = commit_context.transaction_guard.as_ref();
             rollback_futures.push(async move {
+                if transaction_guard.is_some_and(|guard| guard.is_lock_lost()) {
+                    return;
+                }
                 if should_rollback {
                     // The dedicated undo path never forwards the marker-only creation flag.
                     if let Err(err) = disk
@@ -9099,6 +9210,7 @@ impl SetDisks {
         }
 
         join_all(rollback_futures).await;
+        commit_context.ensure_transaction_lock_held(bucket, object)?;
         drop(namespace_owner);
         if quorum_result.is_ok()
             && errs.iter().any(Option::is_some)
@@ -9558,9 +9670,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         &self,
         bucket: &str,
         objects: Vec<ObjectToDelete>,
-        opts: ObjectOptions,
+        mut opts: ObjectOptions,
     ) -> (Vec<DeletedObject>, Vec<Option<Error>>, Vec<Option<DeleteAccounting>>) {
-        let current_generations = opts.conditional_delete_generations.as_deref();
         let mut del_objects = vec![DeletedObject::default(); objects.len()];
         let mut accounting = vec![None; objects.len()];
         let delete_config_snapshot = opts
@@ -9655,6 +9766,30 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             }
         }
 
+        let outer_namespace_lock_fence = opts.namespace_lock_fence.clone();
+        let mut transaction_guards = std::collections::BTreeMap::new();
+        if !is_meta_bucketname(bucket) {
+            // Every raw batch lock is acquired first. Canonical sorting also
+            // gives aliases of one disk key exactly one transaction owner.
+            let keys: std::collections::BTreeSet<_> = locked_objects.iter().cloned().collect();
+            for key in keys {
+                match self.acquire_delete_transaction_lock(bucket, &key).await {
+                    Ok(guard) => {
+                        opts.add_namespace_lock_guard(&guard);
+                        transaction_guards.insert(key, guard);
+                    }
+                    Err(error) => {
+                        for (index, lock_name) in lock_names.iter().enumerate() {
+                            if lock_name.as_ref() == key {
+                                del_errs[index] = Some(error.clone());
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        let transaction_guards = Arc::new(transaction_guards);
+        let current_generations = opts.conditional_delete_generations.as_deref();
         let object_lock_config = if is_meta_bucketname(bucket) {
             Arc::new(ObjectLockConfigSnapshot::new(ObjectLockConfigState::ConfirmedAbsent))
         } else {
@@ -9752,6 +9887,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                         &dobj.object_name,
                         DeleteRecoveryContext {
                             dry_run: !opts.no_lock,
+                            transaction_guard: transaction_guards.get(lock_names[i].as_ref()).cloned(),
                             namespace_owner: opts.delete_mutation_owner.clone(),
                             ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &|| {
                                 ensure_delete_commit_locks_held(None, bucket, &dobj.object_name, &opts).is_ok()
@@ -10033,6 +10169,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         let namespace_owner = (!is_meta_bucketname(bucket)).then(|| {
             Arc::new((
                 self.ctx.begin_namespace_commit(),
+                Arc::clone(&transaction_guards),
                 opts.delete_mutation_owner.clone(),
                 opts.object_lock_config_snapshot.clone(),
                 opts.scanner_publication_commit_scope.clone(),
@@ -10208,6 +10345,24 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         }
 
         let results = join_all(futures).await;
+        if transaction_guards.values().any(|guard| guard.is_lock_lost()) {
+            for (index, object) in objects.iter().enumerate() {
+                if del_errs[index].is_none() {
+                    accounting[index] = None;
+                    del_errs[index] = Some(StorageError::NamespaceLockQuorumUnavailable {
+                        mode: "delete_transaction",
+                        bucket: bucket.to_owned(),
+                        object: object.object_name.clone(),
+                        required: 1,
+                        achieved: 0,
+                    });
+                }
+            }
+            if dist_erasure {
+                self.release_dist_delete_object_locks_batch(dist_batch_lock_ids).await;
+            }
+            return (del_objects, del_errs, accounting);
+        }
 
         let mut del_obj_errs: Vec<Vec<Option<DiskError>>> = vec![vec![None; objects.len()]; disks.len()];
 
@@ -10303,6 +10458,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 let group = &vers[index];
                 let recovery_ticket = recovery_tickets[index].clone();
                 let namespace_owner = namespace_owner.clone();
+                let transaction_guard = transaction_guards
+                    .get(object_name_disk_identity(&group.name).as_ref())
+                    .cloned();
                 let disks = &disks;
                 let opts = &opts;
                 async move {
@@ -10323,6 +10481,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                             DeleteRecoveryContext {
                                 recovery_ticket,
                                 namespace_owner,
+                                transaction_guard,
                                 ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &can_commit)
                             },
                         )
@@ -10369,7 +10528,11 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 let object = fi_vers.name.clone();
                 let versions = fi_vers.clone();
                 let disk_namespace_owner = namespace_owner.clone();
+                let transaction_guard = transaction_guards.get(object_name_disk_identity(&object).as_ref()).cloned();
                 rollback_futures.push(async move {
+                    if transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
+                        return;
+                    }
                     if should_rollback {
                         let errs = disk
                             .delete_versions_with_namespace_owner(
@@ -10445,7 +10608,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             if del_errs[index].is_none() {
                 let object = &objects[index].object_name;
                 let result = match ensure_delete_commit_locks_held(None, bucket, object, &opts) {
-                    Ok(()) => self.reclaim_missing_delete_residue(bucket, object, &opts, None).await,
+                    Ok(()) => {
+                        self.reclaim_missing_delete_residue(
+                            bucket,
+                            object,
+                            &opts,
+                            None,
+                            transaction_guards.get(object_name_disk_identity(object).as_ref()),
+                        )
+                        .await
+                    }
                     Err(err) => Err(err),
                 };
                 if let Err(err) = result {
@@ -10454,6 +10626,13 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             }
         }
 
+        // Deferred recovery takes fresh raw and transaction locks. Discard the
+        // completed leases' deadlines while retaining every caller-owned fence.
+        opts.namespace_lock_fence = outer_namespace_lock_fence;
+        // Release inner owners before reacquiring any raw key lock.
+        drop(namespace_owner);
+        drop(transaction_guards);
+
         // TODO(backlog): support partial object deletion for multi-part objects
 
         if dist_erasure {
@@ -10559,7 +10738,30 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
 
         // Guard lock for single object delete
         let _lock_guard = if (!opts.delete_prefix || opts.delete_prefix_object) && !opts.no_lock {
-            Some(Arc::new(self.acquire_write_lock_diag("delete_object", bucket, object).await?))
+            #[cfg(test)]
+            let guard = {
+                pause_delete_object_before_namespace(bucket, object).await;
+                let guard = self
+                    .acquire_write_lock_diag_with_pending_hook("delete_object", bucket, object, || {
+                        notify_delete_object_namespace(bucket, object, false);
+                    })
+                    .await?;
+                notify_delete_object_namespace(bucket, object, true);
+                guard
+            };
+            #[cfg(not(test))]
+            let guard = self.acquire_write_lock_diag("delete_object", bucket, object).await?;
+            Some(Arc::new(guard))
+        } else {
+            None
+        };
+        // Lock order: the caller's raw object lock, then this target set's
+        // transaction key. The inner key remains distinct even when pool
+        // client quorums overlap, and no_lock never bypasses journal ownership.
+        let transaction_guard = if !is_meta_bucketname(bucket) && (!opts.delete_prefix || opts.delete_prefix_object) {
+            let guard = self.acquire_delete_transaction_lock(bucket, object).await?;
+            opts.add_namespace_lock_guard(&guard);
+            Some(guard)
         } else {
             None
         };
@@ -10663,7 +10865,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                                 &delete_request,
                                 false,
                                 None,
-                                DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
+                                DeleteCommitContext::request(&opts, _lock_guard.as_ref())
+                                    .with_transaction_guard(transaction_guard.as_ref()),
                             )
                             .await?;
                             if let Some((_, deleted_object)) = replication_delete {
@@ -10697,7 +10900,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                                 &delete_request,
                                 false,
                                 None,
-                                DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
+                                DeleteCommitContext::request(&opts, _lock_guard.as_ref())
+                                    .with_transaction_guard(transaction_guard.as_ref()),
                             )
                             .await?;
                         }
@@ -10717,7 +10921,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                                 &delete_request,
                                 false,
                                 None,
-                                DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
+                                DeleteCommitContext::request(&opts, _lock_guard.as_ref())
+                                    .with_transaction_guard(transaction_guard.as_ref()),
                             )
                             .await?;
                         }
@@ -10806,6 +11011,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                     bucket,
                     object,
                     DeleteRecoveryContext {
+                        transaction_guard: transaction_guard.clone(),
                         namespace_owner: Some(Arc::new((
                             self.ctx.begin_namespace_commit(),
                             _lock_guard.clone(),
@@ -10846,7 +11052,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 if is_err_object_not_found(err) {
                     ensure_delete_commit_locks_held(_lock_guard.as_deref(), bucket, object, &opts)?;
                     if self
-                        .reclaim_missing_delete_residue(bucket, object, &opts, _lock_guard.as_ref())
+                        .reclaim_missing_delete_residue(bucket, object, &opts, _lock_guard.as_ref(), transaction_guard.as_ref())
                         .await?
                     {
                         if let Some(scope) = scanner_publication_commit_scope.as_ref() {
@@ -10896,7 +11102,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 &dfi,
                 false,
                 None,
-                DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
+                DeleteCommitContext::request(&opts, _lock_guard.as_ref()).with_transaction_guard(transaction_guard.as_ref()),
             )
             .await
             .map_err(|e| to_object_err(e, vec![bucket, object]))?;
@@ -10998,7 +11204,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 &fi,
                 should_force_delete_marker_for_missing_version(&opts),
                 delete_marker_purge.clone(),
-                DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
+                DeleteCommitContext::request(&opts, _lock_guard.as_ref()).with_transaction_guard(transaction_guard.as_ref()),
             )
             .await
             .map_err(|e| to_object_err(e, vec![bucket, object]))?;
@@ -11063,7 +11269,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             &dfi,
             opts.delete_marker,
             delete_marker_purge,
-            DeleteCommitContext::request(&opts, _lock_guard.as_ref()),
+            DeleteCommitContext::request(&opts, _lock_guard.as_ref()).with_transaction_guard(transaction_guard.as_ref()),
         )
         .await
         .map_err(|e| to_object_err(e, vec![bucket, object]))?;
@@ -11838,13 +12044,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         let bucket_lifecycle_guard = if let Some(expected_incarnation_id) = opts.expected_bucket_incarnation_id
             && opts.bucket_lifecycle_lock_fence.is_none()
         {
-            let guard = metadata_sys::object_store_in(&self.ctx)
-                .await?
-                .acquire_bucket_lifecycle_read_lock(bucket)
-                .await?;
-            self.validate_bucket_incarnation(bucket, expected_incarnation_id, opts)
-                .await?;
-            Some(guard)
+            Some(
+                metadata_sys::object_store_in(&self.ctx)
+                    .await?
+                    .acquire_bucket_incarnation_fence_with_options(bucket, expected_incarnation_id, Some(opts))
+                    .await?,
+            )
         } else {
             None
         };
@@ -21484,6 +21689,206 @@ mod put_object_tmp_cleanup_tests {
         .await;
     }
 
+    #[tokio::test]
+    async fn deep_heal_preserves_committed_generation_during_old_data_cleanup() {
+        use rustfs_heal_contracts::heal_channel::{HealOpts, HealScanMode};
+
+        for (disk_count, parity, read_repair) in [(4, 2, false), (4, 2, true), (16, 4, false), (16, 4, true)] {
+            let (dirs, disks, set) = super::hermetic_set_disks_support::hermetic_set_disks_for_pool_with_default_parity_isolated(
+                disk_count, 0, parity,
+            )
+            .await;
+            let bucket = "native-heal-commit-cleanup";
+            let object = format!("generation-{disk_count}-{parity}-{read_repair}.bin");
+            for disk in &disks {
+                disk.make_volume(bucket).await.expect("create actual fixture volume");
+            }
+            let options = ObjectOptions {
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
+                shard_integrity_write_mode: Some(crate::object_api::ShardIntegrityWriteMode::Protected),
+                ..Default::default()
+            };
+            set.put_object(bucket, &object, &mut PutObjReader::from_vec(vec![0x17; TEST_OBJECT_SIZE]), &options)
+                .await
+                .expect("acknowledge original generation on every disk");
+            let old = disks[0]
+                .read_version("", bucket, &object, "", &ReadOptions::default())
+                .await
+                .expect("read actual original generation");
+            let old_dir = old.data_dir.expect("original external data directory");
+            let old_part = dirs[0]
+                .path()
+                .join(bucket)
+                .join(&object)
+                .join(old_dir.to_string())
+                .join("part.1");
+            assert!(tokio::fs::try_exists(&old_part).await.expect("original part exists"));
+
+            // Cleanup slots follow the writer's shuffled erasure distribution,
+            // not the physical fixture order. Pause every real cleanup slot so
+            // no physical old shard can disappear before the proof below.
+            let barriers: Vec<_> = (0..disk_count)
+                .map(|slot| rename_fanout_barrier::arm(&object, slot, rename_fanout_barrier::PHASE_CLEANUP))
+                .collect();
+            let writer_set = Arc::clone(&set);
+            let writer_object = object.clone();
+            let writer_options = options.clone();
+            let payload = vec![0xa3; TEST_OBJECT_SIZE];
+            let writer_payload = payload.clone();
+            let writer = tokio::spawn(async move {
+                writer_set
+                    .put_object(bucket, &writer_object, &mut PutObjReader::from_vec(writer_payload), &writer_options)
+                    .await
+            });
+            tokio::time::timeout(Duration::from_secs(30), async {
+                for barrier in &barriers {
+                    barrier.wait_until_paused().await;
+                }
+            })
+            .await
+            .expect("overwrite commits before its real old-data cleanup pause");
+            assert!(!writer.is_finished(), "tail-drained ACK must remain pending at cleanup");
+            assert!(
+                tokio::fs::try_exists(&old_part)
+                    .await
+                    .expect("old source still present at pause")
+            );
+
+            let mut committed = Vec::new();
+            let mut healthy_metadata = Vec::new();
+            for disk in &disks {
+                let info = disk
+                    .read_version("", bucket, &object, "", &ReadOptions::default())
+                    .await
+                    .expect("read independently committed physical generation without GET");
+                assert_eq!((info.erasure.data_blocks, info.erasure.parity_blocks), (disk_count - parity, parity));
+                assert!(info.data.is_none(), "use actual external parts");
+                assert_ne!(info.data_dir, Some(old_dir));
+                committed.push(info);
+                healthy_metadata.push(
+                    disk.read_all(bucket, &format!("{object}/xl.meta"))
+                        .await
+                        .expect("committed raw metadata"),
+                );
+            }
+            let new_dir = committed[0].data_dir.expect("new committed data directory");
+            assert!(committed.iter().all(|info| info.data_dir == Some(new_dir)));
+            let new_part = dirs[0]
+                .path()
+                .join(bucket)
+                .join(&object)
+                .join(new_dir.to_string())
+                .join("part.1");
+            let expected_part = tokio::fs::read(&new_part).await.expect("exact new source bytes");
+            let mut committed_parts = Vec::new();
+            for dir in &dirs {
+                let original = dir.path().join(bucket).join(&object).join(old_dir.to_string()).join("part.1");
+                assert!(
+                    tokio::fs::try_exists(original)
+                        .await
+                        .expect("every old cleanup slot is paused")
+                );
+                committed_parts.push(
+                    tokio::fs::read(dir.path().join(bucket).join(&object).join(new_dir.to_string()).join("part.1"))
+                        .await
+                        .expect("every new encoded shard before fault"),
+                );
+            }
+            tokio::fs::remove_file(&new_part)
+                .await
+                .expect("remove only the newly committed target shard");
+            println!(
+                "native_g01: geometry={disk_count}/{parity} read_repair={read_repair} old_cleanup_paused new_generation_committed target_part_missing"
+            );
+
+            let (result, error) = tokio::time::timeout(
+                Duration::from_secs(30),
+                set.heal_object(
+                    bucket,
+                    &object,
+                    "",
+                    &HealOpts {
+                        scan_mode: HealScanMode::Deep,
+                        read_repair,
+                        ..Default::default()
+                    },
+                ),
+            )
+            .await
+            .expect("real locked Heal must progress during old cleanup")
+            .expect("complete actual deep Heal");
+            assert!(error.is_none(), "new generation must remain repairable: {error:?}");
+            assert_eq!(result.drives_healed(), Some(1));
+            // Shared data-phase repair deliberately does not certify full integrity.
+            assert_eq!(result.integrity_verified, !read_repair);
+            assert_eq!(tokio::fs::read(&new_part).await.expect("rebuilt part before any GET"), expected_part);
+            let repaired = disks[0]
+                .read_version("", bucket, &object, "", &ReadOptions::default())
+                .await
+                .expect("read actual repaired generation");
+            assert_eq!(repaired.data_dir, Some(new_dir));
+            assert_eq!(repaired.mod_time, committed[0].mod_time);
+            for (disk, expected) in disks.iter().zip(&healthy_metadata).skip(1) {
+                assert_eq!(
+                    disk.read_all(bucket, &format!("{object}/xl.meta"))
+                        .await
+                        .expect("healthy raw metadata after Heal"),
+                    *expected
+                );
+            }
+            println!(
+                "native_g01: geometry={disk_count}/{parity} read_repair={read_repair} locked_heal_completed new_generation_part_rebuilt_before_get"
+            );
+
+            for barrier in &barriers {
+                barrier.release();
+            }
+            let acknowledged = tokio::time::timeout(Duration::from_secs(30), writer)
+                .await
+                .expect("cleanup drains")
+                .expect("writer joins")
+                .expect("overwrite ACK remains successful");
+            assert_eq!(acknowledged.size, i64::try_from(payload.len()).expect("fixture size fits i64"));
+            drop(barriers);
+            for (slot, dir) in dirs.iter().enumerate() {
+                let root = dir.path().join(bucket).join(&object);
+                assert!(
+                    !tokio::fs::try_exists(root.join(old_dir.to_string()))
+                        .await
+                        .expect("old cleanup completed")
+                );
+                assert_eq!(
+                    tokio::fs::read(root.join(new_dir.to_string()).join("part.1"))
+                        .await
+                        .expect("new shard survives cleanup before GET"),
+                    committed_parts[slot],
+                    "old cleanup must preserve every exact committed encoded shard"
+                );
+            }
+            let mut reader = set
+                .get_object_reader(bucket, &object, None, HeaderMap::new(), &options)
+                .await
+                .expect("read final acknowledged generation");
+            let mut body = Vec::new();
+            reader
+                .stream
+                .read_to_end(&mut body)
+                .await
+                .expect("read complete final payload");
+            assert_eq!(body, payload);
+            for disk in &disks {
+                let final_info = disk
+                    .read_version("", bucket, &object, "", &ReadOptions::default())
+                    .await
+                    .expect("final physical generation");
+                assert_eq!(final_info.data_dir, Some(new_dir), "old cleanup must not revert the healed generation");
+            }
+            println!(
+                "native_g01: geometry={disk_count}/{parity} read_repair={read_repair} overwrite_ack_preserved final_physical_generation_and_full_body_verified"
+            );
+        }
+    }
+
     #[tokio::test]
     async fn committed_put_releases_namespace_lock_before_old_data_cleanup() {
         let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
@@ -25406,6 +25811,7 @@ mod delete_objects_lock_gating_tests {
     }
 
     #[tokio::test]
+    #[serial_test::serial]
     async fn delete_object_aborts_when_outer_lock_is_lost_at_commit() {
         let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
         let bucket = "single-lost-outer-lock";
@@ -25626,6 +26032,13 @@ mod delete_objects_lock_gating_tests {
             .as_fast_lock_manager()
             .expect("isolated fixture should use the local lock manager");
         let lock_metrics = lock_manager.get_metrics().shard_metrics;
+        let raw_key = ObjectKey::new(bucket, "obj-a");
+        let raw_holder = lock_manager.get_lock_info(&raw_key).expect("outer holder must be visible");
+        let transaction_key = ObjectKey::with_version(
+            bucket,
+            "obj-a",
+            format!("rustfs:delete-transaction:{}:{}", set_disks.pool_index, set_disks.set_index),
+        );
         assert_eq!(lock_manager.current_lock_counts(), (1, 0));
         let mut delete = Box::pin(set_disks.delete_objects(
             bucket,
@@ -25642,23 +26055,43 @@ mod delete_objects_lock_gating_tests {
         // than a second on CI without waiting for the namespace lock at all.
         let deletion = futures::future::poll_fn(|cx| {
             let result = std::future::Future::poll(delete.as_mut(), cx);
+            let current_holder = lock_manager.get_lock_info(&raw_key).expect("outer holder must remain held");
+            assert_eq!(
+                (current_holder.mode, &current_holder.owner, current_holder.acquired_at),
+                (raw_holder.mode, &raw_holder.owner, raw_holder.acquired_at),
+                "no_lock must preserve the original raw object holder"
+            );
+            let transaction_holder = lock_manager.get_lock_info(&transaction_key);
+            if let Some(holder) = &transaction_holder {
+                assert_eq!(holder.mode, rustfs_lock::LockMode::Exclusive);
+            }
             assert_eq!(
                 lock_manager.current_lock_counts(),
-                (1, 0),
-                "no_lock batch delete must preserve the outer holder without adding lock waiters"
+                (1 + u64::from(transaction_holder.is_some()), 0),
+                "only the distinct transaction owner may be added, without raw-key waiters"
             );
             let current = lock_manager.get_metrics().shard_metrics;
-            assert_eq!(
-                (current.total_acquisitions(), current.timeouts),
-                (lock_metrics.total_acquisitions(), lock_metrics.timeouts),
-                "no_lock batch delete must not acquire or time out on another namespace lock"
+            assert!(
+                current.total_acquisitions() <= lock_metrics.total_acquisitions() + 1,
+                "the transaction owner must be reused by inline recovery"
             );
+            assert_eq!(current.timeouts, lock_metrics.timeouts, "the raw key must never be reacquired");
             result
         });
         let (deleted, errs) = tokio::time::timeout(Duration::from_secs(30), deletion)
             .await
             .expect("no_lock batch delete disk work should complete");
 
+        assert_eq!(lock_manager.current_lock_counts(), (1, 0));
+        assert!(
+            lock_manager.get_lock_info(&transaction_key).is_none(),
+            "the transaction owner must be released"
+        );
+        assert_eq!(
+            lock_manager.get_metrics().shard_metrics.total_acquisitions(),
+            lock_metrics.total_acquisitions() + 1,
+            "no_lock still requires exactly one distinct transaction owner"
+        );
         assert!(errs[0].is_none(), "no_lock batch delete should not fail with a lock error: {:?}", errs[0]);
         assert!(deleted[0].found, "existing object must still be deleted");
         for disk in &disk_stores {

crates/ecstore/src/set_disk/read.rs

@@ -1818,7 +1818,10 @@ impl SetDisks {
                         "Create bitrot reader failed read quorum"
                     );
                     record_get_object_pipeline_failure(GET_STAGE_READER_SETUP, reason);
-                    return Err(to_object_err(read_err.into(), vec![bucket, object]));
+                    // The metadata selected this object, but the part cannot
+                    // be reconstructed. A dominant disk I/O/not-found error
+                    // must not erase the object's read-quorum failure.
+                    return Err(to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]));
                 }
                 let reason = GetObjectFailureReason::ReadQuorum;
                 error!(
@@ -1843,7 +1846,7 @@ impl SetDisks {
                     "Create bitrot reader did not have enough disks"
                 );
                 record_get_object_pipeline_failure(GET_STAGE_READER_SETUP, reason);
-                return Err(Error::other(format!("not enough disks to read: {:?}", reader_setup.errors)));
+                return Err(to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]));
             }
 
             // Check if we have missing shards even though we can read successfully
@@ -2378,10 +2381,10 @@ impl SetDisks {
             if let Some(read_err) = reduce_read_quorum_errs(&reader_setup.errors, OBJECT_OP_IGNORED_ERRS, erasure.data_shards) {
                 let reason = classify_disk_error(&read_err);
                 record_get_object_pipeline_failure_for_path(metrics_path, GET_STAGE_READER_SETUP, reason);
-                return Err(to_object_err(read_err.into(), vec![bucket, object]));
+                return Err(to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]));
             }
             record_get_object_pipeline_failure_for_path(metrics_path, GET_STAGE_READER_SETUP, GetObjectFailureReason::ReadQuorum);
-            return Err(Error::other(format!("not enough disks to read: {:?}", reader_setup.errors)));
+            return Err(to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]));
         }
 
         let missing_shards = reader_setup.completed_failed_shards();
@@ -8192,6 +8195,105 @@ mod tests {
         (data_dir, files)
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn external_missing_shards_keep_typed_read_quorum() {
+        use tokio::io::AsyncReadExt;
+
+        let payload = vec![0x37; 256 * 1024];
+        for readable_shards in [1, 2, 4] {
+            let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
+            let mut fi = codec_streaming_test_fileinfo(i64::try_from(payload.len()).expect("fixture size fits i64"), 1);
+            fi.erasure.data_blocks = erasure.data_shards;
+            fi.erasure.parity_blocks = erasure.parity_shards;
+            fi.erasure.block_size = erasure.block_size;
+            fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
+            let (dirs, disks) = local_test_disks(erasure.total_shard_count(), CODEC_STREAMING_TEST_BUCKET).await;
+            let (data_dir, files) = codec_streaming_external_files(&erasure, &payload, &disks).await;
+            fi.data_dir = Some(data_dir);
+            for directory in dirs.iter().skip(readable_shards) {
+                tokio::fs::remove_file(
+                    directory
+                        .path()
+                        .join(CODEC_STREAMING_TEST_BUCKET)
+                        .join(CODEC_STREAMING_TEST_OBJECT)
+                        .join(data_dir.to_string())
+                        .join("part.1"),
+                )
+                .await
+                .expect("remove real part while retaining its selected metadata");
+            }
+
+            let mut output = Vec::new();
+            let legacy = SetDisks::get_object_with_fileinfo(
+                CODEC_STREAMING_TEST_BUCKET,
+                CODEC_STREAMING_TEST_OBJECT,
+                Arc::new(ErasureCache::new()),
+                0,
+                fi.size,
+                &mut output,
+                fi.clone(),
+                files.clone(),
+                &disks,
+                0,
+                0,
+                false,
+                true,
+                false,
+                false,
+                GET_OBJECT_PATH_LEGACY_DUPLEX,
+                "plain_single_part",
+                "le_1mib",
+            )
+            .await;
+            let streaming = SetDisks::get_object_mid_size_reader_with_fileinfo(
+                CODEC_STREAMING_TEST_BUCKET,
+                CODEC_STREAMING_TEST_OBJECT,
+                Arc::new(ErasureCache::new()),
+                &fi,
+                &files,
+                &disks,
+                0,
+                0,
+                false,
+                true,
+                "plain_single_part",
+                "le_1mib",
+                false,
+            )
+            .await;
+            if readable_shards < erasure.data_shards {
+                for error in [
+                    legacy.expect_err("legacy read must reject quorum minus one"),
+                    streaming.err().expect("streaming setup must reject quorum minus one"),
+                ] {
+                    assert_eq!(
+                        error,
+                        Error::InsufficientReadQuorum(
+                            CODEC_STREAMING_TEST_BUCKET.to_owned(),
+                            CODEC_STREAMING_TEST_OBJECT.to_owned(),
+                        )
+                    );
+                }
+                assert!(output.is_empty(), "quorum rejection must precede all response bytes");
+            } else {
+                legacy.expect("legacy reader retains reconstructable quorum");
+                assert_eq!(output, payload, "legacy read returns every original byte at quorum");
+                let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) =
+                    streaming.expect("streaming quorum remains readable")
+                else {
+                    panic!("mid-size quorum must yield an actual reader");
+                };
+                let mut decoded = Vec::new();
+                reader
+                    .read_to_end(&mut decoded)
+                    .await
+                    .expect("streaming reader returns the complete body");
+                assert_eq!(decoded, payload, "streaming read returns every original byte at quorum");
+            }
+        }
+    }
+
     #[test]
     #[serial]
     fn payload_read_repair_is_suppressed_only_for_read_only_requests() {

crates/ecstore/src/store/bucket.rs

@@ -448,6 +448,14 @@ impl ECStore {
         await_orphan_bucket_recovery(bucket, &publication_guard, &lifecycle_guard, &metadata_guard, &ns_guard, recover).await
     }
 
+    /// Read quota and bucket identity from one authoritative metadata snapshot.
+    pub async fn bucket_quota_and_incarnation_from_disk(
+        &self,
+        bucket: &str,
+    ) -> Result<(Option<crate::bucket::quota::BucketQuota>, Uuid, OffsetDateTime)> {
+        metadata_sys::get_quota_config_and_incarnation_from_disk_in(&self.ctx, bucket).await
+    }
+
     pub async fn get_bucket_metadata(&self, bucket: &str) -> Result<Arc<BucketMetadata>> {
         let sys = metadata_sys::require_bucket_metadata_sys_in(&self.ctx)?;
         sys.read().await.get(bucket).await
@@ -564,13 +572,83 @@ impl ECStore {
             .await
     }
 
-    pub(super) async fn acquire_bucket_incarnation_fence_with_options(
+    pub(crate) async fn acquire_bucket_incarnation_fence_with_options(
         &self,
         bucket: &str,
         expected: Uuid,
         opts: Option<&ObjectOptions>,
     ) -> Result<super::BucketIncarnationFenceGuard> {
+        Ok(self
+            .acquire_bucket_incarnation_fence_for_options(bucket, Some(expected), opts)
+            .await?
+            .0)
+    }
+
+    pub(crate) async fn acquire_bucket_incarnation_fence_for_options(
+        &self,
+        bucket: &str,
+        expected: Option<Uuid>,
+        opts: Option<&ObjectOptions>,
+    ) -> Result<(super::BucketIncarnationFenceGuard, Uuid)> {
         let inner = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
+        self.validate_bucket_incarnation_fence_guard_with_options(bucket, inner, expected, opts)
+            .await
+    }
+
+    /// Acquire lexicographically ordered bucket lifecycle read locks for a copy
+    /// operation and validate both bucket incarnations through [`super::BucketFenceRegistry`].
+    pub async fn acquire_copy_bucket_incarnation_fences(
+        &self,
+        source_bucket: &str,
+        expected_source: Option<Uuid>,
+        destination_bucket: &str,
+        expected_destination: Option<Uuid>,
+    ) -> Result<(super::BucketIncarnationFenceGuard, Uuid, Option<super::BucketIncarnationFenceGuard>, Uuid)> {
+        let (source_guard, destination_guard) = if source_bucket == destination_bucket {
+            (self.acquire_bucket_lifecycle_read_lock(source_bucket).await?, None)
+        } else if source_bucket < destination_bucket {
+            let source_guard = self.acquire_bucket_lifecycle_read_lock(source_bucket).await?;
+            let destination_guard = self.acquire_bucket_lifecycle_read_lock(destination_bucket).await?;
+            (source_guard, Some(destination_guard))
+        } else {
+            let destination_guard = self.acquire_bucket_lifecycle_read_lock(destination_bucket).await?;
+            let source_guard = self.acquire_bucket_lifecycle_read_lock(source_bucket).await?;
+            (source_guard, Some(destination_guard))
+        };
+        let (source_guard, current_source_incarnation_id) = self
+            .validate_bucket_incarnation_fence_guard_with_options(source_bucket, source_guard, expected_source, None)
+            .await?;
+        let (destination_guard, current_destination_incarnation_id) = if let Some(destination_guard) = destination_guard {
+            let (destination_guard, destination_incarnation_id) = self
+                .validate_bucket_incarnation_fence_guard_with_options(
+                    destination_bucket,
+                    destination_guard,
+                    expected_destination,
+                    None,
+                )
+                .await?;
+            (Some(destination_guard), destination_incarnation_id)
+        } else {
+            if expected_destination.is_some_and(|expected| expected != current_source_incarnation_id) {
+                return Err(StorageError::BucketNotFound(destination_bucket.to_owned()));
+            }
+            (None, current_source_incarnation_id)
+        };
+        Ok((
+            source_guard,
+            current_source_incarnation_id,
+            destination_guard,
+            current_destination_incarnation_id,
+        ))
+    }
+
+    async fn validate_bucket_incarnation_fence_guard_with_options(
+        &self,
+        bucket: &str,
+        inner: rustfs_lock::NamespaceLockGuard,
+        expected: Option<Uuid>,
+        opts: Option<&ObjectOptions>,
+    ) -> Result<(super::BucketIncarnationFenceGuard, Uuid)> {
         let mut guard = super::BucketIncarnationFenceGuard::new(
             Arc::clone(&self.bucket_fence_registry),
             bucket,
@@ -586,10 +664,10 @@ impl ECStore {
                 }
             })
             .await?;
-        if current != expected {
+        if expected.is_some_and(|expected| current != expected) {
             return Err(StorageError::BucketNotFound(bucket.to_owned()));
         }
-        Ok(guard)
+        Ok((guard, current))
     }
 
     /// Hold this guard through recursive-delete authorization and mutation so

crates/ecstore/src/store/bucket_fence.rs

@@ -149,6 +149,18 @@ impl BucketFenceRegistry {
             }
         }
     }
+
+    /// Return the memoized incarnation id for `bucket` when at least one
+    /// registered lifecycle read guard is currently held without lock loss.
+    pub(crate) fn validated_incarnation(&self, bucket: &str) -> Option<Uuid> {
+        let mut entries = self.entries.lock().expect("bucket fence registry poisoned");
+        let entry = entries.get_mut(bucket)?;
+        if entry.guards.iter().any(|guard| guard.loss_probe.is_lost()) {
+            entry.validated = None;
+            return None;
+        }
+        entry.validated
+    }
 }
 
 /// Owns continuous lifecycle coverage and its incarnation validation memo.
@@ -221,7 +233,7 @@ impl BucketIncarnationFenceGuard {
         }
     }
 
-    pub(crate) fn is_lock_lost(&self) -> bool {
+    pub fn is_lock_lost(&self) -> bool {
         self.inner.as_ref().is_some_and(NamespaceLockGuard::is_lock_lost)
     }
 
@@ -310,9 +322,16 @@ mod tests {
         let (first_probe, first_lost) = controllable_probe();
         let first = reg.enter("b", first_probe);
         assert_eq!(first.memoized, None);
+        assert_eq!(reg.validated_incarnation("b"), None);
         reg.memoize("b", uuid(7));
+        assert_eq!(reg.validated_incarnation("b"), Some(uuid(7)));
 
         first_lost.store(true, std::sync::atomic::Ordering::SeqCst);
+        assert_eq!(
+            reg.validated_incarnation("b"),
+            None,
+            "live lost guard must clear and block validated_incarnation"
+        );
 
         let second = reg.enter("b", live_probe());
         assert_eq!(second.memoized, None, "live lost guard must force disk revalidation");
@@ -320,9 +339,11 @@ mod tests {
 
         let third = reg.enter("b", live_probe());
         assert_eq!(third.memoized, None, "memo remains blocked while the lost guard is live");
+        assert_eq!(reg.validated_incarnation("b"), None);
 
         reg.exit("b", first.token, true);
         reg.memoize("b", uuid(8));
+        assert_eq!(reg.validated_incarnation("b"), Some(uuid(8)));
         let fourth = reg.enter("b", live_probe());
         assert_eq!(fourth.memoized, Some(uuid(8)), "memo resumes after lost coverage leaves");
 

crates/ecstore/src/store/heal.rs

@@ -2167,6 +2167,210 @@ mod tests {
         shutdown.cancel();
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn bucket_incarnation_fence_survives_caller_abort_at_real_repair_commit() {
+        use crate::set_disk::ReadRepairPauseScope;
+
+        let (_root, store, shutdown) = multi_pool_heal_store().await;
+        let bucket = format!("heal-commit-owner-{}", Uuid::new_v4().simple());
+        let object = "history.bin";
+        let version = Uuid::new_v4().to_string();
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("original bucket");
+        let old = store
+            .bucket_incarnation_id_from_disk(&bucket)
+            .await
+            .expect("original identity");
+        let set = store.pools[0].get_disks(0);
+        let mut reader = PutObjReader::from_vec(vec![0x17; 1024 * 1024]);
+        set.put_object(
+            &bucket,
+            object,
+            &mut reader,
+            &ObjectOptions {
+                versioned: true,
+                version_id: Some(version.clone()),
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("original acknowledged version");
+        let missing = set.disks.read().await[0].clone().expect("repair target");
+        let healthy = set.disks.read().await[1].clone().expect("healthy target");
+        let baseline = healthy
+            .read_all(&bucket, &format!("{object}/xl.meta"))
+            .await
+            .expect("healthy metadata baseline");
+        let original = healthy
+            .read_version("", &bucket, object, &version, &Default::default())
+            .await
+            .expect("original version metadata");
+        assert!(original.data.is_none(), "the commit barrier fixture must use actual external data shards");
+        missing
+            .delete(
+                &bucket,
+                object,
+                DeleteOptions {
+                    recursive: true,
+                    immediate: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("remove one physical shard");
+
+        // The scope pauses the real repair after reconstruction, immediately
+        // before its generation check and final shard rename.
+        let (_pause, arrived, release) = ReadRepairPauseScope::install_incarnation_commit(&bucket, object);
+        let (drained_tx, drained_rx) = tokio::sync::oneshot::channel();
+        let worker_store = store.clone();
+        let worker_bucket = bucket.clone();
+        let worker_version = version.clone();
+        let opts = HealOpts {
+            pool: Some(0),
+            set: Some(0),
+            scan_mode: rustfs_heal_contracts::heal_channel::HealScanMode::Deep,
+            ..Default::default()
+        };
+        let mut waiter = tokio::spawn(async move {
+            worker_store
+                .run_bucket_heal_at_incarnation(&worker_bucket, old, &opts, move |store, bucket, opts| async move {
+                    let result = store.heal_object_with_proof(&bucket, object, &worker_version, &opts).await?;
+                    drained_tx
+                        .send(result)
+                        .expect("retain the actual storage result after caller cancellation");
+                    Ok(())
+                })
+                .await
+        });
+        tokio::select! {
+            paused = tokio::time::timeout(std::time::Duration::from_secs(10), arrived.notified()) => {
+                paused.expect("real missing-shard repair reaches the final commit pause");
+            }
+            completed = &mut waiter => panic!("repair ended before the actual final commit barrier: {completed:?}"),
+        }
+        assert!(
+            missing.read_xl(&bucket, object, false).await.is_err(),
+            "final shard is not published at the pause"
+        );
+        println!("incarnation_commit_barrier: final_rename_paused");
+        waiter.abort();
+        assert!(waiter.await.expect_err("caller aborted").is_cancelled());
+        let mut writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket));
+        assert!(
+            futures::poll!(&mut writer).is_pending(),
+            "bucket deletion/recreation cannot acquire its generation writer while the cancelled repair still owns commit"
+        );
+        println!("incarnation_commit_barrier: caller_aborted_writer_pending");
+        release.notify_one();
+        let result = tokio::time::timeout(std::time::Duration::from_secs(10), drained_rx)
+            .await
+            .expect("actual repair drains after release")
+            .expect("retained repair result");
+        assert!(
+            result.error.is_none(),
+            "physical repair must succeed before lifecycle handoff: {:?}",
+            result.error
+        );
+        let guard = tokio::time::timeout(std::time::Duration::from_secs(10), writer)
+            .await
+            .expect("generation writer acquires only after physical repair drains")
+            .expect("generation writer");
+        assert!(
+            missing.read_xl(&bucket, object, false).await.is_ok(),
+            "the physical shard is published before writer admission"
+        );
+        let repaired = missing
+            .read_version("", &bucket, object, &version, &Default::default())
+            .await
+            .expect("repaired exact version metadata");
+        assert_eq!(repaired.version_id, original.version_id);
+        assert_eq!(repaired.data_dir, original.data_dir);
+        let checked = missing
+            .check_parts(&bucket, object, &repaired)
+            .await
+            .expect("verify the published physical shard");
+        assert_eq!(
+            checked.results,
+            vec![crate::disk::CHECK_PART_SUCCESS],
+            "the actual published part passes bitrot verification"
+        );
+        assert_eq!(
+            healthy
+                .read_all(&bucket, &format!("{object}/xl.meta"))
+                .await
+                .expect("healthy after repair"),
+            baseline,
+            "repair does not overwrite the surviving version"
+        );
+        println!("incarnation_commit_barrier: repair_drained_writer_acquired");
+        drop(guard);
+
+        set.delete_object(
+            &bucket,
+            object,
+            ObjectOptions {
+                versioned: true,
+                version_id: Some(version.clone()),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("normal explicit deletion of the old version");
+        store
+            .delete_bucket(&bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("normal empty bucket deletion");
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("same-name successor bucket");
+        let new = store
+            .bucket_incarnation_id_from_disk(&bucket)
+            .await
+            .expect("successor identity");
+        assert_ne!(old, new);
+        let successor_version = Uuid::new_v4().to_string();
+        let mut successor_reader = PutObjReader::from_vec(vec![0x29; 512 * 1024]);
+        set.put_object(
+            &bucket,
+            object,
+            &mut successor_reader,
+            &ObjectOptions {
+                versioned: true,
+                version_id: Some(successor_version),
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("acknowledged successor version");
+        let successor = healthy
+            .read_all(&bucket, &format!("{object}/xl.meta"))
+            .await
+            .expect("successor metadata");
+        assert!(
+            store
+                .heal_object_at_incarnation(&bucket, object, &version, old, &opts)
+                .await
+                .is_err()
+        );
+        assert_eq!(
+            healthy
+                .read_all(&bucket, &format!("{object}/xl.meta"))
+                .await
+                .expect("successor after obsolete admission"),
+            successor,
+            "obsolete admission cannot change the same-name successor"
+        );
+        println!("incarnation_commit_barrier: successor_rejects_obsolete_admission");
+        shutdown.cancel();
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn bucket_incarnation_heal_owner_survives_cancelled_waiter_and_queued_writer() {

crates/ecstore/src/store/init.rs

@@ -11991,6 +11991,7 @@ mod tests {
             ..Default::default()
         });
 
+        let mut stage_started = std::time::Instant::now();
         futures::stream::iter(0..RECEIPT_COUNT)
             .map(|index| {
                 let store = store.clone();
@@ -12024,15 +12025,21 @@ mod tests {
             .try_collect::<Vec<_>>()
             .await
             .expect("more than one receipt page should persist");
+        eprintln!("receipt fixture: seed {:.3}s", stage_started.elapsed().as_secs_f64());
+        stage_started = std::time::Instant::now();
         store
             .persist_decommission_durable_ilm_manifest_for_test(0)
             .await
             .expect("paginated source receipts should produce a manifest");
+        eprintln!("receipt fixture: manifest {:.3}s", stage_started.elapsed().as_secs_f64());
+        stage_started = std::time::Instant::now();
 
         let target_receipts = store
             .decommission_durable_ilm_receipt_paths_for_test(0)
             .await
             .expect("paginated target receipts should list");
+        eprintln!("receipt fixture: list {:.3}s", stage_started.elapsed().as_secs_f64());
+        stage_started = std::time::Instant::now();
         assert_eq!(target_receipts.len(), RECEIPT_COUNT);
         let (target_pool_idx, second_page_path) = target_receipts
             .get(1000)
@@ -12050,6 +12057,8 @@ mod tests {
             .await
             .expect_err("a missing second page receipt must block completion")
             .to_string();
+        eprintln!("receipt fixture: missing {:.3}s", stage_started.elapsed().as_secs_f64());
+        stage_started = std::time::Instant::now();
         assert!(missing.contains(&second_page_path));
         assert!(
             !store.pool_meta.read().await.pools[0]
@@ -12076,6 +12085,7 @@ mod tests {
             .await
             .expect_err("a corrupt second page receipt must block completion")
             .to_string();
+        eprintln!("receipt fixture: corrupt {:.3}s", stage_started.elapsed().as_secs_f64());
         assert!(corrupt.contains(&second_page_path));
         assert!(corrupt.contains("invalid"));
         assert!(
@@ -21384,6 +21394,141 @@ mod tests {
         shutdown.cancel();
     }
 
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn put_object_metadata_and_copy_object_reuse_overlapping_incarnation_fence() {
+        let temp = tempfile::tempdir().expect("create metadata/copy fence store");
+        let (_ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp.path(), "metadata-copy-fence-reuse", &[4])).await;
+        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("metadata-copy-fence-{}", Uuid::new_v4());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+        let mut seed_reader = PutObjReader::from_vec(b"seed payload".to_vec());
+        let versioned_opts = ObjectOptions {
+            versioned: true,
+            ..Default::default()
+        };
+        let mut src_info = store
+            .put_object(&bucket, "src.bin", &mut seed_reader, &versioned_opts)
+            .await
+            .expect("seed source object");
+        let src_version_id = src_info
+            .version_id
+            .expect("versioned PUT should return a version ID")
+            .to_string();
+        let expected = store.bucket_incarnation_id(&bucket).await.expect("read incarnation");
+        let coverage = store
+            .acquire_bucket_incarnation_fence(&bucket, expected)
+            .await
+            .expect("validate continuous coverage");
+        let metadata_writer = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
+            .await
+            .expect("hold metadata writer during covered metadata and copy operations");
+
+        let updated = tokio::time::timeout(
+            Duration::from_secs(10),
+            store.put_object_metadata(
+                &bucket,
+                "src.bin",
+                &ObjectOptions {
+                    version_id: Some(src_version_id.clone()),
+                    versioned: true,
+                    eval_metadata: Some(HashMap::from([("x-amz-meta-fenced".to_string(), "true".to_string())])),
+                    ..Default::default()
+                },
+            ),
+        )
+        .await
+        .expect("covered put_object_metadata must not reacquire the metadata transaction lock")
+        .expect("update object metadata using authoritative memo");
+        assert_eq!(updated.user_defined.get("x-amz-meta-fenced").map(String::as_str), Some("true"));
+
+        Arc::make_mut(&mut src_info.user_defined).insert("x-amz-meta-copied".to_string(), "true".to_string());
+        let src_copy_opts = ObjectOptions {
+            version_id: Some(src_version_id),
+            versioned: true,
+            ..Default::default()
+        };
+        let copied = tokio::time::timeout(
+            Duration::from_secs(10),
+            store.copy_object(&bucket, "src.bin", &bucket, "src.bin", &mut src_info, &src_copy_opts, &versioned_opts),
+        )
+        .await
+        .expect("covered versioned copy_object must not reacquire the metadata transaction lock")
+        .expect("versioned copy using authoritative memo");
+        assert_eq!(copied.user_defined.get("x-amz-meta-copied").map(String::as_str), Some("true"));
+
+        let (src_guard, src_incarnation, dst_guard, dst_incarnation) = tokio::time::timeout(
+            Duration::from_secs(10),
+            store.acquire_copy_bucket_incarnation_fences(&bucket, Some(expected), &bucket, Some(expected)),
+        )
+        .await
+        .expect("covered acquire_copy_bucket_incarnation_fences must not reacquire the metadata transaction lock")
+        .expect("validate same-bucket copy fences using authoritative memo");
+        assert!(dst_guard.is_none());
+        assert_eq!(src_incarnation, expected);
+        assert_eq!(dst_incarnation, expected);
+        let mut borrowed_fence_opts = ObjectOptions {
+            expected_bucket_incarnation_id: Some(expected),
+            ..Default::default()
+        };
+        src_guard.attach_to_object_options(&mut borrowed_fence_opts);
+        let fenced_incarnation = tokio::time::timeout(
+            Duration::from_secs(10),
+            crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&store.ctx, &bucket, &borrowed_fence_opts),
+        )
+        .await
+        .expect("borrowed fence under active coverage must not reacquire the metadata transaction lock")
+        .expect("resolve incarnation under borrowed fence");
+        assert_eq!(fenced_incarnation, expected);
+        assert!(
+            matches!(
+                store
+                    .acquire_copy_bucket_incarnation_fences(&bucket, Some(expected), &bucket, Some(Uuid::new_v4()))
+                    .await,
+                Err(StorageError::BucketNotFound(_))
+            ),
+            "same-bucket copy fence must reject a mismatched destination incarnation"
+        );
+        drop(src_guard);
+
+        let stale_opts = ObjectOptions {
+            expected_bucket_incarnation_id: Some(Uuid::new_v4()),
+            versioned: true,
+            ..Default::default()
+        };
+        assert!(
+            matches!(
+                store.put_object_metadata(&bucket, "src.bin", &stale_opts).await,
+                Err(StorageError::BucketNotFound(_))
+            ),
+            "put_object_metadata must reject a stale expected incarnation even under warm coverage"
+        );
+        assert!(
+            matches!(
+                store
+                    .copy_object(
+                        &bucket,
+                        "src.bin",
+                        &bucket,
+                        "src.bin",
+                        &mut src_info,
+                        &ObjectOptions::default(),
+                        &stale_opts,
+                    )
+                    .await,
+                Err(StorageError::BucketNotFound(_))
+            ),
+            "copy_object must reject a stale expected incarnation even under warm coverage"
+        );
+        drop(metadata_writer);
+        drop(coverage);
+        shutdown.cancel();
+    }
+
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]
     async fn multipart_snapshot_bypasses_initializer_queued_behind_metadata_writer() {
@@ -21469,10 +21614,25 @@ mod tests {
             .acquire_bucket_incarnation_fence(&bucket, old)
             .await
             .expect("validate with no-op lock");
+        let snapshot = store
+            .object_lock_config_snapshot_under_lifecycle_fence(&bucket, &crate::object_api::NamespaceLockFence::new())
+            .await
+            .expect("capture snapshot under no-op lock");
         let current = Uuid::new_v4();
         crate::bucket::metadata::save_bucket_incarnation(store.clone(), &bucket, current)
             .await
             .expect("replace authority while no-op guard provides no exclusion");
+        let snapshot_opts = ObjectOptions {
+            expected_bucket_incarnation_id: Some(old),
+            object_lock_config_snapshot: Some(snapshot),
+            ..Default::default()
+        };
+        assert_eq!(
+            crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&store.ctx, &bucket, &snapshot_opts)
+                .await
+                .expect("disabled lock manager must re-read incarnation from disk instead of trusting snapshot"),
+            current
+        );
         assert!(
             matches!(
                 store.acquire_bucket_incarnation_fence(&bucket, old).await,

crates/ecstore/src/store/list_objects.rs

@@ -1030,16 +1030,19 @@ impl ECStore {
         // Lock order matches writes: bucket lifecycle, metadata validation,
         // then object namespace. Borrow fences supplied by an outer operation.
         let bucket_guard = if opts.bucket_lifecycle_lock_fence.is_none() {
-            let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
-            opts.add_bucket_lifecycle_lock_guard(&guard);
+            let guard = self
+                .acquire_bucket_incarnation_fence_with_options(bucket, bucket_incarnation_id, Some(&opts))
+                .await?;
+            guard.attach_to_object_options(&mut opts);
             Some(guard)
         } else {
+            let incarnation =
+                crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
+            if incarnation != bucket_incarnation_id {
+                return Err(Error::BucketNotFound(bucket.to_owned()));
+            }
             None
         };
-        let incarnation = crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
-        if incarnation != bucket_incarnation_id {
-            return Err(Error::BucketNotFound(bucket.to_owned()));
-        }
         let metadata_sys = crate::bucket::metadata_sys::bucket_metadata_sys_of(&self.ctx)?;
         let metadata_sys = metadata_sys.read().await.clone();
         let (versioning, _) = metadata_sys.get_versioning_config(bucket).await?;

crates/ecstore/src/store/multipart.rs

@@ -412,7 +412,7 @@ impl ECStore {
 
     /// Multipart lock order is bucket lifecycle, generation validation, then
     /// object/upload locks in the selected set.
-    async fn guard_multipart_bucket_incarnation(
+    pub(super) async fn guard_multipart_bucket_incarnation(
         &self,
         bucket: &str,
         opts: &ObjectOptions,

crates/ecstore/src/store/object.rs

@@ -30,8 +30,8 @@ use crate::bucket::lifecycle::{
     },
 };
 use crate::bucket::metadata_sys::{
-    acquire_bucket_metadata_transaction_read_lock_in, get_bucket_incarnation_id_for_options_in,
-    get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in, get_object_lock_config_and_incarnation_from_disk_in,
+    acquire_bucket_metadata_transaction_read_lock_in, get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in,
+    get_object_lock_config_and_incarnation_from_disk_in,
 };
 use crate::bucket::object_lock::objectlock_sys::{
     check_object_lock_for_deletion_with_state, ensure_recursive_force_delete_allowed_for_state,
@@ -4883,24 +4883,20 @@ impl ECStore {
         if !is_meta_bucketname(dst_bucket) && dst_opts.expected_bucket_incarnation_id.is_none() {
             dst_opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(dst_bucket).await?);
         }
-        let _bucket_lifecycle_guard = if is_meta_bucketname(dst_bucket) || dst_opts.bucket_lifecycle_lock_fence.is_some() {
-            None
-        } else {
-            Some(self.acquire_bucket_lifecycle_read_lock(dst_bucket).await?)
-        };
-        let current_bucket_incarnation_id = if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
-            dst_opts.add_bucket_lifecycle_lock_guard(guard);
-            let current_incarnation_id = get_bucket_incarnation_id_for_options_in(&self.ctx, dst_bucket, &dst_opts).await?;
-            if dst_opts
-                .expected_bucket_incarnation_id
-                .is_some_and(|expected| expected != current_incarnation_id)
-            {
-                return Err(StorageError::BucketNotFound(dst_bucket.to_string()));
-            }
-            Some(current_incarnation_id)
-        } else {
-            dst_opts.expected_bucket_incarnation_id
-        };
+        let (_bucket_lifecycle_guard, current_bucket_incarnation_id) =
+            if is_meta_bucketname(dst_bucket) || dst_opts.bucket_lifecycle_lock_fence.is_some() {
+                (None, dst_opts.expected_bucket_incarnation_id)
+            } else {
+                let (guard, current_incarnation_id) = self
+                    .acquire_bucket_incarnation_fence_for_options(
+                        dst_bucket,
+                        dst_opts.expected_bucket_incarnation_id,
+                        Some(&dst_opts),
+                    )
+                    .await?;
+                guard.attach_to_object_options(&mut dst_opts);
+                (Some(guard), Some(current_incarnation_id))
+            };
         if dst_opts
             .bucket_lifecycle_lock_fence
             .as_ref()
@@ -6504,24 +6500,8 @@ impl ECStore {
         opts: &ObjectOptions,
     ) -> Result<()> {
         let object = encode_dir_object(object);
-        let mut opts = transition_restore_pool_opts(opts);
-        if !is_meta_bucketname(bucket) && opts.expected_bucket_incarnation_id.is_none() {
-            opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(bucket).await?);
-        }
-        let bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
-            None
-        } else {
-            Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
-        };
-        if let Some(guard) = bucket_lifecycle_guard.as_ref() {
-            opts.add_bucket_lifecycle_lock_guard(guard);
-        }
-        if !is_meta_bucketname(bucket) {
-            let current_incarnation_id = get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
-            if opts.expected_bucket_incarnation_id != Some(current_incarnation_id) {
-                return Err(StorageError::BucketNotFound(bucket.to_string()));
-            }
-        }
+        let opts = transition_restore_pool_opts(opts);
+        let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, &opts).await?;
         if opts.overwrites_existing_version() && !is_meta_bucketname(bucket) {
             let expected_incarnation_id = opts
                 .expected_bucket_incarnation_id
@@ -6589,16 +6569,11 @@ impl ECStore {
         let bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
             None
         } else {
-            let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
-            let current_incarnation_id = get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?;
-            if opts
-                .expected_bucket_incarnation_id
-                .is_some_and(|expected| expected != current_incarnation_id)
-            {
-                return Err(StorageError::BucketNotFound(bucket.to_string()));
-            }
+            let (guard, current_incarnation_id) = self
+                .acquire_bucket_incarnation_fence_for_options(bucket, opts.expected_bucket_incarnation_id, Some(&opts))
+                .await?;
             opts.expected_bucket_incarnation_id = Some(current_incarnation_id);
-            opts.add_bucket_lifecycle_lock_guard(&guard);
+            guard.attach_to_object_options(&mut opts);
             if guard.is_lock_lost() {
                 return Err(Error::other("bucket lifecycle lock was lost before the metadata update"));
             }
@@ -13167,6 +13142,202 @@ mod tests {
         })
         .await;
     }
+
+    async fn assert_delete_excludes_recovery_across_pool_lock_domains(topology: &str, batch: bool) {
+        let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
+        let (_first_dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+        let (_second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+        let mut store = new_prepared_reader_test_store_with_ctx(&[first_set, second_set], Arc::clone(&ctx)).await;
+        let clients: Vec<Arc<dyn rustfs_lock::LockClient>> = (0..4)
+            .map(|_| {
+                Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(
+                    Arc::new(rustfs_lock::FastObjectLockManager::new()),
+                )))) as Arc<dyn rustfs_lock::LockClient>
+            })
+            .collect();
+        let domains = match topology {
+            "disjoint" => [vec![clients[0].clone()], vec![clients[1].clone()]],
+            "identical" => [clients[..3].to_vec(), clients[..3].to_vec()],
+            "overlapping" => [clients[..3].to_vec(), clients[1..].to_vec()],
+            _ => panic!("unknown lock topology"),
+        };
+        for (pool, lockers) in store.pools.iter_mut().zip(domains) {
+            let mut set = (*pool.disk_set[0]).clone();
+            set.lockers = lockers;
+            Arc::make_mut(pool).disk_set[0] = Arc::new(set);
+        }
+        let store = Arc::new(store);
+        crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
+        let bucket = "delete-nonzero-pool-lock";
+        let object = "object.bin";
+        store
+            .handle_make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create the bucket in both pools");
+        if batch {
+            // Latest-marker routing sends this batch only to the nonzero pool,
+            // so the cleanup barrier observes its prepared journal directly.
+            store
+                .update_bucket_metadata_config(
+                    bucket,
+                    crate::bucket::metadata::BUCKET_VERSIONING_CONFIG,
+                    b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
+                )
+                .await
+                .expect("enable versioning for the routed batch");
+        }
+        let target = store.pools[1].get_disks_by_key(object);
+        let source = target
+            .put_object(
+                bucket,
+                object,
+                &mut PutObjReader::from_vec(b"delete lock regression".to_vec()),
+                &ObjectOptions {
+                    versioned: batch,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("seed an object only in the nonzero pool");
+        ctx.update_erasure_type(SetupType::DistErasure).await;
+
+        let object_barrier = (!batch).then(|| crate::set_disk::DeleteObjectCommitBarrier::install(bucket, object));
+        let batch_barrier = batch.then(|| crate::set_disk::DeleteCleanupBarrier::install(bucket));
+        let delete_store = Arc::clone(&store);
+        let delete = tokio::spawn(async move {
+            if batch {
+                let (_, errors) = delete_store
+                    .delete_objects(
+                        bucket,
+                        vec![ObjectToDelete {
+                            object_name: object.to_owned(),
+                            ..Default::default()
+                        }],
+                        ObjectOptions::default(),
+                    )
+                    .await;
+                match errors.into_iter().flatten().next() {
+                    Some(error) => Err(error),
+                    None => Ok(()),
+                }
+            } else {
+                delete_store
+                    .delete_object(bucket, object, ObjectOptions::default())
+                    .await
+                    .map(|_| ())
+            }
+        });
+        if let Some(barrier) = &object_barrier {
+            barrier.wait_until_paused().await;
+        }
+        if let Some(barrier) = &batch_barrier {
+            barrier.wait_until_paused().await;
+        }
+        let recovery_store = Arc::clone(&store);
+        let mut recovery = tokio::spawn(async move {
+            recovery_store
+                .recover_delete_transactions_at_incarnation(
+                    bucket,
+                    Some(object),
+                    None,
+                    &HealOpts {
+                        pool: Some(1),
+                        set: Some(0),
+                        ..Default::default()
+                    },
+                )
+                .await
+        });
+        let early_recovery = tokio::time::timeout(Duration::from_millis(100), &mut recovery).await;
+        let recovery_was_blocked = early_recovery.is_err();
+        if let Some(barrier) = &object_barrier {
+            barrier.release();
+        }
+        if let Some(barrier) = &batch_barrier {
+            barrier.release();
+        }
+        tokio::time::timeout(Duration::from_secs(10), delete)
+            .await
+            .expect("the delete should finish after releasing its commit barrier")
+            .expect("the delete task should join")
+            .expect("the delete should succeed");
+        let recovered = match early_recovery {
+            Ok(result) => result,
+            Err(_) => tokio::time::timeout(Duration::from_secs(10), recovery)
+                .await
+                .expect("recovery should finish after the frontend delete"),
+        };
+        recovered
+            .expect("the recovery task should join")
+            .expect("recovery should succeed");
+        assert!(
+            recovery_was_blocked,
+            "recovery must not inspect an in-flight frontend delete with {topology} pool lock clients"
+        );
+        if batch {
+            let marker = target
+                .get_object_info(bucket, object, &ObjectOptions::default())
+                .await
+                .expect("the versioned batch must publish a current delete marker");
+            assert!(marker.delete_marker && marker.is_latest);
+            assert_eq!(marker.num_versions, 2, "the source version must remain beside the new marker");
+            assert_ne!(marker.version_id, source.version_id);
+            let mut reader = target
+                .get_object_reader(
+                    bucket,
+                    object,
+                    None,
+                    HeaderMap::new(),
+                    &ObjectOptions {
+                        version_id: Some(source.version_id.expect("versioned source identity").to_string()),
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("the original source version must remain readable");
+            let mut body = Vec::new();
+            reader
+                .stream
+                .read_to_end(&mut body)
+                .await
+                .expect("read the retained source version");
+            assert_eq!(body, b"delete lock regression");
+        } else {
+            let error = target
+                .get_object_info(bucket, object, &ObjectOptions::default())
+                .await
+                .expect_err("the deleted object must be absent");
+            assert!(is_err_object_not_found(&error));
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    #[serial_test::serial(storage_class_env)]
+    async fn batch_delete_excludes_recovery_on_nonzero_pool() {
+        assert_delete_excludes_recovery_across_pool_lock_domains("disjoint", true).await;
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    #[serial_test::serial(storage_class_env)]
+    async fn delete_excludes_recovery_with_disjoint_pool_lock_clients() {
+        assert_delete_excludes_recovery_across_pool_lock_domains("disjoint", false).await;
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    #[serial_test::serial(storage_class_env)]
+    async fn delete_excludes_recovery_with_identical_pool_lock_clients() {
+        assert_delete_excludes_recovery_across_pool_lock_domains("identical", false).await;
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    #[serial_test::serial(storage_class_env)]
+    async fn delete_excludes_recovery_with_overlapping_pool_lock_clients() {
+        assert_delete_excludes_recovery_across_pool_lock_domains("overlapping", false).await;
+    }
 }
 
 #[cfg(test)]

crates/filemeta/src/filemeta.rs

@@ -2429,6 +2429,52 @@ mod test {
         }
     }
 
+    #[test]
+    fn truncated_inline_xlmeta_is_file_corrupt_only_for_supported_formats() {
+        let mut meta = FileMeta::new();
+        meta.data
+            .replace("fixture", vec![0x5a; 4095])
+            .expect("seed real inline framing");
+        let inline_len = meta.data.as_slice().len();
+        let original = meta.marshal_msg().expect("serialize valid inline metadata");
+        let inline_start = original.len() - inline_len;
+        let parsed = FileMeta::load(&original).expect("complete inline metadata must decode");
+        assert_eq!(parsed.data, meta.data);
+        for cut in inline_start + 1..original.len() {
+            assert_eq!(
+                FileMeta::load(&original[..cut]).expect_err("incomplete inline framing must fail"),
+                Error::FileCorrupt,
+                "inline truncation at byte {cut} must remain repairable"
+            );
+        }
+
+        for inline_version in [0, 2] {
+            let mut unsupported = original[..original.len() / 2].to_vec();
+            unsupported[inline_start] = inline_version;
+            assert_ne!(
+                FileMeta::load(&unsupported).expect_err("malformed unsupported inline format must fail"),
+                Error::FileCorrupt,
+                "unknown inline version must not authorize repair"
+            );
+        }
+        for (header_ver, meta_ver) in [
+            (XL_HEADER_VERSION + 1, XL_META_VERSION),
+            (XL_HEADER_VERSION, XL_META_VERSION + 1),
+        ] {
+            let mut headers = Vec::new();
+            rmp::encode::write_uint(&mut headers, u64::from(header_ver)).expect("write future header");
+            rmp::encode::write_uint(&mut headers, u64::from(meta_ver)).expect("write future metadata version");
+            rmp::encode::write_uint(&mut headers, 0).expect("write empty version count");
+            let mut unsupported = build_xl_buffer(&headers);
+            unsupported.extend_from_slice(&meta.data.as_slice()[..inline_len / 2]);
+            assert_ne!(
+                FileMeta::load(&unsupported).expect_err("unsupported metadata with truncated inline data must fail"),
+                Error::FileCorrupt,
+                "future metadata schema must not authorize repair"
+            );
+        }
+    }
+
     #[test]
     fn truncated_xlmeta_index_and_format_reads_are_file_corrupt() {
         let buf = FileMeta::default()

crates/filemeta/src/filemeta/codec.rs

@@ -188,7 +188,13 @@ impl FileMeta {
         if !buf.is_empty() {
             InlineData::validate_slice(buf).map_err(|e| {
                 error!("data validation failed: {}", e);
-                e
+                // Validation only examines bytes already read. Classify damage
+                // as repairable corruption only for formats we understand.
+                if InlineData::version_ok_slice(buf) && Self::decode_xl_headers(meta).is_ok() {
+                    Error::FileCorrupt
+                } else {
+                    e
+                }
             })?;
         }
 

crates/filemeta/src/filemeta_inline.rs

@@ -168,11 +168,11 @@ impl InlineData {
         self.0.as_slice()
     }
     pub fn version_ok(&self) -> bool {
-        if self.0.is_empty() {
-            return true;
-        }
+        Self::version_ok_slice(&self.0)
+    }
 
-        self.0[0] > 0 && self.0[0] <= INLINE_DATA_VER
+    pub(crate) fn version_ok_slice(raw: &[u8]) -> bool {
+        raw.first().is_none_or(|version| *version > 0 && *version <= INLINE_DATA_VER)
     }
 
     pub fn after_version(&self) -> &[u8] {

crates/heal-contracts/Cargo.toml

@@ -29,9 +29,8 @@ workspace = true
 
 [dependencies]
 serde = { workspace = true, features = ["derive"] }
-thiserror.workspace = true
 tokio = { workspace = true, features = ["sync"] }
-uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
+uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
 
 [dev-dependencies]
 serde_json = { workspace = true }

crates/heal-contracts/src/completion.rs

@@ -18,25 +18,12 @@ use crate::{
     heal_channel::HealScanMode,
     outcome::{HealExecutionOutcome, HealObjectReceipt},
 };
-use std::sync::{
-    Arc, OnceLock,
-    atomic::{AtomicU64, Ordering},
-};
+use std::sync::{Arc, OnceLock};
 use tokio::sync::broadcast;
 
 pub const MAX_COMPLETION_IDENTITY_BYTES: usize = 4096;
 const COMPLETION_CHANNEL_CAPACITY: usize = 1024;
 
-/// A process-local observation identity. It is never persisted or sent over RPC.
-pub type HealObservationId = u64;
-
-/// Allocate one identity per accepted owner. Exhaustion is reported as unknown.
-pub fn new_heal_observation_id() -> Option<HealObservationId> {
-    static NEXT: AtomicU64 = AtomicU64::new(1);
-    NEXT.try_update(Ordering::Relaxed, Ordering::Relaxed, |value| value.checked_add(1))
-        .ok()
-}
-
 #[derive(Debug, Clone)]
 pub struct HealCompletion {
     /// The accepted owner, never a discarded merged request ID.
@@ -49,240 +36,70 @@ pub struct HealCompletion {
     pub object: Option<HealObjectReceipt>,
 }
 
-/// A completion paired with an optional in-process owner identity.
-#[derive(Debug, Clone)]
-pub struct ObservedHealCompletion {
-    pub completion: Arc<HealCompletion>,
-    pub observation_id: Option<HealObservationId>,
-}
-
-impl std::ops::Deref for ObservedHealCompletion {
-    type Target = HealCompletion;
-
-    fn deref(&self) -> &Self::Target {
-        &self.completion
-    }
-}
-
-struct CompletionBroadcaster {
-    legacy: broadcast::Sender<Arc<HealCompletion>>,
-    observed: broadcast::Sender<Arc<ObservedHealCompletion>>,
-}
-
-impl CompletionBroadcaster {
-    fn new(capacity: usize) -> Self {
-        Self {
-            legacy: broadcast::channel(capacity).0,
-            observed: broadcast::channel(capacity).0,
-        }
-    }
-
-    fn has_observers(&self) -> bool {
-        self.legacy.receiver_count() > 0 || self.observed.receiver_count() > 0
-    }
-
-    fn publish(&self, completion: HealCompletion, observation_id: Option<HealObservationId>) {
-        let identity_bytes = completion.object.as_ref().map_or(0, |object| {
-            object
-                .identity
-                .bucket
-                .capacity()
-                .saturating_add(object.identity.object.capacity())
-                .saturating_add(object.identity.version_id.as_ref().map_or(0, String::capacity))
-        });
-        if completion.task_id.capacity().saturating_add(identity_bytes) <= MAX_COMPLETION_IDENTITY_BYTES {
-            let completion = Arc::new(completion);
-            let _ = self.legacy.send(Arc::clone(&completion));
-            let _ = self.observed.send(Arc::new(ObservedHealCompletion {
-                completion,
-                observation_id,
-            }));
-        }
-    }
-}
-
-fn broadcaster() -> &'static CompletionBroadcaster {
-    static BROADCASTER: OnceLock<CompletionBroadcaster> = OnceLock::new();
-    BROADCASTER.get_or_init(|| CompletionBroadcaster::new(COMPLETION_CHANNEL_CAPACITY))
+fn sender() -> &'static broadcast::Sender<Arc<HealCompletion>> {
+    static SENDER: OnceLock<broadcast::Sender<Arc<HealCompletion>>> = OnceLock::new();
+    SENDER.get_or_init(|| broadcast::channel(COMPLETION_CHANNEL_CAPACITY).0)
 }
 
 pub fn has_heal_completion_observers() -> bool {
-    broadcaster().has_observers()
+    sender().receiver_count() > 0
 }
 
 /// Subscribe before admission so a fast worker cannot precede subscription.
 pub fn subscribe_heal_completions() -> broadcast::Receiver<Arc<HealCompletion>> {
-    broadcaster().legacy.subscribe()
-}
-
-pub fn subscribe_observed_heal_completions() -> broadcast::Receiver<Arc<ObservedHealCompletion>> {
-    broadcaster().observed.subscribe()
+    sender().subscribe()
 }
 
 /// Oversized diagnostics and lagging consumers lose observations, never repairs.
 pub fn publish_heal_completion(completion: HealCompletion) {
-    publish_observed_heal_completion(completion, None);
-}
-
-/// Each channel is FIFO independently; relative delivery across channels is not ordered.
-pub fn publish_observed_heal_completion(completion: HealCompletion, observation_id: Option<HealObservationId>) {
-    broadcaster().publish(completion, observation_id);
+    let identity_bytes = completion.object.as_ref().map_or(0, |object| {
+        object
+            .identity
+            .bucket
+            .capacity()
+            .saturating_add(object.identity.object.capacity())
+            .saturating_add(object.identity.version_id.as_ref().map_or(0, String::capacity))
+    });
+    if completion.task_id.capacity().saturating_add(identity_bytes) <= MAX_COMPLETION_IDENTITY_BYTES {
+        let _ = sender().send(Arc::new(completion));
+    }
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
 
-    #[test]
-    fn public_completion_facades_share_receipts_and_preserve_missing_tokens() {
-        // Other completion tests use private broadcasters, keeping this global
-        // facade test isolated from their capacity and lag checks.
-        let mut legacy = subscribe_heal_completions();
-        let mut observed = subscribe_observed_heal_completions();
-        let completion = HealCompletion {
-            task_id: uuid::Uuid::new_v4().to_string(),
-            scan_mode: HealScanMode::Deep,
-            dry_run: false,
-            execution: None,
-            object: None,
-        };
-        publish_heal_completion(completion.clone());
-        let legacy_event = legacy.try_recv().expect("legacy subscriber receives old API event");
-        let observed_event = observed.try_recv().expect("observer receives old API event");
-        assert!(Arc::ptr_eq(&legacy_event, &observed_event.completion));
-        assert_eq!(legacy_event.task_id, completion.task_id);
-        assert_eq!(observed_event.observation_id, None);
-
-        publish_observed_heal_completion(completion, Some(51));
-        let legacy_event = legacy.try_recv().expect("legacy subscriber receives observed API event");
-        let observed_event = observed.try_recv().expect("observer receives observed API event");
-        assert!(Arc::ptr_eq(&legacy_event, &observed_event.completion));
-        assert_eq!(observed_event.observation_id, Some(51));
-    }
-
-    #[test]
-    fn legacy_and_observed_subscribers_receive_the_same_completion_in_each_channel_order() {
-        let id = uuid::Uuid::new_v4().to_string();
-        let bus = CompletionBroadcaster::new(COMPLETION_CHANNEL_CAPACITY);
-        let mut legacy = bus.legacy.subscribe();
-        let mut observed = bus.observed.subscribe();
-        assert!(bus.has_observers());
-        for (suffix, token) in [("first", Some(41)), ("second", None)] {
-            bus.publish(
-                HealCompletion {
-                    task_id: format!("{id}-{suffix}"),
-                    scan_mode: HealScanMode::Normal,
-                    dry_run: false,
-                    execution: None,
-                    object: None,
-                },
-                token,
-            );
-        }
-        for (suffix, token) in [("first", Some(41)), ("second", None)] {
-            let expected = format!("{id}-{suffix}");
-            let legacy_event = loop {
-                let event = legacy.try_recv().expect("legacy completion");
-                if event.task_id == expected {
-                    break event;
-                }
-            };
-            let observed_event = loop {
-                let event = observed.try_recv().expect("observed completion");
-                if event.task_id == expected {
-                    break event;
-                }
-            };
-            assert!(Arc::ptr_eq(&legacy_event, &observed_event.completion));
-            assert_eq!(observed_event.observation_id, token);
-        }
-        drop(legacy);
-        assert!(bus.has_observers(), "observed-only subscribers require completion construction");
-        let completion = HealCompletion {
-            task_id: id.clone(),
-            scan_mode: HealScanMode::Deep,
-            dry_run: false,
-            execution: None,
-            object: None,
-        };
-        bus.publish(completion, None);
-        let observed_event = loop {
-            let event = observed.try_recv().expect("legacy publication reaches observed channel");
-            if event.task_id == id {
-                break event;
-            }
-        };
-        assert_eq!(observed_event.observation_id, None);
-    }
-
-    #[test]
-    fn legacy_and_observed_channels_report_lag_independently() {
-        let bus = CompletionBroadcaster::new(COMPLETION_CHANNEL_CAPACITY);
-        let mut legacy = bus.legacy.subscribe();
-        let mut observed = bus.observed.subscribe();
-        for index in 0..=COMPLETION_CHANNEL_CAPACITY {
-            bus.publish(
-                HealCompletion {
-                    task_id: format!("dual-lag-{index}"),
-                    scan_mode: HealScanMode::Normal,
-                    dry_run: false,
-                    execution: None,
-                    object: None,
-                },
-                Some(42),
-            );
-        }
-        assert!(matches!(legacy.try_recv(), Err(broadcast::error::TryRecvError::Lagged(_))));
-        assert!(matches!(observed.try_recv(), Err(broadcast::error::TryRecvError::Lagged(_))));
-    }
-
     #[test]
     fn completion_delivery_is_bounded_and_does_not_require_a_consumer() {
-        let bus = CompletionBroadcaster::new(COMPLETION_CHANNEL_CAPACITY);
-        assert!(!bus.has_observers());
-        let mut receiver = bus.legacy.subscribe();
-        assert!(bus.has_observers());
+        let mut receiver = subscribe_heal_completions();
         for index in 0..=COMPLETION_CHANNEL_CAPACITY {
-            bus.publish(
-                HealCompletion {
-                    task_id: format!("task-{index}"),
-                    scan_mode: HealScanMode::Deep,
-                    dry_run: false,
-                    execution: Some(HealExecutionOutcome::Completed),
-                    object: None,
-                },
-                None,
-            );
+            publish_heal_completion(HealCompletion {
+                task_id: format!("task-{index}"),
+                scan_mode: HealScanMode::Deep,
+                dry_run: false,
+                execution: Some(HealExecutionOutcome::Completed),
+                object: None,
+            });
         }
         assert!(matches!(receiver.try_recv(), Err(broadcast::error::TryRecvError::Lagged(1))));
         while receiver.try_recv().is_ok() {}
         let mut oversized = String::with_capacity(MAX_COMPLETION_IDENTITY_BYTES + 1);
         oversized.push_str("short-but-large-allocation");
-        let mut observed = bus.observed.subscribe();
-        bus.publish(
-            HealCompletion {
-                task_id: oversized,
-                scan_mode: HealScanMode::Deep,
-                dry_run: false,
-                execution: None,
-                object: None,
-            },
-            None,
-        );
+        publish_heal_completion(HealCompletion {
+            task_id: oversized,
+            scan_mode: HealScanMode::Deep,
+            dry_run: false,
+            execution: None,
+            object: None,
+        });
         assert!(matches!(receiver.try_recv(), Err(broadcast::error::TryRecvError::Empty)));
-        assert!(matches!(observed.try_recv(), Err(broadcast::error::TryRecvError::Empty)));
         drop(receiver);
-        drop(observed);
-        assert!(!bus.has_observers());
-        bus.publish(
-            HealCompletion {
-                task_id: "no-consumer".into(),
-                scan_mode: HealScanMode::Deep,
-                dry_run: false,
-                execution: Some(HealExecutionOutcome::Completed),
-                object: None,
-            },
-            None,
-        );
+        publish_heal_completion(HealCompletion {
+            task_id: "no-consumer".into(),
+            scan_mode: HealScanMode::Deep,
+            dry_run: false,
+            execution: Some(HealExecutionOutcome::Completed),
+            object: None,
+        });
     }
 }

crates/heal-contracts/src/heal_channel.rs

@@ -271,39 +271,6 @@ pub struct HealAdmissionReceipt {
     pub task_id: String,
 }
 
-/// Optional process-local observation metadata for a canonical receipt.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct ObservedHealAdmissionReceipt {
-    pub receipt: HealAdmissionReceipt,
-    /// Process-local owner identity; absent for remote or unproven owners.
-    pub observation_id: Option<crate::completion::HealObservationId>,
-}
-
-/// Errors from process-local observed admission and its channel initialization.
-#[derive(Debug, thiserror::Error)]
-pub enum HealObservationChannelError {
-    #[error("Heal channel sender already initialized")]
-    AlreadyInitialized,
-    #[error("Heal channel not initialized")]
-    NotInitialized,
-    #[error("Heal receipt command channel closed")]
-    LegacyCommandClosed,
-    #[error("Failed to receive heal admission response: {0}")]
-    LegacyResponseClosed(#[source] oneshot::error::RecvError),
-    #[error("Failed to receive observed heal admission response: {0}")]
-    ObservedResponseClosed(#[source] oneshot::error::RecvError),
-    #[error("{0}")]
-    AdmissionFailed(String),
-}
-
-impl std::ops::Deref for ObservedHealAdmissionReceipt {
-    type Target = HealAdmissionReceipt;
-
-    fn deref(&self) -> &Self::Target {
-        &self.receipt
-    }
-}
-
 impl HealAdmissionResult {
     pub fn result_label(self) -> &'static str {
         match self {
@@ -480,19 +447,9 @@ pub struct HealReceiptCommand {
 /// Canonical-receipt command receiver.
 pub type HealReceiptReceiver = mpsc::UnboundedReceiver<HealReceiptCommand>;
 
-/// Process-local admission command carrying an observed receipt.
-#[derive(Debug)]
-pub struct ObservedHealReceiptCommand {
-    pub request: HealChannelRequest,
-    pub response_tx: oneshot::Sender<Result<ObservedHealAdmissionReceipt, HealObservationChannelError>>,
-}
-
-pub type ObservedHealReceiptReceiver = mpsc::UnboundedReceiver<ObservedHealReceiptCommand>;
-
 struct HealChannelSenders {
     command: HealChannelSender,
     receipt: mpsc::UnboundedSender<HealReceiptCommand>,
-    observed: Option<mpsc::UnboundedSender<ObservedHealReceiptCommand>>,
 }
 
 /// Global heal channel sender
@@ -512,31 +469,12 @@ pub fn init_heal_channel() -> Result<HealChannelReceiver, &'static str> {
 
 /// Initialize the legacy command and canonical-receipt channels atomically.
 pub fn init_heal_channels() -> Result<(HealChannelReceiver, HealReceiptReceiver), &'static str> {
-    let (command_receiver, receipt_receiver, _) =
-        init_heal_channels_inner(false).map_err(|_| "Heal channel sender already initialized")?;
-    Ok((command_receiver, receipt_receiver))
-}
-
-/// Initialize an additional local observation channel while preserving the legacy channels.
-pub fn init_heal_channels_with_observations()
--> Result<(HealChannelReceiver, HealReceiptReceiver, ObservedHealReceiptReceiver), HealObservationChannelError> {
-    init_heal_channels_inner(true)
-}
-
-fn init_heal_channels_inner(
-    with_observations: bool,
-) -> Result<(HealChannelReceiver, HealReceiptReceiver, ObservedHealReceiptReceiver), HealObservationChannelError> {
     let (command, command_receiver) = mpsc::unbounded_channel();
     let (receipt, receipt_receiver) = mpsc::unbounded_channel();
-    let (observed, observed_receiver) = mpsc::unbounded_channel();
     GLOBAL_HEAL_CHANNEL_SENDERS
-        .set(HealChannelSenders {
-            command,
-            receipt,
-            observed: with_observations.then_some(observed),
-        })
-        .map_err(|_| HealObservationChannelError::AlreadyInitialized)?;
-    Ok((command_receiver, receipt_receiver, observed_receiver))
+        .set(HealChannelSenders { command, receipt })
+        .map_err(|_| "Heal channel sender already initialized")?;
+    Ok((command_receiver, receipt_receiver))
 }
 
 /// Get global heal channel sender
@@ -577,17 +515,10 @@ pub fn subscribe_heal_responses() -> broadcast::Receiver<HealChannelResponse> {
 
 /// Send heal start request and wait for structured admission feedback.
 pub async fn send_heal_request_with_receipt(request: HealChannelRequest) -> Result<HealAdmissionReceipt, String> {
+    let (response_tx, response_rx) = oneshot::channel();
     let senders = GLOBAL_HEAL_CHANNEL_SENDERS
         .get()
         .ok_or_else(|| "Heal channel not initialized".to_string())?;
-    send_heal_request_with_receipt_on(senders, request).await
-}
-
-async fn send_heal_request_with_receipt_on(
-    senders: &HealChannelSenders,
-    request: HealChannelRequest,
-) -> Result<HealAdmissionReceipt, String> {
-    let (response_tx, response_rx) = oneshot::channel();
     senders
         .receipt
         .send(HealReceiptCommand { request, response_tx })
@@ -597,46 +528,6 @@ async fn send_heal_request_with_receipt_on(
         .map_err(|e| format!("Failed to receive heal admission response: {e}"))?
 }
 
-/// A missing or closed observation channel falls back only before a command is accepted.
-pub async fn send_heal_request_with_observation(
-    request: HealChannelRequest,
-) -> Result<ObservedHealAdmissionReceipt, HealObservationChannelError> {
-    let senders = GLOBAL_HEAL_CHANNEL_SENDERS
-        .get()
-        .ok_or(HealObservationChannelError::NotInitialized)?;
-    send_heal_request_with_observation_on(senders, request).await
-}
-
-async fn send_heal_request_with_observation_on(
-    senders: &HealChannelSenders,
-    mut request: HealChannelRequest,
-) -> Result<ObservedHealAdmissionReceipt, HealObservationChannelError> {
-    if let Some(observed) = &senders.observed {
-        let (response_tx, response_rx) = oneshot::channel();
-        match observed.send(ObservedHealReceiptCommand { request, response_tx }) {
-            Ok(()) => {
-                return response_rx
-                    .await
-                    .map_err(HealObservationChannelError::ObservedResponseClosed)?;
-            }
-            Err(error) => request = error.0.request,
-        }
-    }
-    let (response_tx, response_rx) = oneshot::channel();
-    senders
-        .receipt
-        .send(HealReceiptCommand { request, response_tx })
-        .map_err(|_| HealObservationChannelError::LegacyCommandClosed)?;
-    let receipt = response_rx
-        .await
-        .map_err(HealObservationChannelError::LegacyResponseClosed)?
-        .map_err(HealObservationChannelError::AdmissionFailed)?;
-    Ok(ObservedHealAdmissionReceipt {
-        receipt,
-        observation_id: None,
-    })
-}
-
 /// Send heal start request and wait for structured admission feedback.
 pub async fn send_heal_request_with_admission(request: HealChannelRequest) -> Result<HealAdmissionResult, String> {
     let (response_tx, response_rx) = oneshot::channel();
@@ -884,109 +775,6 @@ mod auto_heal_disk_request_tests {
 mod tests {
     use super::*;
 
-    #[tokio::test]
-    async fn observed_receipt_falls_back_only_when_no_observed_command_was_accepted() {
-        for closed in [false, true] {
-            let (command, _command_rx) = mpsc::unbounded_channel();
-            let (receipt, mut receipt_rx) = mpsc::unbounded_channel();
-            let observed = if closed {
-                let (sender, receiver) = mpsc::unbounded_channel();
-                drop(receiver);
-                Some(sender)
-            } else {
-                None
-            };
-            let senders = HealChannelSenders {
-                command,
-                receipt,
-                observed,
-            };
-            let request = HealChannelRequest::new();
-            let task = tokio::spawn(async move { send_heal_request_with_observation_on(&senders, request).await });
-            let command = receipt_rx.recv().await.expect("legacy receipt command");
-            command
-                .response_tx
-                .send(Ok(HealAdmissionReceipt {
-                    result: HealAdmissionResult::Accepted,
-                    task_id: "owner".into(),
-                }))
-                .expect("caller waiting");
-            let received = task.await.expect("sender task").expect("legacy admission");
-            assert_eq!(received.task_id, "owner");
-            assert_eq!(received.observation_id, None);
-        }
-    }
-
-    #[tokio::test]
-    async fn lost_observed_reply_never_resends_the_request() {
-        let (command, _command_rx) = mpsc::unbounded_channel();
-        let (receipt, mut receipt_rx) = mpsc::unbounded_channel();
-        let keep_receipt_open = receipt.clone();
-        let (observed, mut observed_rx) = mpsc::unbounded_channel();
-        let senders = HealChannelSenders {
-            command,
-            receipt,
-            observed: Some(observed),
-        };
-        let task = tokio::spawn(async move { send_heal_request_with_observation_on(&senders, HealChannelRequest::new()).await });
-        let command = observed_rx.recv().await.expect("accepted observed command");
-        drop(command.response_tx);
-        let error = task
-            .await
-            .expect("sender task")
-            .expect_err("lost response must remain an error");
-        assert!(matches!(&error, HealObservationChannelError::ObservedResponseClosed(_)));
-        assert!(std::error::Error::source(&error).is_some());
-        assert!(matches!(receipt_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)));
-        drop(keep_receipt_open);
-    }
-
-    #[tokio::test]
-    async fn fallback_reports_a_typed_error_if_legacy_channel_is_closed() {
-        let (command, _command_rx) = mpsc::unbounded_channel();
-        let (receipt, receipt_rx) = mpsc::unbounded_channel();
-        drop(receipt_rx);
-        let senders = HealChannelSenders {
-            command,
-            receipt,
-            observed: None,
-        };
-        let error = send_heal_request_with_observation_on(&senders, HealChannelRequest::new())
-            .await
-            .expect_err("closed legacy channel must not accept work");
-        assert!(matches!(error, HealObservationChannelError::LegacyCommandClosed));
-    }
-
-    #[tokio::test]
-    async fn observed_receipt_keeps_the_owner_token_from_the_processor() {
-        let (command, _command_rx) = mpsc::unbounded_channel();
-        let (receipt, mut receipt_rx) = mpsc::unbounded_channel();
-        let keep_receipt_open = receipt.clone();
-        let (observed, mut observed_rx) = mpsc::unbounded_channel();
-        let senders = HealChannelSenders {
-            command,
-            receipt,
-            observed: Some(observed),
-        };
-        let task = tokio::spawn(async move { send_heal_request_with_observation_on(&senders, HealChannelRequest::new()).await });
-        let command = observed_rx.recv().await.expect("observed command");
-        command
-            .response_tx
-            .send(Ok(ObservedHealAdmissionReceipt {
-                receipt: HealAdmissionReceipt {
-                    result: HealAdmissionResult::Merged,
-                    task_id: "owner".into(),
-                },
-                observation_id: Some(37),
-            }))
-            .expect("caller waiting");
-        let received = task.await.expect("sender task").expect("observed admission");
-        assert_eq!(received.task_id, "owner");
-        assert_eq!(received.observation_id, Some(37));
-        assert!(matches!(receipt_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)));
-        drop(keep_receipt_open);
-    }
-
     #[test]
     fn heal_request_source_labels_are_stable() {
         assert_eq!(HealRequestSource::Scanner.as_str(), "scanner");

crates/heal/src/error.rs

@@ -16,7 +16,7 @@ use rustfs_io_metrics::internode_metrics::INTERNODE_OPERATION_PUT_FILE_STREAM;
 use rustfs_rio::{InternodeHttpError, InternodeHttpErrorKind};
 use thiserror::Error;
 
-use super::heal::{DiskError, EcstoreError};
+use super::heal::{DiskError, EcstoreError, storage_api::owner::EcstorePoolMetadataFailure};
 
 const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
 pub(crate) const MRF_MISSING_PROOF_PREFIX: &str = "Missing exact storage proof for durable MRF repair ";
@@ -236,6 +236,14 @@ pub(crate) fn classify_heal_error(error: &Error) -> HealErrorClass {
 /// with object absence, the conflation that once let an offline drive be
 /// recorded as absent and skipped (backlog#856 / #799 B7).
 pub(crate) fn classify_storage_error(error: &EcstoreError) -> HealErrorClass {
+    if let Some(failure) = error.pool_metadata_failure() {
+        return match failure.kind {
+            EcstorePoolMetadataFailure::ReadUnavailable | EcstorePoolMetadataFailure::TransactionUnknown => {
+                HealErrorClass::Transient
+            }
+            EcstorePoolMetadataFailure::RecoveryRequired | EcstorePoolMetadataFailure::FenceLost => HealErrorClass::Permanent,
+        };
+    }
     if let EcstoreError::Lock(lock_error) = error {
         return if lock_error.is_fatal() {
             HealErrorClass::Permanent
@@ -435,6 +443,34 @@ mod tests {
     use super::Error;
     use crate::heal::{DiskError, EcstoreError};
 
+    #[test]
+    fn pool_metadata_failures_keep_their_typed_recovery_classification() {
+        use crate::heal::storage_api::owner::{EcstorePoolMetadataError, EcstorePoolMetadataFailure};
+
+        for (kind, retryable) in [
+            (EcstorePoolMetadataFailure::ReadUnavailable, true),
+            (EcstorePoolMetadataFailure::TransactionUnknown, true),
+            (EcstorePoolMetadataFailure::RecoveryRequired, false),
+            (EcstorePoolMetadataFailure::FenceLost, false),
+        ] {
+            let error = EcstoreError::other(EcstorePoolMetadataError {
+                kind,
+                operation: "bucket heal".to_owned(),
+                phase: "native_heal",
+                since: std::time::UNIX_EPOCH.into(),
+                source: Some(std::sync::Arc::new(EcstoreError::SlowDown)),
+            });
+            let text = error.to_string();
+            for error in [error.clone(), error] {
+                assert_eq!(Error::Storage(error).is_recoverable_heal(), retryable, "{kind:?}");
+            }
+            assert!(
+                !Error::Storage(EcstoreError::other(text)).is_recoverable_heal(),
+                "display text must not grant recovery authority"
+            );
+        }
+    }
+
     #[test]
     fn namespace_lock_quorum_is_recoverable_across_io_error_conversions() {
         for (required, achieved) in [(2, 1), (1, 0)] {

crates/heal/src/heal/channel.rs

@@ -21,8 +21,8 @@ use crate::heal::{
 use crate::{Error, Result};
 use rustfs_heal_contracts::heal_channel::{
     HealAdmissionReceipt, HealAdmissionResult, HealChannelCommand, HealChannelPriority, HealChannelReceiver, HealChannelRequest,
-    HealChannelResponse, HealObservationChannelError, HealOpts, HealReceiptCommand, HealReceiptReceiver, HealRequestSource,
-    HealScanMode, ObservedHealAdmissionReceipt, ObservedHealReceiptCommand, ObservedHealReceiptReceiver, publish_heal_response,
+    HealChannelResponse, HealOpts, HealReceiptCommand, HealReceiptReceiver, HealRequestSource, HealScanMode,
+    publish_heal_response,
 };
 use rustfs_madmin::heal_commands::HealResultItem;
 use serde::Serialize;
@@ -30,24 +30,6 @@ use std::sync::Arc;
 use tokio::sync::{mpsc, oneshot};
 use tracing::{debug, error, info};
 
-enum StartResponseSender {
-    Legacy(oneshot::Sender<std::result::Result<HealAdmissionReceipt, String>>),
-    Observed(oneshot::Sender<std::result::Result<ObservedHealAdmissionReceipt, HealObservationChannelError>>),
-}
-
-impl StartResponseSender {
-    fn send(self, result: std::result::Result<ObservedHealAdmissionReceipt, String>) {
-        match self {
-            Self::Legacy(sender) => {
-                let _ = sender.send(result.map(|observed| observed.receipt));
-            }
-            Self::Observed(sender) => {
-                let _ = sender.send(result.map_err(HealObservationChannelError::AdmissionFailed));
-            }
-        }
-    }
-}
-
 const LOG_COMPONENT_HEAL: &str = "heal";
 const LOG_SUBSYSTEM_CHANNEL: &str = "channel";
 const EVENT_HEAL_CHANNEL_STATE: &str = "heal_channel_state";
@@ -214,6 +196,20 @@ impl HealChannelProcessor {
             .map_err(Error::other)
     }
 
+    /// Query a local automatic task; other tokens require coordinator routing.
+    pub async fn execute_local_automatic_query(
+        &self,
+        heal_path: &str,
+        client_token: &str,
+        since_seq: Option<u64>,
+    ) -> Result<Option<HealChannelResponse>> {
+        self.heal_manager
+            .get_local_automatic_task_report(heal_path, client_token, since_seq)
+            .await?
+            .map(|report| Self::query_report_response(client_token.to_owned(), Ok(report)))
+            .transpose()
+    }
+
     /// Execute cancellation directly against the manager.
     pub async fn execute_cancel_request(&self, heal_path: String, client_token: String) -> Result<HealChannelResponse> {
         let (response_tx, response_rx) = oneshot::channel();
@@ -233,22 +229,9 @@ impl HealChannelProcessor {
 
     /// Start processing legacy and canonical-receipt heal channel requests.
     pub async fn start_with_receipts(
-        &mut self,
-        receiver: HealChannelReceiver,
-        receipt_receiver: HealReceiptReceiver,
-    ) -> Result<()> {
-        let (sender, receiver_observed) = mpsc::unbounded_channel();
-        drop(sender);
-        self.start_with_observed_receipts(receiver, receipt_receiver, receiver_observed)
-            .await
-    }
-
-    /// The three command channels are FIFO independently, with no ordering across channels.
-    pub async fn start_with_observed_receipts(
         &mut self,
         mut receiver: HealChannelReceiver,
         mut receipt_receiver: HealReceiptReceiver,
-        mut observed_receiver: ObservedHealReceiptReceiver,
     ) -> Result<()> {
         info!(
             target: "rustfs::heal::channel",
@@ -260,7 +243,6 @@ impl HealChannelProcessor {
         );
 
         let mut receipt_channel_open = true;
-        let mut observed_channel_open = true;
         loop {
             tokio::select! {
                 command = receiver.recv() => {
@@ -308,23 +290,6 @@ impl HealChannelProcessor {
                         );
                     }
                 }
-                command = observed_receiver.recv(), if observed_channel_open => {
-                    let Some(ObservedHealReceiptCommand { request, response_tx }) = command else {
-                        observed_channel_open = false;
-                        continue;
-                    };
-                    if let Err(e) = self.process_start_request_inner(request, false, true, StartResponseSender::Observed(response_tx)).await {
-                        error!(
-                            target: "rustfs::heal::channel",
-                            event = EVENT_HEAL_CHANNEL_REQUEST,
-                            component = LOG_COMPONENT_HEAL,
-                            subsystem = LOG_SUBSYSTEM_CHANNEL,
-                            state = "receipt_process_failed",
-                            error = %e,
-                            "Heal receipt request processing failed"
-                        );
-                    }
-                }
                 response = self.response_receiver.recv() => {
                     if let Some(response) = response {
                         // Handle response if needed
@@ -392,22 +357,11 @@ impl HealChannelProcessor {
 
     /// Process start request
     async fn process_start_request(
-        &self,
-        request: HealChannelRequest,
-        preserve_alias: bool,
-        publish_canonical_id: bool,
-        response_tx: oneshot::Sender<std::result::Result<HealAdmissionReceipt, String>>,
-    ) -> Result<()> {
-        self.process_start_request_inner(request, preserve_alias, publish_canonical_id, StartResponseSender::Legacy(response_tx))
-            .await
-    }
-
-    async fn process_start_request_inner(
         &self,
         mut request: HealChannelRequest,
         preserve_alias: bool,
         publish_canonical_id: bool,
-        response_tx: StartResponseSender,
+        response_tx: oneshot::Sender<std::result::Result<HealAdmissionReceipt, String>>,
     ) -> Result<()> {
         if request.id.is_empty() {
             request.id = uuid::Uuid::new_v4().to_string();
@@ -429,7 +383,7 @@ impl HealChannelProcessor {
             Ok(heal_request) => heal_request,
             Err(err) => {
                 let error_text = err.to_string();
-                response_tx.send(Err(error_text.clone()));
+                let _ = response_tx.send(Err(error_text.clone()));
                 self.publish_response(HealChannelResponse {
                     request_id: request.id,
                     success: false,
@@ -443,7 +397,7 @@ impl HealChannelProcessor {
         // Submit to heal manager
         match self
             .heal_manager
-            .submit_heal_request_with_observation_and_alias(heal_request, preserve_alias)
+            .submit_heal_request_with_receipt_and_alias(heal_request, preserve_alias)
             .await
         {
             Ok(receipt) => {
@@ -465,7 +419,7 @@ impl HealChannelProcessor {
                     request.id.clone()
                 };
                 self.publish_response(admission_response(response_id, receipt.result));
-                response_tx.send(Ok(receipt));
+                let _ = response_tx.send(Ok(receipt));
             }
             Err(e) => {
                 let error_text = e.to_string();
@@ -479,7 +433,7 @@ impl HealChannelProcessor {
                     error = %error_text,
                     "Heal start submission failed"
                 );
-                response_tx.send(Err(error_text.clone()));
+                let _ = response_tx.send(Err(error_text.clone()));
 
                 // Send error response
                 let response = HealChannelResponse {
@@ -523,6 +477,13 @@ impl HealChannelProcessor {
                 .await
         };
 
+        let response = Self::query_report_response(client_token, report)?;
+        let _ = response_tx.send(Ok(response.clone()));
+        self.publish_response(response);
+        Ok(())
+    }
+
+    fn query_report_response(client_token: String, report: Result<HealTaskReport>) -> Result<HealChannelResponse> {
         let outcome = report.as_ref().ok().and_then(|report| report.outcome.clone());
         let settings = report
             .as_ref()
@@ -641,27 +602,21 @@ impl HealChannelProcessor {
                 0,
             ),
             Err(crate::Error::InvalidClientToken) => {
-                let response = HealChannelResponse {
+                return Ok(HealChannelResponse {
                     request_id: client_token,
                     success: false,
                     data: None,
                     error: Some("invalid heal client token".to_string()),
-                };
-                let _ = response_tx.send(Ok(response.clone()));
-                self.publish_response(response);
-                return Ok(());
+                });
             }
             Err(err) => {
                 let error_text = err.to_string();
-                let response = HealChannelResponse {
+                return Ok(HealChannelResponse {
                     request_id: client_token,
                     success: false,
                     data: None,
                     error: Some(error_text),
-                };
-                let _ = response_tx.send(Ok(response.clone()));
-                self.publish_response(response);
-                return Ok(());
+                });
             }
         };
 
@@ -678,17 +633,12 @@ impl HealChannelProcessor {
             },
         )?;
 
-        let response = HealChannelResponse {
+        Ok(HealChannelResponse {
             request_id: client_token,
             success: true,
             data: Some(data),
             error: detail,
-        };
-
-        let _ = response_tx.send(Ok(response.clone()));
-        self.publish_response(response);
-
-        Ok(())
+        })
     }
 
     /// Process cancel request
@@ -1808,84 +1758,6 @@ mod tests {
             .expect("legacy alias should cancel the canonical task");
     }
 
-    #[tokio::test]
-    async fn observed_and_legacy_channels_share_the_real_admission_owner() {
-        let manager = create_test_heal_manager();
-        let mut processor = HealChannelProcessor::new(manager);
-        let (command_tx, command_rx) = mpsc::unbounded_channel();
-        let (receipt_tx, receipt_rx) = mpsc::unbounded_channel();
-        let (observed_tx, observed_rx) = mpsc::unbounded_channel();
-        let task = tokio::spawn(async move {
-            processor
-                .start_with_observed_receipts(command_rx, receipt_rx, observed_rx)
-                .await
-        });
-        let request = HealChannelRequest {
-            id: "observed-owner".into(),
-            bucket: "observed-owner-bucket".into(),
-            object_prefix: Some("object".into()),
-            source: HealRequestSource::Internal,
-            ..Default::default()
-        };
-        let (response_tx, response_rx) = oneshot::channel();
-        observed_tx
-            .send(ObservedHealReceiptCommand {
-                request: request.clone(),
-                response_tx,
-            })
-            .expect("observed command");
-        let first = tokio::time::timeout(Duration::from_secs(1), response_rx)
-            .await
-            .expect("observed response")
-            .expect("sender")
-            .expect("admitted");
-        assert_eq!(first.result, HealAdmissionResult::Accepted);
-        assert!(first.observation_id.is_some());
-
-        let mut duplicate = request.clone();
-        duplicate.id = "legacy-observed-duplicate".into();
-        let (response_tx, response_rx) = oneshot::channel();
-        receipt_tx
-            .send(HealReceiptCommand {
-                request: duplicate,
-                response_tx,
-            })
-            .expect("legacy command");
-        let legacy = tokio::time::timeout(Duration::from_secs(1), response_rx)
-            .await
-            .expect("legacy response")
-            .expect("sender")
-            .expect("merged");
-        assert_eq!(legacy.result, HealAdmissionResult::Merged);
-        assert_eq!(legacy.task_id, first.task_id);
-
-        let mut duplicate = request;
-        duplicate.id = "observed-duplicate".into();
-        let (response_tx, response_rx) = oneshot::channel();
-        observed_tx
-            .send(ObservedHealReceiptCommand {
-                request: duplicate,
-                response_tx,
-            })
-            .expect("observed duplicate");
-        let merged = tokio::time::timeout(Duration::from_secs(1), response_rx)
-            .await
-            .expect("observed response")
-            .expect("sender")
-            .expect("merged");
-        assert_eq!(merged.result, HealAdmissionResult::Merged);
-        assert_eq!(merged.task_id, first.task_id);
-        assert_eq!(merged.observation_id, first.observation_id);
-        drop(command_tx);
-        assert!(
-            tokio::time::timeout(Duration::from_secs(1), task)
-                .await
-                .expect("processor stops")
-                .expect("join")
-                .is_ok()
-        );
-    }
-
     #[tokio::test]
     async fn test_public_legacy_start_processes_commands_with_receipt_channel_closed() {
         let manager = create_test_heal_manager();

crates/heal/src/heal/manager.rs

@@ -29,7 +29,7 @@ use rustfs_concurrency::workload::{ForegroundPressure, foreground_pressure};
 #[cfg(test)]
 use rustfs_concurrency::{AdmissionState, WorkloadClass};
 use rustfs_heal_contracts::heal_channel::{
-    HealAdmissionDropReason, HealAdmissionReceipt, HealAdmissionResult, HealRequestSource, ObservedHealAdmissionReceipt,
+    HealAdmissionDropReason, HealAdmissionReceipt, HealAdmissionResult, HealRequestSource,
 };
 use rustfs_madmin::heal_commands::HealResultItem;
 #[cfg(test)]
@@ -234,6 +234,8 @@ fn record_displaced_terminal(
     request: &HealRequest,
 ) -> Arc<CompletedHealStatus> {
     let terminal = Arc::new(CompletedHealStatus {
+        task_id: request.id.clone(),
+        source: request.source,
         outcome: None,
         progress: None,
         retained_bytes: std::sync::OnceLock::new(),
@@ -297,7 +299,6 @@ async fn publish_completed_heal(
     task_id: &str,
     completed: CompletedHealStatus,
     terminal: bool,
-    observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
 ) {
     let completed = Arc::new(completed);
     completed.retained_bytes();
@@ -329,16 +330,13 @@ async fn publish_completed_heal(
                 identity: object.identity.clone(),
                 disposition: object.disposition.clone(),
             });
-        rustfs_heal_contracts::completion::publish_observed_heal_completion(
-            rustfs_heal_contracts::completion::HealCompletion {
-                task_id: task_id.to_owned(),
-                scan_mode: completed.options.scan_mode,
-                dry_run: completed.options.dry_run,
-                execution: outcome.map(|outcome| outcome.execution),
-                object,
-            },
-            observation_id,
-        );
+        rustfs_heal_contracts::completion::publish_heal_completion(rustfs_heal_contracts::completion::HealCompletion {
+            task_id: task_id.to_owned(),
+            scan_mode: completed.options.scan_mode,
+            dry_run: completed.options.dry_run,
+            execution: outcome.map(|outcome| outcome.execution),
+            object,
+        });
     }
 }
 
@@ -629,38 +627,6 @@ fn request_matches_request(request: &HealRequest, existing: &HealRequest) -> boo
         && request.heal_endpoints == existing.heal_endpoints
 }
 
-fn lookup_request_id_admission(
-    request: &HealRequest,
-    active_heals: &HashMap<String, Arc<HealTask>>,
-    queue: &PriorityHealQueue,
-    retrying_heals: &HashMap<String, RetryingHeal>,
-    durable_owners: &[HealRequest],
-) -> Option<(bool, &'static str, Option<rustfs_heal_contracts::completion::HealObservationId>)> {
-    active_heals
-        .get(&request.id)
-        .map(|task| (request_matches_task(request, task), "active", task.observation_id))
-        .or_else(|| {
-            // A request ID can name multiple queued sequences; the token is unknown.
-            queue
-                .requests()
-                .find(|queued| queued.id == request.id)
-                .map(|queued| (request_matches_request(request, queued), "queued", None))
-        })
-        .or_else(|| {
-            retrying_heals
-                .get(&request.id)
-                .map(|retrying| (request_matches_request(request, &retrying.request), "retrying", retrying.observation_id))
-        })
-        // A durable-only owner must be resumed by recovery, not overwritten
-        // with a new execution budget by a replay before recovery completes.
-        .or_else(|| {
-            durable_owners
-                .iter()
-                .find(|owner| owner.id == request.id)
-                .map(|_| (false, "durable", None))
-        })
-}
-
 fn retrying_heal_for_dedup_key(retrying_heals: &HashMap<String, RetryingHeal>, key: &str) -> Option<(String, HealType)> {
     retrying_heals
         .iter()
@@ -705,7 +671,10 @@ fn retry_budget_for_result(task: &HealTask, result: &Result<()>, retryable_batch
     if task.retry_attempts >= MAX_RECOVERABLE_HEAL_RETRIES {
         return None;
     }
-    if task.has_batch_failure() && !retryable_batch_failure {
+    // Admin traversals already exhaust their per-object retries. Restarting
+    // the whole traversal under the same token discards those receipts and
+    // reports previously repaired objects as unchanged on the next scan.
+    if task.has_batch_failure() && (task.source == HealRequestSource::Admin || !retryable_batch_failure) {
         return None;
     }
 
@@ -1107,9 +1076,9 @@ pub struct HealManager {
 /// cascade without re-locking.
 enum TaskStateLookup {
     Active(Arc<HealTask>),
-    Retrying(HealTaskStatus, HealOptions),
+    Retrying(HealTaskStatus, HealOptions, HealRequestSource),
     Completed(Arc<CompletedHealStatus>),
-    Queued(HealOptions),
+    Queued(HealOptions, HealRequestSource),
     /// A durable root heal record that cannot be admitted or quarantined;
     /// it answers status queries with the reason until an operator repairs it.
     Unrunnable(String),
@@ -1328,7 +1297,6 @@ impl HealManager {
         config: &HealConfig,
         context: &'static str,
         replacement_targets_frozen: bool,
-        observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
     ) -> HealAdmissionDecision {
         let replacement_targets_frozen = replacement_targets_frozen || request.retry_attempts != 0;
         let queue_len = queue.len();
@@ -1365,7 +1333,7 @@ impl HealManager {
                 let priority = request.priority;
                 let source = request.source;
                 if let Some(displaced) =
-                    queue.push_displacing_lower_priority_where(request, Self::queued_request_can_be_displaced, observation_id)
+                    queue.push_displacing_lower_priority_where(request, Self::queued_request_can_be_displaced)
                 {
                     publish_heal_queue_length(queue);
                     Self::record_admission_metric(source, HealAdmissionResult::Accepted, context);
@@ -1516,7 +1484,7 @@ impl HealManager {
         let priority = request.priority;
         let source = request.source;
 
-        match queue.push_with_replacement_scope(request, replacement_targets_frozen, observation_id) {
+        match queue.push_with_replacement_scope(request, replacement_targets_frozen) {
             QueuePushOutcome::Accepted => {
                 publish_heal_queue_length(queue);
                 Self::record_admission_metric(source, HealAdmissionResult::Accepted, context);
@@ -1988,17 +1956,6 @@ impl HealManager {
         request: HealRequest,
         preserve_alias: bool,
     ) -> Result<HealAdmissionReceipt> {
-        Ok(self
-            .submit_heal_request_with_observation_and_alias(request, preserve_alias)
-            .await?
-            .receipt)
-    }
-
-    pub(crate) async fn submit_heal_request_with_observation_and_alias(
-        &self,
-        request: HealRequest,
-        preserve_alias: bool,
-    ) -> Result<ObservedHealAdmissionReceipt> {
         self.submit_heal_request_with_receipt_alias_and_mrf_notice(request, preserve_alias, true, None)
             .await
     }
@@ -2039,10 +1996,8 @@ impl HealManager {
         mrf_notice_target: MrfRepairNoticeTarget,
     ) -> Result<HealAdmissionReceipt> {
         let preserve_alias = !mrf_notice_target.kind.is_durable();
-        Ok(self
-            .submit_heal_request_with_receipt_alias_and_mrf_notice(request, preserve_alias, true, Some(mrf_notice_target))
-            .await?
-            .receipt)
+        self.submit_heal_request_with_receipt_alias_and_mrf_notice(request, preserve_alias, true, Some(mrf_notice_target))
+            .await
     }
 
     #[cfg(test)]
@@ -2108,7 +2063,7 @@ impl HealManager {
         preserve_alias: bool,
         accept_same_request_id_replay: bool,
         mrf_notice_target: Option<MrfRepairNoticeTarget>,
-    ) -> Result<ObservedHealAdmissionReceipt> {
+    ) -> Result<HealAdmissionReceipt> {
         let admission_start = Instant::now();
         if request.id.is_empty() {
             request.id = uuid::Uuid::new_v4().to_string();
@@ -2169,10 +2124,77 @@ impl HealManager {
         let mut queue = self.heal_queue.lock().await;
         let mut retrying_heals = self.retrying_heals.lock().await;
 
-        let mut request_id_admission =
-            lookup_request_id_admission(&request, &active_heals, &queue, &retrying_heals, &durable_owners);
+        let request_id_admission = active_heals
+            .get(&request.id)
+            .map(|task| (request_matches_task(&request, task), "active"))
+            .or_else(|| {
+                queue
+                    .requests()
+                    .find(|queued| queued.id == request.id)
+                    .map(|queued| (request_matches_request(&request, queued), "queued"))
+            })
+            .or_else(|| {
+                retrying_heals
+                    .get(&request.id)
+                    .map(|retrying| (request_matches_request(&request, &retrying.request), "retrying"))
+            })
+            // A durable-only owner must be resumed by recovery, not overwritten
+            // with a new execution budget by a replay before recovery completes.
+            .or_else(|| {
+                durable_owners
+                    .iter()
+                    .find(|owner| owner.id == request.id)
+                    .map(|_| (false, "durable"))
+            });
+        if let Some((matches_existing, duplicate_state)) = request_id_admission {
+            let admission = if matches_existing {
+                if accept_same_request_id_replay {
+                    HealAdmissionResult::Accepted
+                } else {
+                    Self::duplicate_admission_for_request(&request, &config)
+                }
+            } else {
+                HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
+            };
+            if matches!(admission, HealAdmissionResult::Accepted | HealAdmissionResult::Merged)
+                && let Some(target) = mrf_notice_target
+            {
+                let mut targets = lock_mrf_repair_notice_targets(&self.mrf_repair_notice_targets);
+                Self::insert_mrf_repair_notice_target(&mut targets, &request.id, target);
+            }
+            drop(retrying_heals);
+            drop(queue);
+            drop(active_heals);
+            let lock_timing =
+                HealAdmissionLockTiming::finish(lock_phase_start, active_lock_acquired, active_lock_wait, active_lock_hold);
+            Self::record_admission_metric(request.source, admission, "duplicate");
+            self.record_admission_observation(HealAdmissionObservation {
+                source,
+                result: admission,
+                context: "duplicate",
+                force_start,
+                displaced: false,
+                start_duration: admission_start.elapsed(),
+                lock_timing,
+            });
+            debug!(
+                target: "rustfs::heal::manager",
+                event = EVENT_HEAL_QUEUE_ADMISSION,
+                component = LOG_COMPONENT_HEAL,
+                subsystem = LOG_SUBSYSTEM_MANAGER,
+                request_id = %request.id,
+                duplicate_state,
+                result = admission.result_label(),
+                reason = admission.reason_label(),
+                "Heal queue admission reused an existing request id"
+            );
+            return Ok(HealAdmissionReceipt {
+                result: admission,
+                task_id: request.id,
+            });
+        }
 
-        if request_id_admission.is_none() && source == HealRequestSource::Admin && force_start {
+        if source == HealRequestSource::Admin && force_start {
             let overlapping = active_heals
                 .values()
                 .map(|task| (&task.id, &task.heal_type, &task.options, task.source))
@@ -2218,59 +2240,6 @@ impl HealManager {
             active_lock_wait += active_lock_acquired.duration_since(reacquire_start);
             queue = self.heal_queue.lock().await;
             retrying_heals = self.retrying_heals.lock().await;
-            // Non-admin admission can claim this id while cancellation drops the state locks.
-            request_id_admission = lookup_request_id_admission(&request, &active_heals, &queue, &retrying_heals, &durable_owners);
-        }
-
-        if let Some((matches_existing, duplicate_state, observation_id)) = request_id_admission {
-            let admission = if matches_existing {
-                if accept_same_request_id_replay {
-                    HealAdmissionResult::Accepted
-                } else {
-                    Self::duplicate_admission_for_request(&request, &config)
-                }
-            } else {
-                HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
-            };
-            if matches!(admission, HealAdmissionResult::Accepted | HealAdmissionResult::Merged)
-                && let Some(target) = mrf_notice_target
-            {
-                let mut targets = lock_mrf_repair_notice_targets(&self.mrf_repair_notice_targets);
-                Self::insert_mrf_repair_notice_target(&mut targets, &request.id, target);
-            }
-            drop(retrying_heals);
-            drop(queue);
-            drop(active_heals);
-            let lock_timing =
-                HealAdmissionLockTiming::finish(lock_phase_start, active_lock_acquired, active_lock_wait, active_lock_hold);
-            Self::record_admission_metric(request.source, admission, "duplicate");
-            self.record_admission_observation(HealAdmissionObservation {
-                source,
-                result: admission,
-                context: "duplicate",
-                force_start,
-                displaced: false,
-                start_duration: admission_start.elapsed(),
-                lock_timing,
-            });
-            debug!(
-                target: "rustfs::heal::manager",
-                event = EVENT_HEAL_QUEUE_ADMISSION,
-                component = LOG_COMPONENT_HEAL,
-                subsystem = LOG_SUBSYSTEM_MANAGER,
-                request_id = %request.id,
-                duplicate_state,
-                result = admission.result_label(),
-                reason = admission.reason_label(),
-                "Heal queue admission reused an existing request id"
-            );
-            return Ok(ObservedHealAdmissionReceipt {
-                receipt: HealAdmissionReceipt {
-                    result: admission,
-                    task_id: request.id,
-                },
-                observation_id,
-            });
         }
 
         let rejection = if source == HealRequestSource::Admin {
@@ -2318,45 +2287,26 @@ impl HealManager {
             None
         };
 
-        let scanner_queued_owner_required = request.source == HealRequestSource::Scanner
-            && mrf_notice_target.is_none()
-            && request.expected_mrf_bucket_incarnation_id.is_none();
         let duplicate = (!request.force_start && rejection.is_none()).then(|| {
             active_heal_for_dedup_key(&active_heals, &dedup_key)
-                .map(|(task_id, _)| {
-                    let observation_id = active_heals.get(&task_id).and_then(|task| task.observation_id);
-                    (task_id, "active", observation_id)
-                })
+                .map(|(task_id, _)| (task_id, "active"))
                 .or_else(|| {
-                    let owner = if scanner_queued_owner_required {
-                        queue.queued_scanner_owner_for_dedup_key(&dedup_key)
-                    } else {
-                        queue.queued_legacy_owner_for_dedup_key(&dedup_key)
-                    };
-                    owner
-                        .map(|(id, observation_id)| (id.to_string(), "queued", observation_id))
-                        .or_else(|| {
-                            (scanner_queued_owner_required && queue.dedup_keys.contains_key(&dedup_key))
-                                .then(|| (request.id.clone(), "queued_owner_unknown", None))
-                        })
-                })
-                .or_else(|| {
-                    retrying_heal_for_dedup_key(&retrying_heals, &dedup_key).map(|(task_id, _)| {
-                        let observation_id = retrying_heals.get(&task_id).and_then(|retry| retry.observation_id);
-                        (task_id, "retrying", observation_id)
-                    })
+                    queue
+                        .queued_request_id_for_dedup_key(&dedup_key)
+                        .map(|queued_id| (queued_id.to_string(), "queued"))
                 })
+                .or_else(|| retrying_heal_for_dedup_key(&retrying_heals, &dedup_key).map(|(task_id, _)| (task_id, "retrying")))
         });
-        if let Some((merged_task_id, duplicate_state, observation_id)) = duplicate.flatten() {
+        if let Some((merged_task_id, duplicate_state)) = duplicate.flatten() {
             // HS-06: under the minio_error overlap policy an exact duplicate
             // admin start reports the typed AlreadyRunning rejection instead
             // of the silent merge (MinIO's ErrHealAlreadyRunning).
-            let mut admission = if duplicate_state == "queued_owner_unknown"
-                || (request.source == HealRequestSource::Admin && config.overlap_policy == HealOverlapPolicy::MinioError)
+            let mut admission = if (request.source == HealRequestSource::Admin
+                && config.overlap_policy == HealOverlapPolicy::MinioError)
                 || (duplicate_state != "queued" && mrf_notice_target.as_ref().is_some_and(|target| target.kind.is_durable()))
             {
-                // An unknown queued owner cannot certify a receipt. A running or
-                // retrying task may have observed the object before this write.
+                // A running/retrying task may have observed the object before
+                // this write committed. Its receipt cannot prove the new lease.
                 HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
             } else {
                 Self::duplicate_admission_for_request(&request, &config)
@@ -2455,12 +2405,9 @@ impl HealManager {
                 HealAdmissionResult::Accepted | HealAdmissionResult::Full => {}
             }
 
-            return Ok(ObservedHealAdmissionReceipt {
-                receipt: HealAdmissionReceipt {
-                    result: admission,
-                    task_id: merged_task_id,
-                },
-                observation_id,
+            return Ok(HealAdmissionReceipt {
+                result: admission,
+                task_id: merged_task_id,
             });
         }
 
@@ -2491,12 +2438,9 @@ impl HealManager {
                 result = "overlap_rejected",
                 "Admin heal start rejected by overlap policy"
             );
-            return Ok(ObservedHealAdmissionReceipt {
-                receipt: HealAdmissionReceipt {
-                    result: HealAdmissionResult::Dropped(reason),
-                    task_id: overlap_task_id,
-                },
-                observation_id: None,
+            return Ok(HealAdmissionReceipt {
+                result: HealAdmissionResult::Dropped(reason),
+                task_id: overlap_task_id,
             });
         }
 
@@ -2538,12 +2482,9 @@ impl HealManager {
                     result = "dropped_by_disposition",
                     "Heal queue admission answered from the disposition memory"
                 );
-                return Ok(ObservedHealAdmissionReceipt {
-                    receipt: HealAdmissionReceipt {
-                        result: admission,
-                        task_id: recorded_task_id,
-                    },
-                    observation_id: None,
+                return Ok(HealAdmissionReceipt {
+                    result: admission,
+                    task_id: recorded_task_id,
                 });
             }
         }
@@ -2571,31 +2512,17 @@ impl HealManager {
                 .lock()
                 .unwrap_or_else(|poisoned| poisoned.into_inner())
                 .contains_key(&request.id);
-        let mut observation_id = rustfs_heal_contracts::completion::new_heal_observation_id();
-        let admission_decision =
-            Self::admit_request_to_queue(&mut queue, request, &config, "submit", recovered_replacement, observation_id);
+        let admission_decision = Self::admit_request_to_queue(&mut queue, request, &config, "submit", recovered_replacement);
         let admission = admission_decision.result;
         if durable_handoff_required && !admission.is_admitted() {
             self.root_recovery
                 .remove(&request_id, &request_heal_type, request_source)
                 .await?;
         }
-        if admission == HealAdmissionResult::Merged {
-            observation_id = None;
-            let owner = if scanner_queued_owner_required {
-                queue.queued_scanner_owner_for_dedup_key(&dedup_key)
-            } else {
-                queue.queued_legacy_owner_for_dedup_key(&dedup_key)
-            };
-            if let Some((id, owner_observation_id)) = owner {
-                task_id = id.to_owned();
-                observation_id = owner_observation_id;
-            } else if scanner_queued_owner_required {
-                return Err(Error::Other("Scanner queued owner is unavailable".to_owned()));
-            }
-        }
-        if !admission.is_admitted() {
-            observation_id = None;
+        if admission == HealAdmissionResult::Merged
+            && let Some(queued_id) = queue.queued_request_id_for_dedup_key(&dedup_key)
+        {
+            task_id = queued_id.to_owned();
         }
         let should_notify = matches!(admission, HealAdmissionResult::Accepted) && config.event_driven_scheduler_enable;
         let displaced_task_id = admission_decision.displaced_task_id().map(ToOwned::to_owned);
@@ -2641,12 +2568,9 @@ impl HealManager {
             lock_timing,
         });
 
-        Ok(ObservedHealAdmissionReceipt {
-            receipt: HealAdmissionReceipt {
-                result: admission,
-                task_id,
-            },
-            observation_id,
+        Ok(HealAdmissionReceipt {
+            result: admission,
+            task_id,
         })
     }
 
@@ -2667,6 +2591,23 @@ impl HealManager {
     /// completed entry. `heal_path` additionally constrains the map matches
     /// the way the `*_for_path` variants always have.
     async fn lookup_task_state(&self, canonical_task_id: &str, heal_path: Option<&str>) -> Result<TaskStateLookup> {
+        let state = self.lookup_memory_task_state(canonical_task_id, heal_path).await;
+        if !matches!(state, TaskStateLookup::NotFound) {
+            return Ok(state);
+        }
+        if let Some(completed) = self.root_recovery.completed(canonical_task_id).await?
+            && heal_path.is_none_or(|path| heal_type_matches_path(&completed.heal_type, path))
+        {
+            return Ok(TaskStateLookup::Completed(Arc::new(completed)));
+        }
+        // A durable record that cannot run still answers with its reason.
+        if let Some(reason) = self.root_recovery.record_issue(canonical_task_id).await {
+            return Ok(TaskStateLookup::Unrunnable(reason));
+        }
+        Ok(TaskStateLookup::NotFound)
+    }
+
+    async fn lookup_memory_task_state(&self, canonical_task_id: &str, heal_path: Option<&str>) -> TaskStateLookup {
         let matches_path = |heal_type: &HealType| heal_path.is_none_or(|path| heal_type_matches_path(heal_type, path));
 
         {
@@ -2675,7 +2616,7 @@ impl HealManager {
                 .get(canonical_task_id)
                 .filter(|task| matches_path(&task.heal_type))
             {
-                return Ok(TaskStateLookup::Active(Arc::clone(task)));
+                return TaskStateLookup::Active(Arc::clone(task));
             }
         }
 
@@ -2685,7 +2626,7 @@ impl HealManager {
                 .get(canonical_task_id)
                 .filter(|retrying| matches_path(&retrying.request.heal_type))
             {
-                return Ok(TaskStateLookup::Retrying(retrying.status(), retrying.request.options.clone()));
+                return TaskStateLookup::Retrying(retrying.status(), retrying.request.options.clone(), retrying.request.source);
             }
         }
 
@@ -2698,7 +2639,7 @@ impl HealManager {
             prune_completed_heal_statuses(&mut completed_heals);
             if let Some(completed) = completed_heals.get(canonical_task_id).filter(|c| matches_path(&c.heal_type)) {
                 if completed_status_is_retrying(&completed.status) {
-                    return Ok(TaskStateLookup::Completed(Arc::clone(completed)));
+                    return TaskStateLookup::Completed(Arc::clone(completed));
                 }
                 terminal_completed = Some(Arc::clone(completed));
             }
@@ -2707,7 +2648,7 @@ impl HealManager {
         {
             let queue = self.heal_queue.lock().await;
             if let Some(request) = queue.request_matching_id_and_path(canonical_task_id, heal_path) {
-                return Ok(TaskStateLookup::Queued(request.options.clone()));
+                return TaskStateLookup::Queued(request.options.clone(), request.source);
             }
         }
 
@@ -2720,22 +2661,10 @@ impl HealManager {
                 .cloned();
         }
 
-        if terminal_completed.is_none()
-            && let Some(completed) = self.root_recovery.completed(canonical_task_id).await?
-            && matches_path(&completed.heal_type)
-        {
-            terminal_completed = Some(Arc::new(completed));
-        }
-
         if let Some(completed) = terminal_completed {
-            return Ok(TaskStateLookup::Completed(completed));
+            return TaskStateLookup::Completed(completed);
         }
-        // A durable root record that can neither run nor be quarantined still
-        // answers status queries with its reason.
-        if let Some(reason) = self.root_recovery.record_issue(canonical_task_id).await {
-            return Ok(TaskStateLookup::Unrunnable(reason));
-        }
-        Ok(TaskStateLookup::NotFound)
+        TaskStateLookup::NotFound
     }
 
     async fn publish_admin_terminal(
@@ -2759,6 +2688,8 @@ impl HealManager {
     ) -> Result<bool> {
         let previous = self.completed_heals.lock().await.get(task_id).cloned();
         let mut completed = CompletedHealStatus {
+            task_id: task_id.to_owned(),
+            source,
             outcome: None,
             progress: None,
             retained_bytes: std::sync::OnceLock::new(),
@@ -2794,9 +2725,9 @@ impl HealManager {
         let canonical_task_id = self.canonical_task_id(task_id).await;
         match self.lookup_task_state(&canonical_task_id, None).await? {
             TaskStateLookup::Active(task) => Ok(task.get_status().await),
-            TaskStateLookup::Retrying(status, _) => Ok(status),
+            TaskStateLookup::Retrying(status, _, _) => Ok(status),
             TaskStateLookup::Completed(completed) => Ok(completed.status.clone()),
-            TaskStateLookup::Queued(_) => Ok(HealTaskStatus::Pending),
+            TaskStateLookup::Queued(_, _) => Ok(HealTaskStatus::Pending),
             TaskStateLookup::Unrunnable(error) => Ok(HealTaskStatus::Failed { error }),
             TaskStateLookup::NotFound => Err(Error::TaskNotFound {
                 task_id: task_id.to_string(),
@@ -2815,9 +2746,9 @@ impl HealManager {
         let canonical_task_id = self.canonical_task_id(task_id).await;
         match self.lookup_task_state(&canonical_task_id, None).await? {
             TaskStateLookup::Active(task) => Ok(active_task_report(&task, since).await),
-            TaskStateLookup::Retrying(status, options) => Ok(empty_task_report_with_options(status, options)),
+            TaskStateLookup::Retrying(status, options, _) => Ok(empty_task_report_with_options(status, options)),
             TaskStateLookup::Completed(completed) => Ok(completed_task_report(&completed, since)),
-            TaskStateLookup::Queued(options) => Ok(empty_task_report_with_options(HealTaskStatus::Pending, options)),
+            TaskStateLookup::Queued(options, _) => Ok(empty_task_report_with_options(HealTaskStatus::Pending, options)),
             TaskStateLookup::Unrunnable(error) => {
                 Ok(empty_task_report_with_options(HealTaskStatus::Failed { error }, HealOptions::default()))
             }
@@ -2831,6 +2762,42 @@ impl HealManager {
         self.get_task_report_for_path_since(heal_path, task_id, None).await
     }
 
+    /// Resolve a node-owned automatic task without granting local control over
+    /// Admin tasks. The source and report come from the same state snapshot.
+    pub async fn get_local_automatic_task_report(
+        &self,
+        heal_path: &str,
+        task_id: &str,
+        since: Option<u64>,
+    ) -> Result<Option<HealTaskReport>> {
+        let canonical_task_id = self.canonical_task_id(task_id).await;
+        if canonical_task_id != task_id {
+            return Ok(None);
+        }
+        let state = self.lookup_memory_task_state(&canonical_task_id, Some(heal_path)).await;
+        let source = match &state {
+            TaskStateLookup::Active(task) => task.source,
+            TaskStateLookup::Retrying(_, _, source) | TaskStateLookup::Queued(_, source) => *source,
+            TaskStateLookup::Completed(completed) if completed.task_id == task_id => completed.source,
+            TaskStateLookup::Completed(_) => return Ok(None),
+            TaskStateLookup::Unrunnable(_) | TaskStateLookup::NotFound => return Ok(None),
+        };
+        if !matches!(
+            source,
+            HealRequestSource::Scanner | HealRequestSource::Mrf | HealRequestSource::AutoHeal | HealRequestSource::ReadRepair
+        ) {
+            return Ok(None);
+        }
+        let report = match state {
+            TaskStateLookup::Active(task) => active_task_report(&task, since).await,
+            TaskStateLookup::Retrying(status, options, _) => empty_task_report_with_options(status, options),
+            TaskStateLookup::Completed(completed) => completed_task_report(&completed, since),
+            TaskStateLookup::Queued(options, _) => empty_task_report_with_options(HealTaskStatus::Pending, options),
+            TaskStateLookup::Unrunnable(_) | TaskStateLookup::NotFound => return Ok(None),
+        };
+        Ok(Some(report))
+    }
+
     /// Incremental variant of [`Self::get_task_report_for_path`] (HS-06).
     pub async fn get_task_report_for_path_since(
         &self,
@@ -2841,9 +2808,9 @@ impl HealManager {
         let canonical_task_id = self.canonical_task_id(task_id).await;
         match self.lookup_task_state(&canonical_task_id, Some(heal_path)).await? {
             TaskStateLookup::Active(task) => Ok(active_task_report(&task, since).await),
-            TaskStateLookup::Retrying(status, options) => Ok(empty_task_report_with_options(status, options)),
+            TaskStateLookup::Retrying(status, options, _) => Ok(empty_task_report_with_options(status, options)),
             TaskStateLookup::Completed(completed) => Ok(completed_task_report(&completed, since)),
-            TaskStateLookup::Queued(options) => Ok(empty_task_report_with_options(HealTaskStatus::Pending, options)),
+            TaskStateLookup::Queued(options, _) => Ok(empty_task_report_with_options(HealTaskStatus::Pending, options)),
             TaskStateLookup::Unrunnable(error) => {
                 Ok(empty_task_report_with_options(HealTaskStatus::Failed { error }, HealOptions::default()))
             }
@@ -2867,9 +2834,9 @@ impl HealManager {
         let canonical_task_id = self.canonical_task_id(task_id).await;
         match self.lookup_task_state(&canonical_task_id, Some(heal_path)).await? {
             TaskStateLookup::Active(task) => Ok(task.get_status().await),
-            TaskStateLookup::Retrying(status, _) => Ok(status),
+            TaskStateLookup::Retrying(status, _, _) => Ok(status),
             TaskStateLookup::Completed(completed) => Ok(completed.status.clone()),
-            TaskStateLookup::Queued(_) => Ok(HealTaskStatus::Pending),
+            TaskStateLookup::Queued(_, _) => Ok(HealTaskStatus::Pending),
             TaskStateLookup::Unrunnable(error) => Ok(HealTaskStatus::Failed { error }),
             TaskStateLookup::NotFound => {
                 if self.path_has_task(heal_path).await {
@@ -2968,15 +2935,7 @@ impl HealManager {
                     .await?;
                 self.root_recovery.remove(&task.id, &task.heal_type, task.source).await?;
                 task.cancel().await?;
-                publish_completed_heal(
-                    &self.completed_heals,
-                    &self.task_aliases,
-                    &canonical_task_id,
-                    completed,
-                    true,
-                    task.observation_id,
-                )
-                .await;
+                publish_completed_heal(&self.completed_heals, &self.task_aliases, &canonical_task_id, completed, true).await;
                 active_heals.remove(&canonical_task_id);
                 publish_active_heal_count(&active_heals);
                 info!(
@@ -3087,15 +3046,7 @@ impl HealManager {
                         .await?;
                     self.root_recovery.remove(&task.id, &task.heal_type, task.source).await?;
                     task.cancel().await?;
-                    publish_completed_heal(
-                        &self.completed_heals,
-                        &self.task_aliases,
-                        task_id,
-                        completed,
-                        true,
-                        task.observation_id,
-                    )
-                    .await;
+                    publish_completed_heal(&self.completed_heals, &self.task_aliases, task_id, completed, true).await;
                 }
                 active_heals.remove(task_id);
                 cancelled += 1;

crates/heal/src/heal/manager/auto_scan.rs

@@ -693,7 +693,7 @@ impl HealManager {
                             let config = config.read().await;
                             let mut queue = heal_queue.lock().await;
                             let admission_decision =
-                                Self::admit_request_to_queue(&mut queue, req, &config, "auto_scan", recovered_replacement, rustfs_heal_contracts::completion::new_heal_observation_id());
+                                Self::admit_request_to_queue(&mut queue, req, &config, "auto_scan", recovered_replacement);
                             let admission = admission_decision.result;
                             let should_notify =
                                 matches!(admission, HealAdmissionResult::Accepted) && config.event_driven_scheduler_enable;

crates/heal/src/heal/manager/queue.rs

@@ -13,31 +13,15 @@
 // limitations under the License.
 /// The priority heal queue and its per-key dedup index.
 use super::*;
-use std::collections::BTreeMap;
 
-/// Per-key bookkeeping for the queued-request dedup index. The legacy
-/// representative is retained for existing non-scanner callers; scanner
-/// receipts use the separately tracked queued holders.
+/// Per-key bookkeeping for the queued-request dedup index: how many queued
+/// requests hold the key, and the id of the first request that opened it —
+/// the O(1) stand-in for the former heap scan when a merge receipt needs to
+/// name a queued representative.
 #[derive(Debug)]
 pub(super) struct DedupKeyEntry {
     pub(super) refcount: usize,
     pub(super) representative_request_id: String,
-    representative_sequence: u64,
-    representative_observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
-    pub(super) scanner_queued_owners: ScannerQueuedOwners,
-}
-
-#[derive(Debug)]
-pub(super) struct QueuedOwner {
-    request_id: String,
-    observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
-}
-
-#[derive(Debug)]
-pub(super) enum ScannerQueuedOwners {
-    Single(u64),
-    // Once a key has forced duplicates, keep every live holder until it is released.
-    Indexed(BTreeMap<u64, QueuedOwner>),
 }
 
 /// Priority queue wrapper for heal requests
@@ -86,25 +70,10 @@ pub(super) struct PriorityQueueItem {
     pub(super) sequence: u64,
     pub(super) dedup_key: String,
     pub(super) request: HealRequest,
-    observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
     /// Execution or durable recovery fixed these targets; grace waits can have zero retries.
     pub(super) replacement_targets_frozen: bool,
 }
 
-/// The in-process identity moves with the queue item, including forced duplicates.
-pub(super) struct QueuedHeal {
-    pub(super) request: HealRequest,
-    pub(super) observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
-}
-
-impl std::ops::Deref for QueuedHeal {
-    type Target = HealRequest;
-
-    fn deref(&self) -> &Self::Target {
-        &self.request
-    }
-}
-
 impl Eq for PriorityQueueItem {}
 
 impl PartialEq for PriorityQueueItem {
@@ -140,7 +109,9 @@ pub(super) enum QueuePushOutcome {
 
 #[derive(Debug, Clone)]
 pub(super) struct CompletedHealStatus {
+    pub(super) task_id: String,
     pub(super) heal_type: HealType,
+    pub(super) source: HealRequestSource,
     /// Options used to execute the task, retained for token-scoped status.
     pub(super) options: HealOptions,
     pub(super) status: HealTaskStatus,
@@ -168,6 +139,7 @@ impl CompletedHealStatus {
     fn measure_retained_bytes(&self) -> usize {
         let mut bytes = size_of::<Self>();
         let mut add = |amount: usize| bytes = bytes.saturating_add(amount);
+        add(self.task_id.capacity());
         add(self.outcome.as_ref().map_or(0, |outcome| outcome.retained_bytes()));
         match &self.heal_type {
             HealType::Cluster => {}
@@ -289,7 +261,9 @@ impl CompletedHealStatus {
             outcome.finish(Some(crate::heal::outcome::HealAbortReason::Cancelled));
         }
         let mut snapshot = Self {
+            task_id: task.id.clone(),
             heal_type: task.heal_type.clone(),
+            source: task.source,
             options: task.options.clone(),
             status,
             progress: Some(task.get_progress().await),
@@ -314,7 +288,6 @@ pub(super) struct HealTaskAlias {
 #[derive(Debug, Clone)]
 pub(super) struct RetryingHeal {
     pub(super) request: HealRequest,
-    pub(super) observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
     pub(super) error: String,
     pub(super) cancel_token: CancellationToken,
 }
@@ -338,7 +311,11 @@ impl PriorityHealQueue {
 
     #[cfg(test)]
     pub(super) fn pop_next(&mut self) -> Option<HealRequest> {
-        self.heap.pop().map(|item| self.remove_queued_item(item).request)
+        self.heap.pop().map(|item| {
+            Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
+            self.remove_best_effort_count(item.request.source);
+            item.request
+        })
     }
 
     pub(super) fn is_empty(&self) -> bool {
@@ -346,15 +323,10 @@ impl PriorityHealQueue {
     }
 
     pub(super) fn push(&mut self, request: HealRequest) -> QueuePushOutcome {
-        self.push_with_replacement_scope(request, false, rustfs_heal_contracts::completion::new_heal_observation_id())
+        self.push_with_replacement_scope(request, false)
     }
 
-    pub(super) fn push_with_replacement_scope(
-        &mut self,
-        request: HealRequest,
-        frozen: bool,
-        observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
-    ) -> QueuePushOutcome {
+    pub(super) fn push_with_replacement_scope(&mut self, request: HealRequest, frozen: bool) -> QueuePushOutcome {
         let key = Self::make_dedup_key(&request);
 
         // Check for duplicates unless the caller explicitly forces admission.
@@ -365,60 +337,20 @@ impl PriorityHealQueue {
         // also reserves the dedup key for later non-forced duplicates. The first
         // request that opens the key becomes the named representative for merge
         // receipts (taken before `request` moves into the heap).
+        self.dedup_keys
+            .entry(key.clone())
+            .or_insert_with(|| DedupKeyEntry {
+                refcount: 0,
+                representative_request_id: request.id.clone(),
+            })
+            .refcount += 1;
         self.sequence += 1;
-        let sequence = self.sequence;
-        if let Some(entry) = self.dedup_keys.get_mut(&key) {
-            match &mut entry.scanner_queued_owners {
-                ScannerQueuedOwners::Single(first_sequence) => {
-                    let mut owners = BTreeMap::new();
-                    if entry.refcount == 1 && entry.representative_sequence == *first_sequence {
-                        owners.insert(
-                            *first_sequence,
-                            QueuedOwner {
-                                request_id: entry.representative_request_id.clone(),
-                                observation_id: entry.representative_observation_id,
-                            },
-                        );
-                    }
-                    owners.insert(
-                        sequence,
-                        QueuedOwner {
-                            request_id: request.id.clone(),
-                            observation_id,
-                        },
-                    );
-                    entry.scanner_queued_owners = ScannerQueuedOwners::Indexed(owners);
-                }
-                ScannerQueuedOwners::Indexed(owners) => {
-                    owners.insert(
-                        sequence,
-                        QueuedOwner {
-                            request_id: request.id.clone(),
-                            observation_id,
-                        },
-                    );
-                }
-            }
-            entry.refcount += 1;
-        } else {
-            self.dedup_keys.insert(
-                key.clone(),
-                DedupKeyEntry {
-                    refcount: 1,
-                    representative_request_id: request.id.clone(),
-                    representative_sequence: sequence,
-                    representative_observation_id: observation_id,
-                    scanner_queued_owners: ScannerQueuedOwners::Single(sequence),
-                },
-            );
-        }
         let source = request.source;
         self.heap.push(PriorityQueueItem {
             priority: request.priority,
             sequence: self.sequence,
             dedup_key: key,
             request,
-            observation_id,
             replacement_targets_frozen: frozen,
         });
         if is_best_effort_source(source) {
@@ -444,12 +376,7 @@ impl PriorityHealQueue {
         self.can_displace_lower_priority_where(priority, |_| true)
     }
 
-    pub(super) fn push_displacing_lower_priority_where<F>(
-        &mut self,
-        request: HealRequest,
-        can_displace: F,
-        observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
-    ) -> Option<HealRequest>
+    pub(super) fn push_displacing_lower_priority_where<F>(&mut self, request: HealRequest, can_displace: F) -> Option<HealRequest>
     where
         F: Fn(&HealRequest) -> bool,
     {
@@ -480,7 +407,7 @@ impl PriorityHealQueue {
         self.heap = retained;
 
         let displaced = displaced.map(|item| {
-            Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item);
+            Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
             self.remove_best_effort_count(item.request.source);
             self.refresh_dedup_representative(&item.dedup_key);
             item.request
@@ -491,7 +418,7 @@ impl PriorityHealQueue {
             // into `debug_assert_eq!` — in release builds (`debug_assertions` off) the whole
             // macro, including its argument expression, is compiled out, which would silently
             // drop the new high-priority request after having already evicted a queued item.
-            let outcome = self.push_with_replacement_scope(request, false, observation_id);
+            let outcome = self.push(request);
             debug_assert_eq!(outcome, QueuePushOutcome::Accepted);
         }
 
@@ -500,7 +427,7 @@ impl PriorityHealQueue {
 
     #[cfg(test)]
     pub(super) fn push_displacing_lower_priority(&mut self, request: HealRequest) -> Option<HealRequest> {
-        self.push_displacing_lower_priority_where(request, |_| true, rustfs_heal_contracts::completion::new_heal_observation_id())
+        self.push_displacing_lower_priority_where(request, |_| true)
     }
 
     /// Get statistics about queue contents by priority
@@ -524,10 +451,14 @@ impl PriorityHealQueue {
 
     #[cfg(test)]
     pub(super) fn pop(&mut self) -> Option<HealRequest> {
-        self.heap.pop().map(|item| self.remove_queued_item(item).request)
+        self.heap.pop().map(|item| {
+            Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
+            self.remove_best_effort_count(item.request.source);
+            item.request
+        })
     }
 
-    pub(super) fn pop_runnable_with_skips<F, G>(&mut self, can_run: F, skip_label: G) -> (Option<QueuedHeal>, Vec<String>)
+    pub(super) fn pop_runnable_with_skips<F, G>(&mut self, can_run: F, skip_label: G) -> (Option<HealRequest>, Vec<String>)
     where
         F: Fn(&HealRequest) -> bool,
         G: Fn(&HealRequest) -> Option<String>,
@@ -549,10 +480,17 @@ impl PriorityHealQueue {
 
         self.restore_deferred_items(deferred);
 
-        (selected.map(|item| self.remove_queued_item(item)), skipped)
+        (
+            selected.map(|item| {
+                Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
+                self.remove_best_effort_count(item.request.source);
+                item.request
+            }),
+            skipped,
+        )
     }
 
-    pub(super) fn pop_runnable_with_fairness<F, G>(&mut self, can_run: F, skip_label: G) -> (Option<QueuedHeal>, Vec<String>)
+    pub(super) fn pop_runnable_with_fairness<F, G>(&mut self, can_run: F, skip_label: G) -> (Option<HealRequest>, Vec<String>)
     where
         F: Fn(&HealRequest) -> bool,
         G: Fn(&HealRequest) -> Option<String>,
@@ -621,22 +559,15 @@ impl PriorityHealQueue {
         self.heap = BinaryHeap::from(items);
 
         let chosen = chosen.map(|item| {
-            let request = self.remove_queued_item(item);
+            let request = item.request;
+            Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
+            self.remove_best_effort_count(request.source);
             self.record_dispatch(&request);
             request
         });
         (chosen, skipped)
     }
 
-    fn remove_queued_item(&mut self, item: PriorityQueueItem) -> QueuedHeal {
-        Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item);
-        self.remove_best_effort_count(item.request.source);
-        QueuedHeal {
-            request: item.request,
-            observation_id: item.observation_id,
-        }
-    }
-
     fn record_dispatch(&mut self, request: &HealRequest) {
         if is_best_effort_source(request.source) {
             self.mrf_normal_dispatches_since_best_effort = 0;
@@ -741,34 +672,12 @@ impl PriorityHealQueue {
         }
     }
 
-    fn decrement_or_remove_dedup_key(dedup_keys: &mut HashMap<String, DedupKeyEntry>, item: &PriorityQueueItem) {
-        if let Some(entry) = dedup_keys.get_mut(&item.dedup_key) {
-            match &mut entry.scanner_queued_owners {
-                ScannerQueuedOwners::Single(sequence) if entry.refcount != 1 || *sequence != item.sequence => {
-                    entry.scanner_queued_owners = ScannerQueuedOwners::Indexed(BTreeMap::new());
-                    entry.representative_observation_id = None;
-                }
-                ScannerQueuedOwners::Single(_) => {}
-                ScannerQueuedOwners::Indexed(owners) => {
-                    let matches_holder = owners.len() == entry.refcount
-                        && owners.get(&item.sequence).is_some_and(|owner| {
-                            owner.request_id == item.request.id && owner.observation_id == item.observation_id
-                        });
-                    if matches_holder {
-                        owners.remove(&item.sequence);
-                    } else {
-                        owners.clear();
-                        entry.representative_observation_id = None;
-                    }
-                }
-            }
+    pub(super) fn decrement_or_remove_dedup_key(dedup_keys: &mut HashMap<String, DedupKeyEntry>, key: &str) {
+        if let Some(entry) = dedup_keys.get_mut(key) {
             if entry.refcount <= 1 {
-                dedup_keys.remove(&item.dedup_key);
+                dedup_keys.remove(key);
             } else {
                 entry.refcount -= 1;
-                if entry.representative_sequence == item.sequence {
-                    entry.representative_observation_id = None;
-                }
             }
         }
     }
@@ -840,59 +749,18 @@ impl PriorityHealQueue {
     }
 
     pub(super) fn queued_request_id_for_dedup_key(&self, key: &str) -> Option<&str> {
-        self.queued_legacy_owner_for_dedup_key(key).map(|(id, _)| id)
+        self.dedup_keys.get(key).map(|entry| entry.representative_request_id.as_str())
     }
 
-    pub(super) fn queued_legacy_owner_for_dedup_key(
-        &self,
-        key: &str,
-    ) -> Option<(&str, Option<rustfs_heal_contracts::completion::HealObservationId>)> {
-        let entry = self.dedup_keys.get(key)?;
-        Some((entry.representative_request_id.as_str(), entry.representative_observation_id))
-    }
-
-    pub(super) fn queued_scanner_owner_for_dedup_key(
-        &self,
-        key: &str,
-    ) -> Option<(&str, Option<rustfs_heal_contracts::completion::HealObservationId>)> {
-        let entry = self.dedup_keys.get(key)?;
-        match &entry.scanner_queued_owners {
-            ScannerQueuedOwners::Single(sequence) if entry.refcount == 1 && entry.representative_sequence == *sequence => {
-                Some((entry.representative_request_id.as_str(), entry.representative_observation_id))
-            }
-            ScannerQueuedOwners::Indexed(owners) if owners.len() == entry.refcount => owners
-                .first_key_value()
-                .map(|(_, owner)| (owner.request_id.as_str(), owner.observation_id)),
-            _ => None,
-        }
-    }
-
-    #[cfg(test)]
-    pub(super) fn scanner_queued_request_id_for_dedup_key(&self, key: &str) -> Option<&str> {
-        self.queued_scanner_owner_for_dedup_key(key).map(|(id, _)| id)
-    }
-
-    #[cfg(test)]
-    pub(super) fn queued_observation_for_dedup_key(
-        &self,
-        key: &str,
-    ) -> Option<rustfs_heal_contracts::completion::HealObservationId> {
-        self.queued_legacy_owner_for_dedup_key(key)
-            .and_then(|(_, observation_id)| observation_id)
-    }
-
-    #[cfg(test)]
-    pub(super) fn queued_observation_for_request_id(
-        &self,
-        id: &str,
-    ) -> Option<rustfs_heal_contracts::completion::HealObservationId> {
-        let mut matching = self.heap.iter().filter(|item| item.request.id == id);
-        let observation_id = matching.next()?.observation_id;
-        if matching.next().is_some() { None } else { observation_id }
-    }
-
-    /// Re-elect the legacy representative from the heap after cancellation or
-    /// displacement. Dequeue leaves it unchanged for existing callers.
+    /// Re-elect the representative for `key` from the queue entries holding
+    /// it. Needed after a holder leaves the queue *without* becoming active
+    /// (canceled by id, or displaced): the former opener may be the request
+    /// that just left, and a merge receipt must never name an id that
+    /// resolves nowhere. The scheduler pop path does not need this — the
+    /// popped request surfaces in `active_heals` under the same id and the
+    /// duplicate pre-check consults active heals before the queue. No-op for
+    /// released keys; the survivor scan only runs when a key still has
+    /// holders, which under forced duplicates is the rare admin path.
     pub(super) fn refresh_dedup_representative(&mut self, key: &str) {
         if !self.dedup_keys.contains_key(key) {
             return;
@@ -901,12 +769,10 @@ impl PriorityHealQueue {
             .heap
             .iter()
             .find(|item| item.dedup_key == key)
-            .map(|item| (item.request.id.clone(), item.sequence, item.observation_id))
+            .map(|item| item.request.id.clone())
             && let Some(entry) = self.dedup_keys.get_mut(key)
         {
-            entry.representative_request_id = id.0;
-            entry.representative_sequence = id.1;
-            entry.representative_observation_id = id.2;
+            entry.representative_request_id = id;
         }
     }
 
@@ -925,7 +791,7 @@ impl PriorityHealQueue {
         while let Some(item) = self.heap.pop() {
             if removed.is_none() && item.request.id == request_id {
                 let key = item.dedup_key.clone();
-                Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item);
+                Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &key);
                 self.remove_best_effort_count(item.request.source);
                 affected_key = Some(key);
                 removed = Some(item.request);
@@ -951,7 +817,7 @@ impl PriorityHealQueue {
 
         while let Some(item) = self.heap.pop() {
             if should_remove(&item.request) {
-                Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item);
+                Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
                 self.remove_best_effort_count(item.request.source);
                 affected_keys.push(item.dedup_key);
                 removed.push(item.request);

crates/heal/src/heal/manager/root_recovery.rs

@@ -460,6 +460,8 @@ impl RootHealTerminal {
 
     fn into_completed(self) -> CompletedHealStatus {
         CompletedHealStatus {
+            task_id: self.task_id,
+            source: HealRequestSource::Admin,
             outcome: None,
             progress: self.progress,
             retained_bytes: std::sync::OnceLock::new(),

crates/heal/src/heal/manager/scheduler.rs

@@ -15,27 +15,6 @@ use super::disposition::record_task_disposition;
 /// The heal scheduler: queue consumption loop and its skip/metric helpers.
 use super::*;
 
-pub(super) fn current_task_observation_id(
-    active: &HashMap<String, Arc<HealTask>>,
-    task: &Arc<HealTask>,
-) -> Option<rustfs_heal_contracts::completion::HealObservationId> {
-    active
-        .get(&task.id)
-        .filter(|owner| Arc::ptr_eq(owner, task))
-        .and_then(|_| task.observation_id)
-}
-
-pub(super) fn current_retry_observation_id(
-    retrying: &HashMap<String, RetryingHeal>,
-    task_id: &str,
-    cancel_token: &CancellationToken,
-) -> Option<rustfs_heal_contracts::completion::HealObservationId> {
-    retrying
-        .get(task_id)
-        .filter(|owner| &owner.cancel_token == cancel_token)
-        .and_then(|owner| owner.observation_id)
-}
-
 pub(super) fn retry_admission_delay(
     grace_deadline: Option<u64>,
     ordinary_delay: Duration,
@@ -232,9 +211,7 @@ impl HealManager {
                 queue.pop_runnable_with_fairness(|_| true, |_| None).0
             };
 
-            if let Some(queued) = selected_request {
-                let observation_id = queued.observation_id;
-                let mut request = queued.request;
+            if let Some(mut request) = selected_request {
                 request.options.timeout.get_or_insert(config.task_timeout);
                 let task_priority = request.priority;
                 let task_type_label = heal_request_type_label(&request).to_string();
@@ -252,7 +229,6 @@ impl HealManager {
                 let mainline_pacer = running_mainline_pacer(&request, &config, workload_provider);
                 let task = Arc::new(
                     HealTask::from_replacement_recovery_request(request, storage.clone(), replacement_resume_endpoint)
-                        .with_observation_id(observation_id)
                         .with_mainline_pacer(mainline_pacer)
                         .with_admin_recovery(root_recovery.clone()),
                 );
@@ -360,9 +336,6 @@ impl HealManager {
                     tests::pause_completed_retention_before_publish(&task_id, &completed_status).await;
                     let mut active_heals_guard = active_heals_clone.lock().await;
                     let owns_completion = active_heals_guard.contains_key(&task_id);
-                    // The legacy scheduler checks only the ID; observation needs the
-                    // actual task object because IDs can be reused concurrently.
-                    let observation_id = current_task_observation_id(&active_heals_guard, &task);
                     let cancelled_snapshot = if owns_completion {
                         None
                     } else {
@@ -455,7 +428,6 @@ impl HealManager {
                                 request.id.clone(),
                                 RetryingHeal {
                                     request: request.clone(),
-                                    observation_id,
                                     error: error.clone(),
                                     cancel_token: cancel_token.clone(),
                                 },
@@ -474,7 +446,6 @@ impl HealManager {
                             &task_id,
                             completed_status_entry,
                             terminal_completion,
-                            observation_id,
                         )
                         .await;
                     }
@@ -622,18 +593,8 @@ impl HealManager {
                                 if retry_cancel_token.is_cancelled() || !retrying.contains_key(&retry_request_id) {
                                     return;
                                 }
-                                // A later retry can reuse the ID. Only the same
-                                // cancellation token proves this is its owner.
-                                let retry_observation_id =
-                                    current_retry_observation_id(&retrying, &retry_request_id, &retry_cancel_token);
-                                let admission_decision = Self::admit_request_to_queue(
-                                    &mut queue,
-                                    retry_request.clone(),
-                                    &retry_config,
-                                    "retry",
-                                    true,
-                                    retry_observation_id,
-                                );
+                                let admission_decision =
+                                    Self::admit_request_to_queue(&mut queue, retry_request.clone(), &retry_config, "retry", true);
                                 let admission = admission_decision.result;
                                 let should_notify = matches!(admission, HealAdmissionResult::Accepted)
                                     && retry_config.event_driven_scheduler_enable;

crates/heal/src/heal/manager/tests.rs

@@ -34,7 +34,6 @@ mod auto_scan;
 mod completion;
 mod disposition;
 mod legacy_ambiguity_logging;
-mod queued_representative;
 mod running_mainline;
 
 use super::super::{DiskOption, DiskStore, Endpoint, new_disk, storage_api::status::BucketInfo};
@@ -119,6 +118,8 @@ struct MockStorage;
 
 fn completed_retention_fixture(completed_at: SystemTime) -> CompletedHealStatus {
     CompletedHealStatus {
+        task_id: "fixture-token".to_string(),
+        source: HealRequestSource::Internal,
         outcome: None,
         heal_type: HealType::Cluster,
         options: HealOptions::default(),
@@ -936,7 +937,6 @@ async fn heal_start_retry_scheduler_carries_explicit_budget_and_identity() {
 
 #[tokio::test]
 async fn heal_start_retry_scheduler_carries_explicit_budget_across_retries() {
-    let mut terminals = rustfs_heal_contracts::completion::subscribe_observed_heal_completions();
     let manager = HealManager::new_without_root_recovery_for_test(
         Arc::new(MockStorage),
         Some(HealConfig {
@@ -950,11 +950,10 @@ async fn heal_start_retry_scheduler_carries_explicit_budget_across_retries() {
     request.options.timeout = Some(Duration::from_secs(60));
     let task_id = request.id.clone();
     let created_at = request.created_at;
-    let receipt = manager
-        .submit_heal_request_with_observation_and_alias(request, false)
+    manager
+        .submit_heal_request(request)
         .await
         .expect("admit explicit-budget task");
-    let observation_id = receipt.observation_id.expect("accepted owner has an observation identity");
 
     let mut previous_remaining = Duration::from_secs(60);
     for expected_attempt in 1..=3 {
@@ -970,9 +969,7 @@ async fn heal_start_retry_scheduler_carries_explicit_budget_across_retries() {
         .await
         .expect("recoverable executor failure should enter retry backoff");
 
-        let retry_owner = manager.retrying_heals.lock().await[&task_id].clone();
-        assert_eq!(retry_owner.observation_id, Some(observation_id));
-        let retry = retry_owner.request;
+        let retry = manager.retrying_heals.lock().await[&task_id].request.clone();
         assert_eq!(retry.id, task_id);
         assert_eq!(retry.created_at, created_at);
         assert_eq!(retry.source, HealRequestSource::Admin);
@@ -998,10 +995,6 @@ async fn heal_start_retry_scheduler_carries_explicit_budget_across_retries() {
         .await
         .expect("retry backoff should requeue the same task id");
         assert_eq!(manager.operations_snapshot().await.queue_length, 1);
-        assert_eq!(
-            manager.heal_queue.lock().await.queued_observation_for_request_id(&task_id),
-            Some(observation_id)
-        );
     }
 
     process_manager_queue_once(&manager).await;
@@ -1022,17 +1015,6 @@ async fn heal_start_retry_scheduler_carries_explicit_budget_across_retries() {
     );
     assert_eq!(manager.operations_snapshot().await.queue_length, 0);
     assert_eq!(manager.operations_snapshot().await.retrying_tasks, 0);
-    let terminal = tokio::time::timeout(Duration::from_secs(5), async {
-        loop {
-            let terminal = terminals.recv().await.expect("terminal channel remains open");
-            if terminal.task_id == task_id {
-                break terminal;
-            }
-        }
-    })
-    .await
-    .expect("retry exhaustion emits one terminal");
-    assert_eq!(terminal.observation_id, Some(observation_id));
 }
 
 struct ManagerRecoveryTestHook {
@@ -1226,32 +1208,6 @@ fn test_push_displacing_lower_priority_actually_enqueues_new_request() {
     assert_eq!(queue.len(), 0);
 }
 
-#[test]
-fn queued_observation_tracks_the_holder_sequence_when_ids_repeat() {
-    let mut queue = PriorityHealQueue::new();
-    let first = bucket_request("same-owner-key", HealPriority::Normal, HealRequestSource::Internal);
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let mut second = first.clone();
-    second.force_start = true;
-    assert_eq!(queue.push_with_replacement_scope(first, false, Some(11)), QueuePushOutcome::Accepted);
-    assert_eq!(queue.push_with_replacement_scope(second, false, Some(12)), QueuePushOutcome::Accepted);
-    assert_eq!(queue.queued_observation_for_dedup_key(&key), Some(11));
-    assert_eq!(
-        queue.queued_observation_for_request_id(&queue.heap.peek().expect("holder").request.id),
-        None
-    );
-
-    let (first, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
-    assert_eq!(first.expect("first holder").observation_id, Some(11));
-    assert_eq!(
-        queue.queued_observation_for_dedup_key(&key),
-        None,
-        "a departed representative is unproven"
-    );
-    let (second, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
-    assert_eq!(second.expect("second holder").observation_id, Some(12));
-}
-
 #[test]
 fn queued_request_id_for_dedup_key_tracks_the_representative() {
     let mut queue = PriorityHealQueue::new();
@@ -1260,7 +1216,6 @@ fn queued_request_id_for_dedup_key_tracks_the_representative() {
     let first_id = first.id.clone();
     let first_key = PriorityHealQueue::make_dedup_key(&first);
     assert_eq!(queue.push(first), QueuePushOutcome::Accepted);
-    assert!(queue.queued_observation_for_dedup_key(&first_key).is_some());
 
     // A forced duplicate of the same target opens a second entry under
     // the same key; the representative stays the request that opened it.
@@ -1388,7 +1343,6 @@ async fn durable_mrf_anchor_snapshot_follows_live_owners_and_ignores_stale_notic
     manager.retrying_heals.lock().await.insert(
         "retry-merged-active".into(),
         RetryingHeal {
-            observation_id: None,
             request: retry_request,
             error: "retry pending".into(),
             cancel_token: CancellationToken::new(),
@@ -2638,7 +2592,6 @@ async fn automatic_replacement_frozen_targets_reject_late_disks_but_cover_duplic
                 manager.retrying_heals.lock().await.insert(
                     owner_id.clone(),
                     RetryingHeal {
-                        observation_id: None,
                         request: owner,
                         error: "target unavailable".to_string(),
                         cancel_token: CancellationToken::new(),
@@ -2650,7 +2603,7 @@ async fn automatic_replacement_frozen_targets_reject_late_disks_but_cover_duplic
                 let mut queue = manager.heal_queue.lock().await;
                 let context = if state == "grace" { "retry" } else { "auto_scan" };
                 assert_eq!(
-                    HealManager::admit_request_to_queue(&mut queue, owner, &config, context, true, None).result,
+                    HealManager::admit_request_to_queue(&mut queue, owner, &config, context, true).result,
                     HealAdmissionResult::Accepted
                 );
                 assert!(
@@ -2774,7 +2727,7 @@ fn automatic_replacement_direct_admission_checks_frozen_targets_before_capacity(
     let owner_id = owner.id.clone();
     assert_eq!(queue.push(owner), QueuePushOutcome::Accepted);
     let admit = |queue: &mut PriorityHealQueue, request, frozen| {
-        HealManager::admit_request_to_queue(queue, request, &config, "auto_scan", frozen, None).result
+        HealManager::admit_request_to_queue(queue, request, &config, "auto_scan", frozen).result
     };
     assert_eq!(
         admit(&mut queue, automatic_replacement_request(&["replacement-b"]), false),
@@ -3120,6 +3073,181 @@ async fn test_active_duplicate_token_can_query_and_cancel_original_task() {
     );
 }
 
+#[tokio::test]
+async fn local_automatic_query_preserves_source_path_token_and_state() {
+    use crate::heal::channel::HealChannelProcessor;
+    use crate::heal::outcome::{
+        HealExecutionOutcome, HealObjectDisposition, HealObjectIdentity, HealObjectKind, HealObjectOutcome, HealTaskOutcome,
+        HealTraversalCoverage,
+    };
+
+    for source in [
+        HealRequestSource::Scanner,
+        HealRequestSource::Mrf,
+        HealRequestSource::AutoHeal,
+        HealRequestSource::ReadRepair,
+        HealRequestSource::Admin,
+        HealRequestSource::Internal,
+    ] {
+        for state in ["queued", "active", "retrying", "completed", "displaced"] {
+            let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
+            let manager = Arc::new(HealManager::new_without_root_recovery_for_test(storage.clone(), None));
+            let mut request = HealRequest::object("bucket".to_string(), "object".to_string(), None);
+            request.source = source;
+            let token = request.id.clone();
+            let task = Arc::new(HealTask::from_request(request.clone(), storage));
+            let mut snapshot = CompletedHealStatus::snapshot(&task, HealTaskStatus::Completed).await;
+            assert_eq!(snapshot.source, source, "completion must retain actual task source");
+            let mut outcome = HealTaskOutcome::default();
+            outcome.execution = HealExecutionOutcome::Completed;
+            outcome.coverage = HealTraversalCoverage::Complete;
+            outcome.record(HealObjectOutcome {
+                identity: HealObjectIdentity {
+                    kind: HealObjectKind::Object,
+                    bucket: "bucket".to_string(),
+                    object: "object".to_string(),
+                    version_id: None,
+                    bucket_incarnation_id: Some(uuid::Uuid::nil()),
+                    pool_index: Some(0),
+                    set_index: Some(0),
+                },
+                disposition: HealObjectDisposition::VerifiedHealthy,
+                detail: None,
+            });
+            snapshot.outcome = Some(Arc::new(outcome));
+            snapshot.seqed_items = vec![(
+                3,
+                HealResultItem {
+                    bucket: "bucket".to_string(),
+                    object: "object".to_string(),
+                    ..Default::default()
+                },
+            )];
+            snapshot.min_seq = 3;
+            snapshot.next_seq = 4;
+            let expected_status = match state {
+                "queued" => {
+                    manager.heal_queue.lock().await.push(request.clone());
+                    HealTaskStatus::Pending
+                }
+                "active" => {
+                    manager.active_heals.lock().await.insert(token.clone(), task);
+                    HealTaskStatus::Pending
+                }
+                "retrying" => {
+                    manager.retrying_heals.lock().await.insert(
+                        token.clone(),
+                        RetryingHeal {
+                            request: request.clone(),
+                            error: "transient".to_string(),
+                            cancel_token: CancellationToken::new(),
+                        },
+                    );
+                    HealTaskStatus::Retrying {
+                        error: "transient".to_string(),
+                        retry_attempt: request.retry_attempts,
+                    }
+                }
+                "completed" => {
+                    manager.completed_heals.lock().await.insert(token.clone(), Arc::new(snapshot));
+                    HealTaskStatus::Completed
+                }
+                "displaced" => {
+                    let terminal = record_displaced_terminal(&manager.displaced_terminals, &request);
+                    terminal.status.clone()
+                }
+                _ => unreachable!("fixed test states"),
+            };
+            let automatic = !matches!(source, HealRequestSource::Admin | HealRequestSource::Internal);
+            let report = manager
+                .get_local_automatic_task_report("bucket/object", &token, Some(2))
+                .await
+                .expect("local source lookup");
+            assert_eq!(report.is_some(), automatic, "source={source:?} state={state}");
+            if let Some(report) = report {
+                assert_eq!(report.status, expected_status);
+                if state == "completed" {
+                    let outcome = report.outcome.as_ref().expect("retained structured outcome");
+                    assert_eq!(outcome.objects.len(), 1);
+                    assert_eq!(outcome.objects[0].identity.object, "object");
+                    assert_eq!(outcome.objects[0].identity.pool_index, Some(0));
+                    assert_eq!(outcome.objects[0].identity.set_index, Some(0));
+                    assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::VerifiedHealthy);
+                    assert_eq!(report.min_seq, 3);
+                    assert_eq!(report.next_seq, 4);
+                    assert_eq!(report.result_items.len(), 1);
+                    assert_eq!(report.result_items[0].object, "object");
+                }
+                let processor = HealChannelProcessor::new(manager.clone());
+                let local = processor
+                    .execute_local_automatic_query("bucket/object", &token, Some(2))
+                    .await
+                    .expect("local response")
+                    .expect("automatic task response");
+                let direct = processor
+                    .execute_query_request_since("bucket/object".to_string(), token.clone(), Some(2))
+                    .await
+                    .expect("original query response");
+                assert!(local.success);
+                assert_eq!(local.request_id, direct.request_id);
+                assert_eq!(local.data, direct.data, "same report encoding and cursor contract");
+                assert_eq!(local.error, direct.error);
+                let coordinator = Arc::new(HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None));
+                let missing = HealChannelProcessor::new(coordinator)
+                    .execute_query_request("bucket/object".to_string(), token.clone())
+                    .await
+                    .expect("coordinator query");
+                let body: serde_json::Value =
+                    serde_json::from_slice(missing.data.as_deref().expect("coordinator response")).expect("coordinator JSON");
+                assert_eq!(body["summary"], "notFound", "a remote manager does not own the automatic task");
+                if state == "completed" {
+                    let completed = manager
+                        .completed_heals
+                        .lock()
+                        .await
+                        .get(&token)
+                        .cloned()
+                        .expect("original completion");
+                    manager
+                        .completed_heals
+                        .lock()
+                        .await
+                        .insert("retired-alias".to_string(), completed);
+                    assert!(
+                        manager
+                            .get_local_automatic_task_report("bucket/object", "retired-alias", None)
+                            .await
+                            .expect("retired alias lookup")
+                            .is_none()
+                    );
+                }
+            }
+            assert!(
+                manager
+                    .get_local_automatic_task_report("bucket/other", &token, None)
+                    .await
+                    .expect("wrong path lookup")
+                    .is_none()
+            );
+            assert!(
+                manager
+                    .get_local_automatic_task_report("bucket/object", "unknown", None)
+                    .await
+                    .expect("unknown token lookup")
+                    .is_none()
+            );
+            manager.insert_task_alias("admin-alias", &token).await;
+            assert!(
+                manager
+                    .get_local_automatic_task_report("bucket/object", "admin-alias", None)
+                    .await
+                    .expect("alias lookup")
+                    .is_none()
+            );
+        }
+    }
+}
+
 #[tokio::test]
 async fn test_queued_duplicate_token_can_query_and_cancel_original_request() {
     let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -3935,6 +4063,38 @@ async fn test_retry_request_rescans_batch_when_all_exhausted_objects_are_retryab
     }
 }
 
+#[tokio::test]
+async fn admin_exhausted_batches_do_not_restart_the_traversal() {
+    for heal_type in [
+        HealType::Bucket {
+            bucket: "bucket".to_owned(),
+        },
+        HealType::Prefix {
+            bucket: "bucket".to_owned(),
+            prefix: "prefix/".to_owned(),
+        },
+        HealType::Cluster,
+    ] {
+        let mut request = HealRequest::new(heal_type, HealOptions::default(), HealPriority::Normal);
+        request.source = HealRequestSource::Admin;
+        let task = HealTask::from_request(request, Arc::new(MockStorage));
+        let result = Err(task
+            .record_batch_failure(BatchHealFailure {
+                scope: "completed-admin-traversal".to_owned(),
+                failed: 1,
+                retryable: 1,
+                permanent: 0,
+                first_object: "exhausted-object".to_owned(),
+                first_error: Error::Storage(EcstoreError::SlowDown).to_string(),
+            })
+            .await);
+        assert!(
+            retry_request_for_result_with_budget(&task, &result).await.is_none(),
+            "the exhausted admin batch must retain its original object receipts"
+        );
+    }
+}
+
 #[tokio::test]
 async fn test_retry_request_does_not_rescan_cancelled_or_timed_out_retryable_batch() {
     for terminal_error in [Error::TaskCancelled, Error::TaskTimeout] {
@@ -4007,7 +4167,6 @@ async fn insert_retrying_request(manager: &HealManager, request: HealRequest) ->
     manager.retrying_heals.lock().await.insert(
         task_id.clone(),
         RetryingHeal {
-            observation_id: None,
             request: request.clone(),
             error: "Lock acquisition timeout".to_string(),
             cancel_token: cancel_token.clone(),
@@ -4018,6 +4177,8 @@ async fn insert_retrying_request(manager: &HealManager, request: HealRequest) ->
         Arc::new(CompletedHealStatus {
             progress: None,
             outcome: None,
+            source: request.source,
+            task_id: request.id.clone(),
             retained_bytes: std::sync::OnceLock::new(),
             heal_type: request.heal_type,
             options: request.options.clone(),
@@ -4560,7 +4721,6 @@ async fn test_operations_snapshot_counts_retry_backoff_as_owned_work() {
     manager.retrying_heals.lock().await.insert(
         task_id,
         RetryingHeal {
-            observation_id: None,
             request,
             error: "transient".to_string(),
             cancel_token: CancellationToken::new(),
@@ -4840,7 +5000,6 @@ async fn test_get_task_report_preserves_retrying_options() {
     manager.retrying_heals.lock().await.insert(
         task_id.clone(),
         RetryingHeal {
-            observation_id: None,
             request,
             error: "transient".to_string(),
             cancel_token: CancellationToken::new(),
@@ -4868,6 +5027,8 @@ async fn test_retrying_completion_outranks_the_queue_for_the_same_id() {
         Arc::new(CompletedHealStatus {
             progress: None,
             outcome: None,
+            source: request.source,
+            task_id: request.id.clone(),
             retained_bytes: std::sync::OnceLock::new(),
             heal_type: request.heal_type.clone(),
             options: request.options.clone(),
@@ -4907,6 +5068,8 @@ async fn test_get_task_status_reads_recent_completed_status() {
         Arc::new(CompletedHealStatus {
             progress: None,
             outcome: None,
+            source: HealRequestSource::Internal,
+            task_id: "completed-token".to_string(),
             retained_bytes: std::sync::OnceLock::new(),
             heal_type: HealType::Bucket {
                 bucket: "bucket".to_string(),
@@ -4940,6 +5103,8 @@ async fn test_get_task_report_for_path_reads_completed_items() {
         Arc::new(CompletedHealStatus {
             progress: None,
             outcome: None,
+            source: HealRequestSource::Internal,
+            task_id: "completed-token".to_string(),
             retained_bytes: std::sync::OnceLock::new(),
             heal_type: HealType::Object {
                 bucket: "bucket".to_string(),

crates/heal/src/heal/manager/tests/admin_overlap.rs

@@ -37,7 +37,6 @@ async fn install_owner(manager: &HealManager, request: HealRequest, state: Owner
             manager.retrying_heals.lock().await.insert(
                 id.clone(),
                 RetryingHeal {
-                    observation_id: None,
                     request,
                     error: "recoverable fixture error".to_string(),
                     cancel_token: CancellationToken::new(),
@@ -451,177 +450,6 @@ async fn admin_overlap_normal_start_cannot_enter_force_start_cancellation_window
     assert_eq!(manager.heal_queue.lock().await.len(), 1);
 }
 
-#[tokio::test]
-async fn admin_force_start_rechecks_request_id_after_cancelling_overlaps() {
-    for source in [HealRequestSource::Scanner, HealRequestSource::Internal] {
-        let manager = manager_with_policy(HealOverlapPolicy::Merge);
-        let previous_id = install_owner(&manager, admin_prefix_request("bucket", "scope/child/"), OwnerState::Active).await;
-        let aliases = manager.task_aliases.lock().await;
-        let mut replacement = admin_prefix_request("bucket", "scope/");
-        replacement.force_start = true;
-        let request_id = replacement.id.clone();
-        let mut forced = Box::pin(manager.submit_heal_request_with_receipt(replacement));
-
-        tokio::time::timeout(
-            Duration::from_secs(5),
-            std::future::poll_fn(|cx| {
-                assert!(
-                    std::pin::pin!(tokio::task::unconstrained(forced.as_mut()))
-                        .poll(cx)
-                        .is_pending()
-                );
-                if manager.active_heals.try_lock().is_ok() {
-                    Poll::Ready(())
-                } else {
-                    cx.waker().wake_by_ref();
-                    Poll::Pending
-                }
-            }),
-        )
-        .await
-        .expect("forceStart reaches cancellation with registry locks released");
-
-        let mut competing = HealRequest::object("other-bucket".to_string(), "object".to_string(), None);
-        competing.id = request_id.clone();
-        competing.source = source;
-        competing.options.recreate_missing = false;
-        let receipt = manager
-            .submit_heal_request_with_receipt(competing)
-            .await
-            .expect("non-admin admission during forceStart cancellation");
-        assert_eq!(receipt.result, HealAdmissionResult::Accepted);
-        assert_eq!(receipt.task_id, request_id);
-
-        drop(aliases);
-        let receipt = forced.await.expect("forceStart returns a conflict");
-        assert_eq!(receipt.result, HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning));
-        assert_eq!(receipt.task_id, request_id);
-        let queue = manager.heal_queue.lock().await;
-        let same_id: Vec<_> = queue.requests().filter(|request| request.id == request_id).collect();
-        assert_eq!(same_id.len(), 1, "conflicting requests must not share an id");
-        assert_eq!(same_id[0].source, source);
-        drop(queue);
-        assert!(
-            matches!(manager.get_task_status(&previous_id).await, Ok(HealTaskStatus::Cancelled)),
-            "overlapping admin cancellation precedes the conflict recheck"
-        );
-    }
-}
-
-#[tokio::test]
-async fn forced_id_conflict_preserves_scanner_queued_owner() {
-    for competing_source in [HealRequestSource::Scanner, HealRequestSource::Internal] {
-        let manager = manager_with_policy(HealOverlapPolicy::Merge);
-        let previous_id = install_owner(&manager, admin_object_request("object"), OwnerState::Active).await;
-        let mut queued_ids = Vec::new();
-        for source in [
-            HealRequestSource::Scanner,
-            HealRequestSource::Internal,
-            HealRequestSource::Scanner,
-        ] {
-            let mut request = HealRequest::object("bucket".to_string(), "object".to_string(), None);
-            request.source = source;
-            request.force_start = true;
-            let id = request.id.clone();
-            let receipt = manager
-                .submit_heal_request_with_receipt(request)
-                .await
-                .expect("queue forced work");
-            assert_eq!(receipt.result, HealAdmissionResult::Accepted);
-            assert_eq!(receipt.task_id, id);
-            queued_ids.push(id);
-        }
-
-        // The first holder leaves the queue, leaving two forced tasks queued.
-        let selected = {
-            let mut queue = manager.heal_queue.lock().await;
-            let (selected, skipped) = queue.pop_runnable_with_skips(|_| true, |_| None);
-            assert!(skipped.is_empty());
-            let selected = selected.expect("first queued task should run");
-            assert_eq!(selected.id, queued_ids[0]);
-            assert_eq!(queue.len(), 2);
-            selected
-        };
-        let QueuedHeal { request, observation_id } = selected;
-        let completed_task = HealTask::from_request(request, manager.storage.clone()).with_observation_id(observation_id);
-        let mut completed = CompletedHealStatus::snapshot(&completed_task, HealTaskStatus::Completed).await;
-        Arc::make_mut(completed.outcome.as_mut().expect("completed task outcome")).finish(None);
-        manager
-            .completed_heals
-            .lock()
-            .await
-            .insert(queued_ids[0].clone(), Arc::new(completed));
-
-        let aliases = manager.task_aliases.lock().await;
-        let mut replacement = admin_object_request("object");
-        replacement.force_start = true;
-        let competing_id = replacement.id.clone();
-        let mut forced = Box::pin(manager.submit_heal_request_with_receipt(replacement));
-        tokio::time::timeout(
-            Duration::from_secs(5),
-            std::future::poll_fn(|cx| {
-                assert!(
-                    std::pin::pin!(tokio::task::unconstrained(forced.as_mut()))
-                        .poll(cx)
-                        .is_pending()
-                );
-                if manager.active_heals.try_lock().is_ok() {
-                    Poll::Ready(())
-                } else {
-                    cx.waker().wake_by_ref();
-                    Poll::Pending
-                }
-            }),
-        )
-        .await
-        .expect("forceStart reaches cancellation with registry locks released");
-
-        let mut competing = HealRequest::object("bucket".to_string(), "object".to_string(), None);
-        competing.id = competing_id.clone();
-        competing.source = competing_source;
-        competing.force_start = true;
-        let receipt = manager
-            .submit_heal_request_with_receipt(competing)
-            .await
-            .expect("competing same-key forced work");
-        assert_eq!(receipt.result, HealAdmissionResult::Accepted);
-        assert_eq!(receipt.task_id, competing_id);
-
-        drop(aliases);
-        let receipt = forced.await.expect("forceStart returns an ID conflict");
-        assert_eq!(receipt.result, HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning));
-        assert_eq!(receipt.task_id, competing_id);
-        assert!(!manager.active_heals.lock().await.contains_key(&competing_id));
-        assert!(!manager.retrying_heals.lock().await.contains_key(&competing_id));
-        assert!(matches!(manager.get_task_status(&previous_id).await, Ok(HealTaskStatus::Cancelled)));
-        {
-            let queue = manager.heal_queue.lock().await;
-            let same_id: Vec<_> = queue.requests().filter(|request| request.id == competing_id).collect();
-            assert_eq!(same_id.len(), 1, "the competing ID must have one live owner");
-            assert_eq!(same_id[0].source, competing_source);
-            assert_eq!(queue.len(), 3);
-        }
-
-        let mut scanner = HealRequest::object("bucket".to_string(), "object".to_string(), None);
-        scanner.source = HealRequestSource::Scanner;
-        let receipt = manager
-            .submit_heal_request_with_receipt(scanner)
-            .await
-            .expect("scanner should merge into queued work");
-        assert_eq!(receipt.result, HealAdmissionResult::Merged);
-        assert_ne!(receipt.task_id, queued_ids[0], "the completed holder cannot own a queued receipt");
-        assert!(
-            manager
-                .heal_queue
-                .lock()
-                .await
-                .requests()
-                .any(|request| request.id == receipt.task_id),
-            "the canonical receipt must name a live queued task"
-        );
-    }
-}
-
 #[tokio::test]
 async fn admin_overlap_cancel_cannot_miss_a_queued_to_active_transition() {
     let manager = manager_with_policy(HealOverlapPolicy::Merge);

crates/heal/src/heal/manager/tests/completion.rs

@@ -1,208 +1,6 @@
 use super::*;
 use crate::heal::outcome::{HealObjectIdentity, HealObjectKind, HealObjectOutcome};
-use rustfs_heal_contracts::{
-    completion::{subscribe_heal_completions, subscribe_observed_heal_completions},
-    heal_channel::HealScanMode,
-};
-
-#[test]
-fn observation_requires_the_current_task_or_retry_object() {
-    let request = bucket_request("observation-object-owner", HealPriority::Normal, HealRequestSource::Internal);
-    let id = request.id.clone();
-    let stale = Arc::new(HealTask::from_request(request.clone(), Arc::new(MockStorage)).with_observation_id(Some(301)));
-    let current = Arc::new(HealTask::from_request(request.clone(), Arc::new(MockStorage)).with_observation_id(Some(302)));
-    let active = HashMap::from([(id.clone(), Arc::clone(&current))]);
-    assert_eq!(current_task_observation_id(&active, &stale), None);
-    assert_eq!(current_task_observation_id(&active, &current), Some(302));
-
-    let current_token = CancellationToken::new();
-    let stale_token = CancellationToken::new();
-    let retrying = HashMap::from([(
-        id.clone(),
-        RetryingHeal {
-            request,
-            observation_id: Some(303),
-            error: "retrying".into(),
-            cancel_token: current_token.clone(),
-        },
-    )]);
-    assert_eq!(current_retry_observation_id(&retrying, &id, &stale_token), None);
-    assert_eq!(current_retry_observation_id(&retrying, &id, &current_token), Some(303));
-}
-
-#[tokio::test]
-async fn queued_same_id_replay_is_unknown_but_canonical_merge_retains_owner_identity() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let request = bucket_request("queued-id-replay-observation", HealPriority::Normal, HealRequestSource::Internal);
-    let first = manager
-        .submit_heal_request_with_observation_and_alias(request.clone(), false)
-        .await
-        .expect("first owner");
-    assert!(first.observation_id.is_some());
-    let replay = manager
-        .submit_heal_request_with_observation_and_alias(request, false)
-        .await
-        .expect("same ID replay");
-    assert_eq!(replay.result, HealAdmissionResult::Accepted);
-    assert_eq!(replay.task_id, first.task_id);
-    assert_eq!(replay.observation_id, None);
-
-    let duplicate = bucket_request("queued-id-replay-observation", HealPriority::Normal, HealRequestSource::Scanner);
-    let merged = manager
-        .submit_heal_request_with_observation_and_alias(duplicate, false)
-        .await
-        .expect("canonical merge");
-    assert_eq!(merged.result, HealAdmissionResult::Merged);
-    assert_eq!(merged.task_id, first.task_id);
-    assert_eq!(merged.observation_id, first.observation_id);
-}
-
-#[tokio::test]
-async fn merged_receipt_uses_live_active_or_retrying_owner_identity() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let request = bucket_request("owner-observation-states", HealPriority::Normal, HealRequestSource::Internal);
-    let id = request.id.clone();
-    manager.active_heals.lock().await.insert(
-        id.clone(),
-        Arc::new(HealTask::from_request(request.clone(), Arc::new(MockStorage)).with_observation_id(Some(201))),
-    );
-    let scanner = bucket_request("owner-observation-states", HealPriority::Normal, HealRequestSource::Scanner);
-    let active = manager
-        .submit_heal_request_with_observation_and_alias(scanner.clone(), false)
-        .await
-        .expect("active merge");
-    assert_eq!(active.result, HealAdmissionResult::Merged);
-    assert_eq!(active.task_id, id);
-    assert_eq!(active.observation_id, Some(201));
-    manager.active_heals.lock().await.remove(&id);
-    manager.retrying_heals.lock().await.insert(
-        id.clone(),
-        RetryingHeal {
-            request,
-            observation_id: Some(202),
-            error: "retrying".into(),
-            cancel_token: CancellationToken::new(),
-        },
-    );
-    let mut scanner_retry = scanner;
-    scanner_retry.id = Uuid::new_v4().to_string();
-    let retrying = manager
-        .submit_heal_request_with_observation_and_alias(scanner_retry, false)
-        .await
-        .expect("retry merge");
-    assert_eq!(retrying.result, HealAdmissionResult::Merged);
-    assert_eq!(retrying.task_id, id);
-    assert_eq!(retrying.observation_id, Some(202));
-}
-
-#[tokio::test]
-async fn cancelled_or_displaced_queue_owner_does_not_reuse_its_observation_identity() {
-    for displace in [false, true] {
-        let manager = HealManager::new_without_root_recovery_for_test(
-            Arc::new(MockStorage),
-            Some(HealConfig {
-                queue_size: 1,
-                event_driven_scheduler_enable: false,
-                ..Default::default()
-            }),
-        );
-        let request = bucket_request("removed-observation-owner", HealPriority::Low, HealRequestSource::Scanner);
-        let old = manager
-            .submit_heal_request_with_observation_and_alias(request.clone(), false)
-            .await
-            .expect("first owner");
-        assert_eq!(old.result, HealAdmissionResult::Accepted);
-        assert!(old.observation_id.is_some());
-        if displace {
-            let urgent = bucket_request("displacing-owner", HealPriority::Urgent, HealRequestSource::Internal);
-            let urgent_id = urgent.id.clone();
-            let admitted = manager
-                .submit_heal_request_with_observation_and_alias(urgent, false)
-                .await
-                .expect("displacing owner");
-            assert_eq!(admitted.result, HealAdmissionResult::Accepted);
-            assert!(
-                manager
-                    .heal_queue
-                    .lock()
-                    .await
-                    .requests()
-                    .all(|queued| queued.id != old.task_id)
-            );
-            manager.cancel_task(&urgent_id).await.expect("free queue after displacement");
-        } else {
-            manager.cancel_task(&old.task_id).await.expect("cancel queued owner");
-        }
-        let next = manager
-            .submit_heal_request_with_observation_and_alias(request, false)
-            .await
-            .expect("readmit same id");
-        assert_eq!(next.result, HealAdmissionResult::Accepted);
-        assert_eq!(next.task_id, old.task_id);
-        assert!(next.observation_id.is_some());
-        assert_ne!(next.observation_id, old.observation_id);
-    }
-}
-
-#[tokio::test]
-async fn reused_id_has_distinct_owner_generations_in_receipts_and_terminals() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let mut receiver = subscribe_observed_heal_completions();
-    let request = bucket_request("observation-id-reuse", HealPriority::Normal, HealRequestSource::Internal);
-
-    let first = manager
-        .submit_heal_request_with_observation_and_alias(request.clone(), false)
-        .await
-        .expect("first owner");
-    let mut scanner = bucket_request("observation-id-reuse", HealPriority::Normal, HealRequestSource::Scanner);
-    let merged = manager
-        .submit_heal_request_with_observation_and_alias(scanner.clone(), false)
-        .await
-        .expect("first merge");
-    assert_eq!(merged.result, HealAdmissionResult::Merged);
-    assert_eq!(merged.task_id, first.task_id);
-    assert!(first.observation_id.is_some());
-    assert_eq!(merged.observation_id, first.observation_id);
-    process_manager_queue_once(&manager).await;
-    let first_terminal = tokio::time::timeout(Duration::from_secs(5), async {
-        loop {
-            let terminal = receiver.recv().await.expect("terminal channel");
-            if terminal.task_id == first.task_id {
-                break terminal;
-            }
-        }
-    })
-    .await
-    .expect("first terminal");
-    assert_eq!(first_terminal.observation_id, first.observation_id);
-
-    let second = manager
-        .submit_heal_request_with_observation_and_alias(request, false)
-        .await
-        .expect("reused id");
-    assert_eq!(second.result, HealAdmissionResult::Accepted);
-    assert_eq!(second.task_id, first.task_id);
-    assert_ne!(second.observation_id, first.observation_id);
-    scanner.id = Uuid::new_v4().to_string();
-    let merged_again = manager
-        .submit_heal_request_with_observation_and_alias(scanner, false)
-        .await
-        .expect("second merge");
-    assert_eq!(merged_again.result, HealAdmissionResult::Merged);
-    assert_eq!(merged_again.observation_id, second.observation_id);
-    process_manager_queue_once(&manager).await;
-    let second_terminal = tokio::time::timeout(Duration::from_secs(5), async {
-        loop {
-            let terminal = receiver.recv().await.expect("terminal channel");
-            if terminal.task_id == second.task_id && terminal.observation_id == second.observation_id {
-                break terminal;
-            }
-        }
-    })
-    .await
-    .expect("second terminal");
-    assert_eq!(second_terminal.observation_id, second.observation_id);
-}
+use rustfs_heal_contracts::{completion::subscribe_heal_completions, heal_channel::HealScanMode};
 
 #[tokio::test]
 async fn canonical_completion_publishes_typed_objects_only_for_terminal_untruncated_results() {
@@ -245,12 +43,12 @@ async fn canonical_completion_publishes_typed_objects_only_for_terminal_untrunca
         error: "retry".into(),
         retry_attempt: 1,
     };
-    publish_completed_heal(&completed_heals, &aliases, &id, completed.clone(), false, Some(1)).await;
+    publish_completed_heal(&completed_heals, &aliases, &id, completed.clone(), false).await;
     while let Ok(observed) = receiver.try_recv() {
         assert_ne!(observed.task_id, id, "a retry attempt must not publish a terminal");
     }
     completed.status = HealTaskStatus::Completed;
-    publish_completed_heal(&completed_heals, &aliases, &id, completed.clone(), true, Some(1)).await;
+    publish_completed_heal(&completed_heals, &aliases, &id, completed.clone(), true).await;
     let observed = tokio::time::timeout(Duration::from_secs(1), async {
         loop {
             let event = receiver.recv().await.expect("terminal broadcaster stays open");
@@ -275,7 +73,7 @@ async fn canonical_completion_publishes_typed_objects_only_for_terminal_untrunca
     outcome.objects_truncated = true;
     completed.outcome = Some(Arc::new(outcome));
     let truncated_id = Uuid::new_v4().to_string();
-    publish_completed_heal(&completed_heals, &aliases, &truncated_id, completed, true, Some(1)).await;
+    publish_completed_heal(&completed_heals, &aliases, &truncated_id, completed, true).await;
     let truncated = tokio::time::timeout(Duration::from_secs(1), async {
         loop {
             let event = receiver.recv().await.expect("terminal broadcaster stays open");

crates/heal/src/heal/manager/tests/queued_representative.rs

@@ -1,653 +0,0 @@
-// Copyright 2026 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use super::*;
-
-fn object_request(source: HealRequestSource, priority: HealPriority, force_start: bool) -> HealRequest {
-    let mut request = HealRequest::new(
-        HealType::Object {
-            bucket: "canonical-owner".to_owned(),
-            object: "object".to_owned(),
-            version_id: None,
-        },
-        HealOptions::default(),
-        priority,
-    );
-    request.source = source;
-    request.force_start = force_start;
-    request
-}
-
-fn scoped_object_request(
-    source: HealRequestSource,
-    version_id: &str,
-    pool_index: Option<usize>,
-    set_index: Option<usize>,
-    force_start: bool,
-) -> HealRequest {
-    let mut request = object_request(source, HealPriority::Low, force_start);
-    let HealType::Object { version_id: version, .. } = &mut request.heal_type else {
-        unreachable!("object_request always constructs an object heal")
-    };
-    *version = Some(version_id.to_owned());
-    request.options.pool_index = pool_index;
-    request.options.set_index = set_index;
-    request.options.recreate_missing = source != HealRequestSource::Scanner;
-    request
-}
-
-#[tokio::test]
-async fn scanner_receipt_tracks_scoped_read_repair_forced_holders() {
-    let manager = HealManager::new_without_root_recovery_for_test(
-        Arc::new(MockStorage),
-        Some(HealConfig {
-            queue_size: 16,
-            low_priority_merge_enable: true,
-            ..HealConfig::default()
-        }),
-    );
-    let version = "00000000-0000-4000-8000-000000000001";
-    let original = scoped_object_request(HealRequestSource::ReadRepair, version, Some(0), Some(1), false);
-    let original_id = original.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&original);
-    let admitted = manager
-        .submit_heal_request_with_receipt(original)
-        .await
-        .expect("admit read-repair owner");
-    assert_eq!(admitted.result, HealAdmissionResult::Accepted);
-    assert_eq!(admitted.task_id, original_id);
-
-    let receipt = manager
-        .submit_heal_request_with_receipt(scoped_object_request(HealRequestSource::Scanner, version, Some(0), Some(1), false))
-        .await
-        .expect("scanner merges into the scoped read-repair owner");
-    assert_eq!(receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(receipt.task_id, original_id);
-
-    let first_forced = scoped_object_request(HealRequestSource::Internal, version, Some(0), Some(1), true);
-    let first_forced_id = first_forced.id.clone();
-    let second_forced = scoped_object_request(HealRequestSource::Internal, version, Some(0), Some(1), true);
-    let second_forced_id = second_forced.id.clone();
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        assert_eq!(queue.push(first_forced), QueuePushOutcome::Accepted);
-        assert_eq!(queue.push(second_forced), QueuePushOutcome::Accepted);
-        let (selected, skipped) = queue.pop_runnable_with_skips(|_| true, |_| None);
-        assert!(skipped.is_empty());
-        assert_eq!(selected.expect("read-repair owner should leave first").id, original_id);
-        assert_eq!(queue.queued_request_id_for_dedup_key(&key), Some(original_id.as_str()));
-    }
-
-    let receipt = manager
-        .submit_heal_request_with_receipt(scoped_object_request(HealRequestSource::Scanner, version, Some(0), Some(1), false))
-        .await
-        .expect("scanner merges into the remaining owner");
-    assert_eq!(receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(receipt.task_id, first_forced_id);
-
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        let (selected, _) = queue.pop_runnable_with_skips(|request| request.id == first_forced_id, |_| None);
-        assert_eq!(selected.expect("first forced holder should leave").id, first_forced_id);
-        assert!(queue.contains_request_id(&second_forced_id));
-    }
-    let receipt = manager
-        .submit_heal_request_with_receipt(scoped_object_request(HealRequestSource::Scanner, version, Some(0), Some(1), false))
-        .await
-        .expect("scanner merges into the last queued holder");
-    assert_eq!(receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(receipt.task_id, second_forced_id);
-}
-
-#[tokio::test]
-async fn scanner_receipt_does_not_merge_read_repair_with_a_different_scope_or_version() {
-    let version = "00000000-0000-4000-8000-000000000001";
-    let other_version = "00000000-0000-4000-8000-000000000002";
-    for (candidate_version, pool, set) in [
-        (version, None, None),
-        (version, Some(1), Some(1)),
-        (version, Some(0), Some(2)),
-        (other_version, Some(0), Some(1)),
-    ] {
-        let manager = HealManager::new_without_root_recovery_for_test(
-            Arc::new(MockStorage),
-            Some(HealConfig {
-                queue_size: 16,
-                low_priority_merge_enable: true,
-                ..HealConfig::default()
-            }),
-        );
-        let original = scoped_object_request(HealRequestSource::ReadRepair, version, Some(0), Some(1), false);
-        let original_id = original.id.clone();
-        let initial = manager
-            .submit_heal_request_with_receipt(original)
-            .await
-            .expect("admit read-repair owner");
-        assert_eq!(initial.result, HealAdmissionResult::Accepted);
-        assert_eq!(initial.task_id, original_id);
-
-        let candidate = scoped_object_request(HealRequestSource::Scanner, candidate_version, pool, set, false);
-        let candidate_id = candidate.id.clone();
-        let receipt = manager
-            .submit_heal_request_with_receipt(candidate)
-            .await
-            .expect("admit scanner with a distinct scope or version");
-        assert_eq!(
-            receipt.result,
-            HealAdmissionResult::Accepted,
-            "version={candidate_version}, pool={pool:?}, set={set:?}"
-        );
-        assert_eq!(receipt.task_id, candidate_id);
-        let queue = manager.heal_queue.lock().await;
-        assert!(queue.contains_request_id(&original_id));
-        assert!(queue.contains_request_id(&candidate_id));
-    }
-}
-
-async fn scanner_receipt_tracks_queued_owner(use_fairness: bool) {
-    let manager = HealManager::new_without_root_recovery_for_test(
-        Arc::new(MockStorage),
-        Some(HealConfig {
-            queue_size: 16,
-            low_priority_merge_enable: true,
-            overlap_policy: HealOverlapPolicy::Merge,
-            ..HealConfig::default()
-        }),
-    );
-
-    if use_fairness {
-        let mut queue = manager.heal_queue.lock().await;
-        for index in 0..4 {
-            let request = bucket_request(&format!("internal-{index}"), HealPriority::Normal, HealRequestSource::Internal);
-            assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-        }
-        for _ in 0..4 {
-            let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
-            assert_eq!(request.expect("internal work should run").source, HealRequestSource::Internal);
-        }
-    }
-
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-    let second_id = second.id.clone();
-    for request in [first, second] {
-        assert_eq!(
-            manager
-                .submit_heal_request_with_receipt(request)
-                .await
-                .expect("admit task")
-                .result,
-            HealAdmissionResult::Accepted
-        );
-    }
-
-    let first = {
-        let mut queue = manager.heal_queue.lock().await;
-        let (selected, skipped) = if use_fairness {
-            queue.pop_runnable_with_fairness(|_| true, |_| None)
-        } else {
-            queue.pop_runnable_with_skips(|_| true, |_| None)
-        };
-        assert!(skipped.is_empty());
-        let selected = selected.expect("the first same-priority task should run");
-        assert_eq!(selected.id, first_id);
-        assert!(queue.contains_request_id(&second_id));
-        assert_eq!(
-            queue.queued_request_id_for_dedup_key(&key),
-            Some(first_id.as_str()),
-            "the existing representative for other sources must remain unchanged"
-        );
-        selected
-    };
-    let QueuedHeal { request, observation_id } = first;
-    manager.active_heals.lock().await.insert(
-        first_id.clone(),
-        Arc::new(HealTask::from_request(request, Arc::new(MockStorage)).with_observation_id(observation_id)),
-    );
-
-    let active = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::Low, false))
-        .await
-        .expect("scanner admission while the first task is active");
-    assert_eq!(active.result, HealAdmissionResult::Merged);
-    assert_eq!(active.task_id, first_id, "active ownership keeps precedence");
-
-    let completed_task = manager
-        .active_heals
-        .lock()
-        .await
-        .remove(&first_id)
-        .expect("first task should leave active ownership");
-    let mut retained = CompletedHealStatus::snapshot(&completed_task, HealTaskStatus::Completed).await;
-    Arc::make_mut(retained.outcome.as_mut().expect("retained task outcome")).finish(None);
-    manager
-        .completed_heals
-        .lock()
-        .await
-        .insert(first_id.clone(), Arc::new(retained));
-    let queued = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::Low, false))
-        .await
-        .expect("scanner admission after the first task leaves active state");
-    assert_eq!(queued.result, HealAdmissionResult::Merged);
-    assert_eq!(queued.task_id, second_id, "the receipt must name the remaining queued task");
-}
-
-#[tokio::test]
-async fn scanner_receipt_tracks_queued_owner_after_normal_dequeue() {
-    scanner_receipt_tracks_queued_owner(false).await;
-}
-
-#[tokio::test]
-async fn scanner_receipt_tracks_queued_owner_after_fair_dequeue() {
-    scanner_receipt_tracks_queued_owner(true).await;
-}
-
-#[tokio::test]
-async fn scanner_receipt_keeps_queued_owner_when_a_different_forced_task_runs() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let first = object_request(HealRequestSource::Scanner, HealPriority::Low, false);
-    let first_id = first.id.clone();
-    let second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-    let second_id = second.id.clone();
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        assert_eq!(queue.push(first), QueuePushOutcome::Accepted);
-        assert_eq!(queue.push(second), QueuePushOutcome::Accepted);
-        let (selected, _) = queue.pop_runnable_with_skips(|_| true, |_| None);
-        assert_eq!(selected.expect("higher-priority forced task should run").id, second_id);
-    }
-    let receipt = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::High, false))
-        .await
-        .expect("scanner admission after a non-representative task leaves the queue");
-    assert_eq!(receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(receipt.task_id, first_id);
-}
-
-#[tokio::test]
-async fn scanner_receipt_reselects_after_the_queued_owner_is_cancelled() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-    let second_id = second.id.clone();
-    let third = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    let third_id = third.id.clone();
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        for request in [first, second, third] {
-            assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-        }
-        let (selected, _) = queue.pop_runnable_with_skips(|_| true, |_| None);
-        assert_eq!(selected.expect("first task should run").id, first_id);
-    }
-    let first_receipt = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::High, false))
-        .await
-        .expect("first scanner admission");
-    assert_eq!(first_receipt.result, HealAdmissionResult::Merged);
-    assert!(first_receipt.task_id == second_id || first_receipt.task_id == third_id);
-    let remaining = if first_receipt.task_id == second_id {
-        third_id
-    } else {
-        second_id
-    };
-    manager.heal_queue.lock().await.remove_request_id(&first_receipt.task_id);
-
-    let next_receipt = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::High, false))
-        .await
-        .expect("scanner admission after the queued owner is removed");
-    assert_eq!(next_receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(next_receipt.task_id, remaining);
-}
-
-#[tokio::test]
-async fn scanner_receipt_reselects_after_multiple_forced_dequeues() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-    let second_id = second.id.clone();
-    let third = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    let third_id = third.id.clone();
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        for request in [first, second, third] {
-            assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-        }
-        let (selected, _) = queue.pop_runnable_with_skips(|_| true, |_| None);
-        assert_eq!(selected.expect("first task should run").id, first_id);
-    }
-    let first_receipt = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::High, false))
-        .await
-        .expect("first scanner admission");
-    assert_eq!(first_receipt.result, HealAdmissionResult::Merged);
-    assert!(first_receipt.task_id == second_id || first_receipt.task_id == third_id);
-    let remaining = if first_receipt.task_id == second_id {
-        third_id
-    } else {
-        second_id
-    };
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        let (selected, _) = queue.pop_runnable_with_skips(|request| request.id == first_receipt.task_id, |_| None);
-        assert_eq!(selected.expect("selected queued owner should run").id, first_receipt.task_id);
-    }
-    let next_receipt = manager
-        .submit_heal_request_with_receipt(object_request(HealRequestSource::Scanner, HealPriority::High, false))
-        .await
-        .expect("scanner admission after another queued owner starts");
-    assert_eq!(next_receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(next_receipt.task_id, remaining);
-}
-
-#[test]
-fn scanner_owner_index_survives_a_single_remaining_holder_and_new_forced_work() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-    let second_id = second.id.clone();
-    let third = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    let third_id = third.id.clone();
-
-    assert_eq!(queue.push(first), QueuePushOutcome::Accepted);
-    assert!(matches!(&queue.dedup_keys[&key].scanner_queued_owners, ScannerQueuedOwners::Single(_)));
-    assert_eq!(queue.push(second), QueuePushOutcome::Accepted);
-    assert_eq!(queue.pop().expect("first queued task").id, first_id);
-    assert_eq!(queue.queued_request_id_for_dedup_key(&key), Some(first_id.as_str()));
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(second_id.as_str()));
-
-    assert_eq!(queue.push(third), QueuePushOutcome::Accepted);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(second_id.as_str()));
-    assert_eq!(queue.pop().expect("second queued task").id, second_id);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(third_id.as_str()));
-}
-
-#[test]
-fn scanner_owner_index_distinguishes_holders_with_the_same_request_id() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let shared_id = first.id.clone();
-    let mut second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-    second.id.clone_from(&shared_id);
-    let third = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    let third_id = third.id.clone();
-    for request in [first, second, third] {
-        assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-    }
-
-    assert_eq!(queue.pop().expect("first holder").id, shared_id);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(shared_id.as_str()));
-    assert_eq!(queue.remove_request_id(&shared_id).expect("second holder").id, shared_id);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(third_id.as_str()));
-    assert_eq!(queue.queued_request_id_for_dedup_key(&key), Some(third_id.as_str()));
-}
-
-#[test]
-fn scanner_owner_index_tracks_bulk_removal_and_key_release() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Admin, HealPriority::Normal, true);
-    let second_id = second.id.clone();
-    let third = object_request(HealRequestSource::Internal, HealPriority::Low, true);
-    let third_id = third.id.clone();
-    for request in [first, second, third] {
-        assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-    }
-
-    let removed = queue.remove_matching(|request| request.id == first_id || request.id == second_id);
-    assert_eq!(removed.len(), 2);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(third_id.as_str()));
-    assert_eq!(queue.queued_request_id_for_dedup_key(&key), Some(third_id.as_str()));
-    assert_eq!(queue.remove_matching(|request| request.id == third_id).len(), 1);
-    assert!(!queue.dedup_keys.contains_key(&key));
-
-    let next = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let next_id = next.id.clone();
-    assert_eq!(queue.push(next), QueuePushOutcome::Accepted);
-    assert!(matches!(&queue.dedup_keys[&key].scanner_queued_owners, ScannerQueuedOwners::Single(_)));
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(next_id.as_str()));
-}
-
-#[test]
-fn scanner_owner_index_tracks_displacement_without_changing_legacy_reselection() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::Low, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Admin, HealPriority::Normal, true);
-    let second_id = second.id.clone();
-    assert_eq!(queue.push(first), QueuePushOutcome::Accepted);
-    assert_eq!(queue.push(second), QueuePushOutcome::Accepted);
-
-    let incoming = bucket_request("displacement", HealPriority::Urgent, HealRequestSource::Internal);
-    let displaced = queue
-        .push_displacing_lower_priority(incoming)
-        .expect("lower priority task should be displaced");
-    assert_eq!(displaced.id, first_id);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(second_id.as_str()));
-    assert_eq!(queue.queued_request_id_for_dedup_key(&key), Some(second_id.as_str()));
-}
-
-#[tokio::test]
-async fn scanner_receipt_rejects_a_missing_or_empty_holder_index() {
-    for missing in [true, false] {
-        let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-        let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-        let key = PriorityHealQueue::make_dedup_key(&first);
-        let second = object_request(HealRequestSource::Admin, HealPriority::High, true);
-        {
-            let mut queue = manager.heal_queue.lock().await;
-            assert_eq!(queue.push(first), QueuePushOutcome::Accepted);
-            assert_eq!(queue.push(second), QueuePushOutcome::Accepted);
-            let entry = queue.dedup_keys.get_mut(&key).expect("dedup key should exist");
-            entry.scanner_queued_owners = if missing {
-                ScannerQueuedOwners::Single(0)
-            } else {
-                ScannerQueuedOwners::Indexed(std::collections::BTreeMap::new())
-            };
-        }
-
-        for leave_one in [false, true] {
-            if leave_one {
-                manager.heal_queue.lock().await.pop().expect("one holder should leave");
-            }
-            let request = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-            let request_id = request.id.clone();
-            let receipt = manager
-                .submit_heal_request_with_receipt(request)
-                .await
-                .expect("admission should fail closed");
-            assert_eq!(receipt.result, HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning));
-            assert_eq!(receipt.task_id, request_id);
-        }
-    }
-}
-
-#[test]
-fn scanner_and_legacy_owners_keep_their_distinct_reselection_orders() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Admin, HealPriority::Low, true);
-    let second_id = second.id.clone();
-    let third = object_request(HealRequestSource::Internal, HealPriority::Normal, true);
-    let third_id = third.id.clone();
-    for request in [first, second, third] {
-        assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-    }
-
-    assert_eq!(queue.remove_request_id(&first_id).expect("first holder should be removed").id, first_id);
-    assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(second_id.as_str()));
-    assert_eq!(queue.queued_request_id_for_dedup_key(&key), Some(third_id.as_str()));
-}
-
-#[test]
-fn scanner_owner_index_tracks_a_series_of_forced_holders() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let mut ids = vec![first.id.clone()];
-    assert_eq!(queue.push(first), QueuePushOutcome::Accepted);
-    for _ in 0..128 {
-        let request = object_request(HealRequestSource::Internal, HealPriority::High, true);
-        ids.push(request.id.clone());
-        assert_eq!(queue.push(request), QueuePushOutcome::Accepted);
-    }
-    for id in &ids {
-        assert_eq!(queue.scanner_queued_request_id_for_dedup_key(&key), Some(id.as_str()));
-        assert_eq!(&queue.pop().expect("indexed holder should remain queued").id, id);
-    }
-    assert!(!queue.dedup_keys.contains_key(&key));
-}
-
-#[tokio::test]
-async fn observed_scanner_and_legacy_receipts_pair_their_own_reselected_holders() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Internal, HealPriority::Low, true);
-    let second_id = second.id.clone();
-    let third = object_request(HealRequestSource::Internal, HealPriority::Normal, true);
-    let third_id = third.id.clone();
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        for (request, observation_id) in [(first, 11), (second, 12), (third, 13)] {
-            assert_eq!(
-                queue.push_with_replacement_scope(request, false, Some(observation_id)),
-                QueuePushOutcome::Accepted
-            );
-        }
-        assert_eq!(queue.remove_request_id(&first_id).expect("remove first holder").id, first_id);
-        assert_eq!(queue.queued_scanner_owner_for_dedup_key(&key), Some((second_id.as_str(), Some(12))));
-        assert_eq!(queue.queued_legacy_owner_for_dedup_key(&key), Some((third_id.as_str(), Some(13))));
-    }
-
-    for (source, expected_id, expected_observation) in [
-        (HealRequestSource::Scanner, &second_id, 12),
-        (HealRequestSource::Internal, &third_id, 13),
-    ] {
-        let receipt = manager
-            .submit_heal_request_with_observation_and_alias(object_request(source, HealPriority::High, false), false)
-            .await
-            .expect("merge into queued owner");
-        assert_eq!(receipt.result, HealAdmissionResult::Merged);
-        assert_eq!(&receipt.task_id, expected_id);
-        assert_eq!(receipt.observation_id, Some(expected_observation));
-    }
-}
-
-#[tokio::test]
-async fn observed_same_id_holders_use_the_selected_sequence_not_the_id() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let mut second = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    second.id.clone_from(&id);
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        assert_eq!(queue.push_with_replacement_scope(first, false, Some(21)), QueuePushOutcome::Accepted);
-        assert_eq!(queue.push_with_replacement_scope(second, false, Some(22)), QueuePushOutcome::Accepted);
-        let (selected, _) = queue.pop_runnable_with_skips(|_| true, |_| None);
-        let selected = selected.expect("first holder should run");
-        assert_eq!(selected.id, id);
-        assert_eq!(selected.observation_id, Some(21));
-        assert_eq!(queue.queued_scanner_owner_for_dedup_key(&key), Some((id.as_str(), Some(22))));
-        assert_eq!(queue.queued_legacy_owner_for_dedup_key(&key), Some((id.as_str(), None)));
-    }
-
-    for (source, expected_observation) in [(HealRequestSource::Scanner, Some(22)), (HealRequestSource::Internal, None)] {
-        let receipt = manager
-            .submit_heal_request_with_observation_and_alias(object_request(source, HealPriority::High, false), false)
-            .await
-            .expect("merge into the selected holder");
-        assert_eq!(receipt.result, HealAdmissionResult::Merged);
-        assert_eq!(receipt.task_id, id);
-        assert_eq!(receipt.observation_id, expected_observation);
-    }
-}
-
-#[test]
-fn displaced_holder_reselects_the_remaining_id_and_observation_together() {
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::Low, false);
-    let first_id = first.id.clone();
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Internal, HealPriority::Normal, true);
-    let second_id = second.id.clone();
-    assert_eq!(queue.push_with_replacement_scope(first, false, Some(31)), QueuePushOutcome::Accepted);
-    assert_eq!(queue.push_with_replacement_scope(second, false, Some(32)), QueuePushOutcome::Accepted);
-    let incoming = bucket_request("observation-displacement", HealPriority::Urgent, HealRequestSource::Internal);
-    let displaced = queue
-        .push_displacing_lower_priority_where(incoming, |_| true, Some(33))
-        .expect("lower-priority holder should be displaced");
-    assert_eq!(displaced.id, first_id);
-    assert_eq!(queue.queued_scanner_owner_for_dedup_key(&key), Some((second_id.as_str(), Some(32))));
-    assert_eq!(queue.queued_legacy_owner_for_dedup_key(&key), Some((second_id.as_str(), Some(32))));
-}
-
-#[tokio::test]
-async fn missing_selected_token_or_inconsistent_index_stays_unknown() {
-    let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    let second_id = second.id.clone();
-    {
-        let mut queue = manager.heal_queue.lock().await;
-        assert_eq!(queue.push_with_replacement_scope(first, false, Some(41)), QueuePushOutcome::Accepted);
-        assert_eq!(queue.push_with_replacement_scope(second, false, None), QueuePushOutcome::Accepted);
-        queue.pop().expect("first holder should leave");
-        assert_eq!(queue.queued_scanner_owner_for_dedup_key(&key), Some((second_id.as_str(), None)));
-    }
-    let receipt = manager
-        .submit_heal_request_with_observation_and_alias(
-            object_request(HealRequestSource::Scanner, HealPriority::High, false),
-            false,
-        )
-        .await
-        .expect("merge into owner without an observation token");
-    assert_eq!(receipt.result, HealAdmissionResult::Merged);
-    assert_eq!(receipt.task_id, second_id);
-    assert_eq!(receipt.observation_id, None);
-
-    let mut queue = PriorityHealQueue::new();
-    let first = object_request(HealRequestSource::Scanner, HealPriority::High, false);
-    let key = PriorityHealQueue::make_dedup_key(&first);
-    let second = object_request(HealRequestSource::Internal, HealPriority::High, true);
-    assert_eq!(queue.push_with_replacement_scope(first, false, Some(51)), QueuePushOutcome::Accepted);
-    assert_eq!(queue.push_with_replacement_scope(second, false, Some(52)), QueuePushOutcome::Accepted);
-    let ScannerQueuedOwners::Indexed(owners) = &mut queue.dedup_keys.get_mut(&key).expect("key").scanner_queued_owners else {
-        panic!("forced holder should create an index");
-    };
-    let first_sequence = *owners.first_key_value().expect("first indexed holder").0;
-    assert!(owners.remove(&first_sequence).is_some());
-    assert_eq!(queue.queued_scanner_owner_for_dedup_key(&key), None);
-    queue.pop().expect("first holder should leave");
-    assert_eq!(queue.queued_scanner_owner_for_dedup_key(&key), None);
-    assert_eq!(queue.queued_legacy_owner_for_dedup_key(&key).and_then(|(_, token)| token), None);
-}

crates/heal/src/heal/manager/tests/root_recovery.rs

@@ -117,6 +117,8 @@ fn admin_request(heal_type: HealType) -> HealRequest {
 
 fn completed_admin_status(heal_type: &HealType, completed_at: SystemTime) -> CompletedHealStatus {
     CompletedHealStatus {
+        task_id: "admin-fixture".to_string(),
+        source: HealRequestSource::Admin,
         outcome: None,
         heal_type: heal_type.clone(),
         options: HealOptions::default(),
@@ -448,7 +450,6 @@ async fn root_recovery_retry_cancellation_preserves_previous_attempt_outcome() {
     manager.retrying_heals.lock().await.insert(
         request.id.clone(),
         RetryingHeal {
-            observation_id: None,
             request: request.clone(),
             error: "retryable".to_string(),
             cancel_token: CancellationToken::new(),
@@ -1240,6 +1241,8 @@ async fn root_recovery_completed_non_root_admin_is_queryable_after_restart() {
         .expect("durable bucket responsibility");
     let completed = CompletedHealStatus {
         outcome: None,
+        source: request.source,
+        task_id: request.id.clone(),
         heal_type: request.heal_type.clone(),
         options: request.options.clone(),
         status: HealTaskStatus::Completed,
@@ -2154,30 +2157,6 @@ async fn root_recovery_cluster_rejects_inventory_without_checkpoint_headroom() {
     );
 }
 
-#[tokio::test]
-async fn root_recovery_replayed_queue_owner_gets_a_fresh_in_process_observation_identity() {
-    let (_temp, disk) = recovery_disk().await;
-    let manager = recovery_manager(vec![disk.clone()]);
-    let request = root_request();
-    let first = manager
-        .submit_heal_request_with_observation_and_alias(request.clone(), false)
-        .await
-        .expect("persisted admission");
-    assert_eq!(first.result, HealAdmissionResult::Accepted);
-    assert!(first.observation_id.is_some());
-    drop(manager);
-
-    let replayed = recovery_manager(vec![disk]);
-    replayed.replay_root_heals().await.expect("replay persisted root");
-    let recovered_id = replayed
-        .heal_queue
-        .lock()
-        .await
-        .queued_observation_for_request_id(&request.id);
-    assert!(recovered_id.is_some());
-    assert_ne!(recovered_id, first.observation_id);
-}
-
 #[tokio::test]
 async fn root_recovery_queued_owner_is_not_priority_displaced() {
     let (_temp, disk) = recovery_disk().await;

crates/heal/src/heal/mrf_queue.rs

@@ -42,6 +42,7 @@ use rustfs_heal_contracts::heal_channel::{HealAdmissionDropReason, HealAdmission
 use serde::{Deserialize, Serialize};
 use sha2::Digest;
 use std::collections::{HashMap, HashSet, VecDeque};
+use std::io::{self, Write};
 use std::sync::Arc;
 use std::time::Duration;
 use tokio::sync::mpsc;
@@ -409,11 +410,24 @@ impl MrfQueue {
 
 /// Append one encoded record to `out`.
 pub(crate) fn encode_intent(intent: &MrfIntent, out: &mut Vec<u8>) -> bool {
+    let start = out.len();
+    match encode_intent_to_writer(intent, out) {
+        Ok(true) => true,
+        _ => {
+            out.truncate(start);
+            false
+        }
+    }
+}
+
+// A caller supplying a fallible writer must discard its output after an
+// error; only the Vec adapter above can roll back its own partial record.
+fn encode_intent_to_writer(intent: &MrfIntent, out: &mut impl Write) -> io::Result<bool> {
     let Ok(bucket_len) = u32::try_from(intent.bucket.len()) else {
-        return false;
+        return Ok(false);
     };
     let Ok(object_len) = u32::try_from(intent.object.len()) else {
-        return false;
+        return Ok(false);
     };
     let scope = (!matches!(intent.kind, rustfs_common::mrf_channel::MrfKind::MetadataCorruption))
         .then_some(intent.scope)
@@ -422,62 +436,68 @@ pub(crate) fn encode_intent(intent: &MrfIntent, out: &mut Vec<u8>) -> bool {
     let purge = match intent.kind {
         MrfKind::DeleteMarkerPurge => match intent.delete_marker_purge.as_ref().filter(|payload| payload.is_valid()) {
             Some(payload) if version_id.is_some() && scope.is_some() => Some(payload),
-            _ => return false,
+            _ => return Ok(false),
         },
         _ if intent.delete_marker_purge.is_none() => None,
-        _ => return false,
+        _ => return Ok(false),
     };
-    let start = out.len();
-    out.push(MRF_JOURNAL_FORMAT);
-    out.push(if purge.is_some() {
-        MRF_JOURNAL_VERSION_DELETE_MARKER_PURGE
-    } else if scope.is_some() {
-        MRF_JOURNAL_VERSION_SCOPED
-    } else {
-        MRF_JOURNAL_VERSION
-    });
-    out.push(match intent.kind {
-        rustfs_common::mrf_channel::MrfKind::DecodeFailure => 1,
-        rustfs_common::mrf_channel::MrfKind::MetadataCorruption => 2,
-        rustfs_common::mrf_channel::MrfKind::PartialWrite => 3,
-        rustfs_common::mrf_channel::MrfKind::DeleteMarkerPurge => 4,
-    });
-    out.push(intent.attempts);
-    out.extend_from_slice(&intent.enqueued_at_ms.to_le_bytes());
-    match version_id {
-        Some(bytes) => {
-            out.push(1);
-            out.extend_from_slice(&bytes);
-        }
-        None => out.push(0),
-    }
-    if let Some(scope) = scope {
-        out.extend_from_slice(&scope.pool_index.to_le_bytes());
-        out.extend_from_slice(&scope.set_index.to_le_bytes());
-    }
-    out.extend_from_slice(&bucket_len.to_le_bytes());
-    out.extend_from_slice(&object_len.to_le_bytes());
-    out.extend_from_slice(intent.bucket.as_bytes());
-    out.extend_from_slice(intent.object.as_bytes());
-    if let Some(purge) = purge {
-        let Ok(marker_len) = u32::try_from(purge.marker.len()) else {
-            out.truncate(start);
-            return false;
+    let marker_len = match purge {
+        Some(payload) => match u32::try_from(payload.marker.len()) {
+            Ok(length) => Some(length),
+            Err(_) => return Ok(false),
+        },
+        None => None,
+    };
+    let mut crc = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
+    {
+        let mut write_record = |bytes: &[u8]| -> io::Result<()> {
+            out.write_all(bytes)?;
+            crc.update(bytes);
+            Ok(())
         };
-        out.extend_from_slice(purge.bucket_incarnation_id.as_bytes());
-        out.extend_from_slice(purge.marker_incarnation_id.as_bytes());
-        out.extend_from_slice(&purge.marker_identity);
-        out.extend_from_slice(&marker_len.to_le_bytes());
-        out.extend_from_slice(&purge.marker);
+        write_record(&[MRF_JOURNAL_FORMAT])?;
+        write_record(&[if purge.is_some() {
+            MRF_JOURNAL_VERSION_DELETE_MARKER_PURGE
+        } else if scope.is_some() {
+            MRF_JOURNAL_VERSION_SCOPED
+        } else {
+            MRF_JOURNAL_VERSION
+        }])?;
+        write_record(&[match intent.kind {
+            rustfs_common::mrf_channel::MrfKind::DecodeFailure => 1,
+            rustfs_common::mrf_channel::MrfKind::MetadataCorruption => 2,
+            rustfs_common::mrf_channel::MrfKind::PartialWrite => 3,
+            rustfs_common::mrf_channel::MrfKind::DeleteMarkerPurge => 4,
+        }])?;
+        write_record(&[intent.attempts])?;
+        write_record(&intent.enqueued_at_ms.to_le_bytes())?;
+        match version_id {
+            Some(bytes) => {
+                write_record(&[1])?;
+                write_record(&bytes)?;
+            }
+            None => write_record(&[0])?,
+        }
+        if let Some(scope) = scope {
+            write_record(&scope.pool_index.to_le_bytes())?;
+            write_record(&scope.set_index.to_le_bytes())?;
+        }
+        write_record(&bucket_len.to_le_bytes())?;
+        write_record(&object_len.to_le_bytes())?;
+        write_record(intent.bucket.as_bytes())?;
+        write_record(intent.object.as_bytes())?;
+        if let Some(purge) = purge {
+            let Some(marker_len) = marker_len else { return Ok(false) };
+            write_record(purge.bucket_incarnation_id.as_bytes())?;
+            write_record(purge.marker_incarnation_id.as_bytes())?;
+            write_record(&purge.marker_identity)?;
+            write_record(&marker_len.to_le_bytes())?;
+            write_record(&purge.marker)?;
+        }
     }
-    let mut hasher = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
-    hasher.update(&out[start..]);
-    let Ok(checksum) = u32::try_from(hasher.finalize()) else {
-        out.truncate(start);
-        return false;
-    };
-    out.extend_from_slice(&checksum.to_le_bytes());
-    true
+    let checksum = u32::try_from(crc.finalize()).map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid MRF CRC"))?;
+    out.write_all(&checksum.to_le_bytes())?;
+    Ok(true)
 }
 
 fn decode_one(data: &[u8]) -> Option<(MrfIntent, usize)> {
@@ -745,9 +765,24 @@ fn decode_mrf_lifecycle_checkpoint(data: &[u8], byte_limit: usize) -> Result<Mrf
     Ok(decoded)
 }
 
+struct IntentDigestWriter(sha2::Sha256);
+
+impl Write for IntentDigestWriter {
+    fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
+        self.0.update(bytes);
+        Ok(bytes.len())
+    }
+
+    fn flush(&mut self) -> io::Result<()> {
+        Ok(())
+    }
+}
+
 fn intent_digest(intent: &MrfIntent) -> Option<[u8; 32]> {
-    let mut encoded = Vec::new();
-    encode_intent(intent, &mut encoded).then(|| sha2::Sha256::digest(encoded).into())
+    let mut encoded = IntentDigestWriter(sha2::Sha256::new());
+    encode_intent_to_writer(intent, &mut encoded)
+        .ok()?
+        .then(|| encoded.0.finalize().into())
 }
 
 // ---------------------------------------------------------------------------
@@ -2409,12 +2444,11 @@ mod tests {
     use crate::heal::manager::HealConfig;
     use crate::heal::storage::{ECStoreHealStorage, HealStorageAPI};
     use crate::heal::{DiskError, RUSTFS_META_BUCKET};
-    use rustfs_common::mrf_channel::{MrfIntent, MrfKind, MrfVerifiedRepairDisposition, MrfVerifiedRepairEvent};
+    use rustfs_common::mrf_channel::{MrfIntent, MrfKind, MrfScope, MrfVerifiedRepairDisposition, MrfVerifiedRepairEvent};
     use serde_json::{Map, Value, json};
     use serial_test::serial;
     use std::env;
     use std::fs;
-    use std::io::Write as _;
     use std::path::{Path, PathBuf};
     use std::sync::Arc as StdArc;
     use std::time::{Duration as StdDuration, Instant};
@@ -2539,6 +2573,73 @@ mod tests {
         payload
     }
 
+    #[test]
+    fn streaming_intent_codec_preserves_existing_wire_bytes_and_digest() {
+        let mut plain = intent("b", "o", 0);
+        plain.version_id = None;
+        let mut scoped = plain.clone();
+        scoped.kind = MrfKind::PartialWrite;
+        scoped.version_id = Some([7; 16]);
+        scoped.scope = Some(MrfScope {
+            pool_index: 2,
+            set_index: 3,
+        });
+        let mut purge = scoped.clone();
+        purge.kind = MrfKind::DeleteMarkerPurge;
+        let incarnation = Uuid::from_u128(1);
+        purge.delete_marker_purge = MrfDeleteMarkerPurge::new(incarnation, incarnation, [9; 32], vec![1, 2]);
+        let cases = [
+            (plain, "010101000068e5cf8b010000000100000001000000626f883931f3"),
+            (
+                scoped,
+                "010203000068e5cf8b010000010707070707070707070707070707070702000000030000000100000001000000626f59661dd0",
+            ),
+            (
+                purge,
+                "010304000068e5cf8b010000010707070707070707070707070707070702000000030000000100000001000000626f00000000000000000000000000000001000000000000000000000000000000010909090909090909090909090909090909090909090909090909090909090909020000000102304b263e",
+            ),
+        ];
+        for (intent, expected_hex) in cases {
+            let encoded = encoded_payload(&intent);
+            let actual_hex: String = encoded.iter().map(|byte| format!("{byte:02x}")).collect();
+            assert_eq!(actual_hex, expected_hex);
+            let expected_digest: [u8; 32] = sha2::Sha256::digest(&encoded).into();
+            assert_eq!(intent_digest(&intent), Some(expected_digest));
+        }
+    }
+
+    #[test]
+    fn streaming_intent_codec_propagates_writer_failure_and_rejects_invalid_record() {
+        struct PartialFailure(Vec<u8>);
+        impl Write for PartialFailure {
+            fn write(&mut self, input: &[u8]) -> io::Result<usize> {
+                let remaining = 5usize.saturating_sub(self.0.len());
+                if remaining == 0 {
+                    return Err(io::Error::from(io::ErrorKind::WouldBlock));
+                }
+                let written = remaining.min(input.len());
+                self.0.extend_from_slice(&input[..written]);
+                Ok(written)
+            }
+            fn flush(&mut self) -> io::Result<()> {
+                Ok(())
+            }
+        }
+        let valid = intent("b", "o", 0);
+        let mut partial = PartialFailure(Vec::new());
+        assert_eq!(
+            encode_intent_to_writer(&valid, &mut partial).unwrap_err().kind(),
+            io::ErrorKind::WouldBlock
+        );
+        assert_eq!(partial.0.len(), 5, "the caller must discard a partially written fallible output");
+        let mut invalid = valid;
+        invalid.kind = MrfKind::DeleteMarkerPurge;
+        let mut bytes = b"prefix".to_vec();
+        assert!(!encode_intent(&invalid, &mut bytes));
+        assert_eq!(bytes, b"prefix");
+        assert_eq!(intent_digest(&invalid), None);
+    }
+
     #[test]
     fn lifecycle_checkpoint_is_bound_checksummed_and_strictly_decoded() {
         let owner = Uuid::new_v4();

crates/heal/src/heal/storage_api.rs

@@ -26,7 +26,11 @@ pub(crate) use rustfs_ecstore::api::disk::{
     RUSTFS_META_BUCKET as ECSTORE_RUSTFS_META_BUCKET, ReplacementExecutionLease as EcstoreReplacementExecutionLease,
 };
 pub(crate) use rustfs_ecstore::api::disk::{DiskOption as EcstoreDiskOption, new_disk as ecstore_new_disk};
-pub(crate) use rustfs_ecstore::api::error::{Error as EcstoreErrorType, StorageError as EcstoreStorageError};
+#[cfg(test)]
+pub(crate) use rustfs_ecstore::api::error::PoolMetadataError as EcstorePoolMetadataError;
+pub(crate) use rustfs_ecstore::api::error::{
+    Error as EcstoreErrorType, PoolMetadataFailure as EcstorePoolMetadataFailure, StorageError as EcstoreStorageError,
+};
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::layout::{
     EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints,
@@ -53,6 +57,9 @@ pub(crate) use rustfs_ecstore::api::storage::{
 use rustfs_storage_api as storage_contracts;
 
 pub(crate) mod owner {
+    #[cfg(test)]
+    pub(crate) use super::EcstorePoolMetadataError;
+    pub(crate) use super::EcstorePoolMetadataFailure;
     pub(crate) use super::storage_contracts::{ObjectIO, ObjectOperations};
 
     pub(crate) use super::{

crates/heal/src/heal/task.rs

@@ -37,7 +37,7 @@ use std::{
     collections::VecDeque,
     future::Future,
     sync::{
-        Arc,
+        Arc, OnceLock,
         atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering},
     },
     time::{Duration, Instant, SystemTime},
@@ -59,6 +59,19 @@ pub(crate) struct OutcomeFinishTestHook {
 pub(crate) static OUTCOME_FINISH_TEST_HOOK: std::sync::LazyLock<tokio::sync::Mutex<Option<Arc<OutcomeFinishTestHook>>>> =
     std::sync::LazyLock::new(|| tokio::sync::Mutex::new(None));
 
+#[cfg(test)]
+static STORAGE_RECEIPT_TEST_HOOK: std::sync::LazyLock<tokio::sync::Mutex<Option<Arc<OutcomeFinishTestHook>>>> =
+    std::sync::LazyLock::new(|| tokio::sync::Mutex::new(None));
+
+#[cfg(test)]
+async fn pause_storage_receipt(task_id: &str) {
+    let hook = STORAGE_RECEIPT_TEST_HOOK.lock().await.clone();
+    if let Some(hook) = hook.filter(|hook| hook.task_id == task_id) {
+        hook.reached.notify_one();
+        hook.release.notified().await;
+    }
+}
+
 #[cfg(test)]
 async fn pause_outcome_finish(task_id: &str) {
     let hook = OUTCOME_FINISH_TEST_HOOK.lock().await.clone();
@@ -413,8 +426,6 @@ pub struct HealResultWindow {
 pub struct HealTask {
     /// Task ID
     pub id: String,
-    /// Process-local diagnostic identity; never persisted or part of a request.
-    pub(crate) observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
     /// Heal type
     pub heal_type: HealType,
     pub bucket_incarnation_id: Option<Uuid>,
@@ -438,6 +449,9 @@ pub struct HealTask {
     replacement_start_retry_count: Arc<AtomicU32>,
     /// Set when the locked storage read classified the single target as absent.
     object_absent: Arc<AtomicBool>,
+    /// The first resolved incarnation belongs to this task even if storage
+    /// fails or the bucket is recreated before its terminal outcome is recorded.
+    single_object_incarnation: Arc<OnceLock<Option<Uuid>>>,
     /// Task status
     pub status: Arc<RwLock<HealTaskStatus>>,
     /// Progress tracking
@@ -496,7 +510,6 @@ impl HealTask {
     pub fn from_request(request: HealRequest, storage: Arc<dyn HealStorageAPI>) -> Self {
         Self {
             id: request.id,
-            observation_id: None,
             heal_type: request.heal_type,
             bucket_incarnation_id: request.bucket_incarnation_id,
             expected_mrf_bucket_incarnation_id: request.expected_mrf_bucket_incarnation_id,
@@ -511,6 +524,7 @@ impl HealTask {
             replacement_running: Arc::new(AtomicBool::new(false)),
             replacement_start_retry_count: Arc::new(AtomicU32::new(0)),
             object_absent: Arc::new(AtomicBool::new(false)),
+            single_object_incarnation: Arc::new(OnceLock::new()),
             status: Arc::new(RwLock::new(HealTaskStatus::Pending)),
             progress: Arc::new(RwLock::new(HealProgress::new())),
             result_items: Arc::new(RwLock::new(VecDeque::with_capacity(MAX_RETAINED_HEAL_RESULT_ITEMS))),
@@ -532,14 +546,6 @@ impl HealTask {
         }
     }
 
-    pub(crate) fn with_observation_id(
-        mut self,
-        observation_id: Option<rustfs_heal_contracts::completion::HealObservationId>,
-    ) -> Self {
-        self.observation_id = observation_id;
-        self
-    }
-
     pub fn retry_request(&self) -> HealRequest {
         HealRequest {
             id: self.id.clone(),
@@ -680,7 +686,9 @@ impl HealTask {
             } => (bucket, object, Some(version_id.as_str())),
             _ => return None,
         };
-        Some(self.outcome_identity(bucket, object, version, self.options.pool_index, self.options.set_index))
+        let mut identity = self.outcome_identity(bucket, object, version, self.options.pool_index, self.options.set_index);
+        identity.bucket_incarnation_id = self.single_object_incarnation.get().copied().flatten();
+        Some(identity)
     }
 
     pub(super) async fn record_verified_storage_receipt(

crates/heal/src/heal/task/heal_bucket.rs

@@ -306,9 +306,9 @@ impl HealTask {
                     let mut progress = self.progress.write().await;
                     progress.update_stage(3, 3);
                 }
-                Err(Error::TaskExecutionFailed {
-                    message: format!("Failed to heal bucket {bucket}: {e}"),
-                })
+                // Keep storage/quorum identity for the cluster retry classifier.
+                // The structured log above already carries the bucket context.
+                Err(e)
             }
         }
     }
@@ -520,9 +520,7 @@ impl HealTask {
                     error = %err,
                     "Heal cluster pool metadata failed"
                 );
-                Err(Error::TaskExecutionFailed {
-                    message: format!("Failed to heal cluster pool metadata: {err}"),
-                })
+                Err(err)
             }
             Err(Error::TaskCancelled) => Err(Error::TaskCancelled),
             Err(Error::TaskTimeout) => Err(Error::TaskTimeout),
@@ -539,9 +537,7 @@ impl HealTask {
                     error = %err,
                     "Heal cluster pool metadata failed"
                 );
-                Err(Error::TaskExecutionFailed {
-                    message: format!("Failed to heal cluster pool metadata: {err}"),
-                })
+                Err(err)
             }
         }
     }
@@ -796,21 +792,27 @@ impl HealTask {
                                 match unavailable_recreate_error(&storage_result.item, &heal_opts) {
                                     Some(error) => Some(error),
                                     None => {
-                                        telemetry_unknown |= !increment_counter(&mut healed);
-                                        telemetry_unknown |= !add_bytes(
-                                            &mut bytes,
-                                            u64::try_from(storage_result.item.object_size).unwrap_or(u64::MAX),
-                                        );
+                                        #[cfg(test)]
+                                        super::pause_storage_receipt(&self.id).await;
                                         recorded_authoritative_outcome =
                                             self.record_verified_storage_receipt(&identity, storage_result.receipt).await;
-                                        if !recorded_authoritative_outcome
-                                            && !heal_opts.dry_run
-                                            && !storage_result.item.detail.is_empty()
-                                        {
-                                            detail = Some(storage_result.item.detail.clone());
+                                        if !recorded_authoritative_outcome && self.cancel_token.is_cancelled() {
+                                            Some(Error::TaskCancelled)
+                                        } else {
+                                            telemetry_unknown |= !increment_counter(&mut healed);
+                                            telemetry_unknown |= !add_bytes(
+                                                &mut bytes,
+                                                u64::try_from(storage_result.item.object_size).unwrap_or(u64::MAX),
+                                            );
+                                            if !recorded_authoritative_outcome
+                                                && !heal_opts.dry_run
+                                                && !storage_result.item.detail.is_empty()
+                                            {
+                                                detail = Some(storage_result.item.detail.clone());
+                                            }
+                                            self.record_result_item(storage_result.item).await;
+                                            None
                                         }
-                                        self.record_result_item(storage_result.item).await;
-                                        None
                                     }
                                 }
                             }

crates/heal/src/heal/task/heal_object.rs

@@ -220,6 +220,11 @@ impl HealTask {
         let mut expected_identity =
             self.outcome_identity(bucket, object, version_id, self.options.pool_index, self.options.set_index);
         expected_identity.bucket_incarnation_id = self.outcome_bucket_incarnation_id(bucket, self.options.dry_run).await?;
+        if matches!(self.heal_type, HealType::Object { .. }) {
+            expected_identity.bucket_incarnation_id = *self
+                .single_object_incarnation
+                .get_or_init(|| expected_identity.bucket_incarnation_id);
+        }
 
         let heal_fut = self.storage.heal_object_with_receipt(bucket, object, version_id, &heal_opts);
         let heal_result = if self.source == HealRequestSource::ReadRepair {

crates/heal/src/heal/task/tests.rs

@@ -16,6 +16,7 @@ use super::super::{DiskOption, DiskStore, Endpoint, new_disk};
 use super::*;
 use crate::heal::EcstoreError;
 use crate::heal::storage::HealStorageObjectResult;
+use crate::heal::storage_api::owner::{EcstorePoolMetadataError, EcstorePoolMetadataFailure};
 
 mod absent_verdict;
 mod concurrent_delete;
@@ -305,6 +306,53 @@ mod canonical_outcome {
         assert_eq!(storage.healed_objects.lock().expect("healed objects").len(), 2);
     }
 
+    #[tokio::test]
+    async fn bucket_cancellation_after_storage_success_keeps_typed_cancelled_responsibility() {
+        let incarnation = Uuid::new_v4();
+        let storage = Arc::new(MockStorage {
+            heal_object_receipts: Mutex::new(HashMap::from([(
+                "object-a".to_string(),
+                VecDeque::from([object_receipt("object-a", None, HealObjectDisposition::Repaired, incarnation)]),
+            )])),
+            bucket_incarnation_id: Mutex::new(Some(incarnation)),
+            ..Default::default()
+        });
+        let task = Arc::new(bucket_task(storage.clone()));
+        let hook = Arc::new(OutcomeFinishTestHook {
+            task_id: task.id.clone(),
+            reached: tokio::sync::Notify::new(),
+            release: tokio::sync::Notify::new(),
+        });
+        *STORAGE_RECEIPT_TEST_HOOK.lock().await = Some(hook.clone());
+        let runner = {
+            let task = task.clone();
+            tokio::spawn(async move { task.execute().await })
+        };
+        tokio::time::timeout(Duration::from_secs(10), hook.reached.notified())
+            .await
+            .expect("storage success must reach the receipt boundary");
+        assert_eq!(storage.healed_objects.lock().expect("storage mutations").as_slice(), ["object-a"]);
+        task.cancel().await.expect("cancel before receipt recording");
+        hook.release.notify_one();
+        let result = tokio::time::timeout(Duration::from_secs(10), runner)
+            .await
+            .expect("cancelled worker must stop")
+            .expect("worker must not panic");
+        STORAGE_RECEIPT_TEST_HOOK.lock().await.take();
+        assert!(matches!(result, Err(Error::TaskCancelled)));
+        let outcome = task.get_outcome().await;
+        assert_eq!(outcome.execution, HealExecutionOutcome::Aborted(HealAbortReason::Cancelled));
+        assert_eq!(outcome.coverage, HealTraversalCoverage::Partial);
+        assert_eq!(outcome.counters.processed, 1);
+        assert_eq!(outcome.counters.healed, 0);
+        assert_eq!(outcome.counters.unchanged, 0);
+        assert_eq!(outcome.counters.unknown, 0);
+        assert_eq!(outcome.counters.skipped, 1);
+        assert_eq!(outcome.objects.len(), 1);
+        assert_eq!(outcome.objects[0].identity.object, "object-a");
+        assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::Cancelled);
+    }
+
     #[tokio::test]
     async fn bucket_heal_rejects_stale_receipts_without_double_recording() {
         let expected_incarnation = Uuid::new_v4();
@@ -784,7 +832,10 @@ mod canonical_outcome {
     async fn untraversable_bucket_does_not_claim_complete_cluster_coverage() {
         let storage = Arc::new(MockStorage {
             listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
-            bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["metadata unavailable"]))])),
+            bucket_heal_errors: Mutex::new(HashMap::from([(
+                "bucket-a".to_string(),
+                VecDeque::from([Error::other("metadata unavailable")]),
+            )])),
             ..Default::default()
         });
         let task = HealTask::from_request(
@@ -809,6 +860,85 @@ mod canonical_outcome {
         );
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn cluster_bucket_write_quorum_retries_preserve_budget_and_truthful_coverage() {
+        for (failures, succeeds) in [(1, true), (4, false)] {
+            let storage = Arc::new(MockStorage {
+                listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
+                typed_bucket_heal_errors: Mutex::new(HashMap::from([(
+                    "bucket-a".to_string(),
+                    (0..failures)
+                        .map(|_| Error::Storage(EcstoreError::ErasureWriteQuorum))
+                        .collect(),
+                )])),
+                ..Default::default()
+            });
+            let task = HealTask::from_request(
+                HealRequest::new(
+                    HealType::Cluster,
+                    HealOptions {
+                        recursive: true,
+                        timeout: None,
+                        ..Default::default()
+                    },
+                    HealPriority::Normal,
+                ),
+                storage.clone(),
+            );
+            let result = task.execute().await;
+            assert_eq!(
+                result.is_ok(),
+                succeeds,
+                "original cluster task must honor its bounded bucket retry budget"
+            );
+            let outcome = task.get_outcome().await;
+            if succeeds {
+                assert_eq!(outcome.execution, HealExecutionOutcome::Completed);
+                assert_eq!(outcome.coverage, HealTraversalCoverage::Complete);
+            } else {
+                assert_eq!(outcome.execution, HealExecutionOutcome::Aborted(HealAbortReason::Untraversable));
+                assert_eq!(outcome.coverage, HealTraversalCoverage::Partial);
+                let failure = task.take_batch_failure().await.expect("exhausted bucket retains its failure");
+                assert_eq!((failure.failed, failure.retryable, failure.permanent), (1, 1, 0));
+            }
+            let calls = storage.bucket_heal_calls.lock().expect("original bucket calls");
+            assert_eq!(
+                calls.iter().filter(|bucket| bucket.as_str() == "bucket-a").count(),
+                if succeeds { 2 } else { 4 }
+            );
+            assert_eq!(calls.last().map(String::as_str), Some("bucket-b"), "later buckets must still be visited");
+        }
+    }
+
+    #[tokio::test]
+    async fn bucket_prepass_preserves_typed_quorum_and_terminal_access_denial() {
+        for error in [
+            Error::Storage(EcstoreError::ErasureWriteQuorum),
+            Error::Disk(DiskError::FileAccessDenied),
+        ] {
+            let retryable = error.is_recoverable_heal();
+            let storage = Arc::new(MockStorage {
+                typed_bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from([error]))])),
+                ..Default::default()
+            });
+            let task = bucket_task(storage.clone());
+            let observed = task
+                .heal_bucket("bucket-a")
+                .await
+                .expect_err("bucket prepass must retain the storage error");
+            assert!(matches!(
+                observed,
+                Error::Storage(EcstoreError::ErasureWriteQuorum) | Error::Disk(DiskError::FileAccessDenied)
+            ));
+            assert_eq!(observed.is_recoverable_heal(), retryable);
+            assert_eq!(storage.bucket_heal_calls.lock().expect("bucket calls").as_slice(), ["bucket-a"]);
+            assert!(
+                storage.heal_object_calls.lock().expect("object calls").is_empty(),
+                "failed prepass must not heal objects"
+            );
+        }
+    }
+
     #[tokio::test(start_paused = true)]
     async fn cancellation_and_deadline_leave_partial_coverage() {
         for cancel in [false, true] {
@@ -1566,7 +1696,8 @@ struct MockStorage {
     include_object_dir_candidate: Mutex<bool>,
     listed_buckets: Mutex<Option<Vec<String>>>,
     bucket_discovery_incomplete: bool,
-    bucket_heal_errors: Mutex<HashMap<String, VecDeque<&'static str>>>,
+    bucket_heal_errors: Mutex<HashMap<String, VecDeque<Error>>>,
+    typed_bucket_heal_errors: Mutex<HashMap<String, VecDeque<Error>>>,
     bucket_heal_calls: Mutex<Vec<String>>,
     delete_recovery_results: Mutex<HashMap<String, Result<Option<HealResultItem>>>>,
     delete_recovery_calls: Mutex<Vec<String>>,
@@ -1815,6 +1946,127 @@ async fn failed_object_heal_rejects_matching_positive_storage_receipt() {
                 | HealObjectDisposition::AuthoritativelyAbsent
         )
     }));
+    assert_eq!(outcome.objects.len(), 1);
+    assert_eq!(outcome.objects[0].identity.bucket_incarnation_id, Some(incarnation));
+}
+
+#[tokio::test]
+async fn object_failure_retains_incarnation_for_inner_and_outer_storage_errors() {
+    for failure in [
+        MockHealObjectOutcome::PermissionDenied,
+        MockHealObjectOutcome::RetryableReadQuorum,
+        MockHealObjectOutcome::RetryableLockTimeout,
+        MockHealObjectOutcome::ErrOther("backend failed"),
+    ] {
+        let incarnation = Uuid::new_v4();
+        let storage = Arc::new(MockStorage {
+            bucket_incarnation_id: Mutex::new(Some(incarnation)),
+            heal_object_outcome: Mutex::new(Some(failure)),
+            ..Default::default()
+        });
+        let mut request = HealRequest::object("bucket-a".to_owned(), "object-a".to_owned(), Some("version-a".to_owned()));
+        request.options.pool_index = Some(2);
+        request.options.set_index = Some(3);
+        let task = HealTask::from_request(request, storage);
+        task.execute()
+            .await
+            .expect_err("the failed storage operation must remain failed");
+        let outcome = task.get_outcome().await;
+        assert_eq!(outcome.objects.len(), 1);
+        assert_eq!(
+            (outcome.counters.processed, outcome.counters.failed, outcome.counters.attempt_failures),
+            (1, 1, 1)
+        );
+        assert_eq!((outcome.counters.healed, outcome.counters.unchanged, outcome.counters.unknown), (0, 0, 0));
+        let object = &outcome.objects[0];
+        assert_eq!(object.identity.bucket_incarnation_id, Some(incarnation));
+        assert_eq!(object.identity.bucket, "bucket-a");
+        assert_eq!(object.identity.object, "object-a");
+        assert_eq!(object.identity.version_id.as_deref(), Some("version-a"));
+        assert_eq!((object.identity.pool_index, object.identity.set_index), (Some(2), Some(3)));
+        assert!(matches!(object.disposition, HealObjectDisposition::Failed(_)));
+    }
+}
+
+#[tokio::test(start_paused = true)]
+async fn object_abort_keeps_pre_storage_incarnation_after_bucket_recreation() {
+    for cancel in [false, true] {
+        let original = Uuid::new_v4();
+        let storage = Arc::new(MockStorage {
+            bucket_incarnation_id: Mutex::new(Some(original)),
+            block_heal_object: Mutex::new(true),
+            ..Default::default()
+        });
+        let mut request = HealRequest::object("bucket-a".to_owned(), "object-a".to_owned(), None);
+        request.options.timeout = Some(Duration::from_secs(10));
+        let task = HealTask::from_request(request, storage.clone());
+        let execution = task.execute();
+        tokio::pin!(execution);
+        assert!(futures::poll!(&mut execution).is_pending());
+        assert_eq!(storage.heal_object_calls.lock().expect("storage boundary").as_slice(), ["object-a"]);
+        *storage.bucket_incarnation_id.lock().expect("recreated bucket") = Some(Uuid::new_v4());
+        if cancel {
+            task.cancel().await.expect("cancel blocked storage");
+        } else {
+            tokio::time::advance(Duration::from_secs(11)).await;
+        }
+        let result = execution.await;
+        assert!(matches!(
+            (&result, cancel),
+            (Err(Error::TaskCancelled), true) | (Err(Error::TaskTimeout), false)
+        ));
+        let outcome = task.get_outcome().await;
+        assert_eq!(outcome.objects.len(), 1);
+        assert_eq!(outcome.objects[0].identity.bucket_incarnation_id, Some(original));
+        assert_eq!((outcome.counters.processed, outcome.counters.skipped), (1, 1));
+        assert_eq!((outcome.counters.failed, outcome.counters.healed, outcome.counters.unknown), (0, 0, 0));
+        assert_eq!(
+            outcome.objects[0].disposition,
+            if cancel {
+                HealObjectDisposition::Cancelled
+            } else {
+                HealObjectDisposition::Deferred {
+                    reason: HealDeferredReason::Deadline,
+                    retry_not_before: None,
+                }
+            }
+        );
+    }
+}
+
+#[tokio::test]
+async fn object_unverified_outcome_keeps_original_incarnation_without_claiming_repair() {
+    let original = Uuid::new_v4();
+    let successor = Uuid::new_v4();
+    let storage = Arc::new(MockStorage {
+        bucket_incarnation_id: Mutex::new(Some(original)),
+        bucket_incarnation_after_object_heal: Mutex::new(Some(successor)),
+        ..Default::default()
+    });
+    let task = HealTask::from_request(HealRequest::object("bucket-a".to_owned(), "object-a".to_owned(), None), storage.clone());
+    task.execute().await.expect("legacy storage keeps its unverified result");
+    let outcome = task.get_outcome().await;
+    assert_eq!(outcome.objects[0].identity.bucket_incarnation_id, Some(original));
+    assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::Unknown);
+    assert_eq!((outcome.counters.healed, outcome.counters.unknown), (0, 1));
+    assert_eq!(*storage.bucket_incarnation_id.lock().expect("successor"), Some(successor));
+}
+
+#[tokio::test]
+async fn object_dry_run_does_not_adopt_bucket_incarnation() {
+    let storage = Arc::new(MockStorage {
+        bucket_incarnation_id: Mutex::new(Some(Uuid::new_v4())),
+        ..Default::default()
+    });
+    let mut request = HealRequest::object("bucket-a".to_owned(), "object-a".to_owned(), None);
+    request.options.dry_run = true;
+    let task = HealTask::from_request(request, storage);
+    task.execute().await.expect("dry-run observation");
+    let outcome = task.get_outcome().await;
+    assert_eq!(outcome.objects.len(), 1);
+    assert_eq!(outcome.objects[0].identity.bucket_incarnation_id, None);
+    assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::DryRunObserved);
+    assert_eq!(outcome.counters.healed, 0);
 }
 
 #[tokio::test]
@@ -1990,6 +2242,7 @@ fn replacement_identity(
 
 #[derive(Clone)]
 enum MockHealObjectOutcome {
+    PoolMetadata { kind: EcstorePoolMetadataFailure, outer: bool },
     LegacyAmbiguous { outer: bool },
     MissingObject { outer: bool },
     MissingVersion,
@@ -2009,6 +2262,16 @@ enum MockHealObjectOutcome {
     PermanentOther(&'static str),
 }
 
+fn pool_metadata_error(kind: EcstorePoolMetadataFailure) -> Error {
+    Error::Storage(EcstoreError::other(EcstorePoolMetadataError {
+        kind,
+        operation: "bucket heal".to_owned(),
+        phase: "native_heal",
+        since: std::time::UNIX_EPOCH.into(),
+        source: None,
+    }))
+}
+
 fn unavailable_drive_heal_result(state: DriveState) -> (HealResultItem, Option<Error>) {
     (
         HealResultItem {
@@ -2180,6 +2443,14 @@ impl HealStorageAPI for MockStorage {
             .and_then(VecDeque::pop_front)
         {
             return match outcome {
+                MockHealObjectOutcome::PoolMetadata { kind, outer } => {
+                    let error = pool_metadata_error(kind);
+                    if outer {
+                        Err(error)
+                    } else {
+                        Ok((HealResultItem::default(), Some(error)))
+                    }
+                }
                 MockHealObjectOutcome::LegacyAmbiguous { outer } => {
                     let error = Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous);
                     if outer {
@@ -2256,6 +2527,14 @@ impl HealStorageAPI for MockStorage {
         }
         if let Some(outcome) = self.heal_object_outcome.lock().unwrap().take() {
             return match outcome {
+                MockHealObjectOutcome::PoolMetadata { kind, outer } => {
+                    let error = pool_metadata_error(kind);
+                    if outer {
+                        Err(error)
+                    } else {
+                        Ok((HealResultItem::default(), Some(error)))
+                    }
+                }
                 MockHealObjectOutcome::LegacyAmbiguous { outer } => {
                     let error = Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous);
                     if outer {
@@ -2387,14 +2666,23 @@ impl HealStorageAPI for MockStorage {
     async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
         self.bucket_heal_calls.lock().unwrap().push(bucket.to_string());
         self.bucket_heal_opts.lock().unwrap().push(*opts);
-        if let Some(message) = self
+        if let Some(error) = self
+            .typed_bucket_heal_errors
+            .lock()
+            .unwrap()
+            .get_mut(bucket)
+            .and_then(VecDeque::pop_front)
+        {
+            return Err(error);
+        }
+        if let Some(error) = self
             .bucket_heal_errors
             .lock()
             .unwrap()
             .get_mut(bucket)
             .and_then(VecDeque::pop_front)
         {
-            return Err(Error::other(message));
+            return Err(error);
         }
         Ok(HealResultItem::default())
     }
@@ -3429,13 +3717,70 @@ async fn cluster_recreate_fails_when_pool_metadata_heal_fails() {
         .await
         .expect_err("cluster recreate heal must not hide pool metadata failure");
 
-    assert!(matches!(err, Error::TaskExecutionFailed { .. }));
+    assert!(matches!(err, Error::Other(ref message) if message == "pool metadata missing"));
     assert_eq!(
         storage.heal_object_calls.lock().expect("object calls").as_slice(),
         ["object-a".to_string(), "object-b".to_string(), POOL_META_NAME.to_string()]
     );
 }
 
+#[tokio::test]
+async fn cluster_pool_metadata_failures_preserve_recovery_classification_in_both_response_shapes() {
+    for outer in [false, true] {
+        for (kind, retryable) in [
+            (EcstorePoolMetadataFailure::ReadUnavailable, true),
+            (EcstorePoolMetadataFailure::TransactionUnknown, true),
+            (EcstorePoolMetadataFailure::RecoveryRequired, false),
+            (EcstorePoolMetadataFailure::FenceLost, false),
+        ] {
+            let storage = Arc::new(MockStorage {
+                listed_buckets: Mutex::new(Some(Vec::new())),
+                ..Default::default()
+            });
+            storage.heal_object_outcomes.lock().expect("object outcomes").insert(
+                POOL_META_NAME.to_string(),
+                VecDeque::from([MockHealObjectOutcome::PoolMetadata { kind, outer }]),
+            );
+            let mut request = HealRequest::new(
+                HealType::Cluster,
+                HealOptions {
+                    recreate_missing: true,
+                    timeout: None,
+                    ..Default::default()
+                },
+                HealPriority::Normal,
+            );
+            request.source = HealRequestSource::Admin;
+            let task = HealTask::from_request(request, storage.clone());
+            let error = task
+                .execute()
+                .await
+                .expect_err("pool metadata failure must reach the scheduler");
+            assert!(
+                matches!(&error, Error::Storage(error) if error.pool_metadata_failure().is_some_and(|failure| failure.kind == kind)),
+                "the original typed pool metadata state must survive either response shape: {error}"
+            );
+            assert_eq!(error.is_recoverable_heal(), retryable, "{kind:?}, outer={outer}");
+            assert_eq!(storage.heal_object_calls.lock().expect("object calls").as_slice(), [POOL_META_NAME]);
+            if retryable {
+                let retry = task
+                    .retry_request_with_remaining_timeout()
+                    .await
+                    .expect("existing retry request");
+                assert_eq!(retry.id, task.id, "recovery retains the accepted task token");
+                HealTask::from_request(retry, storage.clone())
+                    .execute()
+                    .await
+                    .expect("the same task should complete after pool metadata recovery");
+                assert_eq!(
+                    storage.heal_object_calls.lock().expect("object calls").as_slice(),
+                    [POOL_META_NAME, POOL_META_NAME]
+                );
+            }
+        }
+    }
+}
+
 #[tokio::test]
 async fn cluster_dry_run_does_not_heal_pool_metadata() {
     let storage = Arc::new(MockStorage::default());
@@ -3799,7 +4144,10 @@ async fn test_recursive_bucket_heal_reports_typed_exhausted_and_permanent_failur
 async fn test_cluster_heal_continues_after_bucket_failure() {
     let storage = Arc::new(MockStorage {
         listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
-        bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["metadata unavailable"]))])),
+        bucket_heal_errors: Mutex::new(HashMap::from([(
+            "bucket-a".to_string(),
+            VecDeque::from([Error::other("metadata unavailable")]),
+        )])),
         ..Default::default()
     });
     let request = HealRequest::new(
@@ -3831,7 +4179,10 @@ async fn test_cluster_heal_continues_after_bucket_failure() {
 async fn test_cluster_heal_retries_only_recoverable_bucket() {
     let storage = Arc::new(MockStorage {
         listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
-        bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["lock acquisition timeout"]))])),
+        bucket_heal_errors: Mutex::new(HashMap::from([(
+            "bucket-a".to_string(),
+            VecDeque::from([Error::other("lock acquisition timeout")]),
+        )])),
         ..Default::default()
     });
     let request = HealRequest::new(
@@ -3855,6 +4206,47 @@ async fn test_cluster_heal_retries_only_recoverable_bucket() {
     );
 }
 
+#[tokio::test(start_paused = true)]
+async fn cluster_bucket_heal_retries_only_recoverable_pool_metadata_failures() {
+    for (kind, retryable) in [
+        (EcstorePoolMetadataFailure::ReadUnavailable, true),
+        (EcstorePoolMetadataFailure::TransactionUnknown, true),
+        (EcstorePoolMetadataFailure::RecoveryRequired, false),
+        (EcstorePoolMetadataFailure::FenceLost, false),
+    ] {
+        let storage = Arc::new(MockStorage {
+            bucket_incarnation_id: Mutex::new(Some(Uuid::new_v4())),
+            bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_owned(), VecDeque::from([pool_metadata_error(kind)]))])),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::Cluster,
+            HealOptions {
+                recursive: true,
+                timeout: None,
+                ..Default::default()
+            },
+            HealPriority::Normal,
+        );
+        request.source = HealRequestSource::Admin;
+        let task = HealTask::from_request(request, storage.clone());
+        let result = task.execute().await;
+        if retryable {
+            result.expect("a recoverable pool metadata state must not terminate the root heal");
+            assert_eq!(task.get_status().await, HealTaskStatus::Completed);
+            assert_eq!(
+                storage.bucket_heal_calls.lock().expect("bucket calls").as_slice(),
+                ["bucket-a", "bucket-a"]
+            );
+        } else {
+            result.expect_err("operator recovery and lost fences must remain terminal");
+            let failure = task.take_batch_failure().await.expect("permanent cluster failure");
+            assert_eq!((failure.retryable, failure.permanent), (0, 1));
+            assert_eq!(storage.bucket_heal_calls.lock().expect("bucket calls").as_slice(), ["bucket-a"]);
+        }
+    }
+}
+
 #[tokio::test]
 async fn test_recursive_bucket_heal_does_not_remove_bucket_metadata() {
     let storage = Arc::new(MockStorage::default());
@@ -4617,7 +5009,7 @@ async fn erasure_set_bucket_prepass_failure_stops_before_object_heal() {
     let storage = Arc::new(MockStorage {
         bucket_heal_errors: Mutex::new(HashMap::from([(
             "bucket-a".to_string(),
-            VecDeque::from(["injected bucket prepass failure"]),
+            VecDeque::from([Error::other("injected bucket prepass failure")]),
         )])),
         resume_disk: Mutex::new(Some(disk)),
         ..Default::default()

crates/heal/src/lib.rs

@@ -280,12 +280,12 @@ pub async fn init_heal_manager_with_workload_provider(
             .is_some_and(|hook| hook.fail_channel_init.load(Ordering::SeqCst));
         #[cfg(not(test))]
         let force_channel_failure = false;
-        if force_channel_failure {
-            stop_initializing_manager(&heal_manager).await?;
-            return Err(Error::Config("forced heal channel initialization failure".to_string()));
-        }
-        let channel_receiver = rustfs_heal_contracts::heal_channel::init_heal_channels_with_observations();
-        let (receiver, receipt_receiver, observed_receiver) = match channel_receiver {
+        let channel_receiver = if force_channel_failure {
+            Err("forced heal channel initialization failure")
+        } else {
+            rustfs_heal_contracts::heal_channel::init_heal_channels()
+        };
+        let (receiver, receipt_receiver) = match channel_receiver {
             Ok(receivers) => receivers,
             Err(err) => {
                 stop_initializing_manager(&heal_manager).await?;
@@ -302,10 +302,7 @@ pub async fn init_heal_manager_with_workload_provider(
 
         tokio::spawn(async move {
             let mut processor = channel_processor.lock().await;
-            if let Err(e) = processor
-                .start_with_observed_receipts(receiver, receipt_receiver, observed_receiver)
-                .await
-            {
+            if let Err(e) = processor.start_with_receipts(receiver, receipt_receiver).await {
                 error!(
                     target: "rustfs::heal",
                     event = EVENT_HEAL_RUNTIME_STATE,

crates/heal/tests/shard_identity_receipt_test.rs

@@ -550,3 +550,159 @@ fn admin_normal_protected_repairs_report_repaired_for_data_and_parity() {
         .join()
         .expect("heal fixture result");
 }
+
+#[test]
+#[serial]
+fn admin_mixed_batch_retains_repairs_after_object_retry_exhaustion() {
+    std::thread::Builder::new()
+        .stack_size(32 * 1024 * 1024)
+        .spawn(|| {
+            tokio::runtime::Builder::new_current_thread()
+                .enable_all()
+                .build()
+                .expect("mixed batch runtime")
+                .block_on(async {
+                    use rustfs_heal::heal::{
+                        manager::{HealConfig, HealManager},
+                        outcome::{HealExecutionOutcome, HealFailureClass, HealTraversalCoverage},
+                        task::{HealOptions, HealPriority, HealRequest, HealTaskStatus, HealType},
+                    };
+                    use rustfs_heal_contracts::heal_channel::HealRequestSource;
+                    use std::sync::Arc;
+
+                    let root = tempfile::tempdir().expect("mixed batch fixture");
+                    let env = TestECStoreEnv::builder().base_dir(root.path()).build().await;
+                    let bucket = "admin-mixed-batch";
+                    env.make_bucket(bucket, false).await;
+                    let set = env.ecstore.pools[0].get_disks(0);
+                    let disks = set.disks.read().await.iter().flatten().cloned().collect::<Vec<_>>();
+                    let mut originals = Vec::new();
+                    for (object, missing, byte) in [("healthy", 0, 0x11), ("repaired", 1, 0x52), ("failed", 3, 0xa3)] {
+                        env.ecstore
+                            .put_object(
+                                bucket,
+                                object,
+                                &mut PutObjReader::from_vec(vec![byte; 5 * 1024 * 1024 + 123]),
+                                &ObjectOptions {
+                                    shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
+                                    write_completion: WriteCompletion::TailDrained,
+                                    ..Default::default()
+                                },
+                            )
+                            .await
+                            .expect("commit complete fixture");
+                        for (slot, disk) in disks.iter().enumerate() {
+                            let meta = disk
+                                .read_version("", bucket, object, "", &ReadOptions::default())
+                                .await
+                                .expect("fixture metadata");
+                            assert_eq!(meta.erasure.data_blocks, 2, "failed fixture must retain R-1 shards");
+                            let path = env.disk_paths[slot]
+                                .join(bucket)
+                                .join(object)
+                                .join(meta.data_dir.expect("external shard").to_string())
+                                .join("part.1");
+                            let bytes = tokio::fs::read(&path).await.expect("original part");
+                            if slot < missing {
+                                tokio::fs::remove_file(&path).await.expect("inject missing part");
+                            }
+                            originals.push((object, path, bytes, slot < missing));
+                        }
+                    }
+                    let storage = Arc::new(ECStoreHealStorage::new(env.ecstore.clone()));
+                    let manager = HealManager::new(
+                        storage.clone(),
+                        Some(HealConfig {
+                            enable_auto_heal: false,
+                            ..Default::default()
+                        }),
+                    );
+                    manager.start().await.expect("start mixed batch manager");
+                    let mut request = HealRequest::new(
+                        HealType::Bucket {
+                            bucket: bucket.to_owned(),
+                        },
+                        HealOptions {
+                            recursive: true,
+                            scan_mode: HealScanMode::Deep,
+                            pool_index: Some(0),
+                            set_index: Some(0),
+                            timeout: Some(Duration::from_secs(150)),
+                            ..Default::default()
+                        },
+                        HealPriority::High,
+                    );
+                    request.source = HealRequestSource::Admin;
+                    let token = request.id.clone();
+                    manager.submit_heal_request(request).await.expect("admit original mixed task");
+                    let report = tokio::time::timeout(Duration::from_secs(150), async {
+                        loop {
+                            let report = manager.get_task_report(&token).await.expect("same-token report");
+                            if matches!(
+                                report.status,
+                                HealTaskStatus::Completed | HealTaskStatus::Failed { .. } | HealTaskStatus::Timeout
+                            ) {
+                                break report;
+                            }
+                            tokio::time::sleep(Duration::from_millis(20)).await;
+                        }
+                    })
+                    .await
+                    .expect("mixed task terminal deadline");
+                    manager.stop().await.expect("stop mixed task manager");
+                    for (object, path, original, missing) in originals {
+                        if object == "failed" && missing {
+                            assert!(!path.exists(), "below-R failure must not invent a shard");
+                        } else {
+                            assert_eq!(tokio::fs::read(&path).await.expect("preserved or repaired shard"), original);
+                        }
+                    }
+                    assert!(matches!(report.status, HealTaskStatus::Failed { .. }), "{report:?}");
+                    let outcome = report.outcome.as_ref().expect("canonical mixed outcome");
+                    assert_eq!(outcome.execution, HealExecutionOutcome::CompletedWithErrors);
+                    assert_eq!(outcome.coverage, HealTraversalCoverage::Complete);
+                    assert_eq!(
+                        (
+                            outcome.counters.processed,
+                            outcome.counters.healed,
+                            outcome.counters.unchanged,
+                            outcome.counters.failed,
+                            outcome.counters.skipped,
+                            outcome.counters.unknown,
+                            outcome.counters.attempt_failures
+                        ),
+                        (3, 1, 1, 1, 0, 0, 4),
+                        "{outcome:?}"
+                    );
+                    for (object, disposition) in [
+                        ("healthy", HealObjectDisposition::VerifiedHealthy),
+                        ("repaired", HealObjectDisposition::Repaired),
+                        ("failed", HealObjectDisposition::Failed(HealFailureClass::RetryExhausted)),
+                    ] {
+                        let receipt = outcome
+                            .objects
+                            .iter()
+                            .find(|item| item.identity.object == object)
+                            .expect("exact object receipt");
+                        assert_eq!(receipt.disposition, disposition);
+                    }
+                    drop(manager);
+                    let restarted = HealManager::new(
+                        storage,
+                        Some(HealConfig {
+                            enable_auto_heal: false,
+                            ..Default::default()
+                        }),
+                    );
+                    restarted.start().await.expect("restart with retained failed batch");
+                    let restored = restarted.get_task_report(&token).await.expect("original token after restart");
+                    assert_eq!(restored.status, report.status);
+                    assert_eq!(restored.outcome, report.outcome);
+                    assert_eq!(restarted.get_queue_length().await, 0, "terminal batch must not replay");
+                    restarted.stop().await.expect("stop restarted mixed batch manager");
+                });
+        })
+        .expect("mixed batch thread")
+        .join()
+        .expect("mixed batch result");
+}

crates/iam/src/delegation.rs

@@ -0,0 +1,313 @@
+// Copyright 2026 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+//! Durable restrictions owned by an authenticated user. A record is not an IAM
+//! grant or credential; issuing credentials still requires current IAM checks.
+use crate::{error::is_err_config_not_found, store::Store};
+use serde::{Deserialize, Serialize};
+
+const PREFIX: &str = "config/iam/plugin-delegations/v1";
+
+#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(deny_unknown_fields)]
+pub struct Restrictions {
+    #[serde(default, skip_serializing_if = "Option::is_none", deserialize_with = "present_value")]
+    pub version: Option<u32>,
+    #[serde(default, skip_serializing_if = "Option::is_none", deserialize_with = "present_value")]
+    pub output_bucket: Option<String>,
+    pub deployment: String,
+    pub bucket: String,
+    pub input_prefix: String,
+    pub output_prefix: String,
+    pub expires_at_unix_seconds: u64,
+}
+
+// An absent v1 field is compatible; an explicitly null v2 field is invalid.
+fn present_value<'de, T: Deserialize<'de>, D: serde::Deserializer<'de>>(deserializer: D) -> Result<Option<T>, D::Error> {
+    T::deserialize(deserializer).map(Some)
+}
+
+impl Restrictions {
+    pub fn output_bucket(&self) -> Result<&str, DelegationError> {
+        match (self.version, self.output_bucket.as_deref()) {
+            (None, None) => Ok(&self.bucket),
+            (Some(2), Some(output)) => Ok(output),
+            _ => Err(DelegationError::Invalid),
+        }
+    }
+
+    pub fn validate(&self, now: u64) -> Result<(), DelegationError> {
+        let output_bucket = self.output_bucket()?;
+        let names: [&str; 5] = [
+            &self.deployment,
+            &self.bucket,
+            &self.input_prefix,
+            &self.output_prefix,
+            output_bucket,
+        ];
+        if names
+            .iter()
+            .any(|s| s.is_empty() || s.len() > 1024 || s.contains("${") || s.chars().any(char::is_control))
+            || self.bucket.contains('/')
+            || self.bucket.contains('*')
+            || self.bucket.contains('?')
+            || output_bucket.contains('/')
+            || output_bucket.contains('*')
+            || output_bucket.contains('?')
+            || self.input_prefix.contains('*')
+            || self.input_prefix.contains('?')
+            || self.output_prefix.contains('*')
+            || self.output_prefix.contains('?')
+            || self.expires_at_unix_seconds <= now
+            || self.expires_at_unix_seconds.saturating_sub(now) > 86400
+        {
+            return Err(DelegationError::Invalid);
+        }
+        Ok(())
+    }
+}
+
+#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(deny_unknown_fields)]
+pub struct Delegation {
+    pub version: u32,
+    pub id: String,
+    pub revision: String,
+    pub owner_access_key: String,
+    pub created_at_unix_seconds: u64,
+    pub restrictions: Restrictions,
+}
+
+#[derive(Serialize, Deserialize, PartialEq, Eq)]
+#[serde(deny_unknown_fields)]
+struct Tombstone {
+    version: u32,
+    id: String,
+    revision: String,
+    owner_access_key: String,
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum DelegationError {
+    #[error("invalid delegation restriction")]
+    Invalid,
+    #[error("delegation not found")]
+    NotFound,
+    #[error("delegation owner rejected")]
+    OwnerRejected,
+    #[error("delegation unavailable")]
+    Unavailable,
+    #[error("delegation revoked")]
+    Revoked,
+    #[error("delegation expired")]
+    Expired,
+}
+
+fn record_path(id: &str) -> Result<String, DelegationError> {
+    if id.len() != 32 || !id.bytes().all(|c| c.is_ascii_hexdigit()) {
+        return Err(DelegationError::Invalid);
+    }
+    Ok(format!("{PREFIX}/{id}/record.json"))
+}
+
+async fn load<T: Store>(store: &T, id: &str) -> Result<Delegation, DelegationError> {
+    let record: Delegation = store.load_iam_config_strict(record_path(id)?).await.map_err(|error| {
+        if is_err_config_not_found(&error) {
+            DelegationError::NotFound
+        } else {
+            DelegationError::Unavailable
+        }
+    })?;
+    if !matches!(record.version, 1 | 2)
+        || record.version != record.restrictions.version.unwrap_or(1)
+        || record.id != id
+        || record.revision != id
+        || record.owner_access_key.is_empty()
+    {
+        return Err(DelegationError::Unavailable);
+    }
+    record
+        .restrictions
+        .validate(record.created_at_unix_seconds)
+        .map_err(|_| DelegationError::Unavailable)?;
+    Ok(record)
+}
+
+pub(crate) async fn create<T: Store>(
+    store: &T,
+    owner: &str,
+    restrictions: Restrictions,
+    now: u64,
+) -> Result<Delegation, DelegationError> {
+    restrictions.validate(now)?;
+    if owner.is_empty() {
+        return Err(DelegationError::OwnerRejected);
+    }
+    // IDs are generated exclusively here; no update operation accepts an ID.
+    let id = format!("{:032x}", rand::random::<u128>());
+    let path = record_path(&id)?;
+    match store.load_iam_config_strict::<Delegation>(&path).await {
+        Err(error) if is_err_config_not_found(&error) => {}
+        _ => return Err(DelegationError::Unavailable),
+    }
+    let record = Delegation {
+        version: restrictions.version.unwrap_or(1),
+        revision: id.clone(),
+        id,
+        owner_access_key: owner.to_owned(),
+        created_at_unix_seconds: now,
+        restrictions,
+    };
+    store
+        .save_iam_config(&record, path)
+        .await
+        .map_err(|_| DelegationError::Unavailable)?;
+    if load(store, &record.id).await? != record {
+        return Err(DelegationError::Unavailable);
+    }
+    Ok(record)
+}
+
+pub(crate) async fn revoke<T: Store>(store: &T, owner: &str, id: &str) -> Result<(), DelegationError> {
+    let record = load(store, id).await?;
+    if record.owner_access_key != owner {
+        return Err(DelegationError::OwnerRejected);
+    }
+    let tombstone = Tombstone {
+        version: 1,
+        id: record.id,
+        revision: record.revision,
+        owner_access_key: record.owner_access_key,
+    };
+    let path = format!("{PREFIX}/{id}/revoked.json");
+    // Every retry writes exactly the same revocation; active records never change.
+    store
+        .save_iam_config(&tombstone, &path)
+        .await
+        .map_err(|_| DelegationError::Unavailable)?;
+    let saved: Tombstone = store
+        .load_iam_config_strict(path)
+        .await
+        .map_err(|_| DelegationError::Unavailable)?;
+    if saved != tombstone {
+        return Err(DelegationError::Unavailable);
+    }
+    Ok(())
+}
+
+pub(crate) async fn read_active<T: Store>(store: &T, owner: &str, id: &str, now: u64) -> Result<Delegation, DelegationError> {
+    let record = load(store, id).await?;
+    if record.owner_access_key != owner {
+        return Err(DelegationError::OwnerRejected);
+    }
+    check_active(store, record, now).await
+}
+
+pub(crate) async fn read_owned<T: Store>(store: &T, owner: &str, id: &str) -> Result<Delegation, DelegationError> {
+    let record = load(store, id).await?;
+    if record.owner_access_key != owner {
+        return Err(DelegationError::OwnerRejected);
+    }
+    Ok(record)
+}
+
+pub(crate) async fn read_active_for_broker<T: Store>(store: &T, id: &str, now: u64) -> Result<Delegation, DelegationError> {
+    let record = load(store, id).await?;
+    check_active(store, record, now).await
+}
+async fn check_active<T: Store>(store: &T, record: Delegation, now: u64) -> Result<Delegation, DelegationError> {
+    let id = &record.id;
+    match store
+        .load_iam_config_strict::<Tombstone>(format!("{PREFIX}/{id}/revoked.json"))
+        .await
+    {
+        Ok(_) => return Err(DelegationError::Revoked),
+        Err(error) if is_err_config_not_found(&error) => {}
+        Err(_) => return Err(DelegationError::Unavailable),
+    }
+    if now >= record.restrictions.expires_at_unix_seconds {
+        return Err(DelegationError::Expired);
+    }
+    Ok(record)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    #[test]
+    fn delegation_v1_json_remains_compatible() {
+        let legacy = serde_json::json!({"deployment":"runner-a","bucket":"photos",
+            "input_prefix":"inputs/","output_prefix":"outputs/","expires_at_unix_seconds":200});
+        let restriction: Restrictions = serde_json::from_value(legacy.clone()).expect("legacy restriction");
+        restriction.validate(100).expect("legacy validation");
+        assert_eq!(restriction.output_bucket().expect("v1 output"), "photos");
+        assert_eq!(serde_json::to_value(restriction).expect("legacy serialization"), legacy);
+        let record = serde_json::json!({"version":1,"id":"a".repeat(32),"revision":"a".repeat(32),
+            "owner_access_key":"owner","created_at_unix_seconds":100,"restrictions":legacy});
+        let decoded: Delegation = serde_json::from_value(record.clone()).expect("legacy record");
+        assert_eq!(serde_json::to_value(decoded).expect("legacy record serialization"), record);
+    }
+
+    #[test]
+    fn delegation_v2_requires_explicit_output_bucket() {
+        let valid = serde_json::json!({"version":2,"deployment":"runner-a","bucket":"photos",
+            "input_prefix":"inputs/","output_bucket":"artifacts","output_prefix":"outputs/",
+            "expires_at_unix_seconds":200});
+        let restriction: Restrictions = serde_json::from_value(valid.clone()).expect("v2 restriction");
+        restriction.validate(100).expect("explicit output bucket");
+        assert_eq!(restriction.output_bucket().expect("v2 output"), "artifacts");
+        assert_eq!(serde_json::to_value(restriction).expect("v2 serialization"), valid);
+        for key in ["version", "output_bucket"] {
+            let mut bad = valid.clone();
+            bad.as_object_mut().expect("object").remove(key);
+            assert!(
+                serde_json::from_value::<Restrictions>(bad).map_or(true, |r| r.validate(100).is_err()),
+                "v2 required {key}"
+            );
+        }
+        for (key, value) in [
+            ("version", serde_json::json!(1)),
+            ("version", serde_json::json!(3)),
+            ("version", serde_json::Value::Null),
+            ("output_bucket", serde_json::Value::Null),
+            ("output_bucket", serde_json::json!("")),
+            ("output_bucket", serde_json::json!("other/*")),
+            ("output_bucket", serde_json::json!("${aws:username}")),
+            ("output_bucket", serde_json::json!("other?")),
+            ("output_bucket", serde_json::json!("other\n")),
+            ("unknown", serde_json::json!(true)),
+        ] {
+            let mut bad = valid.clone();
+            bad[key] = value;
+            assert!(
+                serde_json::from_value::<Restrictions>(bad).map_or(true, |r| r.validate(100).is_err()),
+                "invalid v2 {key}"
+            );
+        }
+    }
+
+    #[test]
+    fn restrictions_are_literal_bounded_and_expiring() {
+        let mut restriction = Restrictions {
+            version: None,
+            output_bucket: None,
+            deployment: "runner-a".into(),
+            bucket: "photos".into(),
+            input_prefix: "inputs/".into(),
+            output_prefix: "outputs/".into(),
+            expires_at_unix_seconds: 200,
+        };
+        assert!(restriction.validate(100).is_ok());
+        restriction.output_prefix = "outputs/*".into();
+        assert!(matches!(restriction.validate(100), Err(DelegationError::Invalid)));
+        restriction.output_prefix = "outputs/".into();
+        assert!(matches!(restriction.validate(200), Err(DelegationError::Invalid)));
+        restriction.expires_at_unix_seconds = 200;
+        restriction.input_prefix = "inputs/${aws:username}/".into();
+        assert!(
+            matches!(restriction.validate(100), Err(DelegationError::Invalid)),
+            "C07_LITERAL_SCOPE_BOUND: policy variables are not literal restrictions"
+        );
+        assert!(record_path("../../users").is_err());
+    }
+}

crates/iam/src/lib.rs

@@ -28,6 +28,7 @@ pub(crate) const EVENT_OIDC_DIAGNOSTICS: &str = "oidc_diagnostics";
 const EVENT_OIDC_STATE: &str = "oidc_state";
 
 pub mod cache;
+pub mod delegation;
 pub mod error;
 pub mod federation;
 pub mod keyring;

crates/iam/src/oidc/provider_tests.rs

@@ -234,6 +234,69 @@ fn workload_verifier_preserves_algorithm_secret_and_audience_policy() {
         }
     }
 }
+#[test]
+fn configured_verifier_rejects_malformed_hmac_signature_lengths() {
+    let secret = "synthetic-oidc-hmac-regression-secret-0123456789-abcdefghijklmnopqrstuvwxyz";
+    let mut config = build_mocked_oidc_provider_config("hmac", "https://issuer.example.com");
+    config.client_secret = Some(secret.to_owned());
+    let now = time::OffsetDateTime::now_utc().unix_timestamp();
+    let payload = serde_json::json!({
+        "iss": config.config_url, "sub": "synthetic-subject", "aud": config.client_id,
+        "iat": now, "exp": now + 300,
+    });
+
+    for (algorithm, name, signature_length) in [
+        (Algorithm::HS256, "HS256", 32),
+        (Algorithm::HS384, "HS384", 48),
+        (Algorithm::HS512, "HS512", 64),
+    ] {
+        let signed = jsonwebtoken::encode(&Header::new(algorithm), &payload, &EncodingKey::from_secret(secret.as_bytes()))
+            .expect("sign a valid HMAC control token");
+        let valid: CoreIdToken = signed.parse().expect("parse valid HMAC token");
+        let (signing_input, signature) = signed.rsplit_once('.').expect("JWT signature segment");
+        assert_eq!(
+            base64_simd::URL_SAFE_NO_PAD
+                .decode_to_vec(signature)
+                .expect("decode valid signature")
+                .len(),
+            signature_length
+        );
+
+        for workload in [false, true] {
+            let mut document = serde_json::json!({
+                "issuer": config.config_url, "jwks_uri": "https://issuer.example.com/jwks",
+                "response_types_supported": ["id_token"], "subject_types_supported": ["public"],
+                "id_token_signing_alg_values_supported": [name],
+            });
+            if !workload {
+                document["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize");
+            }
+            let metadata = DiscoveredProviderMetadata::parse(&serde_json::to_vec(&document).expect("encode metadata"), true)
+                .expect("parse configured provider metadata");
+            let verifier = metadata.verifier(&config);
+            assert!(
+                valid.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_ok(),
+                "valid {name} token must pass for workload={workload}"
+            );
+
+            for length in [1, 0, signature_length - 1, signature_length, signature_length + 1, 128] {
+                let signature = base64_simd::URL_SAFE_NO_PAD.encode_to_string(vec![0_u8; length]);
+                let malformed: CoreIdToken = format!("{signing_input}.{signature}")
+                    .parse()
+                    .expect("invalid signature length must still reach cryptographic verification");
+                assert!(
+                    malformed.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_err(),
+                    "{name} signature length {length} must return an error for workload={workload}"
+                );
+            }
+            assert!(
+                valid.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_ok(),
+                "invalid inputs must not prevent later valid {name} verification"
+            );
+        }
+    }
+}
+
 #[tokio::test]
 async fn workload_discovery_stops_after_forbidden_jwks() {
     let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0));

crates/iam/src/storage_api.rs

@@ -91,6 +91,7 @@ pub(crate) mod crate_boundary {
 
 pub(crate) mod object_store {
     pub(crate) use super::storage_contracts::{HTTPPreconditions, ListOperations, ObjectInfoOrErr, ObjectOperations};
+    pub(crate) use rustfs_ecstore::api::config::com::read_config_preserve_empty_with_metadata_opts;
 }
 
 pub(crate) mod runtime {

crates/iam/src/store.rs

@@ -26,6 +26,11 @@ pub trait Store: Clone + Send + Sync + 'static {
     fn has_watcher(&self) -> bool;
     async fn save_iam_config<Item: Serialize + Send>(&self, item: Item, path: impl AsRef<str> + Send) -> Result<()>;
     async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item>;
+    /// Only confirmed storage absence may return ConfigNotFound. Corrupt or
+    /// undecryptable objects must remain errors, especially for revocations.
+    async fn load_iam_config_strict<Item: DeserializeOwned>(&self, _path: impl AsRef<str> + Send) -> Result<Item> {
+        Err(crate::error::Error::IamSysNotInitialized)
+    }
     async fn delete_iam_config(&self, path: impl AsRef<str> + Send) -> Result<()>;
 
     async fn save_user_identity(&self, name: &str, user_type: UserType, item: UserIdentity, ttl: Option<usize>) -> Result<()>;

crates/iam/src/store/object.rs

@@ -42,6 +42,7 @@ use tracing::{debug, error, warn};
 
 use crate::storage_api::object_store::{
     HTTPPreconditions, ListOperations, ObjectInfoOrErr as StorageObjectInfoOrErr, ObjectOperations,
+    read_config_preserve_empty_with_metadata_opts,
 };
 
 pub static IAM_CONFIG_PREFIX: LazyLock<String> = LazyLock::new(|| format!("{IAM_CONFIG_ROOT_PREFIX}/iam"));
@@ -611,6 +612,11 @@ impl ObjectStore {
         Ok(serde_json::from_slice(&outcome.plain)?)
     }
 
+    fn decode_iam_config_strict<Item: DeserializeOwned>(data: &[u8]) -> Result<Item> {
+        let outcome = Self::decrypt_data_with_source(data)?;
+        Ok(serde_json::from_slice(&outcome.plain)?)
+    }
+
     /// Parameterized core of [`Store::load_user_identity`].
     async fn load_user_identity_with(&self, name: &str, user_type: UserType, mode: LoadMode) -> Result<UserIdentity> {
         let mut u: UserIdentity = self
@@ -824,6 +830,12 @@ impl Store for ObjectStore {
     async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item> {
         self.load_iam_config_with(path, LoadMode::Locked).await
     }
+    async fn load_iam_config_strict<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item> {
+        let (data, _) =
+            read_config_preserve_empty_with_metadata_opts(self.object_api.clone(), path.as_ref(), &LoadMode::Locked.read_opts())
+                .await?;
+        Self::decode_iam_config_strict(&data)
+    }
     /// Saves IAM configuration with a retry mechanism on failure.
     ///
     /// Attempts to save the IAM configuration up to 5 times if the storage layer is not ready,
@@ -1458,6 +1470,40 @@ mod tests {
         assert!(result.is_err(), "corrupt stream_io data should fail decrypt");
     }
 
+    #[test]
+    fn strict_revocation_load_keeps_real_crypto_failure_distinct_from_absence() {
+        let cred = test_cred();
+        let key = format!("{}:{}", cred.access_key, cred.secret_key);
+        let plain = br#"{"version":1,"id":"revoked"}"#;
+        let mut encrypted = rustfs_crypto::encrypt_stream_io(key.as_bytes(), plain).expect("real stored ciphertext");
+        // Exercise the exact decoder used after the actual object read.
+        assert!(ObjectStore::decode_iam_config_strict::<serde_json::Value>(&encrypted).is_ok());
+        encrypted[50] ^= 0xFF;
+        let error = ObjectStore::decode_iam_config_strict::<serde_json::Value>(&encrypted)
+            .expect_err("corrupt tombstone must not become missing");
+        assert!(
+            matches!(error, Error::CryptoError(_)),
+            "C07_REVOCATION_FAIL_CLOSED: preserve actual crypto error"
+        );
+        assert!(
+            !crate::error::is_err_config_not_found(&error),
+            "C07_REVOCATION_FAIL_CLOSED: corrupted revocation is not absence"
+        );
+        // The preserve-empty reader keeps an existing zero-byte object as
+        // Ok(empty); ecstore's read_config_preserve_empty_distinguishes_empty_object
+        // covers that storage boundary. Its payload must fail here, not become
+        // ConfigNotFound and reactivate a revoked delegation.
+        let empty =
+            ObjectStore::decode_iam_config_strict::<serde_json::Value>(&[]).expect_err("an existing empty tombstone is corrupt");
+        assert!(
+            !crate::error::is_err_config_not_found(&empty),
+            "C07_REVOCATION_FAIL_CLOSED: empty existing revocation is not absence"
+        );
+        let invalid_json =
+            ObjectStore::decode_iam_config_strict::<serde_json::Value>(br#"{"version": }"#).expect_err("invalid stored JSON");
+        assert!(!crate::error::is_err_config_not_found(&invalid_json));
+    }
+
     #[test]
     fn test_decrypt_data_short_data_fails() {
         let short = &[0x00u8; 40]; // less than 41-byte stream_io header, not valid JSON

crates/iam/src/sys.rs

@@ -75,6 +75,7 @@ pub const STATUS_ENABLED: &str = "enabled";
 pub const STATUS_DISABLED: &str = "disabled";
 
 pub const POLICYNAME: &str = "policy";
+pub const PLUGIN_ATTEMPT_CREDENTIAL_CLAIM: &str = "rustfs:plugin-attempt-v1";
 pub const SESSION_POLICY_NAME: &str = "sessionPolicy";
 pub const SESSION_POLICY_NAME_EXTRACTED: &str = "sessionPolicy-extracted";
 const VERIFIED_FEDERATED_POLICY_CLAIM: &str = "x-rustfs-internal-federated-policy";
@@ -168,6 +169,12 @@ enum PreparedServicePolicyMode {
 
 #[derive(Clone)]
 enum PreparedIamMode {
+    TerminalDeny,
+    PluginAttemptSts {
+        is_owner: bool,
+        combined_policy: Policy,
+        session_policy: PreparedSessionPolicy,
+    },
     Opa,
     Owner,
     Deny,
@@ -202,7 +209,7 @@ impl PreparedIamAuth {
     pub async fn needs_existing_object_tag_for_args(&self, args: &Args<'_>) -> bool {
         match &self.mode {
             PreparedIamMode::Opa => true,
-            PreparedIamMode::Owner | PreparedIamMode::Deny => false,
+            PreparedIamMode::Owner | PreparedIamMode::Deny | PreparedIamMode::TerminalDeny => false,
             PreparedIamMode::Regular { combined_policy } => {
                 policy_needs_existing_object_tag_for_args(combined_policy, args).await
             }
@@ -210,6 +217,11 @@ impl PreparedIamAuth {
                 combined_policy,
                 session_policy,
                 ..
+            }
+            | PreparedIamMode::PluginAttemptSts {
+                combined_policy,
+                session_policy,
+                ..
             } => {
                 policy_needs_existing_object_tag_for_args(combined_policy, args).await
                     || prepared_session_policy_needs_existing_object_tag_for_args(session_policy, args).await
@@ -239,14 +251,56 @@ impl PreparedIamAuth {
     pub fn combined_policy_for_view(&self) -> Option<&Policy> {
         match &self.mode {
             PreparedIamMode::Regular { combined_policy } => Some(combined_policy),
-            PreparedIamMode::Sts { combined_policy, .. } => Some(combined_policy),
+            PreparedIamMode::Sts { combined_policy, .. } | PreparedIamMode::PluginAttemptSts { combined_policy, .. } => {
+                Some(combined_policy)
+            }
             PreparedIamMode::ServiceAccount { combined_policy, .. } => Some(combined_policy),
-            PreparedIamMode::Opa | PreparedIamMode::Owner | PreparedIamMode::Deny => None,
+            PreparedIamMode::Opa | PreparedIamMode::Owner | PreparedIamMode::Deny | PreparedIamMode::TerminalDeny => None,
         }
     }
 }
 
 impl<T: Store> IamSys<T> {
+    /// For the authenticated, explicitly configured broker handler only.
+    pub async fn read_active_plugin_delegation_for_broker(
+        &self,
+        id: &str,
+        now: u64,
+    ) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
+        crate::delegation::read_active_for_broker(&self.store.api, id, now).await
+    }
+    pub async fn create_plugin_delegation(
+        &self,
+        owner: &str,
+        restrictions: crate::delegation::Restrictions,
+        now: u64,
+    ) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
+        crate::delegation::create(&self.store.api, owner, restrictions, now).await
+    }
+    pub async fn revoke_plugin_delegation(
+        &self,
+        owner: &str,
+        id: &str,
+    ) -> std::result::Result<(), crate::delegation::DelegationError> {
+        crate::delegation::revoke(&self.store.api, owner, id).await
+    }
+    pub async fn read_active_plugin_delegation(
+        &self,
+        owner: &str,
+        id: &str,
+        now: u64,
+    ) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
+        crate::delegation::read_active(&self.store.api, owner, id, now).await
+    }
+    /// Reads an owned delegation record for historical lookups, including expired or revoked records.
+    pub async fn read_owned_plugin_delegation(
+        &self,
+        owner: &str,
+        id: &str,
+    ) -> std::result::Result<crate::delegation::Delegation, crate::delegation::DelegationError> {
+        crate::delegation::read_owned(&self.store.api, owner, id).await
+    }
+
     /// Create a new IamSys instance with the given IamCache store
     ///
     /// # Arguments
@@ -281,6 +335,16 @@ impl<T: Store> IamSys<T> {
         *guard = PolicyPluginState::Ready(client);
     }
 
+    /// Restricted credentials require the built-in parent/session intersection.
+    /// External authorization bypasses that path, so only Disabled is supported.
+    pub async fn ensure_restricted_credential_vending() -> Result<()> {
+        if matches!(Self::policy_plugin_state().await, PolicyPluginState::Disabled) {
+            Ok(())
+        } else {
+            Err(Error::IAMActionNotAllowed)
+        }
+    }
+
     pub async fn get_policy_plugin_client() -> Option<rustfs_policy::policy::opa::AuthZPlugin> {
         let policy_plugin_state = get_policy_plugin_state();
         let guard = policy_plugin_state.read().await;
@@ -1383,17 +1447,50 @@ impl<T: Store> IamSys<T> {
     /// 1) know whether policy evaluation may need `s3:ExistingObjectTag`, and
     /// 2) evaluate with final conditions without re-merging identity policies.
     pub async fn prepare_auth(&self, args: &Args<'_>) -> PreparedIamAuth {
-        if args.is_owner {
+        let marked = args.claims.contains_key(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM);
+        if args.is_owner && !marked {
             return PreparedIamAuth {
                 needs_existing_object_tag: false,
                 mode: PreparedIamMode::Owner,
             };
         }
-
-        if let Some(prepared) = Self::policy_plugin_state().await.prepared_iam_auth() {
+        // One observed state decides this request's path. Marked credentials
+        // never enter OPA, including after a restart or on another node.
+        let state = Self::policy_plugin_state().await;
+        if marked {
+            if args.is_owner
+                || args.claims.get(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM) != Some(&Value::Bool(true))
+                || !matches!(state, PolicyPluginState::Disabled)
+            {
+                return PreparedIamAuth {
+                    needs_existing_object_tag: false,
+                    mode: PreparedIamMode::TerminalDeny,
+                };
+            }
+            let prepared = self.prepare_builtin_auth(args).await;
+            let mode = match prepared.mode {
+                PreparedIamMode::Sts {
+                    is_owner,
+                    combined_policy,
+                    session_policy,
+                } => PreparedIamMode::PluginAttemptSts {
+                    is_owner,
+                    combined_policy,
+                    session_policy,
+                },
+                _ => PreparedIamMode::TerminalDeny,
+            };
+            return PreparedIamAuth {
+                needs_existing_object_tag: prepared.needs_existing_object_tag,
+                mode,
+            };
+        }
+        if let Some(prepared) = state.prepared_iam_auth() {
             return prepared;
         }
-
+        self.prepare_builtin_auth(args).await
+    }
+    async fn prepare_builtin_auth(&self, args: &Args<'_>) -> PreparedIamAuth {
         let Ok((is_svc, parent_user)) = self.is_service_account(args.account).await else {
             return PreparedIamAuth {
                 needs_existing_object_tag: false,
@@ -1425,6 +1522,20 @@ impl<T: Store> IamSys<T> {
     /// bucket policies on an IAM denial. An error must not become an implicit deny.
     pub async fn try_eval_prepared(&self, prepared: &PreparedIamAuth, args: &Args<'_>) -> Result<bool> {
         Ok(match &prepared.mode {
+            PreparedIamMode::TerminalDeny => return Err(Error::IAMActionNotAllowed),
+            PreparedIamMode::PluginAttemptSts {
+                is_owner,
+                combined_policy,
+                session_policy,
+            } => {
+                // A false IAM result permits bucket-policy fallback. Restricted
+                // plugin credentials must instead terminate authorization.
+                let session_allowed = evaluate_prepared_session_policy(session_policy, args).await == Some(true);
+                if !session_allowed || !(*is_owner || combined_policy.is_allowed(args).await) {
+                    return Err(Error::IAMActionNotAllowed);
+                }
+                true
+            }
             PreparedIamMode::Opa => {
                 let Some(opa_enable) = Self::get_policy_plugin_client().await else {
                     tracing::warn!("eval_prepared: OPA mode requested but plugin is unavailable");
@@ -2127,6 +2238,145 @@ mod tests {
         assert!(prepared.combined_policy_for_view().is_none());
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn marked_signed_sts_denial_is_terminal_before_public_bucket_fallback() {
+        let iam = IamSys::new(IamCache::new(StsTestMockStore::new(false)).await.unwrap());
+        let previous = IamSys::<StsTestMockStore>::policy_plugin_state().await;
+        let signing_key = "c07-test-token-signing-key";
+        let mut metadata=HashMap::from([
+            ("parent".into(),Value::String("sts-fallback-test-parent".into())),
+            ("exp".into(),serde_json::json!((OffsetDateTime::now_utc()+time::Duration::hours(1)).unix_timestamp())),
+            (SESSION_POLICY_NAME.into(),Value::String(base64_simd::URL_SAFE_NO_PAD.encode_to_string(br#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::mybucket/inside"]}]}"#))),
+        ]);
+        metadata.insert(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM.into(), Value::Bool(true));
+        let mut marked = get_new_credentials_with_metadata(&metadata, signing_key).unwrap();
+        marked.parent_user = "sts-fallback-test-parent".into();
+        let marked_claims: HashMap<String, Value> = rustfs_policy::utils::extract_claims(&marked.session_token, signing_key)
+            .expect("signed test credentials must decode")
+            .claims;
+        let marked_user = UserIdentity::from(marked.clone());
+        iam.store
+            .cache
+            .add_or_update_sts_account(&marked.access_key, &marked_user, OffsetDateTime::now_utc());
+        metadata.remove(PLUGIN_ATTEMPT_CREDENTIAL_CLAIM);
+        let mut ordinary = get_new_credentials_with_metadata(&metadata, signing_key).unwrap();
+        ordinary.parent_user = marked.parent_user.clone();
+        let ordinary_claims: HashMap<String, Value> = rustfs_policy::utils::extract_claims(&ordinary.session_token, signing_key)
+            .expect("signed test credentials must decode")
+            .claims;
+        iam.store.cache.add_or_update_sts_account(
+            &ordinary.access_key,
+            &UserIdentity::from(ordinary.clone()),
+            OffsetDateTime::now_utc(),
+        );
+        let groups = None;
+        let conditions = HashMap::new();
+        let args = Args {
+            account: &marked.access_key,
+            groups: &groups,
+            action: Action::S3Action(S3Action::GetObjectAction),
+            bucket: "mybucket",
+            object: "outside",
+            conditions: &conditions,
+            is_owner: false,
+            claims: &marked_claims,
+            deny_only: false,
+        };
+        *get_policy_plugin_state().write().await = PolicyPluginState::Disabled;
+        let denied = iam.prepare_auth(&args).await;
+        let disabled_terminal = matches!(iam.try_eval_prepared(&denied, &args).await, Err(Error::IAMActionNotAllowed));
+        let inside = Args {
+            object: "inside",
+            ..args.clone()
+        };
+        let allowed = iam.prepare_auth(&inside).await;
+        let inside_allowed = matches!(iam.try_eval_prepared(&allowed, &inside).await, Ok(true));
+        let unmarked_args = Args {
+            account: &ordinary.access_key,
+            claims: &ordinary_claims,
+            ..args.clone()
+        };
+        let unmarked = iam.prepare_auth(&unmarked_args).await;
+        let ordinary_false = matches!(iam.try_eval_prepared(&unmarked, &unmarked_args).await, Ok(false));
+        let public:rustfs_policy::policy::BucketPolicy=serde_json::from_value(serde_json::json!({"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["*"]},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::mybucket/*"]}]})).unwrap();
+        let public_allows = public
+            .is_allowed(&rustfs_policy::policy::BucketPolicyArgs {
+                account: &marked.access_key,
+                groups: &groups,
+                action: args.action,
+                bucket: args.bucket,
+                object: args.object,
+                conditions: &conditions,
+                is_owner: false,
+            })
+            .await;
+        IamSys::<StsTestMockStore>::set_policy_plugin_client(opa::AuthZPlugin::new(opa::Args {
+            url: "http://127.0.0.1:8181/allow".into(),
+            auth_token: String::new(),
+        }))
+        .await;
+        let opa_marked = iam.prepare_auth(&args).await;
+        let ready_terminal = matches!(iam.try_eval_prepared(&opa_marked, &args).await, Err(Error::IAMActionNotAllowed));
+        let opa_ordinary = iam.prepare_auth(&unmarked_args).await;
+        let ordinary_keeps_opa = matches!(opa_ordinary.mode, PreparedIamMode::Opa);
+        // The captured Disabled preparation stays on built-in intersection even
+        // if an external policy client becomes ready before evaluation.
+        let captured_still_terminal = matches!(iam.try_eval_prepared(&denied, &args).await, Err(Error::IAMActionNotAllowed));
+        *get_policy_plugin_state().write().await = PolicyPluginState::Initializing;
+        let pending = iam.prepare_auth(&args).await;
+        let initializing_terminal = matches!(iam.try_eval_prepared(&pending, &args).await, Err(Error::IAMActionNotAllowed));
+        *get_policy_plugin_state().write().await = PolicyPluginState::Failed;
+        let failed = iam.prepare_auth(&args).await;
+        let failed_terminal = matches!(iam.try_eval_prepared(&failed, &args).await, Err(Error::IAMActionNotAllowed));
+        *get_policy_plugin_state().write().await = previous;
+        assert!(public_allows, "C07_PUBLIC_BUCKET_CONTROL: real public policy permits outside object");
+        assert!(
+            disabled_terminal && captured_still_terminal,
+            "C07_SCOPE_TERMINAL_BOUND: bucket policy must not rescue restricted scope denial"
+        );
+        assert!(
+            ready_terminal && initializing_terminal && failed_terminal,
+            "C07_EXTERNAL_POLICY_BOUND: marked persisted STS cannot enter OPA"
+        );
+        assert!(
+            inside_allowed && ordinary_false && ordinary_keeps_opa,
+            "C07_UNMARKED_COMPAT: preserve ordinary STS and valid marked scope"
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn restricted_vending_rejects_actual_external_policy_plugin_state() {
+        let previous = IamSys::<StsTestMockStore>::policy_plugin_state().await;
+        IamSys::<StsTestMockStore>::set_policy_plugin_client(opa::AuthZPlugin::new(opa::Args {
+            url: "http://127.0.0.1:8181/v1/data/rustfs/authz/allow".into(),
+            auth_token: String::new(),
+        }))
+        .await;
+        let active_rejected = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
+            .await
+            .is_err();
+        *get_policy_plugin_state().write().await = PolicyPluginState::Initializing;
+        let initializing_rejected = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
+            .await
+            .is_err();
+        *get_policy_plugin_state().write().await = PolicyPluginState::Failed;
+        let failed_rejected = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
+            .await
+            .is_err();
+        *get_policy_plugin_state().write().await = PolicyPluginState::Disabled;
+        let disabled_allowed = IamSys::<StsTestMockStore>::ensure_restricted_credential_vending()
+            .await
+            .is_ok();
+        *get_policy_plugin_state().write().await = previous;
+        assert!(
+            active_rejected && initializing_rejected && failed_rejected,
+            "C07_EXTERNAL_POLICY_BOUND: external authorization cannot bypass minted session scope"
+        );
+        assert!(disabled_allowed);
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_prepare_auth_requests_existing_object_tags_in_opa_mode() {

crates/io-metrics/src/sampler/process.rs

@@ -25,7 +25,7 @@ fn current_pid() -> Pid {
 #[derive(Debug)]
 pub struct ProcessSampler {
     pid: Pid,
-    sys: System,
+    sys: Option<System>,
 }
 
 impl Default for ProcessSampler {
@@ -38,8 +38,10 @@ impl ProcessSampler {
     #[inline]
     pub fn new() -> Self {
         let pid = current_pid();
-        let mut sys = System::new();
-        sys.refresh_processes_specifics(ProcessesToUpdate::Some(&[pid]), true, ProcessRefreshKind::everything());
+        let mut sys = System::new().ok();
+        if let Some(sys) = sys.as_mut() {
+            sys.refresh_processes_specifics(ProcessesToUpdate::Some(&[pid]), true, ProcessRefreshKind::everything());
+        }
         Self { pid, sys }
     }
 
@@ -129,13 +131,14 @@ pub fn snapshot_process_resource_and_system() -> (ProcessResourceSnapshot, Proce
 pub fn snapshot_process_resource_and_system_with(
     sampler: &mut ProcessSampler,
 ) -> (ProcessResourceSnapshot, ProcessSystemSnapshot) {
+    let Some(sys) = sampler.sys.as_mut() else {
+        return (ProcessResourceSnapshot::default(), ProcessSystemSnapshot::default());
+    };
     let platform_stats = crate::snapshot_process_platform_stats();
     let lock_snapshot = crate::snapshot_process_lock_counts();
-    sampler
-        .sys
-        .refresh_processes_specifics(ProcessesToUpdate::Some(&[sampler.pid]), true, ProcessRefreshKind::everything());
+    sys.refresh_processes_specifics(ProcessesToUpdate::Some(&[sampler.pid]), true, ProcessRefreshKind::everything());
 
-    if let Some(process) = sampler.sys.process(sampler.pid) {
+    if let Some(process) = sys.process(sampler.pid) {
         let disk_usage = process.disk_usage();
         let status = ProcessStatusSnapshot::from(process.status());
         let uptime_seconds = process.run_time();
@@ -180,6 +183,18 @@ pub fn snapshot_process_resource_and_system_with(
 mod tests {
     use super::*;
 
+    #[test]
+    fn unavailable_process_source_returns_empty_snapshots() {
+        let mut sampler = ProcessSampler {
+            pid: current_pid(),
+            sys: None,
+        };
+        assert_eq!(
+            sampler.snapshot_resource_and_system(),
+            (ProcessResourceSnapshot::default(), ProcessSystemSnapshot::default())
+        );
+    }
+
     #[test]
     fn process_status_snapshot_maps_sysinfo_values() {
         assert_eq!(ProcessStatusSnapshot::from(ProcessStatus::Run), ProcessStatusSnapshot::Running);

crates/madmin/src/health.rs

@@ -64,7 +64,18 @@ pub struct Cpus {
 }
 
 pub fn get_cpus() -> Cpus {
-    let mut sys = System::new_all();
+    let mut sys = match System::new_all() {
+        Ok(sys) => sys,
+        Err(error) => {
+            return Cpus {
+                node_common: NodeCommon {
+                    error: Some(error.to_string()),
+                    ..Default::default()
+                },
+                ..Default::default()
+            };
+        }
+    };
     sys.refresh_cpu_all();
     let cores = available_parallelism_count();
 
@@ -254,7 +265,18 @@ pub struct MemInfo {
 }
 
 pub fn get_mem_info(addr: &str) -> MemInfo {
-    let mut sys = System::new_all();
+    let mut sys = match System::new_all() {
+        Ok(sys) => sys,
+        Err(error) => {
+            return MemInfo {
+                node_common: NodeCommon {
+                    addr: addr.to_string(),
+                    error: Some(error.to_string()),
+                },
+                ..Default::default()
+            };
+        }
+    };
     sys.refresh_memory();
 
     let total = sys.total_memory();
@@ -279,7 +301,9 @@ pub fn get_mem_info(addr: &str) -> MemInfo {
 }
 
 pub fn collect_mem_stats() -> crate::MemStats {
-    let mut sys = System::new_all();
+    let Ok(mut sys) = System::new_all() else {
+        return crate::MemStats::default();
+    };
     sys.refresh_memory();
 
     let process_memory = sys

crates/notify/src/config_manager.rs

@@ -42,13 +42,14 @@ enum NotifyConfigStoreError {
     Read(String),
     Save(String),
     Converge { persisted: bool, error: String },
+    Conflict,
 }
 
 fn notification_convergence_error(persisted: bool, error: impl std::fmt::Display) -> NotificationError {
-    let durable_state = if persisted { "persisted" } else { "unchanged" };
-    NotificationError::Configuration(format!(
-        "configuration was {durable_state} but notification runtime convergence failed: {error}"
-    ))
+    NotificationError::ConfigurationConvergence {
+        persisted,
+        detail: error.to_string(),
+    }
 }
 
 async fn supervise_config_update<T>(
@@ -63,12 +64,62 @@ where
     })?
 }
 
+fn configs_equal(first: Option<&KVS>, second: Option<&KVS>) -> bool {
+    match (first, second) {
+        (None, None) => true,
+        (Some(first), Some(second)) => {
+            if first == second {
+                return true;
+            }
+            let mut first: Vec<_> = first.0.iter().collect();
+            let mut second: Vec<_> = second.0.iter().collect();
+            // Lookup uses the first occurrence of a key, so preserve duplicate-key order.
+            first.sort_by(|a, b| a.key.cmp(&b.key));
+            second.sort_by(|a, b| a.key.cmp(&b.key));
+            first == second
+        }
+        _ => false,
+    }
+}
+
+fn replace_target_if_unchanged(
+    config: &mut Config,
+    target_type: &str,
+    target_name: &str,
+    expected: Option<&KVS>,
+    expected_default: Option<&KVS>,
+    kvs: &KVS,
+) -> Result<bool, NotifyConfigStoreError> {
+    let targets = config.0.get(target_type);
+    let mut matching = targets
+        .into_iter()
+        .flat_map(|targets| targets.iter())
+        .filter(|(name, _)| name.eq_ignore_ascii_case(target_name));
+    let current = matching.next();
+    if matching.next().is_some() {
+        return Err(NotifyConfigStoreError::Conflict);
+    }
+    let stored_name = current
+        .map(|(name, _)| name.clone())
+        .unwrap_or_else(|| target_name.to_owned());
+    let current_default = targets.and_then(|targets| targets.get(rustfs_config::DEFAULT_DELIMITER));
+    if !configs_equal(current.map(|(_, kvs)| kvs), expected) || !configs_equal(current_default, expected_default) {
+        return Err(NotifyConfigStoreError::Conflict);
+    }
+    config
+        .0
+        .entry(target_type.to_owned())
+        .or_default()
+        .insert(stored_name, kvs.clone());
+    Ok(true)
+}
+
 async fn update_server_config<F>(
     mut modifier: F,
     lifecycle: NotifyLifecycleCoordinator,
 ) -> Result<Option<(crate::lifecycle::NotificationLifecycleTransition, bool)>, NotifyConfigStoreError>
 where
-    F: FnMut(&mut Config) -> bool + Send + 'static,
+    F: FnMut(&mut Config, Option<String>) -> Result<bool, NotifyConfigStoreError> + Send + 'static,
 {
     let Some(store) = resolve_notify_object_store_handle() else {
         return Err(NotifyConfigStoreError::StorageNotAvailable);
@@ -79,7 +130,8 @@ where
             .await
             .map_err(NotifyConfigStoreError::Read)?;
         let mut config = snapshot.config.clone();
-        if !modifier(&mut config) {
+        let generation = snapshot.generation().map(|value| value.to_string());
+        if !modifier(&mut config, generation)? {
             return Ok(None);
         }
 
@@ -245,6 +297,30 @@ impl NotifyConfigManager {
         .await
     }
 
+    /// Replace a target only when its persisted configuration is unchanged since validation.
+    pub async fn set_target_config_if_unchanged(
+        &self,
+        target_type: &str,
+        target_name: &str,
+        expected: Option<KVS>,
+        expected_default: Option<KVS>,
+        expected_generation: Option<String>,
+        kvs: KVS,
+    ) -> Result<(), NotificationError> {
+        let ttype = target_type.to_lowercase();
+        let tname = target_name.to_lowercase();
+        self.update_config_and_reload_checked(move |config, generation| {
+            if expected_generation
+                .as_ref()
+                .is_some_and(|expected| generation.as_ref() != Some(expected))
+            {
+                return Err(NotifyConfigStoreError::Conflict);
+            }
+            replace_target_if_unchanged(config, &ttype, &tname, expected.as_ref(), expected_default.as_ref(), &kvs)
+        })
+        .await
+    }
+
     pub async fn remove_target_config(&self, target_type: &str, target_name: &str) -> Result<(), NotificationError> {
         debug!(
             event = EVENT_NOTIFY_CONFIG_UPDATE,
@@ -338,9 +414,17 @@ impl NotifyConfigManager {
         Ok(())
     }
 
-    async fn update_config_and_reload<F>(&self, modifier: F) -> Result<(), NotificationError>
+    async fn update_config_and_reload<F>(&self, mut modifier: F) -> Result<(), NotificationError>
     where
         F: FnMut(&mut Config) -> bool + Send + 'static,
+    {
+        self.update_config_and_reload_checked(move |config, _generation| Ok(modifier(config)))
+            .await
+    }
+
+    async fn update_config_and_reload_checked<F>(&self, modifier: F) -> Result<(), NotificationError>
+    where
+        F: FnMut(&mut Config, Option<String>) -> Result<bool, NotifyConfigStoreError> + Send + 'static,
     {
         if self.lifecycle.state() == NotificationRuntimeState::Terminated {
             return Err(NotificationError::Initialization("Notification runtime has terminated".to_string()));
@@ -356,6 +440,7 @@ impl NotifyConfigManager {
                     NotifyConfigStoreError::Read(err) => NotificationError::ReadConfig(err),
                     NotifyConfigStoreError::Save(err) => NotificationError::SaveConfig(err),
                     NotifyConfigStoreError::Converge { persisted, error } => notification_convergence_error(persisted, error),
+                    NotifyConfigStoreError::Conflict => NotificationError::ConfigurationConflict,
                 })?
         else {
             debug!(
@@ -394,6 +479,126 @@ mod tests {
     use crate::rules::RulesMap;
     use crate::{NotificationError, NotificationRuntimeState};
 
+    #[test]
+    fn config_comparison_preserves_duplicates_and_hidden_flags() {
+        use rustfs_config::server_config::{KV, KVS};
+        let field = |key: &str, value: &str, hidden_if_empty: bool| KV {
+            key: key.into(),
+            value: value.into(),
+            hidden_if_empty,
+        };
+        let original = KVS(vec![
+            field("password", "secret", false),
+            field("token", "", true),
+            field("password", "secret", false),
+        ]);
+        let reordered = KVS(vec![
+            field("password", "secret", false),
+            field("password", "secret", false),
+            field("token", "", true),
+        ]);
+        let missing_duplicate = KVS(vec![field("password", "secret", false), field("token", "", true)]);
+        let changed_flag = KVS(vec![
+            field("password", "secret", false),
+            field("password", "secret", false),
+            field("token", "", false),
+        ]);
+        assert!(super::configs_equal(Some(&original), Some(&reordered)));
+        assert!(!super::configs_equal(Some(&original), Some(&missing_duplicate)));
+        assert!(!super::configs_equal(Some(&original), Some(&changed_flag)));
+        assert!(!super::configs_equal(Some(&original), None));
+        let first_value = KVS(vec![
+            field("password", "old", false),
+            field("token", "", true),
+            field("password", "new", false),
+        ]);
+        let reordered_keys = KVS(vec![
+            field("token", "", true),
+            field("password", "old", false),
+            field("password", "new", false),
+        ]);
+        let reordered_duplicates = KVS(vec![
+            field("token", "", true),
+            field("password", "new", false),
+            field("password", "old", false),
+        ]);
+        assert!(super::configs_equal(Some(&first_value), Some(&reordered_keys)));
+        assert!(!super::configs_equal(Some(&first_value), Some(&reordered_duplicates)));
+    }
+
+    #[test]
+    fn conditional_target_update_rejects_rotation_and_deletion() {
+        use rustfs_config::server_config::{Config, KVS};
+        let mut old = KVS::new();
+        old.insert("password".into(), "old".into());
+        let mut rotated = KVS::new();
+        rotated.insert("password".into(), "new".into());
+        let mut config = Config::new();
+        config
+            .0
+            .entry("notify_nats".into())
+            .or_default()
+            .insert("primary".into(), rotated.clone());
+        let result = super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), None, &old);
+        assert!(matches!(result, Err(super::NotifyConfigStoreError::Conflict)));
+        assert_eq!(config.0["notify_nats"]["primary"], rotated);
+        config.0.get_mut("notify_nats").expect("subsystem exists").remove("primary");
+        assert!(matches!(
+            super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), None, &old),
+            Err(super::NotifyConfigStoreError::Conflict)
+        ));
+        assert!(
+            super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", None, None, &old)
+                .expect("create unchanged target")
+        );
+        assert!(
+            super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), None, &rotated)
+                .expect("replace unchanged target")
+        );
+        assert_eq!(config.0["notify_nats"]["primary"], rotated);
+
+        let mut first = KVS::new();
+        first.insert("url".into(), "nats://example.com".into());
+        first.insert("token".into(), "secret".into());
+        let mut reordered = KVS::new();
+        reordered.insert("token".into(), "secret".into());
+        reordered.insert("url".into(), "nats://example.com".into());
+        config
+            .0
+            .get_mut("notify_nats")
+            .expect("subsystem")
+            .insert("primary".into(), first);
+        assert!(
+            super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&reordered), None, &rotated)
+                .expect("key order does not signal a change")
+        );
+        let targets = config.0.get_mut("notify_nats").expect("subsystem");
+        targets.remove("primary");
+        targets.insert("Primary".into(), rotated.clone());
+        assert!(
+            super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&rotated), None, &old)
+                .expect("case-insensitive match")
+        );
+        let targets = &config.0["notify_nats"];
+        assert!(targets.contains_key("Primary"));
+        assert!(!targets.contains_key("primary"));
+        let mut old_default = KVS::new();
+        old_default.insert("password".into(), "old-default".into());
+        let mut new_default = KVS::new();
+        new_default.insert("password".into(), "new-default".into());
+        config
+            .0
+            .get_mut("notify_nats")
+            .expect("subsystem")
+            .insert("_".into(), new_default.clone());
+        assert!(matches!(
+            super::replace_target_if_unchanged(&mut config, "notify_nats", "primary", Some(&old), Some(&old_default), &rotated),
+            Err(super::NotifyConfigStoreError::Conflict)
+        ));
+        assert_eq!(config.0["notify_nats"]["Primary"], old);
+        assert_eq!(config.0["notify_nats"]["_"], new_default);
+    }
+
     #[tokio::test]
     async fn supervised_config_update_survives_waiter_cancellation() {
         let (started_tx, started_rx) = tokio::sync::oneshot::channel();
@@ -550,13 +755,9 @@ mod tests {
 
     #[test]
     fn convergence_error_reports_durable_write_state() {
-        for (persisted, expected) in [(true, "configuration was persisted"), (false, "configuration was unchanged")] {
+        for persisted in [true, false] {
             let error = notification_convergence_error(persisted, "injected failure");
-            let NotificationError::Configuration(message) = error else {
-                panic!("convergence error should be a configuration error");
-            };
-            assert!(message.contains(expected), "unexpected convergence error: {message}");
-            assert!(message.contains("injected failure"));
+            assert!(matches!(error, NotificationError::ConfigurationConvergence { persisted: value, .. } if value == persisted));
         }
     }
 }

crates/notify/src/error.rs

@@ -38,6 +38,12 @@ pub enum NotificationError {
     #[error("Configuration error: {0}")]
     Configuration(String),
 
+    #[error("Target configuration changed during validation")]
+    ConfigurationConflict,
+
+    #[error("Notification configuration convergence failed (persisted: {persisted}): {detail}")]
+    ConfigurationConvergence { persisted: bool, detail: String },
+
     #[error("ARN not found: {0}")]
     ARNNotFound(String),
 

crates/notify/src/integration.rs

@@ -298,6 +298,22 @@ impl NotificationSystem {
             .await
     }
 
+    /// Replace a target only if its persisted configuration still matches the validated snapshot.
+    pub async fn set_target_config_if_unchanged(
+        &self,
+        target_type: &str,
+        target_name: &str,
+        expected: Option<KVS>,
+        expected_default: Option<KVS>,
+        expected_generation: Option<String>,
+        kvs: KVS,
+    ) -> Result<(), NotificationError> {
+        self.services
+            .config_manager
+            .set_target_config_if_unchanged(target_type, target_name, expected, expected_default, expected_generation, kvs)
+            .await
+    }
+
     /// Removes all notification configurations for a bucket.
     /// If the configuration is successfully removed, the entire notification system will be automatically reloaded.
     ///