chore(release): prepare 1.0.3

fix(scanner): clear transient cycle recovery on follower lock contention (#8566)

(cherry picked from commit 606d8a3a8a)

Co-authored-by: Ramakrishna Chilaka <49393831+RamakrishnaChilaka@users.noreply.github.com>

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -45,11 +45,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
 - `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
 - `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
+- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
 
 ### S3 object actions, copy, multipart, and upload policy validation
 
 - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
 - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
+- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
+- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
 - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
@@ -60,12 +63,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
 - `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
 - `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
+- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
 
 ### Filesystem paths and object key traversal
 
 - `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
 - `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
 - `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
+- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
 
 ### Secrets, defaults, and cryptographic misuse
 
@@ -106,6 +111,8 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### Object Lock and retention invariants
 
 - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
+- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
+- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
 
 ### Serde deserialization and input validation
 
@@ -135,16 +142,17 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ## Minimum Regression Test Expectations
 
 - Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
-- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
+- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
-- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
+- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
-sha256-linux=9ff0da7406de9b89c0e848ff29f2238b30ba9da89c7e9df97c246ce86631350f
+sha256-darwin=1520cd3027c3a6924afc6a204eddd5475bc30f0109d4c2847456a66177263244
+sha256-linux=26155b7f1c5bf6bf596ff24a4136e6767ea4ee6461c32db7fbe1400791da2c28

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
-sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
+sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
+sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926

.config/e2e-protocols-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
-sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17
+sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
+sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392

.config/ecstore-required-tests.json

@@ -61,6 +61,11 @@
       "suite": "rustfs-ecstore",
       "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
     },
+    {
+      "invariant": "rollback-closed-inspection-channel",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::io_primitives::tests::rename_rollback_closed_heal_receiver_preserves_recovery_material"
+    },
     {
       "invariant": "rollback-cancellation",
       "suite": "rustfs-ecstore",
@@ -86,6 +91,11 @@
       "suite": "rustfs-ecstore",
       "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
     },
+    {
+      "invariant": "seeded-put-tail-pressure",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::seeded_put_completion_pressure_drains_every_real_tail"
+    },
     {
       "invariant": "borrowed-lock-tail",
       "suite": "rustfs-ecstore",
@@ -151,6 +161,11 @@
       "suite": "rustfs-ecstore",
       "name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
     },
+    {
+      "invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
+    },
     {
       "invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
       "suite": "rustfs-ecstore",
@@ -166,6 +181,21 @@
       "suite": "rustfs-ecstore",
       "name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
     },
+    {
+      "invariant": "metadata-cache-real-get-and-range-preserve-body-across-read-modes",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::read::metadata_cache_tests::metadata_cache_real_get_and_range_preserve_body_across_read_modes"
+    },
+    {
+      "invariant": "late-metadata-refresh-rejects-changed-generation-and-mapped-shards",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
+    },
+    {
+      "invariant": "late-full-get-rejects-changed-object-generation",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
+    },
     {
       "invariant": "list-objects-shared-corpus-static-namespace",
       "suite": "rustfs-ecstore",
@@ -260,6 +290,46 @@
       "invariant": "logical-prefix-limit",
       "suite": "rustfs-ecstore",
       "name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
+    },
+    {
+      "invariant": "quota-same-name-owner-and-disk-authority",
+      "suite": "rustfs-ecstore",
+      "name": "bucket::metadata_sys::tests::quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority"
+    },
+    {
+      "invariant": "quota-snapshot-scope-and-owner",
+      "suite": "rustfs-ecstore",
+      "name": "bucket::metadata_sys::tests::quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard"
+    },
+    {
+      "invariant": "quota-snapshot-lifecycle-and-incarnation",
+      "suite": "rustfs-ecstore",
+      "name": "bucket::metadata_sys::tests::quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation"
+    },
+    {
+      "invariant": "marker-purge-receipt-reopen",
+      "suite": "rustfs-ecstore",
+      "name": "set_disk::ops::object::marker_purge_receipt_tests::quorum_marker_purge_receipt_survives_reopen_with_stale_member"
+    },
+    {
+      "invariant": "external-minio-shards",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_minio_shards_decode_and_reconstruct_plaintext"
+    },
+    {
+      "invariant": "external-legacy-shards",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_legacy_shards_decode_and_reconstruct_plaintext"
+    },
+    {
+      "invariant": "external-shard-bitrot",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_erasure_shards_reject_bitrot_corruption"
+    },
+    {
+      "invariant": "external-shard-quorum-codec",
+      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
+      "name": "pinned_erasure_shards_reject_insufficient_quorum_and_wrong_codec"
     }
   ],
   "fixtures": [
@@ -302,6 +372,21 @@
       "path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
       "sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
       "source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/erasure-shards/minio/manifest.json",
+      "sha256": "c398f8dcb76903f6cf8eee07931c76592895ced201ff9d59ed4ff9b0ead2b13c",
+      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/erasure-shards/README.md"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/erasure-shards/legacy/manifest.json",
+      "sha256": "dce397d5b2f42748eb5091b4355bb9d133a50cbba003626fafac28095dbe540b",
+      "source": "RustFS 1.0.0-alpha.37; crates/ecstore/tests/fixtures/erasure-shards/README.md"
+    },
+    {
+      "path": "crates/ecstore/tests/fixtures/erasure-shards/plaintext.bin",
+      "sha256": "b5a83332327961a31b65a8731c64559fd04ff8ac9f6124022201dbbdba5ffa01",
+      "source": "Independent fixed SHA256 counter-stream oracle; crates/ecstore/tests/fixtures/erasure-shards/README.md"
     }
   ]
 }

.config/make/tests.mak

@@ -52,6 +52,9 @@ script-tests: ## Run shell script tests
 	./scripts/check_embedded_secrets.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test

.config/nextest.toml

@@ -88,10 +88,21 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
+# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
+[[profile.default.scripts]]
+filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+setup = 'ecstore-base-stack'
+
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
 
+# These handler/store fixtures and the relocated-pool GET compose deep storage
+# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
+[[profile.default.scripts]]
+filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
+setup = 'ecstore-base-stack'
+
 [[profile.default.scripts]]
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
@@ -170,10 +181,11 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
 test-group = 'embedded-test-ports'
 
 # Registration, inventory delivery, and real drive/object probes include
-# durable filesystem IO in short deadlines. Reserve capacity so sibling test
+# filesystem or HTTP IO in short deadlines. Reserve capacity so sibling test
 # processes cannot exhaust those budgets. Keep deadlines and assertions intact.
+# The short HTTP fixtures are tracked in rustfs/backlog#2714.
 [[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
 threads-required = "num-test-threads"
 
 # This cancellation probe hashes the serving executable before acquiring its
@@ -332,10 +344,19 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
+# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
+[[profile.ci.scripts]]
+filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+setup = 'ecstore-base-stack'
+
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
 
+[[profile.ci.scripts]]
+filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
+setup = 'ecstore-base-stack'
+
 [[profile.ci.scripts]]
 filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
 setup = 'lifecycle-large-stack'
@@ -428,7 +449,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
 test-group = 'embedded-test-ports'
 
 [[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
 threads-required = "num-test-threads"
 
 # Preserve the cancellation probe's startup budget and assertions in CI too.

.docker/observability/grafana/dashboards/rustfs.json

@@ -1748,7 +1748,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1848,7 +1848,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"

.docker/observability/prometheus.yml

@@ -61,6 +61,7 @@ scrape_configs:
 otlp:
   promote_resource_attributes:
     - service.instance.id
+    - process.pid
     - service.name
     - service.namespace
     - cloud.availability_zone

.docker/observability/tests/bucket-usage.test.yml

@@ -0,0 +1,48 @@
+evaluation_interval: 1m
+tests:
+- name: bucket usage keeps clusters separate and deduplicates reporting nodes
+  interval: 1m
+  input_series:
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
+    values: '4096 1024'
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
+    values: '4096 1024'
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
+    values: '65536 65536'
+  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
+    values: '0 0'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
+    values: '3 1'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
+    values: '3 1'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
+    values: '70 70'
+  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
+    values: '0 0'
+  promql_expr_test:
+  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
+    eval_time: 1m
+    exp_samples:
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
+      value: 1024
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
+      value: 0
+  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
+    eval_time: 1m
+    exp_samples:
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
+      value: 1
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
+      value: 0
+  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
+    eval_time: 1m
+    exp_samples:
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
+      value: 1024
+    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
+      value: 0
+    - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
+      value: 65536
+  - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
+    eval_time: 1m
+    exp_samples: []

.github/actions/quick-checks/action.yml

@@ -98,6 +98,10 @@ runs:
       shell: bash
       run: make script-tests
 
+    - name: Check object generation protocol model
+      shell: bash
+      run: python3 ./scripts/test_object_generation_protocol_model.py
+
     - name: Check test wiring
       shell: bash
       run: python3 ./scripts/check_test_wiring.py

.github/actions/setup-docker-compose/action.yml

@@ -0,0 +1,39 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Set up Docker Compose
+description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
+
+runs:
+  using: composite
+  steps:
+    - name: Install verified Docker Compose CLI plugin
+      shell: bash
+      run: |
+        set -euo pipefail
+        [[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
+        plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
+        mkdir -p "$plugin_dir"
+        compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
+        compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
+        trap 'rm -f "$compose_download" "$compose_staged"' EXIT
+        curl --fail --location --retry 3 \
+          https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
+          --output "$compose_download"
+        printf '%s  %s\n' \
+          dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
+          "$compose_download" | sha256sum --check --strict
+        install -m 0755 "$compose_download" "$compose_staged"
+        mv -f "$compose_staged" "$plugin_dir/docker-compose"
+        [[ $(docker compose version --short) == 2.40.3 ]]

.github/actions/setup/action.yml

@@ -36,6 +36,16 @@ inputs:
     description: "Install cross-compilation tools"
     required: false
     default: "false"
+  zig-version:
+    description: >-
+      Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
+      baseline semantics cannot drift with an untested zig release.
+    required: false
+    default: "0.17.0"
+  cargo-zigbuild-version:
+    description: "cargo-zigbuild version installed for cross builds."
+    required: false
+    default: "0.23.4"
   target:
     description: "Target architecture to add"
     required: false
@@ -99,10 +109,14 @@ runs:
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
       uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
+      with:
+        version: ${{ inputs.zig-version }}
 
     - name: Install cargo-zigbuild
       if: inputs.install-cross-tools == 'true'
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
+      with:
+        tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
 
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
@@ -126,5 +140,5 @@ runs:
         # normalises, so cargo fingerprints stay valid.
         cache-all-crates: false
         cache-on-failure: true
-        shared-key: ${{ inputs.cache-shared-key }}
+        shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
         save-if: ${{ inputs.cache-save-if }}

.github/workflows/audit.yml

@@ -88,7 +88,7 @@ jobs:
   # database lookup with a second ignore list to maintain.
   cargo-deny:
     name: Cargo Deny
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     runs-on: sm-standard-2
     timeout-minutes: 20
     steps:
@@ -131,7 +131,7 @@ jobs:
 
   workflow-pin-report:
     name: Workflow Pin Report
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     runs-on: sm-standard-2
     timeout-minutes: 5
     steps:
@@ -203,7 +203,7 @@ jobs:
     # workflow produces, and until now it was only visible to whoever happened to
     # open the Actions tab. Same ci-8 mechanism coverage.yml and
     # e2e-replication-nightly.yml already use.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/build.yml

@@ -213,12 +213,12 @@ jobs:
           fi
 
           all='{"include":[
-            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
-            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
-            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
-            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
+            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
+            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
           ]}'
 
           if [[ "${selected}" == "all" ]]; then
@@ -246,7 +246,8 @@ jobs:
           echo "Selected platforms: ${selected}"
 
   build-rustfs:
-    name: Build RustFS
+    # Acceptance workflows select artifacts by this stable platform name.
+    name: Build RustFS (${{ matrix.target_id }})
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
@@ -273,7 +274,7 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}
+          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -410,35 +411,64 @@ jobs:
         shell: bash
         env:
           BUILD_TARGET: ${{ matrix.target }}
+          BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
         run: |
           # Force rebuild by touching build.rs
           touch rustfs/build.rs
 
-          BINARY_NAMES=(rustfs)
-          if [[ "${{ matrix.platform }}" == "linux" ]]; then
-            BINARY_NAMES+=(rustfs-cli)
-          fi
           FEATURE_ARGS=()
           if [[ -n "${{ matrix.features }}" ]]; then
             FEATURE_ARGS=(--features "${{ matrix.features }}")
           fi
 
-          # Build one binary at a time so release LTO links cannot exhaust a
-          # self-hosted runner by running concurrently.
-          for binary in "${BINARY_NAMES[@]}"; do
-            PROFILE_ARGS=()
-            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
-              # The in-process profiler resolves names from the final executable.
-              # Retain function symbols without enabling DWARF source information.
-              PROFILE_ARGS=(--config 'profile.release.strip="none"')
-            fi
-            if [[ "${{ matrix.cross }}" == "true" ]]; then
-              # All cross targets in the matrix are Linux; zigbuild handles them.
-              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
-            else
-              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
-            fi
-          done
+          # gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
+          # cargo-zigbuild) so release binaries link no symbols newer than it
+          # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
+          TARGET_ARG="$BUILD_TARGET"
+          if [[ -n "${{ matrix.glibc }}" ]]; then
+            TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
+          fi
+
+          PROFILE_ARGS=()
+          if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+            # The in-process profiler resolves names from the final executable.
+            # Retain function symbols without enabling DWARF source information.
+            PROFILE_ARGS=(--config 'profile.release.strip="none"')
+          fi
+          if [[ "${{ matrix.cross }}" == "true" ]]; then
+            cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
+          else
+            cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
+          fi
+
+          # All release targets use same-architecture runners, including GNU
+          # builds that use Zig only to select the glibc ABI floor.
+          bin="target/${{ matrix.target }}/release/rustfs"
+          if [[ -f "${bin}.exe" ]]; then
+            bin="${bin}.exe"
+          fi
+          version_output=$("$bin" --version)
+          if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
+             ! grep -Fxq "build os     : ${BUILD_OS}" <<< "$version_output" || \
+             ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
+            echo "release binary reports an unexpected compile target" >&2
+            exit 1
+          fi
+          if [[ -n "${{ matrix.glibc }}" ]]; then
+            python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
+              "target/${{ matrix.target }}/release/rustfs"
+          fi
+
+      - name: Verify macOS binary has no dynamic liblzma dependency
+        if: matrix.platform == 'macos'
+        shell: bash
+        run: |
+          binary="target/${{ matrix.target }}/release/rustfs"
+          if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
+            echo "macOS release binary must not depend on a dynamic liblzma library" >&2
+            exit 1
+          fi
+          "$binary" --version
 
       - name: Generate the CPU symbol catalog
         if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
@@ -543,9 +573,6 @@ jobs:
           fi
 
           BINARY_FILES=("$BINARY_NAME")
-          if [[ "${{ matrix.platform }}" == "linux" ]]; then
-            BINARY_FILES+=("rustfs-cli")
-          fi
 
           if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
             BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -660,7 +687,7 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged Linux binaries
+      - name: Verify packaged Linux binary
         if: matrix.platform == 'linux'
         shell: bash
         run: |
@@ -668,7 +695,7 @@ jobs:
 
           package_file="${{ steps.package.outputs.package_file }}"
           archive_members="$(unzip -Z1 "$package_file" | sort)"
-          expected_members=$'rustfs\nrustfs-cli'
+          expected_members='rustfs'
           target="${{ matrix.target }}"
           features="${{ matrix.features }}"
           if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -680,13 +707,14 @@ jobs:
             exit 1
           fi
 
-          # Every Linux target builds natively, so executing the binaries here
+          # musl targets build natively; the gnu legs are same-arch
+          # cargo-zigbuild builds pinned to the release glibc baseline. Either
+          # way the binaries run on their runner, so executing them here
           # proves each artifact matches its runner architecture.
-          cli_dir="$(mktemp -d)"
-          trap 'rm -rf "$cli_dir"' EXIT
-          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
-          "$cli_dir/rustfs" --version
-          "$cli_dir/rustfs-cli" --help >/dev/null
+          binary_dir="$(mktemp -d)"
+          trap 'rm -rf "$binary_dir"' EXIT
+          unzip -q "$package_file" rustfs -d "$binary_dir"
+          "$binary_dir/rustfs" --version
 
       - name: Verify packaged console
         if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'

.github/workflows/cache-warm.yml

@@ -99,7 +99,7 @@ env:
   RUST_BACKTRACE: 1
 
 jobs:
-  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
+  # Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
   # e2e-tests, e2e-full.
   warm-ci-dev:
     name: Warm ci-dev

.github/workflows/ci.yml

@@ -36,7 +36,9 @@ on:
       - "flake.lock"
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
-    branches: [ main ]
+    # gateway/integration carries main's ruleset (rustfs/backlog#2734), so its
+    # pull requests must produce the same required checks.
+    branches: [ main, gateway/integration ]
   merge_group:
     types: [ checks_requested ]
   schedule:
@@ -166,13 +168,10 @@ jobs:
             echo "started_at=$(date --utc --iso-8601=seconds)"
           } > artifacts/test-and-lint/run-metadata.txt
 
-      # Clippy runs before the test pass: lint failures are the most common
-      # CI-only breakage and should surface in minutes, not after 20+ minutes
-      # of tests.
-      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
-      # experiment, since --all-targets is check-only for workspace members and
-      # never links the ~100 test binaries the limit exists to throttle.
+      # Keep full Clippy validation on main, merge groups, and scheduled/manual
+      # runs. PRs retain the complete default test suite and compile-free checks.
       - name: Run clippy lints
+        if: github.event_name != 'pull_request'
         run: |
           ./scripts/ci/resource_sampler.sh start clippy
           trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -281,6 +280,49 @@ jobs:
       - name: Verify rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
+      # The workspace already compiled these ignored tests. Keep the same Cargo
+      # package selection to reuse their binaries, then run them serially in
+      # separate processes because they share fixed ports and global state.
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract).
+      - name: Run ignored ILM integration tests serially
+        if: needs.classify-changes.outputs.mode == 'full'
+        env:
+          # Match the preceding workspace build and reuse its feature graph.
+          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+        run: |
+          mkdir -p artifacts/ilm-integration
+          set +e
+          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
+            cargo nextest run --profile default -j1 --run-ignored ignored-only \
+            --all --exclude e2e_test \
+            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
+            --status-level all --final-status-level all \
+            2>&1 | tee artifacts/ilm-integration/nextest.log
+          status=${PIPESTATUS[0]}
+          {
+            echo "exit_status=${status}"
+            echo "finished_at=$(date --utc --iso-8601=seconds)"
+            echo
+            echo "Remaining test-related processes:"
+            pgrep -af 'cargo|nextest|target/.*/deps/' || true
+          } > artifacts/ilm-integration/diagnostics.txt
+          exit "${status}"
+
+      - name: Upload ILM test diagnostics
+        if: always() && needs.classify-changes.outputs.mode == 'full'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
+          path: artifacts/ilm-integration
+          retention-days: 3
+
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
   offline-enrollment-e2e:
@@ -312,83 +354,11 @@ jobs:
       - name: Check offline enrollment E2E root boundary
         run: ./scripts/check_offline_enrollment_e2e.sh
 
-  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
-  # drive the object layer through process-global singletons (the GLOBAL_ENV
-  # ECStore, the global tier-config manager, background-expiry workers) and bind
-  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
-  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
-  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
-  # `-j1` so no two of them share global state or race for a port at once.
-  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
-  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
-  # See rustfs/backlog#1148 (ilm-1) and #1155.
-  test-ilm-integration-serial:
-    name: ILM Integration (serial)
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
-    needs: [ quick-checks, classify-changes ]
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract).
-      - name: Run ignored ILM integration tests serially
-        env:
-          # Match the measured Test and Lint link budget. The default exposed
-          # all 14 pod CPUs and a cold cache spent the full 80m compiling
-          # without starting one ILM test (main run 32982910990).
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
-        run: |
-          mkdir -p artifacts/ilm-integration
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
-            cargo nextest run -j1 --run-ignored ignored-only \
-            -p rustfs-scanner -p rustfs \
-            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/ilm-integration/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-          } > artifacts/ilm-integration/diagnostics.txt
-          exit "${status}"
-
-      - name: Upload ILM test diagnostics
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
-          path: |
-            artifacts/ilm-integration
-            target/nextest/ci/junit.xml
-
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    # Alternate feature configurations run on main, merge groups, and scheduled
+    # or manual validation; the default workspace remains the PR gate.
+    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -504,12 +474,12 @@ jobs:
 
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
     strategy:
-      # Preserve both independent results when one protocol fails, so diagnosis
+      # Preserve independent results when one protocol fails, so diagnosis
       # and a focused fix do not require rebuilding an interrupted sibling.
       fail-fast: false
       matrix:
@@ -518,6 +488,8 @@ jobs:
             flags: "--features swift"
           - name: sftp
             flags: "--features sftp"
+          - name: http3
+            flags: "--features http3,swift"
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -1260,7 +1232,6 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
-      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - connect-short-credential-boundary
       - test-and-lint-protocols
@@ -1294,7 +1265,6 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
-      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - test-and-lint-protocols
       - build-rustfs-debug-binary

.github/workflows/connect-diagnostic-schedule-acceptance.yml

@@ -57,6 +57,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -94,7 +102,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -137,7 +145,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
                   assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -162,7 +170,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule

.github/workflows/connect-health-service-acceptance.yml

@@ -57,6 +57,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -127,7 +135,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -172,7 +180,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -199,7 +207,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=health-service

.github/workflows/connect-performance-drive-acceptance.yml

@@ -73,6 +73,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +115,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive

.github/workflows/connect-performance-network-acceptance.yml

@@ -73,6 +73,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +115,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-network

.github/workflows/connect-profile-cpu-acceptance.yml

@@ -77,6 +77,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -118,7 +126,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.jobs[] | select(.name == $name)] as $matches
@@ -173,7 +181,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
             CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \

.github/workflows/connect-profile-memory-acceptance.yml

@@ -57,6 +57,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -127,7 +135,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -172,7 +180,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -198,7 +206,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory

.github/workflows/connect-profile-threads-acceptance.yml

@@ -81,6 +81,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -148,7 +156,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -201,7 +209,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads

.github/workflows/connect-top-api-acceptance.yml

@@ -73,6 +73,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +115,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-api

.github/workflows/connect-top-disk-acceptance.yml

@@ -68,6 +68,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -138,7 +146,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -183,7 +191,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -213,7 +221,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-disk

.github/workflows/connect-top-locks-acceptance.yml

@@ -73,6 +73,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +115,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-locks

.github/workflows/connect-top-rpc-acceptance.yml

@@ -73,6 +73,14 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
+      - name: Set up Docker Compose
+        uses: ./.github/actions/setup-docker-compose
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -107,7 +115,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc

.github/workflows/e2e-distributed.yml

@@ -78,6 +78,7 @@ concurrency:
 
 jobs:
   distributed:
+    if: github.repository == 'rustfs/rustfs'
     name: Distributed 4-node 4-disk e2e
     # GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
     # ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
@@ -167,11 +168,13 @@ jobs:
           FILTER: ${{ inputs.filter }}
         run: |
           set -euo pipefail
+          rm -f target/nextest/e2e-distributed/junit.xml
           if [ -n "${FILTER}" ]; then
             python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
           else
             python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
           fi
+          test -s target/nextest/e2e-distributed/junit.xml
 
       - name: Upload distributed e2e diagnostics
         if: always()
@@ -200,7 +203,7 @@ jobs:
   alert-on-failure:
     name: Alert on scheduled failure
     needs: [distributed]
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/e2e-replication-nightly.yml

@@ -53,6 +53,7 @@ concurrency:
 
 jobs:
   repl-nightly:
+    if: github.repository == 'rustfs/rustfs'
     name: Replication e2e (nightly)
     # dual-node tests spawn two full rustfs servers each; the extra headroom
     # over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
@@ -119,6 +120,7 @@ jobs:
           if-no-files-found: ignore
 
   cluster-nightly:
+    if: github.repository == 'rustfs/rustfs'
     name: Cluster fault e2e (nightly)
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -168,6 +170,7 @@ jobs:
           if-no-files-found: warn
 
   protocols-nightly:
+    if: github.repository == 'rustfs/rustfs'
     name: Protocol e2e (nightly)
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -229,7 +232,7 @@ jobs:
     # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/e2e-upgrade.yml

@@ -96,38 +96,8 @@ jobs:
           overwrite: true
 
   upgrade:
-    name: ${{ matrix.name }}
+    name: Upgrade compatibility
     needs: build
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          # The two `_from_rc2_` tests keep their names: they assert
-          # release-independent object contracts and pass unchanged against the
-          # newer pinned source, so renaming them would only churn history and
-          # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
-          # single source of truth for which release they actually run against.
-          - name: Direct upgrade from the previous release
-            test: direct_upgrade_from_rc2_preserves_object_contracts
-            artifact: direct-upgrade
-          - name: Mixed-version rolling upgrade from the previous release
-            test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
-            artifact: mixed-version-upgrade
-          - name: Bucket configuration survives the upgrade
-            test: direct_upgrade_from_previous_release_preserves_bucket_configuration
-            artifact: bucket-config-upgrade
-          - name: Rollback reads current bucket metadata
-            test: rollback_to_previous_release_reads_current_bucket_metadata
-            artifact: bucket-config-rollback
-          - name: ODM configuration recovery after rc.5 rollback
-            test: rc5_rollback_requires_restoring_odm_configuration
-            artifact: odm-config-rollback
-          - name: Multipart layouts survive the rc.5 upgrade
-            test: direct_upgrade_from_rc5_preserves_multipart_layouts
-            artifact: multipart-layout-upgrade
-          - name: rc.5 multipart replication baseline
-            test: rc5_baseline_replicates_multipart_layouts
-            artifact: multipart-layout-baseline
     runs-on: sm-standard-2
     timeout-minutes: 60
     env:
@@ -142,8 +112,7 @@ jobs:
         uses: ./.github/actions/setup
         with:
           cache-shared-key: e2e-upgrade-tests
-          # One main-branch consumer warms the shared test-harness dependencies.
-          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
           install-build-packaging-tools: "false"
           install-test-tools: "false"
 
@@ -170,22 +139,49 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      - name: Run upgrade compatibility test
+      # Reuse one compiled harness, while keeping each case in its own process.
+      # Continue after a failure to retain the former matrix's complete verdict.
+      - name: Run upgrade compatibility tests
+        shell: bash
         env:
-          RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           NO_PROXY: 127.0.0.1,localhost,::1
         run: |
           export no_proxy="$NO_PROXY"
-          python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
-            "upgrade_compatibility_test::${{ matrix.test }}" \
-            -- --ignored --exact --nocapture
+          cases=(
+            direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
+            mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
+            bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
+            bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
+            odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
+            multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
+            multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
+          )
+          status=0
+          for entry in "${cases[@]}"; do
+            artifact="${entry%%:*}"
+            test="${entry#*:}"
+            export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
+            export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
+            mkdir -p "$RUSTFS_E2E_LOG_DIR"
+            echo "::group::$test"
+            if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
+              "upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
+              2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
+              echo "PASS: $test"
+            else
+              echo "::error::Upgrade compatibility failed: $test"
+              status=1
+            fi
+            echo "::endgroup::"
+          done
+          exit "$status"
 
       - name: Upload scanner/heal G09 evidence
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
-          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
+          name: upgrade-g09-evidence-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
           if-no-files-found: ignore
           retention-days: 14
 
@@ -193,7 +189,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
+          name: upgrade-server-logs-${{ github.run_number }}
           path: ${{ runner.temp }}/rustfs-upgrade-logs
           if-no-files-found: warn
           retention-days: 14

.github/workflows/functional-chain-health.yml

@@ -11,6 +11,7 @@ concurrency:
   cancel-in-progress: false
 jobs:
   collect:
+    if: github.repository == 'rustfs/rustfs'
     runs-on: smoke-testing
     timeout-minutes: 15
     steps:

.github/workflows/fuzz.yml

@@ -72,9 +72,11 @@ jobs:
   fuzz-build:
     name: Build Fuzz Harness
     if: >
-      (github.event_name == 'pull_request' && github.event.action != 'closed') ||
-      github.event_name == 'schedule' ||
-      github.event_name == 'workflow_dispatch'
+      github.repository == 'rustfs/rustfs' && (
+        (github.event_name == 'pull_request' && github.event.action != 'closed') ||
+        github.event_name == 'schedule' ||
+        github.event_name == 'workflow_dispatch'
+      )
     runs-on: sm-standard-4
     timeout-minutes: 60
     env:
@@ -239,7 +241,7 @@ jobs:
     # `always()` is required: without it this job is skipped when a needed
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
     # ci-8); PR and manual dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/gateway-integration-sync.yml

@@ -0,0 +1,197 @@
+# Copyright 2026 RustFS Team
+# SPDX-License-Identifier: Apache-2.0
+
+name: Gateway Integration Sync
+
+# Merges `main` into the long-lived `gateway/integration` branch every weekday
+# so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736)
+# never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit
+# pushed straight to the branch. A conflicting merge is aborted and surfaced as
+# a pull request from `sync/<source>-<date>`, whose head is the un-merged
+# source commit, so the conflict is resolved the same day, in review, with no
+# other change riding along. The run itself stays green in that case: the pull
+# request is the signal. Any other failure (push rejected, pull request not
+# created) fails the run and, on the schedule, files the shared failure issue.
+#
+# Resolving a conflict pull request: merge `origin/gateway/integration` into
+# the sync branch, resolve, push, get the review. Land it with a fast-forward
+# push of the resolved merge commit to `gateway/integration`, not with the
+# squash button: a squash drops the merge ancestry and the same hunks conflict
+# again on the next run.
+#
+# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and
+# both push targets below are literal refs. The pushes use a one-hour
+# installation token of the organisation's own GitHub App (contents: write)
+# because that app is the only non-admin bypass actor of the
+# `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor,
+# so it cannot push through the required reviews and checks. The conflict pull
+# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the
+# weekly flake.lock pull requests, because this repository keeps "Allow GitHub
+# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is
+# therefore read-only here.
+#
+# `schedule` fires only from the default branch, so the cron is live once this
+# file exists on `main`. `workflow_dispatch` runs from any ref that carries the
+# file and always acts on `gateway/integration`.
+
+on:
+  schedule:
+    - cron: "0 2 * * 1-5" # Weekdays 02:00 UTC
+  workflow_dispatch:
+    inputs:
+      source_ref:
+        description: >-
+          Branch to merge into gateway/integration. The schedule always merges
+          main; only the conflict acceptance test (rustfs/backlog#2736) sets
+          another branch.
+        required: false
+        default: main
+        type: string
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+env:
+  TARGET_BRANCH: gateway/integration
+
+jobs:
+  sync:
+    name: Merge main into gateway/integration
+    if: github.repository == 'rustfs/rustfs'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Create the branch push token
+        id: push-token
+        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
+        with:
+          app-id: ${{ vars.CLA_BOT_APP_ID }}
+          private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
+          owner: ${{ github.repository_owner }}
+          repositories: rustfs
+          permission-contents: write
+
+      - name: Checkout gateway/integration
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          # persist-credentials-exempt: this checkout's token IS the push
+          # credential — the merge commit and the sync branch are pushed with it
+          # below. It is a one-hour installation token scoped to contents.
+          ref: gateway/integration
+          fetch-depth: 0
+          token: ${{ steps.push-token.outputs.token }}
+
+      - name: Configure the bot identity
+        run: |
+          git config user.name "rustfs-cla-assistant[bot]"
+          git config user.email "273121464+rustfs-cla-assistant[bot]@users.noreply.github.com"
+
+      - name: Merge the source branch
+        id: merge
+        env:
+          SOURCE_REF: ${{ inputs.source_ref || 'main' }}
+        run: |
+          set -euo pipefail
+          if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \
+            || [[ "$SOURCE_REF" == *..* ]] \
+            || [[ "$SOURCE_REF" == "$TARGET_BRANCH" ]]; then
+            echo "::error::refusing source_ref '${SOURCE_REF}'"
+            exit 1
+          fi
+          git fetch origin "refs/heads/${SOURCE_REF}:refs/remotes/origin/${SOURCE_REF}"
+          source_sha=$(git rev-parse "origin/${SOURCE_REF}")
+          target_sha=$(git rev-parse HEAD)
+          {
+            echo "source_ref=${SOURCE_REF}"
+            echo "source_sha=${source_sha}"
+            echo "target_sha=${target_sha}"
+          } >> "$GITHUB_OUTPUT"
+          if git merge --no-ff --no-edit "origin/${SOURCE_REF}"; then
+            if [[ "$(git rev-parse HEAD)" == "$target_sha" ]]; then
+              echo "result=up-to-date" >> "$GITHUB_OUTPUT"
+              echo "\`${TARGET_BRANCH}\` already contains \`${SOURCE_REF}\` (${source_sha}); nothing to push." >> "$GITHUB_STEP_SUMMARY"
+            else
+              echo "result=merged" >> "$GITHUB_OUTPUT"
+            fi
+            exit 0
+          fi
+          git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt"
+          git merge --abort
+          if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then
+            echo "::error::git merge failed without reporting conflicting paths; this is not a conflict"
+            exit 1
+          fi
+          echo "result=conflict" >> "$GITHUB_OUTPUT"
+
+      - name: Push the merge commit
+        if: steps.merge.outputs.result == 'merged'
+        env:
+          SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
+          SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
+        run: |
+          set -euo pipefail
+          git push origin HEAD:refs/heads/gateway/integration
+          echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Open the conflict pull request
+        if: steps.merge.outputs.result == 'conflict'
+        env:
+          GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }}
+          SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
+          SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
+          TARGET_SHA: ${{ steps.merge.outputs.target_sha }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          date_utc=$(date -u +%Y-%m-%d)
+          sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}"
+          git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}"
+          {
+            echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged."
+            echo
+            echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution."
+            echo
+            echo "- Run: ${RUN_URL}"
+            echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}"
+            echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}"
+            echo
+            echo "## Conflicting files"
+            echo
+            echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:"
+            echo
+            sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt"
+          } > "$RUNNER_TEMP/pr-body.md"
+          title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)"
+          existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty')
+          if [[ -n "$existing" ]]; then
+            gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md"
+            pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url)
+          else
+            pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md")
+          fi
+          echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY"
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [sync]
+    # Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that
+    # could not push or could not open the conflict pull request must not stay
+    # visible only in the Actions tab.
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}

.github/workflows/nightly-gnu.yml

@@ -43,6 +43,7 @@ env:
 
 jobs:
   resolve-source:
+    if: github.repository == 'rustfs/rustfs'
     name: Resolve nightly source
     runs-on: sm-standard-2
     timeout-minutes: 10
@@ -79,13 +80,24 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu
+          cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
           cache-save-if: 'false'
+          install-cross-tools: 'true'
+          target: x86_64-unknown-linux-gnu
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
       - name: Build RustFS
-        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
+        # Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
+        # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
+        run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
+
+      - name: Verify glibc symbol baseline
+        run: |
+          python3 scripts/release/check_glibc.py 2.31 \
+            target/x86_64-unknown-linux-gnu/release/rustfs \
+            target/x86_64-unknown-linux-gnu/release/rustfs-cli
+          target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
 
       - name: Build DEB package
         id: deb
@@ -496,8 +508,10 @@ jobs:
     name: Alert on scheduled failure
     needs: [build, kms-vault-lane, kms-vault-ha-failover]
     if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      github.repository == 'rustfs/rustfs' && (
+        always() && github.event_name == 'schedule' &&
+        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      )
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/nix-flake-update.yml

@@ -32,6 +32,7 @@ concurrency:
 
 jobs:
   update-flake:
+    if: github.repository == 'rustfs/rustfs'
     name: Update flake.lock
     runs-on: sm-standard-2
     timeout-minutes: 90

.github/workflows/oidc-keycloak.yml

@@ -20,8 +20,9 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc.rs"
+      - "crates/iam/src/oidc/**"
       - "rustfs/src/admin/handlers/oidc.rs"
+      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -31,8 +32,9 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc.rs"
+      - "crates/iam/src/oidc/**"
       - "rustfs/src/admin/handlers/oidc.rs"
+      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"

.github/workflows/performance-ab.yml

@@ -49,6 +49,7 @@ env:
 
 jobs:
   warp-ab:
+    if: github.repository == 'rustfs/rustfs'
     name: Warp A/B budget gate
     runs-on: sm-standard-2
     # A normal nightly restores the last successful binary and builds only the
@@ -364,8 +365,10 @@ jobs:
     # composite action already reports cancelled/timed-out jobs in the issue
     # body.
     if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      github.repository == 'rustfs/rustfs' && (
+        always() && github.event_name == 'schedule' &&
+        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      )
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/runner-hygiene.yml

@@ -42,6 +42,7 @@ concurrency:
 
 jobs:
   check-ephemerality:
+    if: github.repository == 'rustfs/rustfs'
     name: Check runner ephemerality
     runs-on: sm-standard-2
     timeout-minutes: 15
@@ -65,7 +66,7 @@ jobs:
     # Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
     # scheduled runs file a tracking issue, manual dispatch stays quiet so
     # debugging never produces a spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/rustfs-pool-expand-test.yml

@@ -266,7 +266,10 @@ jobs:
         # the previous 45m step cap killed four consecutive runs mid-rebalance
         # with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
         # in ~21m) plus one decommission pass; override per dispatch.
-        timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
+        # fromJSON coerces the string dispatch input to the number that
+        # timeout-minutes requires; the bare || form yielded the string '240'
+        # and the runner rejected the template (Unexpected value '240').
+        timeout-minutes: ${{ fromJSON(inputs.pool_timeout_minutes || '240') }}
         # Case failures keep the run green: the report and the backlog
         # issue manager carry the product signal.
         continue-on-error: true
@@ -583,25 +586,6 @@ jobs:
             echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
             failed=1
           fi
-          # A runner-level kill (step timeout) truncates the suite without a
-          # FAIL verdict or an [ERROR] line, so require every expected
-          # [POOL-STEP] marker to be present with PASS - missing markers mean
-          # the run was truncated, not green. This also closes the gap for
-          # standalone dispatches, where the chain driver's evidence
-          # validation does not run.
-          last_step=6
-          if [ "${{ inputs.pools || '3' }}" = "3" ]; then
-            last_step=8
-            if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
-              last_step=9
-            fi
-          fi
-          for n in $(seq 1 "${last_step}"); do
-            grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
-              echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
-              failed=1
-            }
-          done
           [ "${failed}" -eq 0 ] || exit 1
 
       - name: Upload functional report to dashboard

.github/workflows/scheduled-validation-freshness.yml

@@ -28,6 +28,7 @@ concurrency:
 
 jobs:
   check-freshness:
+    if: github.repository == 'rustfs/rustfs'
     name: Check scheduled validation freshness
     # Stay on GitHub-hosted runners: this monitor must keep running when the
     # self-hosted sm-standard-* pool is down, which is exactly when the

.github/workflows/scheduled-validation-watchdog.yml

@@ -39,6 +39,7 @@ jobs:
   alert-on-incomplete-run:
     name: Alert on incomplete scheduled run
     if: >-
+      github.repository == 'rustfs/rustfs' &&
       github.event.workflow_run.event == 'schedule' &&
       github.event.workflow_run.conclusion != 'success' &&
       github.event.workflow_run.conclusion != 'failure'

.github/workflows/stale.yml

@@ -26,6 +26,7 @@ on:
 
 jobs:
   stale:
+    if: github.repository == 'rustfs/rustfs'
     runs-on: sm-standard-2
     timeout-minutes: 30
     steps:

.github/workflows/star-history.yml

@@ -14,6 +14,7 @@ concurrency:
 
 jobs:
   update:
+    if: github.repository == 'rustfs/rustfs'
     runs-on: sm-standard-2
     timeout-minutes: 10
     steps:

.github/workflows/targets-integration.yml

@@ -40,6 +40,7 @@ env:
 
 jobs:
   targets-live:
+    if: github.repository == 'rustfs/rustfs'
     name: PostgreSQL, MySQL, AMQP, and NATS
     runs-on: sm-standard-2
     timeout-minutes: 90
@@ -175,8 +176,10 @@ jobs:
     name: Alert on scheduled failure
     needs: [targets-live]
     if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      github.repository == 'rustfs/rustfs' && (
+        always() && github.event_name == 'schedule' &&
+        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+      )
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.gitignore

@@ -90,3 +90,4 @@ worktrees/*
 # insta scratch files; the accepted .snap files ARE the assertions and are committed
 *.snap.new
 *.pending-snap
+.mimosa

.vscode/launch.json

@@ -7,7 +7,9 @@
             "name": "Debug(only) executable 'rustfs'",
             "env": {
               "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
-              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on",
+              // Local test volumes share one physical disk.
+              "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
               //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
               // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
               // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -44,6 +46,7 @@
             "env": {
                 "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
                 "RUSTFS_SKIP_BACKGROUND_TASK": "on",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
                 // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
                 // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
@@ -117,7 +120,8 @@
             "env": {
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
-                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
+                "RUSTFS_VOLUMES": "./target/volume/test{1...4}",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -135,41 +139,30 @@
         {
             "type": "lldb",
             "request": "launch",
-            "name": "Debug test_lifecycle_transition_basic",
+            "name": "Debug test_transition_and_restore_flows",
             "cargo": {
                 "args": [
                     "test",
+                    "--no-run",
                     "-p",
                     "rustfs-scanner",
                     "--test",
                     "lifecycle_integration_test",
-                    "serial_tests::test_lifecycle_transition_basic",
                     "-j",
                     "1"
-                ]
+                ],
+                "filter": {
+                    "name": "lifecycle_integration_test",
+                    "kind": "test"
+                }
             },
-            "args": [],
-            "cwd": "${workspaceFolder}"
-        },
-        {
-            "name": "Debug executable target/debug/test",
-            "type":  "lldb",
-            "request": "launch",
-            "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
             "args": [
-              "--skip",
-              "test_lifecycle_expiry_basic",
-              "--skip",
-              "test_lifecycle_expiry_deletemarker",
-              //"--skip",
-              //"test_lifecycle_transition_basic",
-            ],
-            "cwd": "${workspaceFolder}",
-            //"stopAtEntry": false,
-            //"preLaunchTask": "cargo build",
-            "sourceLanguages": [
-                "rust"
+                "serial_tests::test_transition_and_restore_flows",
+                "--exact",
+                "--ignored",
+                "--test-threads=1"
             ],
+            "cwd": "${workspaceFolder}"
         },
         {
             "name": "Debug executable target/debug/rustfs with sse kms",
@@ -285,7 +278,7 @@
                 "RUSTFS_COMPRESSION_ENABLED": "true",
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },
@@ -307,6 +300,7 @@
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
                 "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_REGION": "us-east-1",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
@@ -315,7 +309,7 @@
 
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },

Cargo.toml

@@ -73,8 +73,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.98.1"
-version = "1.0.1"
+rust-version = "1.99.0"
+version = "1.0.3"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -91,56 +91,56 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.1" }
-rustfs-heal = { path = "crates/heal", version = "1.0.1" }
-rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" }
-rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" }
-rustfs-audit = { path = "crates/audit", version = "1.0.1" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.1" }
-rustfs-common = { path = "crates/common", version = "1.0.1" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" }
-rustfs-config = { path = "./crates/config", version = "1.0.1" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.1" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.1" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" }
-rustfs-iam = { path = "crates/iam", version = "1.0.1" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.1" }
-rustfs-license = { path = "crates/license", version = "1.0.1" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" }
-rustfs-kms = { path = "crates/kms", version = "1.0.1" }
-rustfs-lock = { path = "crates/lock", version = "1.0.1" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.1" }
-rustfs-notify = { path = "crates/notify", version = "1.0.1" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.1" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" }
-rustfs-obs = { path = "crates/obs", version = "1.0.1" }
-rustfs-policy = { path = "crates/policy", version = "1.0.1" }
-rustfs-protos = { path = "crates/protos", version = "1.0.1" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.1" }
-rustfs-replication = { path = "crates/replication", version = "1.0.1" }
-rustfs-rio = { path = "crates/rio", version = "1.0.1" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" }
-rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.1" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" }
-rustfs-signer = { path = "crates/signer", version = "1.0.1" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" }
-rustfs-targets = { path = "crates/targets", version = "1.0.1" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" }
-rustfs-utils = { path = "crates/utils", version = "1.0.1" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
+rustfs = { path = "./rustfs", version = "1.0.3" }
+rustfs-heal = { path = "crates/heal", version = "1.0.3" }
+rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.3" }
+rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.3" }
+rustfs-audit = { path = "crates/audit", version = "1.0.3" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.3" }
+rustfs-common = { path = "crates/common", version = "1.0.3" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.3" }
+rustfs-config = { path = "./crates/config", version = "1.0.3" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.3" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.3" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.3" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.3" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.3" }
+rustfs-iam = { path = "crates/iam", version = "1.0.3" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.3" }
+rustfs-license = { path = "crates/license", version = "1.0.3" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.3" }
+rustfs-kms = { path = "crates/kms", version = "1.0.3" }
+rustfs-lock = { path = "crates/lock", version = "1.0.3" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.3" }
+rustfs-notify = { path = "crates/notify", version = "1.0.3" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.3" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.3" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.3" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.3", default-features = false }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.3" }
+rustfs-obs = { path = "crates/obs", version = "1.0.3" }
+rustfs-policy = { path = "crates/policy", version = "1.0.3" }
+rustfs-protos = { path = "crates/protos", version = "1.0.3" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.3" }
+rustfs-replication = { path = "crates/replication", version = "1.0.3" }
+rustfs-rio = { path = "crates/rio", version = "1.0.3" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.3" }
+rustfs-s3-client = { path = "crates/s3-client", version = "1.0.3" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.3" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.3" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.3" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.3" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.3" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.3" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.3" }
+rustfs-signer = { path = "crates/signer", version = "1.0.3" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.3" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.3" }
+rustfs-targets = { path = "crates/targets", version = "1.0.3" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.3" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.3" }
+rustfs-utils = { path = "crates/utils", version = "1.0.3" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.3" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -156,8 +156,9 @@ futures-lite = "2.6.1"
 futures-util = "0.3.34"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.9.0" }
-lapin = { default-features = false, version = "4.12.0" }
-hyper = { version = "1.11.1" }
+lapin = { default-features = false, version = "4.12.2" }
+hyper = { version = "1.12.0" }
+h2 = { version = "0.4.20" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
 hyper-util = { version = "0.1.21" }
 http = "1.5.0"
@@ -166,18 +167,25 @@ http-body-util = "0.1.5"
 memchr = "2.8.3"
 minlz = "1.2.3"
 reqwest = "0.13.5"
-rustfs-kafka-async = { version = "1.3.1" }
+rustfs-kafka-async = { version = "1.4.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.53.1" }
+tokio = { version = "1.53.3" }
 tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
-tokio-util = { version = "0.7.19" }
+tokio-util = { version = "0.7.20" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
+h3 = "0.0.8"
+h3-quinn = "0.0.10"
+quinn = { version = "0.11.12", default-features = false, features = [
+      "futures-io",
+      "runtime-tokio",
+      "rustls-aws-lc-rs",
+] }
 
 # Serialization and Data Formats
 apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
@@ -193,7 +201,7 @@ rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
 serde_ignored = { version = "0.1" }
 serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
-serde_json = { version = "1.0.151" }
+serde_json = { version = "1.0.154" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -217,15 +225,15 @@ rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
 x509-parser = "0.18.1"
 sha1 = "0.11.0"
-sha2 = "0.11.0"
+sha2 = "0.11.1"
 subtle = "2.6"
-zeroize = { version = "1.9.0" }
+zeroize = { version = "1.9.1" }
 proptest = "1"
 
 # Time and Date
 chrono = { version = "0.4.45" }
 humantime = "2.4.0"
-jiff = { version = "0.2.37" }
+jiff = { version = "0.2.38" }
 time = { version = "0.3.55" }
 
 # Database
@@ -244,9 +252,9 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.122.0" }
-aws-sdk-s3 = { default-features = false, version = "1.151.0" }
-aws-sdk-sts = { default-features = false, version = "1.118.0" }
+aws-sdk-kms = { default-features = false, version = "1.123.0" }
+aws-sdk-s3 = { default-features = false, version = "1.152.0" }
+aws-sdk-sts = { default-features = false, version = "1.119.0" }
 aws-smithy-async = { version = "1.3.0" }
 aws-smithy-http-client = { default-features = false, version = "1.5.0" }
 aws-smithy-runtime-api = { version = "1.19.0" }
@@ -261,7 +269,7 @@ crossbeam-queue = "0.3.14"
 crossbeam-channel = "0.5.17"
 crossbeam-deque = "0.8.8"
 crossbeam-utils = "0.8.23"
-datafusion = { default-features = false, version = "55.1.0" }
+datafusion = { default-features = false, version = "55.2.0" }
 derive_builder = "0.20.2"
 ed25519-dalek = "3.0.0"
 enumset = "1.1.14"
@@ -279,7 +287,7 @@ highway = { version = "1.3.0" }
 hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
 lazy_static = "1.5.1"
-libc = "0.2.189"
+libc = "0.2.190"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
 log = "0.4"
@@ -299,7 +307,7 @@ percent-encoding = "2.3.2"
 # Server-side QR rendering for TOTP enrollment, so neither the console nor the
 # CLI needs its own QR encoder. No default features: the image/render backends
 # pull in an image stack this only needs SVG and text output from.
-qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
+qrcode-rs = { version = "2.2.0", default-features = false, features = ["std", "svg"] }
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.3" }
@@ -312,17 +320,17 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" }
 redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
-rust-embed = { version = "8.12.0" }
+rust-embed = { version = "8.13.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { version = "0.17.0", features = ["minio"] }
-s3s-sigv4 = "0.17.0"
+s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
+s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
-smallvec = { version = "1.16.2" }
+smallvec = { version = "1.16.3" }
 compact_str = "0.10.0"
 snap = "1.1.2"
-starshard = { version = "2.3.0" }
+starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
@@ -333,12 +341,12 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.34" }
+tracing-opentelemetry = { version = "0.34.0" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.26.1" }
+uuid = { version = "1.28.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
@@ -346,28 +354,28 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "8.6.0"
-zstd = "0.14.0"
+zip = "9.0.3"
+zstd = "0.14.2"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.2", default-features = false }
-opentelemetry = { version = "0.33.0" }
-opentelemetry-appender-tracing = { version = "0.33.0" }
-opentelemetry-otlp = { version = "0.33.0" }
-opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.33.0" }
-opentelemetry-semantic-conventions = { version = "0.33.0" }
-opentelemetry-stdout = { version = "0.33.0" }
-pyroscope = { version = "2.1.1" }
+dial9 = { version = "0.5.4", default-features = false }
+opentelemetry = { version = "0.33.1" }
+opentelemetry-appender-tracing = { version = "0.33.1" }
+opentelemetry-otlp = { version = "0.33.1" }
+opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.1" }
+opentelemetry-semantic-conventions = { version = "0.33.1" }
+opentelemetry-stdout = { version = "0.33.1" }
+pyroscope = { version = "2.1.2" }
 
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.1.1" }
+suppaftp = { version = "12.2.0" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.63.3" }
+russh = { version = "0.64.1" }
 russh-sftp = "3.0.1"
 
 # WebDAV
@@ -376,9 +384,9 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.28.0", default-features = false }
+hotpath = { version = "0.28.6", default-features = false }
 # Snapshot testing for output format regression detection
-insta = { version = "1.48" }
+insta = { version = "1.49" }
 
 [workspace.metadata.cargo-shear]
 ignored = ["hotpath", "rustfs"]

_typos.toml

@@ -51,6 +51,8 @@ unparseable = "unparseable"
 # Disaster-recovery objectives: recovery time and recovery point.
 RTO = "RTO"
 rto = "rto"
+# rustix OFlags variant (POSIX O_WRONLY equivalent).
+WRONLY = "WRONLY"
 
 [files]
 extend-exclude = []

crates/common/Cargo.toml

@@ -34,6 +34,7 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
 hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 
 [dependencies]
+arc-swap.workspace = true
 hotpath.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }

crates/common/src/globals.rs

@@ -12,9 +12,10 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use arc_swap::ArcSwap;
 use std::collections::HashMap;
-use std::sync::LazyLock;
 use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, LazyLock};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -27,6 +28,42 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 
+#[derive(Clone, Default)]
+pub struct OutboundTlsSnapshot {
+    pub generation: u64,
+    pub root_ca_pem: Option<Vec<u8>>,
+    pub mtls_identity: Option<MtlsIdentityPem>,
+}
+
+static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
+    LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
+
+pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
+    GLOBAL_OUTBOUND_TLS_STATE.load_full()
+}
+
+/// Publish identity, trust and generation together; the individual globals are legacy views.
+pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        if current.generation != 0
+            && current.root_ca_pem == snapshot.root_ca_pem
+            && current.mtls_identity == snapshot.mtls_identity
+        {
+            return Arc::clone(current);
+        }
+        Arc::new(OutboundTlsSnapshot {
+            generation: snapshot.generation.max(current.generation.saturating_add(1)),
+            root_ca_pem: snapshot.root_ca_pem.clone(),
+            mtls_identity: snapshot.mtls_identity.clone(),
+        })
+    });
+    let accepted = get_global_outbound_tls_snapshot();
+    *GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
+    *GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
+    GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
+    accepted.generation
+}
+
 /// Log level to use when reporting cached gRPC connection eviction.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub enum ConnectionEvictionLogLevel {
@@ -85,17 +122,27 @@ pub async fn get_global_addr() -> String {
 /// # Arguments
 /// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
 pub async fn set_global_root_cert(cert: Vec<u8>) {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.root_ca_pem = Some(cert.clone());
+        Arc::new(next)
+    });
     *GLOBAL_ROOT_CERT.write().await = Some(cert);
 }
 
 /// Clear the global root CA certificate for outbound gRPC clients.
 pub async fn clear_global_root_cert() {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.root_ca_pem = None;
+        Arc::new(next)
+    });
     *GLOBAL_ROOT_CERT.write().await = None;
 }
 
 /// Get the global root CA certificate for outbound gRPC clients.
 pub async fn get_global_root_cert() -> Option<Vec<u8>> {
-    GLOBAL_ROOT_CERT.read().await.clone()
+    get_global_outbound_tls_snapshot().root_ca_pem.clone()
 }
 
 /// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -105,22 +152,32 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
 /// # Arguments
 /// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
 pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.mtls_identity = identity.clone();
+        Arc::new(next)
+    });
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
 /// Get the global mTLS identity for outbound gRPC clients.
 pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
-    GLOBAL_MTLS_IDENTITY.read().await.clone()
+    get_global_outbound_tls_snapshot().mtls_identity.clone()
 }
 
 /// Set the global outbound TLS generation.
 pub fn set_global_outbound_tls_generation(generation: u64) {
+    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
+        let mut next = (**current).clone();
+        next.generation = generation;
+        Arc::new(next)
+    });
     GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
 }
 
 /// Get the global outbound TLS generation.
 pub fn get_global_outbound_tls_generation() -> u64 {
-    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
+    get_global_outbound_tls_snapshot().generation
 }
 
 /// Evict a stale/dead connection from the global connection cache.
@@ -194,8 +251,77 @@ pub async fn clear_all_connections() {
 ///
 /// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
 /// When absent, clients proceed with standard server-authenticated TLS.
-#[derive(Clone, Debug)]
+#[derive(Clone, PartialEq, Eq)]
 pub struct MtlsIdentityPem {
     pub cert_pem: Vec<u8>,
     pub key_pem: Vec<u8>,
 }
+
+impl std::fmt::Debug for MtlsIdentityPem {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("MtlsIdentityPem")
+            .field("cert_bytes", &self.cert_pem.len())
+            .field("key_pem", &"[REDACTED]")
+            .finish()
+    }
+}
+
+#[cfg(test)]
+mod tls_snapshot_tests {
+    use super::*;
+
+    #[tokio::test]
+    async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
+        let start = get_global_outbound_tls_generation().saturating_add(1);
+        let snapshot = OutboundTlsSnapshot {
+            generation: start,
+            root_ca_pem: Some(vec![1]),
+            mtls_identity: Some(MtlsIdentityPem {
+                cert_pem: vec![1],
+                key_pem: vec![1],
+            }),
+        };
+        let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
+        assert_eq!(
+            publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
+                generation: first.saturating_add(1),
+                ..snapshot
+            })
+            .await,
+            first
+        );
+        let writer = tokio::spawn(async move {
+            for tag in 2_u8..=255 {
+                publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
+                    generation: start.saturating_add(u64::from(tag)),
+                    root_ca_pem: Some(vec![tag]),
+                    mtls_identity: Some(MtlsIdentityPem {
+                        cert_pem: vec![tag],
+                        key_pem: vec![tag],
+                    }),
+                })
+                .await;
+                tokio::task::yield_now().await;
+            }
+        });
+        while !writer.is_finished() {
+            let state = get_global_outbound_tls_snapshot();
+            let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
+            assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
+            assert_eq!(identity.cert_pem, identity.key_pem);
+            tokio::task::yield_now().await;
+        }
+        writer.await.expect("writer completes");
+    }
+
+    #[test]
+    fn mtls_identity_debug_redacts_private_key() {
+        let identity = MtlsIdentityPem {
+            cert_pem: b"public certificate".to_vec(),
+            key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
+        };
+        let debug = format!("{identity:?}");
+        assert!(!debug.contains("PRIVATE-KEY-MARKER"));
+        assert!(debug.contains("[REDACTED]"));
+    }
+}

crates/config/src/constants/app.rs

@@ -18,10 +18,8 @@ use const_str::concat;
 /// Default value: RustFS
 /// Environment variable: RUSTFS_APP_NAME
 pub const APP_NAME: &str = "RustFS";
-/// Application version
-/// Default value: 1.0.0
-/// Environment variable: RUSTFS_VERSION
-pub const VERSION: &str = "1.0.0";
+/// Cargo package version for this workspace member.
+pub const VERSION: &str = env!("CARGO_PKG_VERSION");
 
 /// Default configuration logger level
 /// Default value: error
@@ -39,12 +37,6 @@ pub const SAMPLE_RATIO: f64 = 1.0;
 /// Default value: 30
 pub const METER_INTERVAL: u64 = 30;
 
-/// Default configuration service version
-/// Default value: 1.0.0
-/// Environment variable: RUSTFS_OBS_SERVICE_VERSION
-/// Uses the same value as VERSION constant
-pub const SERVICE_VERSION: &str = "1.0.0";
-
 /// Default configuration environment
 /// Default value: production
 pub const ENVIRONMENT: &str = "production";
@@ -410,7 +402,6 @@ mod tests {
             APP_NAME,
             VERSION,
             DEFAULT_LOG_LEVEL,
-            SERVICE_VERSION,
             ENVIRONMENT,
             RUSTFS_TLS_KEY,
             RUSTFS_TLS_CERT,
@@ -446,8 +437,7 @@ mod tests {
     fn test_configuration_consistency() {
         // Test configuration consistency
 
-        // Version consistency
-        assert_eq!(VERSION, SERVICE_VERSION, "Application version should match service version");
+        assert_eq!(VERSION, env!("CARGO_PKG_VERSION"));
 
         // Port conflict check
         let ports = [DEFAULT_PORT, DEFAULT_CONSOLE_PORT];

crates/config/src/constants/tls.rs

@@ -207,3 +207,9 @@ pub const ENV_TLS_RELOAD_INTERVAL: &str = "RUSTFS_TLS_RELOAD_INTERVAL";
 
 /// Default interval for TLS certificate reload check
 pub const DEFAULT_TLS_RELOAD_INTERVAL: u64 = 30;
+
+/// Environment variable to enable experimental HTTP/3 support.
+pub const ENV_HTTP3_ENABLE: &str = "RUSTFS_HTTP3_ENABLE";
+
+/// HTTP/3 is disabled unless explicitly enabled.
+pub const DEFAULT_HTTP3_ENABLE: bool = false;

crates/config/src/observability/mod.rs

@@ -51,6 +51,11 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_
 /// PUT records multiple timers and histograms when attribution is active.
 pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
 
+/// Enables sampled in-process GetObject stage summaries without OTLP histograms.
+pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
+pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
+pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
+
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -159,6 +164,15 @@ mod tests {
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
+        assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED");
+        assert_eq!(
+            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE,
+            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"
+        );
+        assert_eq!(
+            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
+            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
+        );
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");

crates/e2e_test/src/cluster_multidrive_pool_test.rs

@@ -33,6 +33,10 @@
 //! tracked separately.
 
 use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use crate::fault_proxy::FaultMode;
+use aws_sdk_s3::Client;
+use std::time::Duration;
+use tokio::time::{Instant, timeout};
 
 type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
 
@@ -144,7 +148,99 @@ async fn cluster_volume_fault_proxy_pass_smoke() -> TestResult {
         cluster.start().await?;
         cluster.create_test_bucket(BUCKET).await?;
         let payload = vec![0x6Du8; 256 * 1024];
-        put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await
+        put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await?;
+
+        // Node 1 owns only two disks; a successful three-vote write must
+        // authenticate at least one disk mutation through node 0's proxy.
+        let client = cluster.create_s3_client(1)?;
+        let client = Client::from_conf(
+            client
+                .config()
+                .to_builder()
+                .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                .build(),
+        );
+        let forwarded_before = proxy.forwarded_bytes();
+        client
+            .put_object()
+            .bucket(BUCKET)
+            .key("volume-proxy/authenticated")
+            .body(bytes::Bytes::copy_from_slice(&payload).into())
+            .send()
+            .await?;
+        assert!(
+            proxy.forwarded_bytes() > forwarded_before,
+            "successful peer-disk publication must traverse the proxy"
+        );
+
+        let dropped_before = proxy.dropped_bytes();
+        proxy.set_mode(FaultMode::Blackhole);
+        assert_eq!(proxy.mode(), FaultMode::Blackhole);
+        let error = timeout(
+            Duration::from_secs(30),
+            client
+                .put_object()
+                .bucket(BUCKET)
+                .key("volume-proxy/rejected")
+                .body(bytes::Bytes::copy_from_slice(&payload).into())
+                .send(),
+        )
+        .await?
+        .expect_err("two local disks cannot commit a write while the peer is blackholed");
+        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+        assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
+        assert!(
+            proxy.dropped_bytes() > dropped_before,
+            "the injected blackhole must discard actual peer traffic"
+        );
+
+        proxy.set_mode(FaultMode::Pass);
+        assert_eq!(proxy.mode(), FaultMode::Pass);
+        let http = reqwest::Client::builder()
+            .no_proxy()
+            .timeout(Duration::from_secs(3))
+            .build()?;
+        let deadline = Instant::now() + Duration::from_secs(30);
+        loop {
+            let response = http.get(format!("{}/health/ready", cluster.nodes[1].url)).send().await?;
+            let status = response.status();
+            let body: serde_json::Value = response.json().await?;
+            // Reachable peers can still have Returning disk handles that reject
+            // writes. Wait for this node's live writer inventory to be Online.
+            if status.as_u16() == 200
+                && body["details"]["storage"]["source"] == "local_runtime"
+                && body["details"]["storage"]["writeQuorum"] == true
+                && body["details"]["storage"]["unavailableDrives"]
+                    .as_array()
+                    .is_some_and(Vec::is_empty)
+                && body["details"]["lock"]["ready"] == true
+            {
+                break;
+            }
+            assert!(Instant::now() < deadline, "peer write quorum did not recover: HTTP {status}, {body}");
+            tokio::time::sleep(Duration::from_millis(200)).await;
+        }
+        client
+            .put_object()
+            .bucket(BUCKET)
+            .key("volume-proxy/recovered")
+            .body(bytes::Bytes::copy_from_slice(&payload).into())
+            .send()
+            .await?;
+        for key in ["volume-proxy/object", "volume-proxy/authenticated", "volume-proxy/recovered"] {
+            let got = client.get_object().bucket(BUCKET).key(key).send().await?;
+            let body = got.body.collect().await?.into_bytes();
+            assert_eq!(body.as_ref(), payload, "proxy recovery must preserve the full object body for {key}");
+        }
+        let rejected = client
+            .head_object()
+            .bucket(BUCKET)
+            .key("volume-proxy/rejected")
+            .send()
+            .await
+            .expect_err("a rejected proxied write must never become visible after recovery");
+        assert_eq!(rejected.raw_response().map(|response| response.status().as_u16()), Some(404));
+        Ok(())
     }
     .await;
 

crates/e2e_test/src/common.rs

@@ -35,7 +35,7 @@ use s3s::Body;
 use serde_json;
 use std::fs as stdfs;
 use std::io::ErrorKind;
-use std::net::SocketAddr;
+use std::net::{Ipv6Addr, SocketAddr};
 use std::path::{Path, PathBuf};
 use std::process::{Child, Command, Stdio};
 use std::sync::Once;
@@ -1442,7 +1442,11 @@ impl RustFSTestClusterEnvironment {
             return Err(format!("a volume proxy is already configured for node {node_idx}").into());
         }
         let target = self.nodes[node_idx].address.parse::<SocketAddr>()?;
-        let proxy = crate::fault_proxy::FaultProxy::start(target).await?;
+        // Endpoint locality and RPC audience both use the advertised authority.
+        // Keep the node's port while the second loopback IP separates the proxy
+        // listener from the direct IPv4 server and S3 client address.
+        let listen = SocketAddr::from((Ipv6Addr::LOCALHOST, target.port()));
+        let proxy = crate::fault_proxy::FaultProxy::start_on(listen, target).await?;
         self.volume_proxy_addresses[node_idx] = Some(proxy.local_addr());
         Ok(proxy)
     }
@@ -1538,6 +1542,15 @@ impl RustFSTestClusterEnvironment {
         self.start_node_from_binary(node_idx, &binary_path).await
     }
 
+    /// Start a process and wait only for TCP, allowing tests to bring up a
+    /// quorum one node at a time before the S3 service can become ready.
+    pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        let binary_path = rustfs_binary_path();
+        let volumes_arg = self.build_volumes_arg();
+        self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
+        self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
+    }
+
     /// Start one stopped cluster node with a specific RustFS binary while
     /// preserving the cluster's volume layout and that node's data directory.
     pub async fn start_node_from_binary(
@@ -2467,6 +2480,20 @@ mod tests {
 
         assert!(volumes.contains(&proxied), "volumes must use the proxy address: {volumes}");
         assert!(!volumes.contains(&direct), "volumes must not retain the direct address: {volumes}");
+        assert_eq!(proxy.local_addr().port(), proxy.target_addr().port());
+        assert_ne!(proxy.local_addr().ip(), proxy.target_addr().ip());
+        assert_eq!(env.nodes[0].address, direct, "S3 clients must retain the direct address");
+        let mut endpoint =
+            crate::storage_api::Endpoint::try_from(volumes.as_str()).expect("proxied volume should be a valid endpoint");
+        endpoint
+            .update_is_local(proxy.target_addr().port())
+            .expect("proxied endpoint locality should resolve");
+        assert!(endpoint.is_local, "the proxied node must keep its local disk and first-disk authority");
+        assert_eq!(
+            endpoint.host_port(),
+            proxied,
+            "the local RPC audience must be the advertised proxy authority"
+        );
 
         proxy.shutdown().await;
     }

crates/e2e_test/src/distributed/harness.rs

@@ -860,10 +860,12 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T
         if status.is_success() {
             break response;
         }
-        if status == StatusCode::INTERNAL_SERVER_ERROR
-            && response.contains("requires a live fleet capability proof")
-            && Instant::now() < deadline
-        {
+        // The init stage answers 503 + Retry-After while the cluster capability
+        // proof is still missing; older binaries surfaced the same condition as
+        // a 500 carrying the proof marker.
+        let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE
+            || (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof"));
+        if waiting_for_readiness && Instant::now() < deadline {
             sleep(Duration::from_secs(1)).await;
             continue;
         }

crates/e2e_test/src/distributed/upgrade_test.rs

@@ -31,6 +31,7 @@ use super::harness::{
 use crate::common::{
     AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
 };
+use crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness;
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::ProvideErrorMetadata;
 use std::path::{Path, PathBuf};
@@ -400,6 +401,19 @@ async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> Tes
     Ok(())
 }
 
+async fn replace_node_and_wait_for_write_readiness(dist: &mut DistCluster, seed: &UpgradeSeed, node_idx: usize) -> TestResult {
+    dist.replace_node_with_current_binary(node_idx).await?;
+    // Node readiness proves reads can succeed; a rolling writer must also
+    // recover before replacing the next peer or checking upgrade contracts.
+    wait_for_upgrade_write_readiness(
+        &dist.clients()?,
+        &seed.history_bucket,
+        &format!("replaced-node-{node_idx}"),
+        CREDENTIAL_TIMEOUT,
+    )
+    .await
+}
+
 #[tokio::test]
 async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> TestResult {
     init_logging();
@@ -410,15 +424,15 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     let seed = seed_history_and_iam(&dist).await?;
 
-    dist.replace_node_with_current_binary(0).await?;
+    replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
     assert_history_and_iam(&dist, &seed, "one-current-node").await?;
 
     for node_idx in [1, 2] {
-        dist.replace_node_with_current_binary(node_idx).await?;
+        replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
     }
     assert_history_and_iam(&dist, &seed, "one-previous-node").await?;
 
-    dist.replace_node_with_current_binary(3).await?;
+    replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
     assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
     Ok(())
 }

crates/e2e_test/src/fake_s3_target/mod.rs

@@ -50,8 +50,8 @@ use std::collections::{BTreeMap, HashMap, VecDeque};
 use std::error::Error;
 use std::io;
 use std::net::SocketAddr;
-use std::sync::{Arc, Mutex, MutexGuard};
-use std::time::{Duration, SystemTime};
+use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
+use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
 use tokio::net::TcpListener;
 use tokio::sync::{OwnedSemaphorePermit, Semaphore, watch};
 use tokio::task::JoinHandle;
@@ -363,6 +363,14 @@ pub struct RequestRecord {
     pub upload_id: Option<String>,
     pub part_number: Option<i32>,
     pub content_length: Option<u64>,
+    pub source_version_id: Option<String>,
+    pub source_mtime: Option<String>,
+    pub source_etag: Option<String>,
+    pub source_replication_request: bool,
+    pub journaled_at_unix_millis: Option<u64>,
+    /// Computed by the target, before response stall/disconnect faults. This
+    /// does not prove that any response bytes reached the client.
+    pub prepared_response: Option<PreparedResponse>,
     pub consumed_bytes: Option<usize>,
     pub replication_timestamps: ReplicationTimestampHeaders,
     pub proxy_headers: ProxyHeaderSnapshot,
@@ -381,6 +389,20 @@ pub struct RequestRecord {
     pub fault: Option<FaultAction>,
 }
 
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct PreparedResponse {
+    pub status: u16,
+    pub version_id: Option<String>,
+    pub etag: Option<String>,
+    pub last_modified: Option<String>,
+    pub content_length: Option<u64>,
+    pub prepared_at_unix_millis: Option<u64>,
+    pub elapsed: Duration,
+}
+
+#[derive(Clone, Default)]
+struct RequestSequence(Arc<OnceLock<u64>>);
+
 /// Integrity and framing headers of an upload. A remote target's acceptance
 /// rules key off exactly these (rustfs#6853: `aws-chunked` framing stored
 /// verbatim; rustfs#7082: Object Lock PUTs need `Content-MD5` or
@@ -430,6 +452,10 @@ impl TransportSnapshot {
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
 struct JournaledHeaders {
     content_length: Option<u64>,
+    source_version_id: Option<String>,
+    source_mtime: Option<String>,
+    source_etag: Option<String>,
+    source_replication_request: bool,
     replication_timestamps: ReplicationTimestampHeaders,
     proxy_headers: ProxyHeaderSnapshot,
     transport: TransportSnapshot,
@@ -444,6 +470,10 @@ impl JournaledHeaders {
                 .get(CONTENT_LENGTH)
                 .and_then(|value| value.to_str().ok())
                 .and_then(|value| value.parse().ok()),
+            source_version_id: header_value(headers, &SOURCE_VERSION_ID_HEADERS).map(bounded_journal_value),
+            source_mtime: header_value(headers, &SOURCE_MTIME_HEADERS).map(bounded_journal_value),
+            source_etag: header_value(headers, &SOURCE_ETAG_HEADERS).map(bounded_journal_value),
+            source_replication_request: header_value(headers, &SOURCE_REPLICATION_REQUEST_HEADERS).as_deref() == Some("true"),
             replication_timestamps: ReplicationTimestampHeaders::from_headers(headers),
             proxy_headers: ProxyHeaderSnapshot::from_headers(headers),
             transport: TransportSnapshot::from_headers(headers),
@@ -511,6 +541,10 @@ struct StoreState {
     /// version id the target never had answers 404 `NoSuchVersion` instead of
     /// the idempotent 204 RustFS/MinIO give.
     reject_unknown_version_deletes: bool,
+    /// Models MinIO and its derivatives (Storj): `X-Minio-Internal-` is a reserved
+    /// metadata prefix, so a PutObject or UploadPart carrying any such request
+    /// header is refused with `InvalidArgument` (rustfs#8362).
+    reject_reserved_internal_headers: bool,
     limits: StoreLimits,
     buckets: HashMap<String, BucketState>,
     uploads: HashMap<String, MultipartState>,
@@ -738,6 +772,7 @@ impl FakeS3Target {
         let connection_limit = Arc::new(Semaphore::new(MAX_CONNECTIONS));
         let (active_connections_tx, active_connections) = watch::channel(0usize);
         let task_connection_limit = Arc::clone(&connection_limit);
+        let task_control = Arc::clone(&control);
 
         let task = tokio::spawn(async move {
             let mut connections = tokio::task::JoinSet::new();
@@ -751,6 +786,7 @@ impl FakeS3Target {
                         };
                         active_connections_tx.send_modify(|count| *count += 1);
                         let service = service.clone();
+                        let control = Arc::clone(&task_control);
                         let active_connections = active_connections_tx.clone();
                         let mut connection_gate = connection_gate.clone();
                         connections.spawn(async move {
@@ -764,7 +800,7 @@ impl FakeS3Target {
                             {
                                 return;
                             }
-                            let handler = service_fn(move |request| handle_request(request, service.clone()));
+                            let handler = service_fn(move |request| handle_request(request, service.clone(), Arc::clone(&control)));
                             let mut connection = http1::Builder::new();
                             connection
                                 .keep_alive(false)
@@ -975,6 +1011,43 @@ impl FakeS3Target {
         lock(&self.backend.store).reject_unknown_version_deletes = enabled;
     }
 
+    /// Refuse every PutObject / UploadPart that carries an `x-minio-internal-*`
+    /// request header, the way MinIO and Storj do.
+    pub fn reject_reserved_internal_headers(&self, enabled: bool) {
+        lock(&self.backend.store).reject_reserved_internal_headers = enabled;
+    }
+
+    /// Keys of every stored, non-delete object in the bucket, sorted.
+    pub fn stored_keys(&self, bucket: &str) -> Vec<String> {
+        let state = lock(&self.backend.store);
+        let mut keys: Vec<String> = state
+            .buckets
+            .get(bucket)
+            .map(|bucket| {
+                bucket
+                    .objects
+                    .iter()
+                    .filter(|(_, versions)| versions.last().is_some_and(|version| !version.delete_marker))
+                    .map(|(key, _)| key.clone())
+                    .collect()
+            })
+            .unwrap_or_default();
+        keys.sort();
+        keys
+    }
+
+    /// Body and user metadata (names as stored, without the `x-amz-meta-` prefix)
+    /// of the latest non-delete version of an object.
+    pub fn stored_object(&self, bucket: &str, key: &str) -> Option<(Bytes, HashMap<String, String>)> {
+        lock(&self.backend.store)
+            .buckets
+            .get(bucket)
+            .and_then(|bucket| bucket.objects.get(key))
+            .and_then(|versions| versions.last())
+            .filter(|version| !version.delete_marker)
+            .map(|version| (version.body.clone(), version.metadata.clone().unwrap_or_default()))
+    }
+
     pub fn require_checksum_for_object_lock(&self, enabled: bool) {
         lock(&self.backend.store).require_checksum_for_object_lock = enabled;
     }
@@ -1161,7 +1234,10 @@ impl S3Access for FaultAccess {
         let operation = operation_from_s3_name(context.s3_op().name());
         let journaled = JournaledHeaders::from_headers(context.headers());
         let aws_chunked = journaled.transport.aws_chunked;
-        let fault = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
+        let (sequence, fault) = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
+        if let Some(slot) = context.extensions_mut().get::<RequestSequence>() {
+            let _ = slot.0.set(sequence);
+        }
         if let Some(status) = fault.as_ref().and_then(|fault| scripted_status(&fault.action)) {
             return Err(scripted_status_error(status));
         }
@@ -1174,6 +1250,15 @@ impl S3Access for FaultAccess {
                 "this target does not decode aws-chunked request bodies; send a plain signed payload with an exact Content-Length"
             ));
         }
+        if matches!(operation, Operation::PutObject | Operation::UploadPart)
+            && lock(&self.store).reject_reserved_internal_headers
+            && context
+                .headers()
+                .keys()
+                .any(|name| name.as_str().starts_with("x-minio-internal-"))
+        {
+            return Err(s3s::s3_error!(InvalidArgument, "Your metadata headers are not supported."));
+        }
         let prebody_permit = if operation == Operation::CompleteMultipartUpload {
             Some(
                 timeout(
@@ -1228,7 +1313,7 @@ fn record_request(
     method: Method,
     parsed: ParsedRequest,
     headers: JournaledHeaders,
-) -> Option<RequestFault> {
+) -> (u64, Option<RequestFault>) {
     let mut state = lock(control);
     let action = parsed
         .key
@@ -1251,6 +1336,12 @@ fn record_request(
         upload_id: parsed.upload_id.map(bounded_journal_value),
         part_number: parsed.part_number,
         content_length: headers.content_length,
+        source_version_id: headers.source_version_id,
+        source_mtime: headers.source_mtime,
+        source_etag: headers.source_etag,
+        source_replication_request: headers.source_replication_request,
+        journaled_at_unix_millis: unix_millis(),
+        prepared_response: None,
         consumed_bytes: None,
         replication_timestamps: headers.replication_timestamps,
         proxy_headers: headers.proxy_headers,
@@ -1261,7 +1352,14 @@ fn record_request(
         continuation_token: parsed.continuation_token.map(bounded_journal_value),
         fault: action.clone(),
     });
-    action.map(|action| RequestFault { sequence, action })
+    (sequence, action.map(|action| RequestFault { sequence, action }))
+}
+
+fn unix_millis() -> Option<u64> {
+    SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .ok()
+        .and_then(|elapsed| u64::try_from(elapsed.as_millis()).ok())
 }
 
 fn bounded_journal_value(mut value: String) -> String {
@@ -1269,14 +1367,31 @@ fn bounded_journal_value(mut value: String) -> String {
     while !value.is_char_boundary(end) {
         end -= 1;
     }
-    value.truncate(end);
-    value
+    if value.capacity() > MAX_RETAINED_IDENTIFIER_BYTES {
+        value[..end].to_owned()
+    } else {
+        value.truncate(end);
+        value
+    }
 }
 
-async fn handle_request(request: Request<Incoming>, service: S3Service) -> Result<Response<Body>, BoxError> {
-    let mut response = timeout(MAX_REQUEST_DURATION, call_s3(service, request.map(Body::from)))
-        .await
-        .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
+async fn handle_request(
+    mut request: Request<Incoming>,
+    service: S3Service,
+    control: Arc<Mutex<ControlState>>,
+) -> Result<Response<Body>, BoxError> {
+    let started = Instant::now();
+    let sequence = RequestSequence::default();
+    request.extensions_mut().insert(sequence.clone());
+    let mut response = timeout(MAX_REQUEST_DURATION, async {
+        let result = call_s3(service, request.map(Body::from)).await;
+        if let (Ok(response), Some(sequence)) = (&result, sequence.0.get()) {
+            update_prepared_response(&control, *sequence, response, started.elapsed());
+        }
+        result
+    })
+    .await
+    .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
     if response.headers_mut().remove(DISCONNECT_HEADER).is_some() {
         return Err(io::Error::new(io::ErrorKind::ConnectionAborted, "fake target scripted disconnect").into());
     }
@@ -1531,6 +1646,25 @@ fn update_consumed(control: &Mutex<ControlState>, sequence: u64, consumed: usize
     }
 }
 
+fn update_prepared_response(control: &Mutex<ControlState>, sequence: u64, response: &Response<Body>, elapsed: Duration) {
+    let prepared = PreparedResponse {
+        status: response.status().as_u16(),
+        version_id: header_value(response.headers(), &["x-amz-version-id"]).map(bounded_journal_value),
+        etag: header_value(response.headers(), &[ETAG.as_str()]).map(bounded_journal_value),
+        last_modified: header_value(response.headers(), &[LAST_MODIFIED.as_str()]).map(bounded_journal_value),
+        content_length: response
+            .headers()
+            .get(CONTENT_LENGTH)
+            .and_then(|value| value.to_str().ok())
+            .and_then(|value| value.parse().ok()),
+        prepared_at_unix_millis: unix_millis(),
+        elapsed,
+    };
+    if let Some(record) = lock(control).requests.iter_mut().find(|record| record.sequence == sequence) {
+        record.prepared_response = Some(prepared);
+    }
+}
+
 async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<ControlState>) -> S3Result<()> {
     match fault.map(|fault| &fault.action) {
         Some(action @ (FaultAction::Status(_) | FaultAction::ResponseStatus(_))) => {
@@ -3230,10 +3364,38 @@ mod tests {
         let retry = client.clone();
         requests.spawn(async move { get_bytes(&retry, bucket, "held", None).await });
         timeout(Duration::from_secs(2), gate.entered.wait_for(|count| *count == 2)).await??;
+        let held_retry = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("held"))
+            .expect("the held retry must be journaled");
+        assert_eq!(held_retry.prepared_response, None);
         assert_eq!(
             timeout(Duration::from_secs(2), get_bytes(&client, bucket, "unrelated", None)).await??,
             Bytes::from_static(b"payload")
         );
+        let unrelated = target
+            .requests()
+            .into_iter()
+            .find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("unrelated"))
+            .expect("the unrelated GET must be journaled");
+        let unrelated_response = unrelated.prepared_response.expect("the unrelated GET prepared a response");
+        assert_eq!(unrelated_response.status, 200);
+        assert_eq!(
+            unrelated_response.version_id.as_deref(),
+            Some(target.stored_versions(bucket, "unrelated")[0].0.as_str())
+        );
+        assert!(held_retry.sequence < unrelated.sequence);
+        assert_eq!(
+            target
+                .requests()
+                .into_iter()
+                .find(|record| record.sequence == held_retry.sequence)
+                .expect("the held retry remains journaled")
+                .prepared_response,
+            None
+        );
         assert!(requests.try_join_next().is_none(), "the retry must remain behind the gate");
         drop(gate);
         assert_eq!(
@@ -3242,6 +3404,18 @@ mod tests {
                 .expect("retried GET task")??,
             Bytes::from_static(b"payload")
         );
+        let held_response = target
+            .requests()
+            .into_iter()
+            .find(|record| record.sequence == held_retry.sequence)
+            .and_then(|record| record.prepared_response)
+            .expect("the released retry prepared its own response");
+        assert_eq!(held_response.status, 200);
+        assert_eq!(
+            held_response.version_id.as_deref(),
+            Some(target.stored_versions(bucket, "held")[0].0.as_str())
+        );
+        assert_ne!(held_response.version_id, unrelated_response.version_id);
         assert_eq!(get_bytes(&client, bucket, "held", None).await?, Bytes::from_static(b"payload"));
         assert_eq!(target.count_requests(Operation::GetObject, "held"), 3);
         Ok(())
@@ -3586,6 +3760,23 @@ mod tests {
             .await?;
         assert_eq!(first_head.content_length(), Some(5));
         assert_eq!(first_head.version_id(), first.version_id.as_deref());
+        let journaled_head = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::HeadObject)
+            .expect("the version HEAD must be journaled");
+        assert_eq!(journaled_head.version_id, first.version_id);
+        assert!(journaled_head.journaled_at_unix_millis.is_some());
+        let prepared_head = journaled_head
+            .prepared_response
+            .expect("the version HEAD prepared a response");
+        assert_eq!(prepared_head.status, 200);
+        assert_eq!(prepared_head.version_id.as_deref(), first_head.version_id());
+        assert_eq!(prepared_head.etag.as_deref(), first_head.e_tag());
+        assert_eq!(prepared_head.content_length, Some(5));
+        assert!(prepared_head.last_modified.is_some());
+        assert!(prepared_head.prepared_at_unix_millis.is_some());
         assert_eq!(
             get_bytes(&client, "target-bucket", "nested/key", None).await?,
             Bytes::from_static(b"second")
@@ -3611,6 +3802,20 @@ mod tests {
             .await
             .expect_err("deleted version must not remain readable");
         assert_sdk_error!(deleted, 404, "NoSuchVersion");
+        let deleted_record = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::GetObject)
+            .expect("the deleted version GET must be journaled");
+        assert_eq!(deleted_record.version_id, first.version_id);
+        assert_eq!(
+            deleted_record
+                .prepared_response
+                .expect("the error response was prepared")
+                .status,
+            404
+        );
         let marker = client
             .delete_object()
             .bucket("target-bucket")
@@ -4042,7 +4247,21 @@ mod tests {
             .await
             .expect_err("wrong part ETag must fail without consuming the upload");
         assert_sdk_error!(invalid_complete, 400, "InvalidPart");
-        client
+        let rejected_complete = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::CompleteMultipartUpload)
+            .expect("the rejected completion must be journaled");
+        assert_eq!(rejected_complete.upload_id.as_deref(), Some(upload_id.as_str()));
+        assert_eq!(
+            rejected_complete
+                .prepared_response
+                .expect("the rejection was prepared")
+                .status,
+            400
+        );
+        let successful_complete = client
             .complete_multipart_upload()
             .bucket("target-bucket")
             .key("multipart")
@@ -4050,6 +4269,18 @@ mod tests {
             .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(parts)).build())
             .send()
             .await?;
+        let completed_record = target
+            .requests()
+            .into_iter()
+            .rev()
+            .find(|record| record.operation == Operation::CompleteMultipartUpload)
+            .expect("the successful completion must be journaled");
+        assert_eq!(completed_record.upload_id.as_deref(), Some(upload_id.as_str()));
+        let prepared_complete = completed_record
+            .prepared_response
+            .expect("the completion response was prepared");
+        assert_eq!(prepared_complete.status, 200);
+        assert_eq!(prepared_complete.version_id.as_deref(), successful_complete.version_id());
         let completed = get_bytes(&client, "target-bucket", "multipart", None).await?;
         assert_eq!(completed.len(), MIN_MULTIPART_PART_BYTES + 5);
         assert_eq!(&completed[..MIN_MULTIPART_PART_BYTES], first_body.as_ref());
@@ -4430,6 +4661,32 @@ mod tests {
             .send()
             .await?;
         assert_eq!(completed.version_id(), Some(completed_version.as_str()));
+        let receipt_records = target.requests();
+        let receipt_create = receipt_records
+            .iter()
+            .find(|record| {
+                record.operation == Operation::CreateMultipartUpload && record.key.as_deref() == Some("multipart-receipt-time")
+            })
+            .expect("the source version must be retained on MPU creation");
+        assert_eq!(receipt_create.source_version_id.as_deref(), Some(completed_version.as_str()));
+        let receipt_complete = receipt_records
+            .iter()
+            .find(|record| {
+                record.operation == Operation::CompleteMultipartUpload
+                    && record.upload_id.as_deref() == Some(receipt_upload.as_str())
+            })
+            .expect("the source mtime must be retained on completion");
+        assert_eq!(receipt_complete.source_mtime.as_deref(), Some("1970-01-01T00:00:00Z"));
+        assert!(receipt_complete.source_replication_request);
+        assert_eq!(
+            receipt_complete
+                .prepared_response
+                .as_ref()
+                .expect("the receipt-time response was prepared")
+                .version_id
+                .as_deref(),
+            completed.version_id()
+        );
         assert_eq!(
             get_bytes(&client, "target-bucket", "multipart-receipt-time", None).await?,
             Bytes::from_static(b"completed"),
@@ -4648,6 +4905,23 @@ mod tests {
             .expect_err("disconnected request must not store an object");
         assert_sdk_error!(disconnected, 404, "NotFound");
 
+        target.inject_for_key(Operation::PutObject, "lost-response", FaultAction::DisconnectAfterResponse, 1);
+        client
+            .put_object()
+            .bucket("target-bucket")
+            .key("lost-response")
+            .body(ByteStream::from_static(b"stored"))
+            .send()
+            .await
+            .expect_err("a prepared response does not prove client delivery");
+        let lost_response_head = client
+            .head_object()
+            .bucket("target-bucket")
+            .key("lost-response")
+            .send()
+            .await?;
+        assert_eq!(lost_response_head.content_length(), Some(6));
+
         assert_eq!(
             parse_request(&Method::PUT, &"/target-bucket/key?uploadId=upload&partNumber=bad".parse::<Uri>()?).operation,
             Operation::Unknown
@@ -4661,6 +4935,28 @@ mod tests {
                 .count(),
             4
         );
+        assert!(
+            requests
+                .iter()
+                .filter(|request| request.fault == Some(FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE)))
+                .all(|request| {
+                    request
+                        .prepared_response
+                        .as_ref()
+                        .is_some_and(|response| response.status == 503)
+                })
+        );
+        let lost_response = requests
+            .iter()
+            .find(|request| request.operation == Operation::PutObject && request.key.as_deref() == Some("lost-response"))
+            .expect("the disconnected response must be journaled");
+        assert_eq!(lost_response.fault, Some(FaultAction::DisconnectAfterResponse));
+        let prepared_lost_response = lost_response
+            .prepared_response
+            .as_ref()
+            .expect("the target prepared the successful response");
+        assert_eq!(prepared_lost_response.status, 200);
+        assert_eq!(prepared_lost_response.version_id.as_deref(), lost_response_head.version_id());
         assert!(
             requests
                 .iter()
@@ -4851,8 +5147,44 @@ mod tests {
             assert_eq!(records.len(), MAX_REQUEST_RECORDS);
             assert_eq!(records.front().map(|record| record.sequence), Some(2));
             assert_eq!(records.back().map(|record| record.sequence), Some((MAX_REQUEST_RECORDS + 1) as u64));
+            let mut oversized_allocation = String::with_capacity(MAX_RETAINED_IDENTIFIER_BYTES * 16);
+            oversized_allocation.push_str("short");
+            let short_value = bounded_journal_value(oversized_allocation);
+            assert_eq!(short_value, "short");
+            assert!(short_value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
+            let mut response = Response::new(Body::empty());
+            for name in ["x-amz-version-id", "etag", "last-modified"] {
+                response.headers_mut().insert(
+                    name,
+                    HeaderValue::from_str(&"r".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
+                );
+            }
+            response.headers_mut().insert(CONTENT_LENGTH, HeaderValue::from_static("17"));
+            response
+                .headers_mut()
+                .insert("authorization", HeaderValue::from_static("unrecorded-response-authorization"));
+            response.headers_mut().insert(
+                "x-amz-server-side-encryption-customer-key",
+                HeaderValue::from_static("unrecorded-response-customer-key"),
+            );
+            update_prepared_response(&control, 1, &response, Duration::ZERO);
+            update_prepared_response(&control, u64::MAX, &response, Duration::ZERO);
+            assert_eq!(lock(&control).requests, records, "misses and evicted records must not be resurrected");
             let bounded_control = Mutex::new(ControlState::default());
             let utf8_boundary = format!("{}é", "a".repeat(MAX_RETAINED_IDENTIFIER_BYTES - 1));
+            let mut source_headers = HeaderMap::new();
+            for name in ["x-rustfs-source-version-id", "x-minio-source-mtime", "x-rustfs-source-etag"] {
+                source_headers.insert(
+                    name,
+                    HeaderValue::from_str(&"s".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
+                );
+            }
+            source_headers.insert("x-minio-source-replication-request", HeaderValue::from_static("true"));
+            source_headers.insert("authorization", HeaderValue::from_static("unrecorded-request-authorization"));
+            source_headers.insert(
+                "x-amz-server-side-encryption-customer-key",
+                HeaderValue::from_static("unrecorded-request-customer-key"),
+            );
             record_request(
                 &bounded_control,
                 Operation::GetObject,
@@ -4865,8 +5197,9 @@ mod tests {
                     upload_id: Some("u".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)),
                     ..Default::default()
                 },
-                JournaledHeaders::default(),
+                JournaledHeaders::from_headers(&source_headers),
             );
+            update_prepared_response(&bounded_control, 1, &response, Duration::from_millis(7));
             {
                 let bounded_records = lock(&bounded_control);
                 let bounded = &bounded_records.requests[0];
@@ -4874,7 +5207,39 @@ mod tests {
                 assert_eq!(bounded.key.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES - 1));
                 assert_eq!(bounded.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
                 assert_eq!(bounded.upload_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(bounded.source_version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(bounded.source_mtime.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(bounded.source_etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                for value in [&bounded.source_version_id, &bounded.source_mtime, &bounded.source_etag]
+                    .into_iter()
+                    .flatten()
+                {
+                    assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
+                }
+                assert!(bounded.source_replication_request);
+                let prepared = bounded
+                    .prepared_response
+                    .as_ref()
+                    .expect("the retained record receives its response");
+                assert_eq!(prepared.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(prepared.etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                assert_eq!(prepared.last_modified.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
+                for value in [&prepared.version_id, &prepared.etag, &prepared.last_modified]
+                    .into_iter()
+                    .flatten()
+                {
+                    assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
+                }
+                assert_eq!(prepared.content_length, Some(17));
+                assert_eq!(prepared.elapsed, Duration::from_millis(7));
+                assert!(
+                    !format!("{bounded:?}").contains("unrecorded-"),
+                    "authorization and customer keys must not enter the journal"
+                );
             }
+            lock(&bounded_control).requests.clear();
+            update_prepared_response(&bounded_control, 1, &response, Duration::ZERO);
+            assert!(lock(&bounded_control).requests.is_empty(), "a drained record must not be resurrected");
             let target = FakeS3Target::start().await.expect("target");
             target.create_bucket("b".repeat(63));
             let empty_bucket = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {

crates/e2e_test/src/fault_proxy.rs

@@ -61,6 +61,8 @@
 
 use std::io;
 use std::net::{Ipv4Addr, SocketAddr};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
 use std::time::Duration;
 
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
@@ -108,19 +110,33 @@ pub struct FaultProxy {
     mode_tx: watch::Sender<FaultMode>,
     shutdown_tx: watch::Sender<bool>,
     accept_task: JoinHandle<()>,
+    traffic: Arc<ProxyTraffic>,
+}
+
+#[derive(Default)]
+struct ProxyTraffic {
+    forwarded_bytes: AtomicUsize,
+    dropped_bytes: AtomicUsize,
 }
 
 impl FaultProxy {
     /// Bind a listener on `127.0.0.1:0` and start forwarding accepted
     /// connections to `target`. Starts in [`FaultMode::Pass`].
     pub async fn start(target: SocketAddr) -> io::Result<Self> {
-        let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+        Self::start_on(SocketAddr::from((Ipv4Addr::LOCALHOST, 0)), target).await
+    }
+
+    /// Bind a specific address so advertised volume endpoints can retain the
+    /// server's port and authenticated RPC identity on another loopback IP.
+    pub async fn start_on(listen: SocketAddr, target: SocketAddr) -> io::Result<Self> {
+        let listener = TcpListener::bind(listen).await?;
         let listen_addr = listener.local_addr()?;
 
         let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
         let (shutdown_tx, shutdown_rx) = watch::channel(false);
 
-        let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
+        let traffic = Arc::new(ProxyTraffic::default());
+        let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx, traffic.clone()));
 
         Ok(Self {
             listen_addr,
@@ -128,9 +144,20 @@ impl FaultProxy {
             mode_tx,
             shutdown_tx,
             accept_task,
+            traffic,
         })
     }
 
+    /// Bytes actually forwarded in either direction, excluding dropped traffic.
+    pub fn forwarded_bytes(&self) -> usize {
+        self.traffic.forwarded_bytes.load(Ordering::Relaxed)
+    }
+
+    /// Bytes actually discarded by a blackhole or one-way partition.
+    pub fn dropped_bytes(&self) -> usize {
+        self.traffic.dropped_bytes.load(Ordering::Relaxed)
+    }
+
     /// Local address the proxy is listening on. Point clients here instead of
     /// the real target to route their traffic through the proxy.
     pub fn local_addr(&self) -> SocketAddr {
@@ -171,6 +198,7 @@ async fn accept_loop(
     target: SocketAddr,
     mode_rx: watch::Receiver<FaultMode>,
     mut shutdown_rx: watch::Receiver<bool>,
+    traffic: Arc<ProxyTraffic>,
 ) {
     loop {
         tokio::select! {
@@ -188,6 +216,7 @@ async fn accept_loop(
                             target,
                             mode_rx.clone(),
                             shutdown_rx.clone(),
+                            traffic.clone(),
                         ));
                     }
                     // A transient accept error should not tear the proxy down.
@@ -205,6 +234,7 @@ async fn handle_connection(
     target: SocketAddr,
     mode_rx: watch::Receiver<FaultMode>,
     shutdown_rx: watch::Receiver<bool>,
+    traffic: Arc<ProxyTraffic>,
 ) {
     let server = match TcpStream::connect(target).await {
         Ok(s) => s,
@@ -221,8 +251,9 @@ async fn handle_connection(
         server_wr,
         mode_rx.clone(),
         shutdown_rx.clone(),
+        traffic.clone(),
     ));
-    let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
+    let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx, traffic));
 
     let _ = up.await;
     let _ = down.await;
@@ -237,6 +268,7 @@ async fn pump<R, W>(
     mut to: W,
     mut mode_rx: watch::Receiver<FaultMode>,
     mut shutdown_rx: watch::Receiver<bool>,
+    traffic: Arc<ProxyTraffic>,
 ) where
     R: AsyncReadExt + Unpin,
     W: AsyncWriteExt + Unpin,
@@ -257,15 +289,22 @@ async fn pump<R, W>(
         let mode = *mode_rx.borrow_and_update();
         match mode {
             // Whole connection is a black hole: drain and drop.
-            FaultMode::Blackhole => continue,
+            FaultMode::Blackhole => {
+                traffic.dropped_bytes.fetch_add(n, Ordering::Relaxed);
+                continue;
+            }
             // This direction is partitioned off: drop its bytes; the peer keeps
             // reading nothing while the other direction may still flow.
-            FaultMode::Partition(blocked) if blocked == dir => continue,
+            FaultMode::Partition(blocked) if blocked == dir => {
+                traffic.dropped_bytes.fetch_add(n, Ordering::Relaxed);
+                continue;
+            }
             FaultMode::Latency(delay) => {
                 tokio::time::sleep(delay).await;
                 if to.write_all(&buf[..n]).await.is_err() {
                     break;
                 }
+                traffic.forwarded_bytes.fetch_add(n, Ordering::Relaxed);
                 if to.flush().await.is_err() {
                     break;
                 }
@@ -274,6 +313,7 @@ async fn pump<R, W>(
                 if to.write_all(&buf[..n]).await.is_err() {
                     break;
                 }
+                traffic.forwarded_bytes.fetch_add(n, Ordering::Relaxed);
                 if to.flush().await.is_err() {
                     break;
                 }
@@ -285,7 +325,6 @@ async fn pump<R, W>(
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::sync::Arc;
     use std::time::Instant;
     use tokio::sync::Mutex;
 
@@ -375,6 +414,8 @@ mod tests {
 
         assert_eq!(got, payload, "pass mode must forward bytes unchanged");
         assert_eq!(echo.received().await, payload, "server must have received the bytes");
+        assert_eq!(proxy.forwarded_bytes(), payload.len() * 2);
+        assert_eq!(proxy.dropped_bytes(), 0);
 
         proxy.shutdown().await;
     }
@@ -414,6 +455,8 @@ mod tests {
 
         assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
         assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
+        assert_eq!(proxy.forwarded_bytes(), 0);
+        assert_eq!(proxy.dropped_bytes(), b"into-the-void".len());
 
         // Proxy is still alive and steerable after a blackholed connection.
         proxy.set_mode(FaultMode::Pass);

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1191,11 +1191,59 @@ mod tests {
         .into())
     }
 
+    fn scanner_cycle_completion_budget(
+        cycle_end: u64,
+        interval: Option<u64>,
+        now: u64,
+        active: bool,
+    ) -> Result<Duration, &'static str> {
+        if cycle_end > now {
+            return Err("scanner cycle timestamp is in the future");
+        }
+        if active {
+            return Ok(Duration::from_secs(60));
+        }
+        let interval = interval
+            .filter(|interval| *interval != 0)
+            .ok_or("scanner schedule has no positive interval")?;
+        let next_cycle = cycle_end
+            .checked_add(interval)
+            .ok_or("scanner schedule timestamp overflows")?;
+        let budget = next_cycle
+            .saturating_sub(now)
+            .checked_add(60)
+            .ok_or("scanner completion budget overflows")?;
+        Ok(Duration::from_secs(budget))
+    }
+
+    #[test]
+    fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
+        assert_eq!(
+            scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
+            Ok(Duration::from_secs(259)),
+        );
+        assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
+    }
+
+    #[test]
+    fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
+        assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
+        assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
+    }
+
+    #[test]
+    fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
+        for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
+            assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
+        }
+    }
+
     async fn wait_for_scanner_cycle_after(
         cluster: &RustFSTestClusterEnvironment,
         previous_cycle_end: u64,
     ) -> Result<u64, Box<dyn Error + Send + Sync>> {
-        let deadline = Instant::now() + Duration::from_secs(60);
+        let mut deadline = Instant::now() + Duration::from_secs(60);
+        let mut schedule_accounted = false;
         loop {
             let mut latest_cycle_end = 0;
             let mut versions_observed = false;
@@ -1237,14 +1285,36 @@ mod tests {
                 let versions_scanned = metrics["versions_scanned"]
                     .as_u64()
                     .ok_or("scanner status is missing its version-coverage counter")?;
-                latest_cycle_end = latest_cycle_end.max(cycle_end);
+                let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
+                if cycle_end > now {
+                    return Err("scanner cycle timestamp is in the future".into());
+                }
+                let schedule = &status["cycle_schedule"];
+                if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
+                    let active = metrics["current_cycle_active"]
+                        .as_bool()
+                        .ok_or("scanner status is missing its active-cycle flag")?;
+                    if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
+                        let interval = schedule["effective_interval_seconds"].as_u64();
+                        let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
+                        deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
+                        schedule_accounted = true;
+                    }
+                }
+                // Superseded walks complete all sets while withholding quota authority.
+                // Deferred and partial attempts do not certify completed scanning.
+                if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
+                    latest_cycle_end = latest_cycle_end.max(cycle_end);
+                }
                 versions_observed |= versions_scanned > 0;
                 observations.push(format!(
-                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
+                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
                     metrics["current_cycle"],
                     metrics["current_cycle_active"],
                     metrics["leader_lock_state"],
                     metrics["last_cycle_result"],
+                    metrics["last_cycle_partial_reason"],
+                    schedule,
                 ));
             }
             // The coordinator records cycle completion, but remote workers
@@ -1555,7 +1625,8 @@ mod tests {
         }
 
         if background_enabled {
-            wait_for_scanner_cycle_after(&cluster, 0).await?;
+            let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
+            wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
         }
 
         cluster.stop_node(1)?;
@@ -1898,11 +1969,16 @@ mod tests {
             let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
             let status: serde_json::Value = serde_json::from_str(&status_body)
                 .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?;
-            if background_enabled || only_admin_heal_is_active(&status) {
+            let admin_active = background_rejoin_heal_evidence
+                || (status["state"].as_str() == Some("active")
+                    && status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1));
+            if admin_active && (background_enabled || only_admin_heal_is_active(&status)) {
                 break status;
             }
             if Instant::now() >= partial_deadline {
-                return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into());
+                return Err(
+                    format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(),
+                );
             }
             sleep(Duration::from_millis(50)).await;
         };
@@ -2562,4 +2638,157 @@ mod tests {
         };
         assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
     }
+    async fn two_node_heal_status(
+        cluster: &RustFSTestClusterEnvironment,
+        bucket: Option<&str>,
+        recreate: bool,
+        scan_mode: u8,
+    ) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
+        let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
+        let options = serde_json::json!({
+            "recursive": true, "dryRun": false, "remove": false,
+            "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
+        });
+        let started =
+            start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
+        let started: serde_json::Value = serde_json::from_str(&started)?;
+        let token = started["clientToken"]
+            .as_str()
+            .ok_or_else(|| format!("missing heal token: {started}"))?;
+        let status_url = format!("{url}?clientToken={token}");
+        let deadline = Instant::now() + Duration::from_secs(240);
+        loop {
+            let body = timeout(
+                Duration::from_secs(20),
+                signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
+            )
+            .await??;
+            let status: serde_json::Value = serde_json::from_str(&body)?;
+            match status["summary"].as_str() {
+                Some("finished" | "stopped") => return Ok(status),
+                Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
+                _ => return Err(format!("heal did not reach a terminal state: {status}").into()),
+            }
+        }
+    }
+
+    async fn verify_two_node_directory_recovery(
+        remove_bucket: bool,
+        scan_mode: u8,
+        bucket_scoped: bool,
+        recreate: bool,
+    ) -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
+        // Isolate explicit administrator recovery from scanner, MRF, and disk healing.
+        for key in [
+            "RUSTFS_HEAL_AUTO_HEAL_ENABLE",
+            "RUSTFS_HEAL_MRF_ENABLE",
+            "RUSTFS_SCANNER_ENABLED",
+            "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
+        ] {
+            cluster.set_env(key, "false");
+        }
+        cluster.start().await?;
+        let source = cluster.create_s3_client(0)?;
+        let target = cluster.create_s3_client(1)?;
+        let bucket = "two-node-directory-recovery";
+        source.create_bucket().bucket(bucket).send().await?;
+        let objects = [
+            ("empty", Vec::new()),
+            ("nested/inline", vec![0x3c; 1024]),
+            ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
+        ];
+        for (key, bytes) in &objects {
+            source
+                .put_object()
+                .bucket(bucket)
+                .key(*key)
+                .body(ByteStream::from(bytes.clone()))
+                .send()
+                .await?;
+        }
+        let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
+        let original = objects
+            .iter()
+            .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
+            .collect::<Result<Vec<_>, _>>()?;
+        for census in &original {
+            assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
+        }
+        assert!(
+            !original[2].present_part_fingerprints.is_empty(),
+            "large fixture must exercise external parts"
+        );
+        let removed = Path::new(&cluster.temp_dir).join("removed");
+        tokio::fs::create_dir(&removed).await?;
+        if remove_bucket {
+            tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
+        } else {
+            for (index, (key, _)) in objects.iter().enumerate() {
+                tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
+            }
+        }
+        let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?;
+        assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}");
+        assert_eq!(status["outcome"]["execution"]["state"], "completed");
+        assert_eq!(status["outcome"]["coverage"], "complete");
+        for ((key, _), before) in objects.iter().zip(&original) {
+            let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
+            assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
+            assert_eq!(
+                after.present_part_fingerprints, before.present_part_fingerprints,
+                "external shard bytes must match: {key}"
+            );
+            assert_eq!(
+                after.inline_data_fingerprint, before.inline_data_fingerprint,
+                "inline shard bytes must match: {key}"
+            );
+        }
+        cluster.stop_node(0)?;
+        for (key, expected) in &objects {
+            let actual = timeout(Duration::from_secs(30), async {
+                target
+                    .get_object()
+                    .bucket(bucket)
+                    .key(*key)
+                    .send()
+                    .await?
+                    .body
+                    .collect()
+                    .await
+                    .map(|body| body.into_bytes())
+                    .map_err(Box::<dyn Error + Send + Sync>::from)
+            })
+            .await??;
+            assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
+            assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
+        }
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(true, 2, true, true).await
+    }
+
+    #[tokio::test]
+    async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(true, 2, false, false).await
+    }
+
+    #[tokio::test]
+    async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(true, 2, true, false).await
+    }
+
+    #[tokio::test]
+    async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(false, 1, false, false).await
+    }
+
+    #[tokio::test]
+    async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
+        verify_two_node_directory_recovery(false, 2, false, false).await
+    }
 }

crates/e2e_test/src/internode_rpc_signature_e2e_test.rs

@@ -264,7 +264,7 @@ async fn call_make_volume_response(
     request: MakeVolumeRequest,
     headers: HeaderMap,
 ) -> Result<Response<MakeVolumeResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(&url.to_string())
+    let mut client = node_service_time_out_client_no_auth(url)
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
 }
 
 async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(&url.to_string())
+    let mut client = node_service_time_out_client_no_auth(url)
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(PingRequest {

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -562,17 +562,23 @@ mod tests {
         create_bucket(&client, bucket).await.expect("Failed to create bucket");
 
         let object_count = 1002;
-        for i in 0..object_count {
-            let key = format!("object{:04}.txt", i);
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(&key)
-                .body(ByteStream::from_static(b"test content"))
-                .send()
-                .await
-                .expect("Failed to put object");
-        }
+        // Finish every durable PUT before listing, with the same bounded fanout as the delimiter fixtures.
+        stream::iter(0..object_count)
+            .for_each_concurrent(16, |i| {
+                let client = &client;
+                async move {
+                    let key = format!("object{i:04}.txt");
+                    client
+                        .put_object()
+                        .bucket(bucket)
+                        .key(&key)
+                        .body(ByteStream::from_static(b"test content"))
+                        .send()
+                        .await
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                }
+            })
+            .await;
 
         eprintln!("Seeded {object_count} objects in {bucket}; starting ListObjectsV2 pagination");
         let deadline = Instant::now() + PAGINATION_TIMEOUT;

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::RustFSTestClusterEnvironment;
+use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::SdkError;
 use bytes::Bytes;
@@ -185,21 +185,113 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
             return Ok(());
         }
         if tokio::time::Instant::now() >= deadline {
-            return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
+            let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
+            return Err(format!(
+                "read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
+            )
+            .into());
         }
         tokio::time::sleep(std::time::Duration::from_millis(100)).await;
     }
 }
 
+#[tokio::test]
+async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult {
+    crate::common::init_logging();
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
+    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
+    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
+    cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false");
+    cluster.start().await?;
+    let clients = cluster
+        .create_all_clients()?
+        .into_iter()
+        .map(|client| {
+            Client::from_conf(
+                client
+                    .config()
+                    .to_builder()
+                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                    .build(),
+            )
+        })
+        .collect::<Vec<_>>();
+    let bucket = "cold-degraded-mutations";
+    clients[0].create_bucket().bucket(bucket).send().await?;
+    // No object mutation may warm a process-wide configuration probe before
+    // the missing peer is introduced. Three shards still satisfy EC:2 writes.
+    cluster.stop_node(3)?;
+    let admission_deadline = Instant::now() + Duration::from_secs(30);
+    for client in &clients[..3] {
+        loop {
+            match client
+                .get_object()
+                .bucket(bucket)
+                .key("never-created-readiness-probe")
+                .send()
+                .await
+            {
+                Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break,
+                Err(error)
+                    if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
+                        && Instant::now() < admission_deadline =>
+                {
+                    tokio::time::sleep(Duration::from_millis(100)).await;
+                }
+                Err(error) => return Err(error.into()),
+                Ok(_) => panic!("a cold readiness probe must not create an object"),
+            }
+        }
+    }
+    let body = b"first write while one peer is offline";
+    clients[0]
+        .put_object()
+        .bucket(bucket)
+        .key("object")
+        .body(Bytes::from_static(body).into())
+        .send()
+        .await?;
+    for client in &clients[..3] {
+        assert_quorum_object_body(client, bucket, "object", body).await?;
+    }
+    // This coordinator has not previously committed an object mutation either.
+    clients[1].delete_object().bucket(bucket).key("object").send().await?;
+    let error = clients[2]
+        .get_object()
+        .bucket(bucket)
+        .key("object")
+        .send()
+        .await
+        .expect_err("acknowledged deletion must be visible at another entrypoint");
+    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey"));
+
+    cluster.stop_node(2)?;
+    let error = clients[0]
+        .put_object()
+        .bucket(bucket)
+        .key("below-quorum")
+        .body(Bytes::from_static(body).into())
+        .send()
+        .await
+        .expect_err("removing the cache gate must not weaken storage write quorum");
+    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
+    cluster.stop();
+    Ok(())
+}
+
 #[tokio::test]
 async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
     crate::common::init_logging();
 
-    for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
+    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
         let read_quorum = node_count - parity;
         let write_quorum = read_quorum + usize::from(read_quorum == parity);
         let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
         cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
+        if parity == 0 {
+            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
+        }
         // Wait for every seed fanout before removing any physical shard.
         cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
         cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -300,17 +392,28 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
         }
 
         cluster.stop_node(read_quorum - 1)?;
+        let lock_read_quorum = node_count - node_count / 2;
+        // EC:2 on six nodes still has three shared lock votes after losing
+        // its fourth data shard. Preserve the distinct data-quorum error.
+        let expected_code = if read_quorum - 1 < lock_read_quorum {
+            "ServiceUnavailable"
+        } else {
+            "SlowDownRead"
+        };
         for client in clients.iter().take(read_quorum - 1) {
             match client.get_object().bucket(&bucket).key("below-quorum").send().await {
                 Ok(response) => assert!(
                     response.body.collect().await.is_err(),
                     "fewer than {read_quorum} valid fragments must not reconstruct an uncached object"
                 ),
-                Err(error) => assert_eq!(
-                    error.as_service_error().and_then(|error| error.meta().code()),
-                    Some("ServiceUnavailable"),
-                    "a quorum loss must not be mistaken for a missing object"
-                ),
+                Err(error) => {
+                    assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+                    assert_eq!(
+                        error.as_service_error().and_then(|error| error.meta().code()),
+                        Some(expected_code),
+                        "a quorum loss must not be mistaken for a missing object"
+                    );
+                }
             }
         }
 
@@ -327,6 +430,277 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
     Ok(())
 }
 
+#[tokio::test]
+async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
+    crate::common::init_logging();
+    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
+        let read_quorum = node_count - parity;
+        let write_quorum = read_quorum + usize::from(read_quorum == parity);
+        let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
+        cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
+        if parity == 0 {
+            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
+        }
+        cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
+        cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
+        cluster.set_env("RUST_LOG", "warn");
+        cluster.start().await?;
+        let clients = cluster
+            .create_all_clients()?
+            .into_iter()
+            .map(|client| {
+                Client::from_conf(
+                    client
+                        .config()
+                        .to_builder()
+                        .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                        .build(),
+                )
+            })
+            .collect::<Vec<_>>();
+        let bucket = format!("cold-read-quorum-{node_count}-{parity}");
+        clients[0].create_bucket().bucket(&bucket).send().await?;
+        let small = b"cold startup follows the stored EC layout".to_vec();
+        let large = vec![73; 256 * 1024];
+        for (key, body) in [
+            ("warm-small", &small),
+            ("warm-large", &large),
+            ("cold-large", &large),
+            ("below-quorum", &large),
+        ] {
+            clients[0]
+                .put_object()
+                .bucket(&bucket)
+                .key(key)
+                .body(Bytes::copy_from_slice(body).into())
+                .send()
+                .await?;
+            for node in &cluster.nodes {
+                let census =
+                    crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
+                assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
+                assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
+            }
+        }
+        for node in 0..node_count {
+            cluster.stop_node(node)?;
+        }
+        // Rebuild every process's in-memory state below read quorum, then
+        // cross the EC boundary. A format bootstrap must not admit S3 reads
+        // merely because half the peers are online (e.g. six disks, EC:2).
+        for node in 0..read_quorum - 1 {
+            cluster.start_node_process(node).await?;
+        }
+        let error = clients[0]
+            .get_object()
+            .bucket(&bucket)
+            .key("below-quorum")
+            .send()
+            .await
+            .expect_err("cold startup below the configured EC read quorum must stay unavailable");
+        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+        cluster.start_node_process(read_quorum - 1).await?;
+        wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
+        for client in clients.iter().take(read_quorum) {
+            assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
+            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
+            let write = client
+                .put_object()
+                .bucket(&bucket)
+                .key("cold-quorum-write")
+                .body(Bytes::copy_from_slice(&small).into())
+                .send()
+                .await;
+            if read_quorum >= write_quorum {
+                write?;
+                assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
+            } else {
+                let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
+                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            }
+        }
+        for node in read_quorum..node_count {
+            cluster.start_node(node).await?;
+        }
+        for client in &clients {
+            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
+        }
+    }
+    Ok(())
+}
+
+#[tokio::test]
+async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
+    crate::common::init_logging();
+    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
+    // Explicit URLs are consumed in order. Place one drive from each node in
+    // every set, matching a four-host, six-drive-per-host deployment.
+    let volumes = (0..6)
+        .flat_map(|drive| {
+            cluster
+                .nodes
+                .iter()
+                .map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
+        })
+        .collect::<Vec<_>>()
+        .join(" ");
+    cluster.set_env("RUSTFS_VOLUMES", volumes);
+    cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
+    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
+    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
+    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
+    cluster.set_env("RUST_LOG", "error");
+    cluster.start().await?;
+    let clients = cluster
+        .create_all_clients()?
+        .into_iter()
+        .map(|client| {
+            Client::from_conf(
+                client
+                    .config()
+                    .to_builder()
+                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                    .build(),
+            )
+        })
+        .collect::<Vec<_>>();
+    let bucket = "cold-start-read-quorum";
+    clients[0].create_bucket().bucket(bucket).send().await?;
+    let mut fixtures = Vec::new();
+    let mut covered = [[false; 2]; 6];
+    for index in 0..128 {
+        let key = match index {
+            0 => "warm-small".to_string(),
+            1 => "warm-large".to_string(),
+            _ => format!("cold-{index}"),
+        };
+        let size_class = index % 2;
+        let size = if size_class == 0 { 71 } else { 256 * 1024 };
+        let body = (0..size)
+            .map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
+            .collect::<Vec<_>>();
+        clients[0]
+            .put_object()
+            .bucket(bucket)
+            .key(&key)
+            .body(Bytes::copy_from_slice(&body).into())
+            .send()
+            .await?;
+        for node in &cluster.nodes {
+            let mut complete_shards = 0;
+            for (drive, path) in node.data_dirs.iter().enumerate() {
+                let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
+                if census.has_xl_meta {
+                    assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
+                    assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
+                    covered[drive][size_class] = true;
+                    complete_shards += 1;
+                }
+            }
+            assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
+        }
+        fixtures.push((key, body));
+        if covered.iter().flatten().all(|covered| *covered) {
+            break;
+        }
+    }
+    assert!(
+        covered.iter().flatten().all(|covered| *covered),
+        "fixtures must cover all six sets with inline and large objects"
+    );
+
+    for first in 0..4 {
+        for second in first + 1..4 {
+            for node in 0..4 {
+                cluster.stop_node(node)?;
+            }
+            cluster.start_node_process(first).await?;
+            let error = clients[first]
+                .get_object()
+                .bucket(bucket)
+                .key("warm-large")
+                .send()
+                .await
+                .expect_err("one node cannot satisfy read quorum after a cold start");
+            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            cluster.start_node_process(second).await?;
+            wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
+            for node in [first, second] {
+                for (key, body) in &fixtures {
+                    assert_quorum_object_body(&clients[node], bucket, key, body).await?;
+                }
+                let range = clients[node]
+                    .get_object()
+                    .bucket(bucket)
+                    .key("warm-large")
+                    .range("bytes=17-1023")
+                    .send()
+                    .await?;
+                assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
+                let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
+                for (key, _) in &fixtures {
+                    assert!(
+                        listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
+                        "listing omitted {key}"
+                    );
+                }
+                let error = clients[node]
+                    .put_object()
+                    .bucket(bucket)
+                    .key("recovery-write")
+                    .body(Bytes::from_static(b"after recovery").into())
+                    .send()
+                    .await
+                    .expect_err("two nodes must not authorize a write");
+                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            }
+            let error = clients[first]
+                .delete_object()
+                .bucket(bucket)
+                .key("warm-large")
+                .send()
+                .await
+                .expect_err("two nodes must not authorize a delete");
+            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
+            assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
+
+            let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
+            cluster.start_node(remaining[0]).await?;
+            let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
+            loop {
+                match clients[first]
+                    .put_object()
+                    .bucket(bucket)
+                    .key("recovery-write")
+                    .body(Bytes::from_static(b"after recovery").into())
+                    .send()
+                    .await
+                {
+                    Ok(_) => break,
+                    Err(error)
+                        if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
+                            && tokio::time::Instant::now() < deadline =>
+                    {
+                        tokio::time::sleep(Duration::from_millis(200)).await
+                    }
+                    Err(error) => return Err(error.into()),
+                }
+            }
+            for node in [first, second, remaining[0]] {
+                assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
+            }
+            cluster.start_node(remaining[1]).await?;
+            for client in &clients {
+                for (key, body) in &fixtures {
+                    assert_quorum_object_body(client, bucket, key, body).await?;
+                }
+                assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
+            }
+        }
+    }
+    Ok(())
+}
+
 async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
     client
         .put_object()
@@ -425,6 +799,60 @@ async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum()
     Ok(())
 }
 
+fn unavailable_readiness_drive_indices(payload: &serde_json::Value) -> Vec<usize> {
+    let mut indices = payload["details"]["storage"]["unavailableDrives"]
+        .as_array()
+        .expect("node readiness must report unavailable drives")
+        .iter()
+        .map(|drive| {
+            usize::try_from(drive["diskIndex"].as_u64().expect("drive index must be unsigned"))
+                .expect("drive index must fit usize")
+        })
+        .collect::<Vec<_>>();
+    indices.sort_unstable();
+    indices
+}
+
+fn assert_node_readiness_contract(payload: &serde_json::Value, survivors: usize) {
+    let read_ready = survivors >= 2;
+    let write_ready = survivors >= 3;
+    let storage = &payload["details"]["storage"];
+    assert_eq!(payload["ready"], read_ready);
+    assert_eq!(storage["ready"], read_ready);
+    assert_eq!(storage["readQuorum"], read_ready);
+    assert_eq!(storage["writeQuorum"], write_ready);
+    assert_eq!(storage["readinessScope"], "read_quorum");
+    assert_eq!(storage["source"], "local_runtime");
+    assert_eq!(storage["status"], if read_ready { "connected" } else { "disconnected" });
+    assert_eq!(payload["details"]["poolMetadata"]["ready"], true);
+    assert_eq!(payload["details"]["iam"]["ready"], true);
+    assert_eq!(payload["details"]["lock"]["ready"], read_ready);
+    assert_eq!(unavailable_readiness_drive_indices(payload), (survivors..4).collect::<Vec<_>>());
+    for drive in storage["unavailableDrives"].as_array().expect("unavailable drives") {
+        assert_eq!(drive["poolIndex"], 0);
+        assert_eq!(drive["setIndex"], 0);
+        let state = drive["runtimeState"].as_str().expect("runtime state must be reported");
+        let host_online = drive["hostOnline"].as_bool().expect("host reachability must be reported");
+        assert!(
+            matches!(state, "online" | "suspect" | "offline" | "returning"),
+            "unknown runtime state: {state}"
+        );
+        assert!(
+            state != "online" || !host_online,
+            "a reachable online drive cannot be excluded from quorum"
+        );
+    }
+    if !read_ready {
+        assert!(
+            payload["degradedReasons"]
+                .as_array()
+                .expect("degraded reasons")
+                .iter()
+                .any(|reason| reason == "storage_and_lock_unavailable")
+        );
+    }
+}
+
 async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvironment) -> TestResult {
     let clients: Vec<_> = cluster
         .create_all_clients()?
@@ -483,27 +911,14 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
                     && payload["details"]["poolMetadata"]["ready"] == true
                     && payload["details"]["iam"]["ready"] == true
                     && payload["details"]["lock"]["ready"] == read_quorum
+                    && unavailable_readiness_drive_indices(&payload) == (survivors..4).collect::<Vec<_>>()
                 {
                     break payload;
                 }
                 assert!(Instant::now() < deadline, "node {idx}, survivors={survivors}: HTTP {status}, {payload}");
                 tokio::time::sleep(Duration::from_millis(200)).await;
             };
-            assert_eq!(payload["details"]["storage"]["readinessScope"], "read_quorum");
-            assert_eq!(payload["details"]["storage"]["source"], "local_runtime");
-            assert_eq!(
-                payload["details"]["storage"]["status"],
-                if node_ready { "connected" } else { "disconnected" }
-            );
-            if !node_ready {
-                assert!(
-                    payload["degradedReasons"]
-                        .as_array()
-                        .expect("degraded reasons")
-                        .iter()
-                        .any(|reason| reason == "storage_and_lock_unavailable")
-                );
-            }
+            assert_node_readiness_contract(&payload, survivors);
 
             for path in ["/health/ready", "/minio/health/ready"] {
                 let head = http.head(format!("{url}{path}")).send().await?;
@@ -512,7 +927,9 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
                 let response = http.get(format!("{url}{path}")).send().await?;
                 assert_eq!(response.status().as_u16(), node_status);
                 let body: serde_json::Value = response.json().await?;
-                assert_eq!(body["details"]["storage"], payload["details"]["storage"]);
+                // Each response samples live diagnostics; suspect -> offline
+                // must not change the quorum contract or excluded topology.
+                assert_node_readiness_contract(&body, survivors);
                 assert_eq!(body["details"]["poolMetadata"], payload["details"]["poolMetadata"]);
             }
             let live = http.get(format!("{url}/health/live")).send().await?;

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -465,6 +465,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn recover_delete_transaction(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
+        Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
+    }
+
     async fn settle_part_transaction(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,

crates/e2e_test/src/replication_extension_test.rs

@@ -3945,7 +3945,15 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         (sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()),
         (large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()),
     ];
-    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
+    // Managed-SSE listings may omit ETags. Check the replicated version's body
+    // before deleting it rather than comparing keyless listing fingerprints.
+    for (key, version_id) in &purged {
+        let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?;
+        let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into();
+        wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?;
+        let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?;
+        assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}");
+    }
 
     for (key, version_id) in &purged {
         source_client
@@ -3956,10 +3964,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
             .send()
             .await?;
     }
-    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
-    let target_state = list_replication_state(&target_client, &target_bucket).await?;
-    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
-
     // The purge state is applied on the source asynchronously after the target
     // acknowledges the delete; only then does the retained version go away and
     // the bucket become deletable. A listing that is empty while DeleteBucket
@@ -3984,6 +3988,9 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         }
     }
 
+    let target_state = list_replication_state(&target_client, &target_bucket).await?;
+    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
+
     Ok(())
 }
 

crates/e2e_test/src/storage_api.rs

@@ -15,7 +15,7 @@
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys;
 #[cfg(test)]
-pub(crate) use rustfs_ecstore::api::disk::{RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
+pub(crate) use rustfs_ecstore::api::disk::{Endpoint, RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
 pub(crate) use rustfs_ecstore::api::rpc::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::rpc::{

crates/e2e_test/src/sts_query_compat_test.rs

@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
         .version_id(version)
         .send()
         .await?;
-    opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
     client.list_objects_v2().bucket(source).send().await?;
     opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
     client.list_object_versions().bucket(source).send().await?;

crates/e2e_test/src/tier_transition_regression_test.rs

@@ -165,4 +165,210 @@ mod tests {
         info!("RT-13c PASS: scanner/config persists across restart");
         Ok(())
     }
+
+    /// RT-14: a transition PUT must not carry reserved `x-minio-internal-*` headers.
+    ///
+    /// Regression pattern: transition forwarded the object's internal metadata as raw
+    /// request headers, so MinIO-derived remotes (Storj) refused every transition with
+    /// `400 InvalidArgument` while `tier add` still succeeded (rustfs#8362). The fake
+    /// target refuses any PutObject carrying such a header, like MinIO and Storj do.
+    #[tokio::test]
+    async fn test_transition_put_omits_reserved_internal_headers() -> TestResult {
+        use crate::fake_s3_target::{BucketMode, FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, Operation};
+        use aws_sdk_s3::primitives::ByteStream;
+        use aws_sdk_s3::types::{
+            BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, LifecycleRuleFilter, Transition,
+            TransitionStorageClass,
+        };
+        use std::time::{Duration, Instant};
+
+        init_logging();
+        info!("RT-14: transition PUT omits reserved internal headers");
+
+        const TIER: &str = "RT14COLD";
+        const TARGET_BUCKET: &str = "rt14-cold";
+        const SOURCE_BUCKET: &str = "rt14-hot";
+
+        let target = FakeS3Target::start().await.expect("start fake target");
+        target.create_bucket_with_mode(TARGET_BUCKET, BucketMode::Unversioned);
+        target.reject_reserved_internal_headers(true);
+
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server_with_env(
+            vec![],
+            &[
+                ("RUSTFS_CONSOLE_ENABLE", "false"),
+                ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true"),
+            ],
+        )
+        .await
+        .expect("start RustFS");
+
+        let tier_body = serde_json::json!({
+            "type": "rustfs",
+            "rustfs": {
+                "name": TIER,
+                "endpoint": target.endpoint(),
+                "accessKey": FAKE_ACCESS_KEY,
+                "secretKey": FAKE_SECRET_KEY,
+                "bucket": TARGET_BUCKET,
+                "prefix": "cold/",
+                "region": "us-east-1",
+                "storageClass": ""
+            }
+        })
+        .to_string();
+        let deadline = Instant::now() + Duration::from_secs(30);
+        loop {
+            match admin_ok(&env, http::Method::PUT, "/rustfs/admin/v3/tier", Some(tier_body.clone())).await {
+                Ok(_) => break,
+                Err(err) if Instant::now() < deadline => {
+                    info!("  add tier not ready yet: {err}");
+                    tokio::time::sleep(Duration::from_millis(500)).await;
+                }
+                Err(err) => panic!("RT-14 FAIL: add tier: {err}"),
+            }
+        }
+
+        let client = env.create_s3_client();
+        client
+            .create_bucket()
+            .bucket(SOURCE_BUCKET)
+            .send()
+            .await
+            .expect("create bucket");
+
+        let objects: Vec<(&str, Vec<u8>)> = vec![
+            ("small.bin", (0..1024u32).map(|i| (i % 251) as u8).collect()),
+            ("medium.bin", (0..200_000u32).map(|i| (i.wrapping_mul(31) % 253) as u8).collect()),
+        ];
+        for (key, body) in &objects {
+            client
+                .put_object()
+                .bucket(SOURCE_BUCKET)
+                .key(*key)
+                .body(ByteStream::from(body.clone()))
+                .send()
+                .await
+                .expect("put source object");
+        }
+
+        let rule = LifecycleRule::builder()
+            .id("to-cold")
+            .status(ExpirationStatus::Enabled)
+            .filter(LifecycleRuleFilter::builder().prefix("").build())
+            .transitions(
+                Transition::builder()
+                    .days(0)
+                    .storage_class(TransitionStorageClass::from(TIER))
+                    .build(),
+            )
+            .build()
+            .expect("build rule");
+        client
+            .put_bucket_lifecycle_configuration()
+            .bucket(SOURCE_BUCKET)
+            .lifecycle_configuration(
+                BucketLifecycleConfiguration::builder()
+                    .rules(rule)
+                    .build()
+                    .expect("build config"),
+            )
+            .send()
+            .await
+            .expect("put lifecycle");
+
+        admin_ok(
+            &env,
+            http::Method::POST,
+            &format!("/rustfs/admin/v3/ilm/transition/run?bucket={SOURCE_BUCKET}&prefix=&tier={TIER}&dryRun=false"),
+            None,
+        )
+        .await
+        .expect("run transition");
+
+        let deadline = Instant::now() + Duration::from_secs(60);
+        while target.stored_keys(TARGET_BUCKET).len() < objects.len() {
+            assert!(
+                Instant::now() < deadline,
+                "RT-14 FAIL: transition did not reach the target; stored keys: {:?}",
+                target.stored_keys(TARGET_BUCKET)
+            );
+            tokio::time::sleep(Duration::from_millis(500)).await;
+        }
+
+        let keys = target.stored_keys(TARGET_BUCKET);
+        assert_eq!(keys.len(), objects.len(), "RT-14 FAIL: unexpected remote objects: {keys:?}");
+
+        let mut stored_bodies = Vec::new();
+        for key in &keys {
+            let (body, metadata) = target.stored_object(TARGET_BUCKET, key).expect("remote object");
+            for name in [
+                "x-rustfs-internal-transition-transaction-id",
+                "x-minio-internal-transition-transaction-id",
+                "x-rustfs-internal-transition-tier-destination-id",
+                "x-minio-internal-transition-tier-destination-id",
+            ] {
+                assert!(
+                    metadata.contains_key(name),
+                    "RT-14 FAIL: remote object {key} lost transition identity {name}: {:?}",
+                    metadata.keys().collect::<Vec<_>>()
+                );
+            }
+            stored_bodies.push(body.to_vec());
+        }
+        let mut expected_bodies: Vec<Vec<u8>> = objects.iter().map(|(_, body)| body.clone()).collect();
+        expected_bodies.sort();
+        stored_bodies.sort();
+        assert_eq!(stored_bodies, expected_bodies, "RT-14 FAIL: remote bytes differ from the source objects");
+
+        let transition_puts = target
+            .requests()
+            .iter()
+            .filter(|record| {
+                record.operation == Operation::PutObject
+                    && record
+                        .key
+                        .as_deref()
+                        .is_some_and(|key| key.contains("transition-transactions"))
+            })
+            .count();
+        assert_eq!(
+            transition_puts,
+            objects.len(),
+            "RT-14 FAIL: transition PUTs were retried or refused by the target"
+        );
+
+        for (key, body) in &objects {
+            let head = client
+                .head_object()
+                .bucket(SOURCE_BUCKET)
+                .key(*key)
+                .send()
+                .await
+                .expect("head transitioned");
+            assert_eq!(
+                head.storage_class().map(|class| class.as_str()),
+                Some(TIER),
+                "RT-14 FAIL: {key} is not reported as transitioned"
+            );
+            let got = client
+                .get_object()
+                .bucket(SOURCE_BUCKET)
+                .key(*key)
+                .send()
+                .await
+                .expect("get transitioned")
+                .body
+                .collect()
+                .await
+                .expect("read body")
+                .into_bytes();
+            assert_eq!(got.as_ref(), body.as_slice(), "RT-14 FAIL: read-back of {key} differs");
+        }
+
+        target.shutdown().await;
+        info!("RT-14 PASS: transition PUT omits reserved internal headers");
+        Ok(())
+    }
 }

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -33,6 +33,7 @@ use aws_sdk_s3::types::{
 };
 use http::{Method, StatusCode};
 use serde_json::Value;
+use std::future::Future;
 use std::io::Write;
 use std::path::{Path, PathBuf};
 use std::time::{Duration, SystemTime, UNIX_EPOCH};
@@ -57,6 +58,10 @@ const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
 const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
 const MULTIPART_WORKERS: usize = 16;
 const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
+// Bound each 64 KiB upload, including completion acknowledgement. A peer
+// that accepts the request but never responds must fail the compatibility
+// check with the writer and object, rather than consume the whole CI job.
+const MIXED_MULTIPART_UPLOAD_TIMEOUT: Duration = Duration::from_secs(30);
 // Peers keep a restarted node's drive in Suspect/Returning for roughly
 // probe_interval (2s) x success_threshold (3) after it comes back; 30s
 // comfortably covers that window plus CI scheduling jitter.
@@ -335,14 +340,21 @@ fn configure_cluster_logs(cluster: &mut RustFSTestClusterEnvironment) -> TestRes
 async fn write_multipart_load(clients: &[Client], phase: &str) -> Result<Vec<String>, Box<dyn std::error::Error + Send + Sync>> {
     let mut tasks = JoinSet::new();
     for worker in 0..MULTIPART_WORKERS {
-        let client = clients[worker % clients.len()].clone();
+        let node = worker % clients.len();
+        let client = clients[node].clone();
         let phase = phase.to_string();
         tasks.spawn(async move {
             let mut keys = Vec::with_capacity(MULTIPART_UPLOADS_PER_WORKER);
             for upload in 0..MULTIPART_UPLOADS_PER_WORKER {
                 let key = format!("{phase}/multipart/{worker:02}/{upload:02}");
                 let part = vec![u8::try_from(worker)?; 64 * 1024];
-                write_multipart(&client, MIXED_BUCKET, &key, &[part]).await?;
+                mixed_multipart_before(
+                    Instant::now() + MIXED_MULTIPART_UPLOAD_TIMEOUT,
+                    node,
+                    &key,
+                    write_multipart(&client, MIXED_BUCKET, &key, &[part]),
+                )
+                .await?;
                 keys.push(key);
             }
             Ok::<_, Box<dyn std::error::Error + Send + Sync>>(keys)
@@ -356,6 +368,172 @@ async fn write_multipart_load(clients: &[Client], phase: &str) -> Result<Vec<Str
     Ok(keys)
 }
 
+async fn mixed_multipart_before(
+    deadline: Instant,
+    node: usize,
+    key: &str,
+    upload: impl Future<Output = TestResult>,
+) -> TestResult {
+    tokio::time::timeout_at(deadline, upload)
+        .await
+        .map_err(|_| MultipartAcknowledgementDeadline {
+            node,
+            key: key.to_owned(),
+        })?
+}
+
+#[derive(Debug)]
+struct MultipartAcknowledgementDeadline {
+    node: usize,
+    key: String,
+}
+
+impl std::fmt::Display for MultipartAcknowledgementDeadline {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(
+            formatter,
+            "node {}: multipart upload {MIXED_BUCKET}/{} acknowledgement deadline exceeded",
+            self.node, self.key
+        )
+    }
+}
+
+impl std::error::Error for MultipartAcknowledgementDeadline {}
+
+// Decode only the public lock snapshot fields. Unknown fields, including
+// credential/configuration payloads, never enter the failure artifact.
+#[derive(serde::Deserialize, serde::Serialize)]
+struct MixedLockSnapshot {
+    total: Option<u64>,
+    truncated: Option<bool>,
+    capability_note: Option<String>,
+    locks: Vec<MixedLockEntry>,
+}
+
+#[derive(serde::Deserialize, serde::Serialize)]
+struct MixedLockEntry {
+    resource: Option<String>,
+    bucket: Option<String>,
+    object: Option<String>,
+    version: Option<String>,
+    owner: Option<String>,
+    #[serde(rename = "type")]
+    lock_type: Option<String>,
+    priority: Option<String>,
+    since: Option<String>,
+    elapsed_secs: Option<u64>,
+    ttl_secs: Option<u64>,
+}
+
+async fn read_mixed_lock_snapshot(response: &mut reqwest::Response) -> Result<MixedLockSnapshot, &'static str> {
+    const MAX_BODY_BYTES: usize = 1024 * 1024;
+    let mut body = Vec::new();
+    while let Some(chunk) = response.chunk().await.map_err(|_| "body_read_error")? {
+        if chunk.len() > MAX_BODY_BYTES - body.len() {
+            return Err("body_limit_exceeded");
+        }
+        body.extend_from_slice(&chunk);
+    }
+    serde_json::from_slice(&body).map_err(|_| "invalid_lock_snapshot")
+}
+
+async fn probe_mixed_multipart_failure(
+    cluster: &RustFSTestClusterEnvironment,
+    node: usize,
+    path: &str,
+    head: bool,
+    deadline: Instant,
+) -> Value {
+    let started = Instant::now();
+    let method = if head { Method::HEAD } else { Method::GET };
+    let mut observation = serde_json::json!({
+        "node": node, "method": method.as_str(), "path": path,
+        "status": null, "observation": "unknown", "timeout": false,
+    });
+    // The shared absolute deadline bounds both headers and a slow/infinite
+    // response body; a send-only timeout would leave the cluster alive forever.
+    let result = tokio::time::timeout_at(deadline, async {
+        let mut response = signed_request(
+            method,
+            &format!("{}{path}", cluster.nodes[node].url.trim_end_matches('/')),
+            &cluster.access_key,
+            &cluster.secret_key,
+            None,
+            None,
+        )
+        .await
+        .map_err(|_| "request_error")?;
+        let status = response.status();
+        observation["status"] = serde_json::json!(status.as_u16());
+        if head {
+            observation["observation"] = serde_json::json!(match status {
+                StatusCode::OK => "visible",
+                StatusCode::NOT_FOUND => "not_visible",
+                _ => "unknown",
+            });
+            for (field, header) in [
+                ("content_length", "content-length"),
+                ("etag", "etag"),
+                ("version", "x-amz-version-id"),
+            ] {
+                observation[field] = serde_json::json!(response.headers().get(header).and_then(|value| value.to_str().ok()));
+            }
+        } else if status == StatusCode::OK {
+            observation["snapshot"] =
+                serde_json::to_value(read_mixed_lock_snapshot(&mut response).await?).map_err(|_| "snapshot_encode_error")?;
+            observation["observation"] = serde_json::json!("captured");
+        }
+        Ok::<_, &'static str>(())
+    })
+    .await;
+    match result {
+        Ok(Ok(())) => {}
+        Ok(Err(reason)) => observation["error"] = serde_json::json!(reason),
+        Err(_) => observation["timeout"] = serde_json::json!(true),
+    }
+    observation["elapsed_secs"] = serde_json::json!(started.elapsed().as_secs_f64());
+    observation
+}
+
+async fn observe_mixed_multipart_failure(
+    cluster: &RustFSTestClusterEnvironment,
+    phase: &str,
+    error: BoxError,
+    log_dir: Option<&Path>,
+) -> TestResult {
+    if let (Some(log_dir), Some(failure)) = (log_dir, error.downcast_ref::<MultipartAcknowledgementDeadline>()) {
+        let started = Instant::now();
+        let deadline = started + Duration::from_secs(5);
+        // Reserve the last second for serialization and bounded artifact I/O.
+        // All probes run directly in this future; no diagnostic task is spawned.
+        let network_deadline = started + Duration::from_secs(4);
+        let path = format!("/{MIXED_BUCKET}/{}", failure.key);
+        let _ = tokio::time::timeout_at(deadline, async {
+            let (heads, locks) = tokio::join!(
+                futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure(
+                    cluster,
+                    node,
+                    &path,
+                    true,
+                    network_deadline
+                ))),
+                probe_mixed_multipart_failure(cluster, 0, "/rustfs/admin/v3/top/locks?count=1000", false, network_deadline),
+            );
+            let snapshot = serde_json::json!({
+                "phase": phase, "bucket": MIXED_BUCKET, "key": failure.key,
+                "writer_node": failure.node, "heads": heads, "top_locks": locks,
+            });
+            let body = serde_json::to_vec_pretty(&snapshot).map_err(|_| ())?;
+            tokio::fs::write(log_dir.join("mixed-multipart-failure.json"), body)
+                .await
+                .map_err(|_| ())
+        })
+        .await;
+    }
+    // Visibility and diagnostic failures cannot acknowledge the original MPU.
+    Err(error)
+}
+
 /// Assert that `client` eventually lists exactly `expected` objects under
 /// `{phase}/`, polling until [`LISTING_CONVERGENCE_TIMEOUT`].
 ///
@@ -400,7 +578,7 @@ async fn exercise_mixed_cluster(
     previous_node: usize,
 ) -> TestResult {
     let clients = cluster.create_all_clients()?;
-    wait_for_upgrade_write_readiness(&clients, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
+    wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
     let current_client = &clients[current_node];
     let previous_client = &clients[previous_node];
 
@@ -426,7 +604,13 @@ async fn exercise_mixed_cluster(
         .await?;
     assert_eq!(read_object(current_client, MIXED_BUCKET, &previous_key, None).await?.1, previous_body);
 
-    let multipart_keys = write_multipart_load(&clients, phase).await?;
+    let multipart_keys = match write_multipart_load(&clients, phase).await {
+        Ok(keys) => keys,
+        Err(error) => {
+            let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR");
+            return observe_mixed_multipart_failure(cluster, phase, error, log_dir.as_deref().map(Path::new)).await;
+        }
+    };
     let expected_count = multipart_keys.len() + 2;
     for (label, client) in [("current", current_client), ("previous", previous_client)] {
         assert_eq!(
@@ -462,7 +646,12 @@ fn upgrade_probe_client(client: &Client) -> Client {
     )
 }
 
-async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budget: Duration) -> TestResult {
+pub(crate) async fn wait_for_upgrade_write_readiness(
+    clients: &[Client],
+    bucket: &str,
+    phase: &str,
+    budget: Duration,
+) -> TestResult {
     // ListBuckets can succeed before peers recover a restarted disk. Cluster
     // health also accepts Returning disks whose write health is still FAULTY.
     // Probe every writer outside the asserted phase prefix; compatibility
@@ -482,7 +671,7 @@ async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budge
                 deadline,
                 client
                     .put_object()
-                    .bucket(MIXED_BUCKET)
+                    .bucket(bucket)
                     .key(&key)
                     .body(ByteStream::from_static(UPGRADE_READINESS_BODY))
                     .send(),
@@ -517,19 +706,20 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro
     // waits for pool metadata (#7473). First prove the elected writer can
     // persist data, then restart each old process once with three peers still
     // readable. This preparation ends before any current binary is started.
-    wait_for_upgrade_write_readiness(&clients[..1], "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
+    wait_for_upgrade_write_readiness(&clients[..1], MIXED_BUCKET, "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
     for node in [1, 2, 3, 0] {
         cluster.stop_node(node)?;
         cluster.start_node_from_binary(node, previous_binary).await?;
         wait_for_upgrade_write_readiness(
             std::slice::from_ref(&clients[node]),
+            MIXED_BUCKET,
             &format!("previous-restart-{node}"),
             LISTING_CONVERGENCE_TIMEOUT,
         )
         .await?;
     }
 
-    wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
+    wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
     for (reader, client) in clients.iter().enumerate() {
         assert_eq!(
             read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
@@ -556,9 +746,10 @@ mod upgrade_write_readiness_tests {
     #[tokio::test]
     async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult {
         let target = FakeS3Target::start().await?;
-        target.create_bucket(MIXED_BUCKET);
+        let bucket = "distributed-upgrade-history";
+        target.create_bucket(bucket);
         let client = fake_source_client(&target);
-        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+        client.head_bucket().bucket(bucket).send().await?;
 
         let phase = "one-previous-node";
         let first_key = format!(".upgrade-readiness/{phase}/node-0");
@@ -578,7 +769,7 @@ mod upgrade_write_readiness_tests {
         // Metadata readiness does not prove that a data write can succeed.
         let premature = client
             .put_object()
-            .bucket(MIXED_BUCKET)
+            .bucket(bucket)
             .key(&first_key)
             .body(ByteStream::from_static(b"upgrade write readiness"))
             .send()
@@ -586,15 +777,15 @@ mod upgrade_write_readiness_tests {
             .expect_err("metadata readiness does not prove write readiness");
         assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503));
 
-        wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], phase, Duration::from_secs(5)).await?;
+        wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], bucket, phase, Duration::from_secs(5)).await?;
         assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3);
         assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2);
-        assert!(target.has_object(MIXED_BUCKET, &first_key));
-        assert!(target.has_object(MIXED_BUCKET, &second_key));
+        assert!(target.has_object(bucket, &first_key));
+        assert!(target.has_object(bucket, &second_key));
         assert!(
             client
                 .list_objects_v2()
-                .bucket(MIXED_BUCKET)
+                .bucket(bucket)
                 .prefix(format!("{phase}/"))
                 .send()
                 .await?
@@ -624,9 +815,10 @@ mod upgrade_write_readiness_tests {
             let phase = format!("permanent-{}", status.as_u16());
             let key = format!(".upgrade-readiness/{phase}/node-0");
             target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Status(status), 1);
-            let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), &phase, Duration::from_secs(5))
-                .await
-                .expect_err("a permanent error must not be retried into success");
+            let error =
+                wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, &phase, Duration::from_secs(5))
+                    .await
+                    .expect_err("a permanent error must not be retried into success");
             assert!(error.to_string().contains("node 0 write readiness failed"), "{error}");
             assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &key), 1);
             assert!(!target.has_object(MIXED_BUCKET, &key));
@@ -649,7 +841,7 @@ mod upgrade_write_readiness_tests {
             FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
             10,
         );
-        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
+        let error = wait_for_upgrade_write_readiness(&[client], MIXED_BUCKET, phase, Duration::from_secs(1))
             .await
             .expect_err("persistent unavailability must exhaust the shared deadline");
         // Transport scheduling consumes the same budget; do not require a
@@ -668,10 +860,134 @@ mod upgrade_write_readiness_tests {
         let phase = "stalled";
         let key = format!(".upgrade-readiness/{phase}/node-0");
         target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Stall(Duration::from_secs(30)), 1);
-        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
+        let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, phase, Duration::from_secs(1))
             .await
             .expect_err("a stalled request must not outlive the readiness deadline");
         assert!(error.to_string().contains("deadline exceeded during PutObject"), "{error}");
+
+        let part = vec![2_u8; 64 * 1024];
+        let successful_key = "stalled/multipart/02/00";
+        mixed_multipart_before(
+            Instant::now() + MIXED_MULTIPART_UPLOAD_TIMEOUT,
+            2,
+            successful_key,
+            write_multipart(&client, MIXED_BUCKET, successful_key, std::slice::from_ref(&part)),
+        )
+        .await?;
+        assert_eq!(read_object(&client, MIXED_BUCKET, successful_key, None).await?.1, part);
+
+        let stalled_key = "stalled/multipart/02/01";
+        target.inject_for_key(
+            FakeTargetOperation::CompleteMultipartUpload,
+            stalled_key,
+            FaultAction::Stall(Duration::from_secs(30)),
+            1,
+        );
+        let error = {
+            let upload = write_multipart(&client, MIXED_BUCKET, stalled_key, std::slice::from_ref(&part));
+            tokio::pin!(upload);
+            tokio::select! {
+                result = &mut upload => panic!("stalled completion unexpectedly returned: {result:?}"),
+                observed = tokio::time::timeout(Duration::from_secs(10), async {
+                    while target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key) == 0
+                        || !target.has_object(MIXED_BUCKET, stalled_key)
+                    {
+                        sleep(Duration::from_millis(10)).await;
+                    }
+                }) => observed.expect("multipart completion must reach and commit on the fake peer"),
+            }
+            // An already committed object is not an acknowledged client success.
+            // Expire only after observing the real completion request, so host
+            // scheduling cannot turn this into a timeout before admission.
+            tokio::time::timeout(Duration::from_secs(1), mixed_multipart_before(Instant::now(), 2, stalled_key, upload))
+                .await
+                .expect("expired deadline must stop the in-flight completion")
+                .expect_err("unacknowledged completion must fail even if the peer committed")
+        };
+        assert!(error.to_string().contains("node 2"), "{error}");
+        assert!(error.to_string().contains(stalled_key), "{error}");
+        assert!(error.to_string().contains("acknowledgement deadline exceeded"), "{error}");
+        assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1);
+
+        let failure = error
+            .downcast_ref::<MultipartAcknowledgementDeadline>()
+            .expect("the timeout must retain typed writer/object context");
+        assert_eq!(failure.node, 2);
+        assert_eq!(failure.key, stalled_key);
+        let original_message = error.to_string();
+        // These four probe addresses share an in-process peer. This verifies
+        // bounded diagnostic fan-out, not a four-node RustFS cluster.
+        let mut probes = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?;
+        probes.access_key = FAKE_ACCESS_KEY.to_owned();
+        probes.secret_key = FAKE_SECRET_KEY.to_owned();
+        for node in &mut probes.nodes {
+            node.url = target.endpoint().to_owned();
+        }
+        let log_dir = tempfile::tempdir()?;
+        target.inject_for_key(
+            FakeTargetOperation::HeadObject,
+            stalled_key,
+            FaultAction::Stall(Duration::from_secs(30)),
+            1,
+        );
+        let error = tokio::time::timeout(
+            Duration::from_secs(7),
+            observe_mixed_multipart_failure(&probes, phase, error, Some(log_dir.path())),
+        )
+        .await
+        .expect("the shared diagnostic budget must include artifact writing")
+        .expect_err("observing committed data cannot acknowledge the stalled upload");
+        assert_eq!(error.to_string(), original_message);
+        assert!(error.downcast_ref::<MultipartAcknowledgementDeadline>().is_some());
+        let snapshot: Value =
+            serde_json::from_slice(&tokio::fs::read(log_dir.path().join("mixed-multipart-failure.json")).await?)?;
+        assert_eq!(snapshot["key"], stalled_key);
+        assert_eq!(snapshot["writer_node"], 2);
+        let heads = snapshot["heads"]
+            .as_array()
+            .expect("failure artifact must retain every probe");
+        assert_eq!(heads.len(), MIXED_NODE_COUNT);
+        assert_eq!(
+            heads
+                .iter()
+                .filter(|head| head["timeout"] == true && head["observation"] == "unknown")
+                .count(),
+            1
+        );
+        assert_eq!(
+            heads
+                .iter()
+                .filter(|head| head["status"] == 200 && head["observation"] == "visible")
+                .count(),
+            3
+        );
+        assert_eq!(target.count_requests(FakeTargetOperation::HeadObject, stalled_key), MIXED_NODE_COUNT);
+        assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1);
+        assert_eq!(snapshot["top_locks"]["observation"], "unknown", "the fake peer has no admin lock API");
+
+        let locks: MixedLockSnapshot = serde_json::from_value(serde_json::json!({
+            "total": 1, "truncated": true, "capability_note": "partial snapshot",
+            "credentials": { "secret_key": "must not persist" },
+            "locks": [{
+                "resource": format!("{MIXED_BUCKET}/{stalled_key}"), "owner": "writer", "type": "WRITE",
+                "elapsed_secs": 30, "ttl_secs": 1, "merged_config": { "secret_key": "must not persist" },
+            }],
+        }))?;
+        let safe_locks = serde_json::to_value(locks)?;
+        assert_eq!(safe_locks["total"], 1);
+        assert_eq!(safe_locks["truncated"], true);
+        assert_eq!(safe_locks["locks"][0]["owner"], "writer");
+        assert_eq!(safe_locks["locks"][0]["elapsed_secs"], 30);
+        assert!(safe_locks.get("credentials").is_none());
+        assert!(safe_locks["locks"][0].get("merged_config").is_none());
+
+        let mut body = br#"{"locks":[]}"#.to_vec();
+        body.resize(1024 * 1024, b' ');
+        let mut at_limit = reqwest::Response::from(http::Response::new(body.clone()));
+        assert!(read_mixed_lock_snapshot(&mut at_limit).await?.locks.is_empty());
+        body.push(b' ');
+        let mut oversized = reqwest::Response::from(http::Response::new(body));
+        assert_eq!(read_mixed_lock_snapshot(&mut oversized).await.err(), Some("body_limit_exceeded"));
         target.shutdown().await;
         Ok(())
     }
@@ -2029,7 +2345,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
     attributes.object_parts().map(|parts| parts.parts().len())
 }
 
-async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
+async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
     let label = case.label();
     let head = layout_head(client, case).await?;
     assert_eq!(
@@ -2046,7 +2362,12 @@ async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &st
     let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
     assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
     assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
+    Ok(())
+}
 
+async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
+    assert_layout_full_body(client, case, context).await?;
+    let label = case.label();
     if case.is_multipart_layout() {
         let first = case.part_sizes[0];
         let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2181,9 +2502,9 @@ struct LayoutTransport {
     uploaded_parts: Vec<i32>,
     single_puts: usize,
     completes: usize,
-    /// Raw per-key journal in target order: (sequence, operation, part number,
-    /// upload id), so duplicate drives can be told apart from retries.
-    journal: Vec<(u64, String, Option<i32>, Option<String>)>,
+    /// Allowlisted request identity and prepared response metadata. A prepared
+    /// response does not prove that the client received it.
+    journal: Vec<Value>,
 }
 
 /// Configure every layout bucket to replicate its existing objects to a fresh
@@ -2245,12 +2566,28 @@ async fn replicate_layouts(
             journal: key_requests
                 .iter()
                 .map(|record| {
-                    (
-                        record.sequence,
-                        format!("{:?}", record.operation),
-                        record.part_number,
-                        record.upload_id.as_ref().map(|id| id.chars().take(12).collect()),
-                    )
+                    serde_json::json!({
+                        "sequence": record.sequence,
+                        "operation": format!("{:?}", record.operation),
+                        "part_number": record.part_number,
+                        "upload_id": record.upload_id,
+                        "requested_version_id": record.version_id,
+                        "source_version_id": record.source_version_id,
+                        "source_mtime": record.source_mtime,
+                        "source_etag": record.source_etag,
+                        "source_replication_request": record.source_replication_request,
+                        "request_content_length": record.content_length,
+                        "journaled_at_unix_millis": record.journaled_at_unix_millis,
+                        "prepared_response": record.prepared_response.as_ref().map(|response| serde_json::json!({
+                            "status": response.status,
+                            "version_id": response.version_id,
+                            "etag": response.etag,
+                            "last_modified": response.last_modified,
+                            "content_length": response.content_length,
+                            "prepared_at_unix_millis": response.prepared_at_unix_millis,
+                            "elapsed_secs": response.elapsed.as_secs_f64(),
+                        })),
+                    })
                 })
                 .collect(),
         };
@@ -2309,10 +2646,11 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
             "rc.5 wrote a legacy layout"
         );
     }
-    // The rc.5 writer must itself still read what it wrote, so a later
-    // failure is attributable to the upgrade rather than to the fixture.
+    // Prove the rc.5 writer persisted every original byte before upgrading.
+    // Its compressed cross-part range reader has a legacy limitation; the
+    // upgraded reader must still satisfy every range and partNumber assertion.
     for case in &cases {
-        assert_layout_readable(&old_client, case, "rc.5").await?;
+        assert_layout_full_body(&old_client, case, "rc.5").await?;
     }
 
     // Upgrade in place.

crates/ecstore/Cargo.toml

@@ -176,13 +176,14 @@ regex = { workspace = true }
 path-absolutize = { workspace = true }
 rmp.workspace = true
 rmp-serde.workspace = true
-tokio-util = { workspace = true, features = ["io", "compat"] }
+tokio-util = { workspace = true, features = ["io", "compat", "rt"] }
 tokio-stream = { workspace = true, features = ["sync"] }
 hmac = { workspace = true }
 sha2 = { workspace = true }
 hex-simd = { workspace = true }
 tempfile.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
+h2 = { workspace = true }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
 hostname.workspace = true
@@ -200,8 +201,8 @@ md-5.workspace = true
 memmap2 = { workspace = true }
 libc.workspace = true
 # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
-# (backlog#1178); "fs" comes from the workspace default.
-rustix = { workspace = true, features = ["process", "fs"] }
+# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
+rustix = { workspace = true, features = ["process", "fs", "thread"] }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }

crates/ecstore/src/api/mod.rs

@@ -199,10 +199,10 @@ pub mod bucket {
             get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
             get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
             get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
-            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
-            update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
-            update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
-            update_under_transaction_lock_at,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
+            remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
+            update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
+            update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
         };
         #[cfg(feature = "test-util")]
         pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,7 +210,8 @@ pub mod bucket {
 
     pub mod migration {
         pub use crate::bucket::migration::{
-            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
+            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
+            try_migrate_iam_config,
         };
     }
 
@@ -240,7 +241,10 @@ pub mod bucket {
     }
 
     pub mod quota {
-        pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
+        pub use crate::bucket::quota::{
+            BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
+            QuotaOperation,
+        };
 
         pub mod checker {
             pub use crate::bucket::quota::checker::QuotaChecker;
@@ -362,10 +366,11 @@ pub mod config {
             ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
             is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
             read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
-            read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
-            save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
-            server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
-            with_server_config_read_lock, with_server_config_write_lock,
+            read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_config_with_opts_quiet,
+            save_server_config, save_server_config_no_lock, save_server_config_snapshot,
+            save_server_config_snapshot_with_generation, server_config_path, try_migrate_server_config,
+            with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
+            with_server_config_write_lock,
         };
     }
 
@@ -418,7 +423,9 @@ pub mod disk {
     pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
+        CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
+        DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
+        DeleteRecoveryRpcReply, DeleteRecoveryState, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
         DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
         PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
         STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
@@ -468,7 +475,7 @@ pub mod event {
 pub mod global {
     pub use crate::runtime::global::{
         set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services,
-        update_erasure_type,
+        update_erasure_type, wait_for_detached_mutations,
     };
 }
 
@@ -515,13 +522,14 @@ pub mod integrity {
 pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
-        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
-        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
+        NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
+        PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
         SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
         ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
         get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
         register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
+        with_get_object_read_condition,
     };
     pub use crate::store::{
         PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -586,6 +594,7 @@ pub mod set_disk {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
+        pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
         pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
 
         /// Keep a namespace commit pending until the returned owner is dropped.

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -87,6 +87,7 @@ use rustfs_filemeta::{
     FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
     is_restored_object_on_disk,
 };
+use rustfs_lock::{NamespaceLockGuard, error::LockError};
 use rustfs_scanner_metrics::metrics::{
     IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
 };
@@ -190,6 +191,8 @@ const TIER_FREE_VERSION_RECOVERY_BASE_INTERVAL: StdDuration = StdDuration::from_
 // before another full sweep can discover work persisted by another node.
 const TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL: StdDuration = StdDuration::from_secs(10 * 60);
 const TIER_FREE_VERSION_RECOVERY_JITTER_PERCENT: u64 = 10;
+const TIER_FREE_VERSION_RECOVERY_LEADER_LOCK: &str = "tier-free-version-recovery/leader.lock";
+const TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT: StdDuration = StdDuration::from_secs(1);
 const DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS: i64 = 5;
 const EXPIRY_WORKER_QUEUE_CAPACITY: usize = 1000;
 const DEFAULT_MANUAL_TRANSITION_JOB_RECOVERY_LIMIT: usize = 100;
@@ -1441,13 +1444,24 @@ pub(crate) async fn enqueue_committed_free_versions(api: &ECStore, free_versions
     queued
 }
 
-async fn enqueue_recovered_free_version_with_state(state: &Arc<RwLock<ExpiryState>>, oi: ObjectInfo) -> bool {
+async fn enqueue_recovered_free_version_with_state(
+    state: &Arc<RwLock<ExpiryState>>,
+    oi: ObjectInfo,
+    cancel: &CancellationToken,
+) -> bool {
     let task = FreeVersionTask(oi);
     let hash = task.op_hash();
     let (wrkr, stats) = {
-        let state = state.read().await;
+        let state = select! {
+            biased;
+            _ = cancel.cancelled() => return false,
+            state = state.read() => state,
+        };
         (state.get_worker_ch(hash), Arc::clone(&state.stats))
     };
+    if cancel.is_cancelled() {
+        return false;
+    }
     let Some(wrkr) = wrkr else {
         stats.increment_missed_freevers_tasks();
         stats.record_scanner_expiry_state();
@@ -1497,9 +1511,13 @@ fn set_recovered_free_version_enqueue_observer(
     RecoveredFreeVersionEnqueueObserverGuard
 }
 
-pub async fn enqueue_recovered_free_version(api: &ECStore, oi: ObjectInfo) -> bool {
+pub(super) async fn enqueue_recovered_free_version_with_cancel(
+    api: &ECStore,
+    oi: ObjectInfo,
+    cancel: &CancellationToken,
+) -> bool {
     let expiry_state = api.ctx.expiry_state();
-    let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi).await;
+    let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi, cancel).await;
 
     #[cfg(test)]
     if let Some(observer) = RECOVERED_FREE_VERSION_ENQUEUE_OBSERVER
@@ -2399,6 +2417,16 @@ fn is_slow_down(err: &Error) -> bool {
     matches!(err, Error::SlowDown)
 }
 
+/// Lifecycle expiry is idempotent when the exact object or version has already
+/// disappeared between the scanner listing and the delete admission.
+///
+/// Keep this predicate narrower than bucket/volume "not found" helpers: an
+/// unavailable bucket or disk is an operational failure and must not be
+/// reported as a completed expiry.
+fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
+    is_err_object_not_found(err) || is_err_version_not_found(err)
+}
+
 /// Resolves the expiry worker count from the single documented knob,
 /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
 /// anything else falls back to `min(cpus, 16)`. The historical
@@ -2921,6 +2949,20 @@ fn resolve_tier_free_version_recovery_enabled(value: Result<String, env::VarErro
 }
 
 fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<()>) -> Option<JoinHandle<()>> {
+    spawn_tier_free_version_recovery_once_with_options(
+        api,
+        started,
+        TIER_FREE_VERSION_RECOVERY_LEADER_LOCK.to_string(),
+        tier_free_version_recovery_leader_retry_delay,
+    )
+}
+
+fn spawn_tier_free_version_recovery_once_with_options(
+    api: Arc<ECStore>,
+    started: &OnceLock<()>,
+    leader_lock_name: String,
+    retry_delay: fn() -> StdDuration,
+) -> Option<JoinHandle<()>> {
     if !tier_free_version_recovery_enabled() {
         warn!(
             event = EVENT_LIFECYCLE_WORKER_STATE,
@@ -2937,20 +2979,43 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<(
 
     Some(tokio::spawn(async move {
         let cancel_token = api.ctx.background_cancel_token().unwrap_or_default();
-        let expiry_state = api.ctx.expiry_state();
-        run_tier_free_version_recovery_loop(
+        let acquire_api = Arc::clone(&api);
+        let acquire_lock_name = Arc::<str>::from(leader_lock_name);
+        run_tier_free_version_recovery_coordinator(
             cancel_token,
-            expiry_state,
-            jitter_tier_free_version_recovery_delay,
-            move |bucket_marker, object_marker, recovery_cancel| {
+            retry_delay,
+            move || {
+                let api = Arc::clone(&acquire_api);
+                let lock_name = Arc::clone(&acquire_lock_name);
+                async move {
+                    let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &lock_name).await?;
+                    match lock.get_write_lock_quiet(TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT).await {
+                        Ok(guard) => Ok(Some(guard)),
+                        Err(LockError::Timeout { .. }) => Ok(None),
+                        Err(err) => Err(err.into()),
+                    }
+                }
+            },
+            move |leader_cancel| {
                 let api = Arc::clone(&api);
                 async move {
-                    recover_tier_free_versions_with_cancel(
-                        api,
-                        DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
-                        bucket_marker,
-                        object_marker,
-                        recovery_cancel,
+                    run_tier_free_version_recovery_loop(
+                        leader_cancel,
+                        api.ctx.expiry_state(),
+                        jitter_tier_free_version_recovery_delay,
+                        move |bucket_marker, object_marker, recovery_cancel| {
+                            let api = Arc::clone(&api);
+                            async move {
+                                recover_tier_free_versions_with_cancel(
+                                    api,
+                                    DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
+                                    bucket_marker,
+                                    object_marker,
+                                    recovery_cancel,
+                                )
+                                .await
+                            }
+                        },
                     )
                     .await
                 }
@@ -2960,6 +3025,86 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<(
     }))
 }
 
+fn tier_free_version_recovery_leader_retry_delay() -> StdDuration {
+    StdDuration::from_secs(rand::rng().random_range(15..=30))
+}
+
+async fn run_tier_free_version_recovery_coordinator<A, AF, R, RF>(
+    cancel: CancellationToken,
+    retry_delay: fn() -> StdDuration,
+    mut acquire: A,
+    mut recover: R,
+) where
+    A: FnMut() -> AF,
+    AF: Future<Output = crate::error::Result<Option<NamespaceLockGuard>>>,
+    R: FnMut(CancellationToken) -> RF,
+    RF: Future<Output = ()>,
+{
+    loop {
+        let acquired = select! {
+            biased;
+            _ = cancel.cancelled() => return,
+            result = acquire() => result,
+        };
+        match acquired {
+            Ok(Some(mut guard)) => {
+                let leader_cancel = cancel.child_token();
+                // Hold ownership across every page, cooldown and idle wait. A
+                // follower taking over starts a fresh sweep of persisted markers.
+                let recovery = recover(leader_cancel.clone());
+                tokio::pin!(recovery);
+                debug!(
+                    event = EVENT_LIFECYCLE_WORKER_STATE,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                    state = "free_version_recovery_leader_acquired",
+                    lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
+                    "Tier free-version recovery leader acquired"
+                );
+                select! {
+                    biased;
+                    _ = cancel.cancelled() => {
+                        leader_cancel.cancel();
+                        recovery.await;
+                    }
+                    _ = guard.lock_lost_notified() => {
+                        debug!(
+                            event = EVENT_LIFECYCLE_WORKER_STATE,
+                            component = LOG_COMPONENT_ECSTORE,
+                            subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                            state = "free_version_recovery_leader_lost",
+                            lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
+                            "Tier free-version recovery leader lost"
+                        );
+                        // Dropping the recovery future would detach its active
+                        // walk. Cancel it and poll the same future through cleanup.
+                        leader_cancel.cancel();
+                        recovery.await;
+                    }
+                    _ = &mut recovery => {}
+                }
+                leader_cancel.cancel();
+                guard.release();
+            }
+            Ok(None) => {}
+            Err(err) => warn!(
+                event = EVENT_LIFECYCLE_WORKER_STATE,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                state = "free_version_recovery_leader_acquire_failed",
+                lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
+                error = %err,
+                "Tier free-version recovery leader acquisition failed"
+            ),
+        }
+        select! {
+            biased;
+            _ = cancel.cancelled() => return,
+            _ = tokio::time::sleep(retry_delay()) => {}
+        }
+    }
+}
+
 async fn run_tier_free_version_recovery_loop<F, Fut>(
     cancel_token: CancellationToken,
     expiry_state: Arc<RwLock<ExpiryState>>,
@@ -2969,7 +3114,11 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
     F: FnMut(Option<String>, Option<String>, CancellationToken) -> Fut,
     Fut: Future<Output = crate::error::Result<FreeVersionRecoveryStats>>,
 {
-    let recovery_notify = Arc::clone(&expiry_state.read().await.recovery_notify);
+    let recovery_notify = select! {
+        biased;
+        _ = cancel_token.cancelled() => return,
+        state = expiry_state.read() => Arc::clone(&state.recovery_notify),
+    };
     let mut schedule = TierFreeVersionRecoverySchedule::default();
 
     loop {
@@ -2992,7 +3141,11 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
                 schedule.record_success(&stats);
                 rustfs_io_metrics::record_stage_duration("lifecycle_free_version_recovery", elapsed.as_secs_f64() * 1000.0);
                 let (pending_tasks, active_tasks) = {
-                    let state = expiry_state.read().await;
+                    let state = select! {
+                        biased;
+                        _ = cancel_token.cancelled() => return,
+                        state = expiry_state.read() => state,
+                    };
                     (state.pending_tasks(), state.stats.active_tasks())
                 };
                 debug!(
@@ -4748,7 +4901,7 @@ async fn expire_transitioned_object_with_lock_lost_signal(
     bucket_incarnation_id: Uuid,
     lock_lost_signal: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
 ) -> Result<ObjectInfo, std::io::Error> {
-    let publication_guard = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id)
+    let publication_guard = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id)
         .await
         .ok_or_else(|| std::io::Error::other("lifecycle expiry is not allowed for this bucket"))?;
     let snapshot = lifecycle_delete_config_snapshot(&api, oi)
@@ -4777,6 +4930,20 @@ async fn expire_transitioned_object_with_lock_lost_signal(
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
+            Err(err) if is_lifecycle_expiry_already_gone(&err) => {
+                debug!(
+                    event = EVENT_LIFECYCLE_DELETE_COMPLETED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    operation = "delete_restored_object",
+                    reason = "already_gone",
+                    "Lifecycle delete target was already absent"
+                );
+                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
+                Ok(oi.clone())
+            }
             Err(err) => Err(std::io::Error::other(err)),
         };
     }
@@ -4787,6 +4954,8 @@ async fn expire_transitioned_object_with_lock_lost_signal(
     // pointing to a remote tier version that has already been removed. If this
     // only creates a delete marker, remote cleanup must be driven by persisted
     // free-version recovery rather than the visible delete result.
+    // Keep remote transitioned not-found errors visible: local metadata
+    // absence alone does not prove that the tier object was cleaned up.
     let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
         Ok(obj) => obj,
         Err(e) => {
@@ -4838,10 +5007,14 @@ pub fn gen_transition_objname(bucket: &str) -> Result<String, Error> {
 pub async fn transition_object(api: Arc<ECStore>, oi: &ObjectInfo, lae: LcAuditEvent) -> Result<(), Error> {
     let time_ilm = Metrics::time_ilm(lae.event.action);
 
+    let (publication_guard, bucket_incarnation_id) = lifecycle_transition_publication_guard(&api, oi)
+        .await
+        .ok_or_else(|| Error::other("lifecycle transition is not allowed for this bucket"))?;
+
     let etag = if let Some(etag) = &oi.etag { etag } else { "" };
     let etag = etag.to_string();
 
-    let opts = ObjectOptions {
+    let mut opts = ObjectOptions {
         transition: TransitionOptions {
             status: lifecycle::TRANSITION_PENDING.to_string(),
             tier: lae.event.storage_class,
@@ -4853,8 +5026,10 @@ pub async fn transition_object(api: Arc<ECStore>, oi: &ObjectInfo, lae: LcAuditE
         versioned: BucketVersioningSys::prefix_enabled(&oi.bucket, &oi.name).await,
         version_suspended: BucketVersioningSys::prefix_suspended(&oi.bucket, &oi.name).await,
         mod_time: oi.mod_time,
+        expected_bucket_incarnation_id: Some(bucket_incarnation_id),
         ..Default::default()
     };
+    opts.add_namespace_lock_guard(&publication_guard);
     let result = api.transition_object(&oi.bucket, &oi.name, &opts).await;
     time_ilm(1)();
     result
@@ -5458,20 +5633,26 @@ pub async fn apply_transition_rule(event: &lifecycle::Event, src: &LcEventSrc, o
         .await
 }
 
-async fn lifecycle_expiry_publication_guard(
+async fn acquire_lifecycle_publication_guard(api: &ECStore, bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard, Error> {
+    let lock = api
+        .new_ns_lock(bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
+        .await?;
+    let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
+    if guard.is_lock_lost() {
+        return Err(Error::other("table-bucket publication lock was lost before lifecycle admission"));
+    }
+    Ok(guard)
+}
+
+pub(crate) async fn lifecycle_expiry_publication_guard(
     api: &ECStore,
-    oi: &ObjectInfo,
+    bucket: &str,
+    object: &str,
     bucket_incarnation_id: Uuid,
 ) -> Option<rustfs_lock::NamespaceLockGuard> {
-    let result = async {
-        let lock = api
-            .new_ns_lock(&oi.bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
-            .await?;
-        let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
-        if guard.is_lock_lost() {
-            return Err(Error::other("table-bucket publication lock was lost before lifecycle delete admission"));
-        }
-        if !metadata_boundary::lifecycle_expiry_allowed(api, &oi.bucket, bucket_incarnation_id).await? {
+    let result: std::result::Result<Option<rustfs_lock::NamespaceLockGuard>, Error> = async {
+        let guard = acquire_lifecycle_publication_guard(api, bucket).await?;
+        if !metadata_boundary::lifecycle_expiry_allowed(api, bucket, bucket_incarnation_id).await? {
             return Ok(None);
         }
         Ok(Some(guard))
@@ -5484,8 +5665,8 @@ async fn lifecycle_expiry_publication_guard(
                 event = EVENT_LIFECYCLE_DELETE_FAILED,
                 component = LOG_COMPONENT_ECSTORE,
                 subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                bucket = %oi.bucket,
-                object = %oi.name,
+                bucket,
+                object,
                 operation = "authorize_lifecycle_expiry",
                 error = %err,
                 "Lifecycle delete admission failed"
@@ -5495,6 +5676,34 @@ async fn lifecycle_expiry_publication_guard(
     }
 }
 
+async fn lifecycle_transition_publication_guard(
+    api: &ECStore,
+    oi: &ObjectInfo,
+) -> Option<(rustfs_lock::NamespaceLockGuard, Uuid)> {
+    let result: std::result::Result<(rustfs_lock::NamespaceLockGuard, Uuid), Error> = async {
+        let guard = acquire_lifecycle_publication_guard(api, &oi.bucket).await?;
+        let configs = metadata_boundary::get_expiry_configs(api, &oi.bucket).await?;
+        Ok((guard, configs.bucket_incarnation_id))
+    }
+    .await;
+    match result {
+        Ok(state) => Some(state),
+        Err(err) => {
+            warn!(
+                event = EVENT_LIFECYCLE_TIER_OPERATION_FAILED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                bucket = %oi.bucket,
+                object = %oi.name,
+                operation = "authorize_lifecycle_transition",
+                error = %err,
+                "Lifecycle transition admission failed"
+            );
+            None
+        }
+    }
+}
+
 pub async fn apply_expiry_on_transitioned_object(
     api: Arc<ECStore>,
     oi: &ObjectInfo,
@@ -5551,7 +5760,8 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
     bucket_incarnation_id: Uuid,
     lock_lost_signal: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
 ) -> bool {
-    let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id).await else {
+    let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id).await
+    else {
         return false;
     };
     let snapshot = match lifecycle_delete_config_snapshot(&api, oi).await {
@@ -5611,11 +5821,27 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
     } else {
         None
     };
+    // The delete owns the publication guard from here on, so a cancelled
+    // expiry worker cannot let table-bucket publication overtake it.
     let mut dobj = match api
-        .delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
+        .delete_object_with_tier_delete_journal_and_guards(&oi.bucket, &encode_dir_object(&oi.name), opts, [publication_guard])
         .await
     {
         Ok(dobj) => dobj,
+        Err(err) if is_lifecycle_expiry_already_gone(&err) => {
+            debug!(
+                event = EVENT_LIFECYCLE_DELETE_COMPLETED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                bucket = %oi.bucket,
+                object = %oi.name,
+                operation = "delete_object",
+                reason = "already_gone",
+                "Lifecycle delete target was already absent"
+            );
+            crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
+            return true;
+        }
         Err(e) => {
             error!(
                 event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -5844,6 +6070,7 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
 #[cfg(test)]
 mod tests {
     use super::expiry_worker_count;
+    use super::lifecycle_transition_publication_guard;
     use super::{
         DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
         DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED, EVENT_LIFECYCLE_EXPIRED_DETECTED,
@@ -5854,15 +6081,16 @@ mod tests {
         cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
         enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
         enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
-        get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
-        lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
-        lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
-        manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
-        manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
-        merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
-        persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
-        resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
-        resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_loop,
+        get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
+        lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
+        lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
+        manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
+        manual_transition_version_marker, manual_transition_worker_failure_reason,
+        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
+        persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
+        recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
+        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
+        resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
         select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
         should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
         wait_for_tier_free_version_recovery,
@@ -5959,6 +6187,231 @@ mod tests {
     use tokio_util::sync::CancellationToken;
     use uuid::Uuid;
 
+    fn recovery_test_lock() -> (Arc<rustfs_lock::NamespaceLock>, Vec<Arc<dyn rustfs_lock::LockClient>>) {
+        let clients: Vec<Arc<dyn rustfs_lock::LockClient>> = (0..3)
+            .map(|_| {
+                Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new())))
+                    as Arc<dyn rustfs_lock::LockClient>
+            })
+            .collect();
+        let lock =
+            rustfs_lock::NamespaceLock::with_clients_and_quorum(format!("recovery-test-{}", Uuid::new_v4()), clients.clone(), 2);
+        (Arc::new(lock), clients)
+    }
+
+    async fn acquire_recovery_test_lock(
+        lock: Arc<rustfs_lock::NamespaceLock>,
+        owner: String,
+    ) -> crate::error::Result<Option<rustfs_lock::NamespaceLockGuard>> {
+        // Keep this lease longer than the virtual-time cooldown/idle advances;
+        // production uses the namespace wrapper's renewable 30-second lease.
+        lock.lock_guard(
+            rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, super::TIER_FREE_VERSION_RECOVERY_LEADER_LOCK),
+            &owner,
+            StdDuration::from_millis(5),
+            StdDuration::from_secs(3600),
+        )
+        .await
+        .map_err(Into::into)
+    }
+
+    fn recovery_test_retry_delay() -> StdDuration {
+        StdDuration::from_secs(15)
+    }
+
+    #[tokio::test(start_paused = true)]
+    #[serial]
+    async fn tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over() {
+        let (lock, _clients) = recovery_test_lock();
+        let (pages_tx, mut pages_rx) = tokio::sync::mpsc::unbounded_channel();
+        let mut tasks = Vec::new();
+        let mut cancels = Vec::new();
+        for node in 0..8 {
+            let cancel = CancellationToken::new();
+            cancels.push(cancel.clone());
+            let lock = Arc::clone(&lock);
+            let pages_tx = pages_tx.clone();
+            tasks.push(tokio::spawn(async move {
+                run_tier_free_version_recovery_coordinator(
+                    cancel,
+                    recovery_test_retry_delay,
+                    move || acquire_recovery_test_lock(Arc::clone(&lock), format!("node-{node}")),
+                    move |leader_cancel| {
+                        let pages_tx = pages_tx.clone();
+                        async move {
+                            let mut pages = 0;
+                            run_tier_free_version_recovery_loop(
+                                leader_cancel,
+                                ExpiryState::new(),
+                                std::convert::identity,
+                                move |bucket, object, _| {
+                                    pages += 1;
+                                    pages_tx.send((node, bucket, object)).expect("page observer should be open");
+                                    std::future::ready(Ok(free_version_recovery_stats(0, 0, pages == 1)))
+                                },
+                            )
+                            .await;
+                        }
+                    },
+                )
+                .await;
+            }));
+        }
+        let (leader, bucket, object) = pages_rx.recv().await.expect("one coordinator should start recovery");
+        assert!(bucket.is_none() && object.is_none());
+        // The same leader must survive both a truncated page's cooldown and
+        // complete idle sweeps. All other coordinators keep probing the lock.
+        let mut observed_pages = 1;
+        for _ in 0..27 {
+            tokio::time::advance(StdDuration::from_secs(60)).await;
+            for _ in 0..100 {
+                tokio::task::yield_now().await;
+            }
+            while let Ok((node, bucket, object)) = pages_rx.try_recv() {
+                assert_eq!(node, leader, "followers must not repeat a namespace sweep");
+                if observed_pages == 1 {
+                    assert_eq!(bucket.as_deref(), Some("bucket"));
+                    assert_eq!(object.as_deref(), Some("object"));
+                } else {
+                    assert!(bucket.is_none() && object.is_none(), "complete sweeps should reset their cursors");
+                }
+                observed_pages += 1;
+            }
+        }
+        assert!(observed_pages >= 7, "the leader should finish a full ten-minute idle wait");
+        cancels[leader].cancel();
+        let (successor, bucket, object) = tokio::time::timeout(StdDuration::from_secs(60), pages_rx.recv())
+            .await
+            .expect("a follower should take over after leader shutdown")
+            .expect("page observer should remain open");
+        assert_ne!(successor, leader);
+        assert!(bucket.is_none() && object.is_none(), "a new leader must restart discovery safely");
+        for cancel in cancels {
+            cancel.cancel();
+        }
+        for task in tasks {
+            task.await.expect("all coordinators should shut down cleanly");
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    #[serial]
+    async fn tier_free_version_recovery_lock_loss_waits_for_active_page_cleanup() {
+        let (lock, _clients) = recovery_test_lock();
+        let cancel = CancellationToken::new();
+        let loop_cancel = cancel.clone();
+        let (page_tx, mut page_rx) = tokio::sync::mpsc::unbounded_channel();
+        let cleanup_started = Arc::new(tokio::sync::Notify::new());
+        let finish_cleanup = Arc::new(tokio::sync::Notify::new());
+        let observed_cleanup = Arc::clone(&cleanup_started);
+        let release_cleanup = Arc::clone(&finish_cleanup);
+        let task = tokio::spawn(async move {
+            run_tier_free_version_recovery_coordinator(
+                loop_cancel,
+                recovery_test_retry_delay,
+                move || acquire_recovery_test_lock(Arc::clone(&lock), "old-leader".to_string()),
+                move |leader_cancel| {
+                    let page_tx = page_tx.clone();
+                    let cleanup_started = Arc::clone(&cleanup_started);
+                    let finish_cleanup = Arc::clone(&finish_cleanup);
+                    async move {
+                        run_tier_free_version_recovery_loop(
+                            leader_cancel,
+                            ExpiryState::new(),
+                            std::convert::identity,
+                            move |_, _, page_cancel| {
+                                let cleanup_started = Arc::clone(&cleanup_started);
+                                let finish_cleanup = Arc::clone(&finish_cleanup);
+                                page_tx.send(()).expect("page observer should remain open");
+                                async move {
+                                    page_cancel.cancelled().await;
+                                    cleanup_started.notify_one();
+                                    finish_cleanup.notified().await;
+                                    Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "cancelled page").into())
+                                }
+                            },
+                        )
+                        .await;
+                    }
+                },
+            )
+            .await;
+        });
+        page_rx.recv().await.expect("leader should start an active page");
+        // Let authoritative backend leases expire before the next heartbeat
+        // can renew them. A failed refresh must cancel the active page.
+        tokio::time::advance(StdDuration::from_secs(3600)).await;
+        tokio::time::timeout(StdDuration::from_secs(1), observed_cleanup.notified())
+            .await
+            .expect("expired backend leases must trigger recovery cancellation");
+        assert!(!task.is_finished(), "lock loss must wait for the active page's cleanup");
+        assert!(page_rx.try_recv().is_err(), "the old leader must not start another page");
+        cancel.cancel();
+        release_cleanup.notify_one();
+        task.await.expect("coordinator should finish cleanup before exiting");
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn tier_free_version_recovery_coordinator_retries_without_scanning_after_lock_error() {
+        let cancel = CancellationToken::new();
+        let loop_cancel = cancel.clone();
+        let attempts = Arc::new(AtomicUsize::new(0));
+        let observed_attempts = Arc::clone(&attempts);
+        let task = tokio::spawn(async move {
+            run_tier_free_version_recovery_coordinator(
+                loop_cancel,
+                recovery_test_retry_delay,
+                move || {
+                    observed_attempts.fetch_add(1, Ordering::SeqCst);
+                    std::future::ready(Err(rustfs_lock::LockError::InsufficientNodes {
+                        required: 2,
+                        available: 1,
+                    }
+                    .into()))
+                },
+                |_| async { panic!("a node without ownership must never start discovery") },
+            )
+            .await;
+        });
+        tokio::task::yield_now().await;
+        assert_eq!(attempts.load(Ordering::SeqCst), 1);
+        tokio::time::advance(StdDuration::from_secs(14)).await;
+        assert_eq!(attempts.load(Ordering::SeqCst), 1, "lock errors must not cause a tight retry loop");
+        tokio::time::advance(StdDuration::from_secs(1)).await;
+        tokio::task::yield_now().await;
+        assert_eq!(attempts.load(Ordering::SeqCst), 2);
+        cancel.cancel();
+        task.await.expect("a follower should respond to shutdown during retry wait");
+    }
+
+    #[tokio::test]
+    async fn tier_free_version_recovery_enqueue_cancels_while_waiting_for_state_lock() {
+        let state = ExpiryState::new();
+        let (tx, mut rx) = tokio::sync::mpsc::channel(1);
+        let mut held = state.write().await;
+        held.tasks_tx.push(tx);
+        let cancel = CancellationToken::new();
+        let enqueue_cancel = cancel.clone();
+        let enqueue_state = Arc::clone(&state);
+        let enqueue = tokio::spawn(async move {
+            enqueue_recovered_free_version_with_state(&enqueue_state, ObjectInfo::default(), &enqueue_cancel).await
+        });
+        tokio::task::yield_now().await;
+        assert!(!enqueue.is_finished(), "enqueue should be blocked on the state lock");
+        cancel.cancel();
+        let queued = tokio::time::timeout(StdDuration::from_secs(1), enqueue)
+            .await
+            .expect("cancellation must not wait for the state write guard")
+            .expect("enqueue should return cleanly");
+        assert!(!queued);
+        assert_eq!(held.pending_tasks(), 0);
+        assert_eq!(held.stats.missed_free_vers_tasks(), 0);
+        assert!(rx.try_recv().is_err(), "a cancelled recovery must not enqueue a task");
+        drop(held);
+        assert!(!enqueue_recovered_free_version_with_state(&state, ObjectInfo::default(), &cancel).await);
+        assert!(rx.try_recv().is_err(), "cancellation must win even when the lock is available");
+    }
+
     fn free_version_recovery_stats(enqueued: usize, failed: usize, truncated: bool) -> FreeVersionRecoveryStats {
         FreeVersionRecoveryStats {
             scanned: enqueued.saturating_add(failed),
@@ -7638,7 +8091,7 @@ mod tests {
         assert_eq!(state.read().await.stats.pending_tasks(), 1);
 
         // The single-slot queue is full for both enqueue paths.
-        assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone()).await);
+        assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await);
         assert_eq!(state.read().await.stats.pending_tasks(), 1);
         assert_eq!(state.read().await.stats.missed_free_vers_tasks(), 1);
 
@@ -7670,7 +8123,7 @@ mod tests {
             ..Default::default()
         };
 
-        let queued = enqueue_recovered_free_version_with_state(&state, oi).await;
+        let queued = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
         let state = state.read().await;
 
         assert!(!queued);
@@ -7799,8 +8252,8 @@ mod tests {
             ..Default::default()
         };
 
-        let first = enqueue_recovered_free_version_with_state(&state, oi.clone()).await;
-        let second = enqueue_recovered_free_version_with_state(&state, oi).await;
+        let first = enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await;
+        let second = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
         let state = state.read().await;
 
         assert!(first);
@@ -7901,7 +8354,7 @@ mod tests {
         };
 
         assert!(
-            super::enqueue_recovered_free_version(&ecstore, oi).await,
+            super::enqueue_recovered_free_version_with_cancel(&ecstore, oi, &CancellationToken::new()).await,
             "the resized production worker queue should accept the task"
         );
         stop_tx.send(None).await.expect("worker stop signal should be delivered");
@@ -10099,6 +10552,26 @@ mod tests {
         );
     }
 
+    #[test]
+    fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
+        assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
+        assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
+            "bucket".to_string(),
+            "object".to_string(),
+        )));
+        assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
+        assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
+            "bucket".to_string(),
+            "object".to_string(),
+            "version".to_string(),
+        )));
+
+        assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
+        assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
+        assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
+        assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
+    }
+
     #[tokio::test]
     async fn manual_transition_counts_already_transitioned_object() {
         let lc = latest_transition_lifecycle();
@@ -12973,7 +13446,9 @@ mod tests {
                 .bucket_incarnation_id_from_disk(&bucket)
                 .await
                 .expect("bucket incarnation");
-            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
+            // Replaying the same scanner snapshot after the delete is an
+            // already-gone success and must not be reported as a failure.
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
                 assert_eq!(
                     super::apply_expiry_on_non_transitioned_objects(
                         ecstore.clone(),
@@ -13028,6 +13503,171 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn queued_noncurrent_batch_expiry_does_not_delete_from_table_bucket() {
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("table-noncurrent-lifecycle-{}", Uuid::new_v4().simple());
+        let object = "tables/table-id/data/part-00001.parquet";
+        create_test_bucket(&ecstore, &bucket).await;
+        metadata_sys::update_in(
+            &ecstore.ctx,
+            &bucket,
+            BUCKET_VERSIONING_CONFIG,
+            b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
+        )
+        .await
+        .expect("bucket versioning should be enabled");
+
+        let mut old_reader = PutObjReader::from_vec(b"old table data".to_vec());
+        let old = ecstore
+            .put_object(
+                &bucket,
+                object,
+                &mut old_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("the historical table version should be created");
+        let old_version_id = old.version_id.expect("the historical version should have an identity");
+        let mut current_reader = PutObjReader::from_vec(b"current table data".to_vec());
+        ecstore
+            .put_object(
+                &bucket,
+                object,
+                &mut current_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("the current table version should be created");
+
+        let incarnation = ecstore
+            .bucket_incarnation_id_from_disk(&bucket)
+            .await
+            .expect("bucket incarnation should be available");
+        let publication_lock = ecstore
+            .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
+            .await
+            .expect("table-bucket publication lock should be created");
+        let enable_guard = publication_lock
+            .get_write_lock(get_lock_acquire_timeout())
+            .await
+            .expect("table-bucket enablement should acquire the publication lock");
+
+        let delete_store = ecstore.clone();
+        let delete_bucket = bucket.clone();
+        let delete_object = object.to_string();
+        let mut delete = tokio::spawn(async move {
+            let target = ObjectToDelete {
+                object_name: delete_object,
+                version_id: Some(old_version_id),
+                ..Default::default()
+            };
+            crate::bucket::lifecycle::object_handlers_common::delete_object_versions(
+                &delete_store,
+                &delete_bucket,
+                std::slice::from_ref(&target),
+                lifecycle::Event::default(),
+                incarnation,
+            )
+            .await
+        });
+        tokio::task::yield_now().await;
+        assert!(
+            tokio::time::timeout(StdDuration::from_millis(100), &mut delete)
+                .await
+                .is_err(),
+            "noncurrent lifecycle cleanup must wait for the table-bucket publication fence"
+        );
+
+        let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system should be initialized");
+        let sys = sys.read().await.clone();
+        let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata should exist")).clone();
+        metadata.table_bucket_config_json = br#"{"enabled":true}"#.to_vec();
+        sys.persist_and_set(metadata)
+            .await
+            .expect("table bucket marker should be persisted");
+        sys.reload_from_store(&bucket)
+            .await
+            .expect("table bucket marker should become authoritative");
+        drop(enable_guard);
+
+        let failed = tokio::time::timeout(StdDuration::from_secs(2), delete)
+            .await
+            .expect("noncurrent lifecycle cleanup should resume after the fence is released")
+            .expect("noncurrent lifecycle cleanup task should join");
+        assert_eq!(failed, 1, "table-bucket noncurrent cleanup must be rejected");
+        assert!(
+            ecstore
+                .get_object_info(
+                    &bucket,
+                    object,
+                    &ObjectOptions {
+                        version_id: Some(old_version_id.to_string()),
+                        versioned: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .is_ok(),
+            "table data must remain after queued noncurrent cleanup is rejected"
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn lifecycle_transition_admission_holds_publication_reader() {
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("lifecycle-transition-{}", Uuid::new_v4().simple());
+        let object = "tables/table-id/data/part-00001.parquet";
+        create_test_bucket(&ecstore, &bucket).await;
+        let object_info = ObjectInfo {
+            bucket: bucket.clone(),
+            name: object.to_string(),
+            ..Default::default()
+        };
+
+        let publication_lock = ecstore
+            .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
+            .await
+            .expect("table-bucket publication lock should be created");
+        let enable_guard = publication_lock
+            .get_write_lock(get_lock_acquire_timeout())
+            .await
+            .expect("publication writer should be acquired");
+        let admission_store = ecstore.clone();
+        let admission_object = object_info.clone();
+        let mut admission =
+            tokio::spawn(async move { lifecycle_transition_publication_guard(&admission_store, &admission_object).await });
+        assert!(
+            tokio::time::timeout(StdDuration::from_millis(100), &mut admission)
+                .await
+                .is_err(),
+            "transition admission must wait for an in-progress table publication"
+        );
+        drop(enable_guard);
+
+        let (transition_guard, incarnation) = tokio::time::timeout(StdDuration::from_secs(2), admission)
+            .await
+            .expect("transition admission should resume after publication")
+            .expect("transition admission task should join")
+            .expect("transition admission should succeed");
+        assert_ne!(incarnation, Uuid::nil(), "transition admission must bind a bucket incarnation");
+        let blocked_writer = publication_lock.get_write_lock(StdDuration::from_millis(100)).await;
+        assert!(blocked_writer.is_err(), "transition admission must retain the publication read lock");
+        drop(transition_guard);
+        publication_lock
+            .get_write_lock(get_lock_acquire_timeout())
+            .await
+            .expect("publication writer should proceed after transition admission releases its guard");
+    }
+
     #[tokio::test]
     #[serial]
     async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {
@@ -13630,6 +14270,10 @@ mod tests {
         .await;
     }
 
+    fn fast_tier_free_version_recovery_leader_retry_delay() -> StdDuration {
+        StdDuration::from_millis(10)
+    }
+
     #[tokio::test]
     #[serial]
     async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
@@ -13645,8 +14289,13 @@ mod tests {
         seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
 
         let started = OnceLock::new();
-        let recovery = super::spawn_tier_free_version_recovery_once(Arc::clone(&ecstore), &started)
-            .expect("production recovery entrypoint should start once");
+        let recovery = super::spawn_tier_free_version_recovery_once_with_options(
+            Arc::clone(&ecstore),
+            &started,
+            format!("tier-free-version-recovery/test/{}", Uuid::new_v4()),
+            fast_tier_free_version_recovery_leader_retry_delay,
+        )
+        .expect("production recovery entrypoint should start once");
         let task = tokio::time::timeout(StdDuration::from_secs(30), async { recovery_rx.lock().await.recv().await })
             .await
             .expect("production recovery entrypoint should enqueue the seeded item")
@@ -13689,13 +14338,14 @@ mod tests {
         };
         let mut recovery_rx = recovery_rx.lock().await;
         assert!(
-            super::enqueue_recovered_free_version(
+            super::enqueue_recovered_free_version_with_cancel(
                 &ecstore,
                 ObjectInfo {
                     bucket: "prefill".to_string(),
                     name: "prefill".to_string(),
                     ..Default::default()
                 },
+                &CancellationToken::new(),
             )
             .await,
             "the production recovery queue should accept its first task"

crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs

@@ -36,6 +36,10 @@ pub async fn delete_object_versions(
     _lc_event: lifecycle::Event,
     bucket_incarnation_id: Uuid,
 ) -> usize {
+    if to_del.is_empty() {
+        return 0;
+    }
+
     if to_del.iter().any(|target| {
         target.version_id.is_none()
             || (target.version_id.is_some_and(|version_id| version_id.is_nil()) && target.expected_identity.is_none())
@@ -52,6 +56,25 @@ pub async fn delete_object_versions(
         return to_del.len();
     }
 
+    let Some(publication_guard) = super::bucket_lifecycle_ops::lifecycle_expiry_publication_guard(
+        api,
+        bucket,
+        to_del[0].object_name.as_str(),
+        bucket_incarnation_id,
+    )
+    .await
+    else {
+        debug!(
+            event = EVENT_LIFECYCLE_CLEANUP_SKIPPED,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+            bucket,
+            target_count = to_del.len(),
+            reason = "publication_admission_rejected",
+            "Skipped lifecycle noncurrent version cleanup"
+        );
+        return to_del.len();
+    };
     let delete_config_snapshot = match ReplicationObjectBridge::delete_request_config(api, bucket).await {
         Ok(snapshot) => Arc::new(snapshot),
         Err(err) => {
@@ -78,17 +101,13 @@ pub async fn delete_object_versions(
             remaining = &[];
         }
 
-        let (mut deleted_objs, errors) = api
-            .delete_objects(
-                bucket,
-                to_del.to_vec(),
-                ObjectOptions {
-                    delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)),
-                    expected_bucket_incarnation_id: Some(bucket_incarnation_id),
-                    ..Default::default()
-                },
-            )
-            .await;
+        let mut delete_options = ObjectOptions {
+            delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)),
+            expected_bucket_incarnation_id: Some(bucket_incarnation_id),
+            ..Default::default()
+        };
+        delete_options.add_namespace_lock_guard(&publication_guard);
+        let (mut deleted_objs, errors) = api.delete_objects(bucket, to_del.to_vec(), delete_options).await;
         failed += errors.iter().filter(|err| err.is_some()).count();
 
         for (i, deleted_obj) in deleted_objs.iter_mut().enumerate() {

crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs

@@ -20,7 +20,7 @@ use std::time::Duration;
 use tokio::sync::mpsc;
 use tokio_util::sync::CancellationToken;
 
-use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version;
+use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version_with_cancel;
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
 use crate::object_api::ObjectInfo;
@@ -201,7 +201,11 @@ pub(super) async fn recover_tier_free_versions_with_cancel(
             return Err(tier_free_version_recovery_cancelled());
         }
         retry_cursor.visit(&oi);
-        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {
+        let queued = enqueue_recovered_free_version_with_cancel(&api, oi, &cancel_token).await;
+        if cancel_token.is_cancelled() {
+            return Err(tier_free_version_recovery_cancelled());
+        }
+        if !record_recovered_free_version_enqueue(&mut stats, queued) {
             let (bucket_marker, object_marker) = retry_cursor.retry_markers();
             stats.truncated = true;
             stats.next_bucket_marker = bucket_marker;

crates/ecstore/src/bucket/metadata.rs

@@ -1210,6 +1210,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.notification_config = None;
         if !self.notification_config_xml.is_empty()
             && let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
                 .map(|c| self.notification_config = Some(c))
@@ -1224,6 +1225,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.lifecycle_config = None;
         if !self.lifecycle_config_xml.is_empty()
             && let Err(e) =
                 deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
@@ -1238,6 +1240,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.object_lock_config = None;
         if !self.object_lock_config_xml.is_empty()
             && let Err(e) =
                 deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
@@ -1267,6 +1270,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.sse_config = None;
         if !self.encryption_config_xml.is_empty()
             && let Err(e) =
                 deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
@@ -1281,6 +1285,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.tagging_config = None;
         if !self.tagging_config_xml.is_empty()
             && let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
         {
@@ -1346,6 +1351,7 @@ impl BucketMetadata {
         } else {
             self.bucket_target_config = Some(BucketTargets::default());
         }
+        self.cors_config = None;
         if !self.cors_config_xml.is_empty()
             && let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
         {
@@ -1359,6 +1365,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.logging_config = None;
         if !self.logging_config_xml.is_empty()
             && let Err(e) = deserialize::<BucketLoggingStatus>(&self.logging_config_xml).map(|c| self.logging_config = Some(c))
         {
@@ -1372,6 +1379,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.website_config = None;
         if !self.website_config_xml.is_empty()
             && let Err(e) = deserialize::<WebsiteConfiguration>(&self.website_config_xml).map(|c| self.website_config = Some(c))
         {
@@ -1385,6 +1393,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.accelerate_config = None;
         if !self.accelerate_config_xml.is_empty()
             && let Err(e) =
                 deserialize::<AccelerateConfiguration>(&self.accelerate_config_xml).map(|c| self.accelerate_config = Some(c))
@@ -1399,6 +1408,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.request_payment_config = None;
         if !self.request_payment_config_xml.is_empty()
             && let Err(e) = deserialize::<RequestPaymentConfiguration>(&self.request_payment_config_xml)
                 .map(|c| self.request_payment_config = Some(c))
@@ -1413,6 +1423,7 @@ impl BucketMetadata {
                 "Failed to parse bucket metadata config"
             );
         }
+        self.public_access_block_config = None;
         if !self.public_access_block_config_xml.is_empty()
             && let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
                 .map(|c| self.public_access_block_config = Some(c))
@@ -1973,6 +1984,156 @@ mod test {
         assert_eq!(bm.lifecycle_config_xml, malformed_xml, "raw bytes are retained");
     }
 
+    const REPARSE_XML_FIXTURES: [(&str, &[u8]); 13] = [
+        (BUCKET_NOTIFICATION_CONFIG, br#"<NotificationConfiguration/>"#),
+        (
+            BUCKET_LIFECYCLE_CONFIG,
+            br#"<LifecycleConfiguration><Rule><ID>keep</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#,
+        ),
+        (
+            OBJECT_LOCK_CONFIG,
+            br#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#,
+        ),
+        (BUCKET_VERSIONING_CONFIG, br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#),
+        (
+            BUCKET_SSECONFIG,
+            br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#,
+        ),
+        (BUCKET_TAGGING_CONFIG, br#"<Tagging><TagSet><Tag><Key>team</Key><Value>blue</Value></Tag></TagSet></Tagging>"#),
+        (
+            BUCKET_REPLICATION_CONFIG,
+            br#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>copy</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#,
+        ),
+        (
+            BUCKET_CORS_CONFIG,
+            br#"<CORSConfiguration><CORSRule><AllowedOrigin>https://example.test</AllowedOrigin><AllowedMethod>GET</AllowedMethod></CORSRule></CORSConfiguration>"#,
+        ),
+        (BUCKET_LOGGING_CONFIG, br#"<BucketLoggingStatus/>"#),
+        (BUCKET_WEBSITE_CONFIG, br#"<WebsiteConfiguration><IndexDocument><Suffix>index.html</Suffix></IndexDocument></WebsiteConfiguration>"#),
+        (BUCKET_ACCELERATE_CONFIG, br#"<AccelerateConfiguration><Status>Enabled</Status></AccelerateConfiguration>"#),
+        (BUCKET_REQUEST_PAYMENT_CONFIG, br#"<RequestPaymentConfiguration><Payer>BucketOwner</Payer></RequestPaymentConfiguration>"#),
+        (
+            BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG,
+            br#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>true</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#,
+        ),
+    ];
+
+    fn reparse_xml_state(bm: &BucketMetadata, config: &str) -> &'static str {
+        fn state<T>(raw: &[u8], parsed: &Option<T>) -> &'static str {
+            match ConfigState::of(raw, parsed) {
+                ConfigState::Absent => "absent",
+                ConfigState::Valid(_) => "valid",
+                ConfigState::Unreadable { .. } => "unreadable",
+            }
+        }
+        match config {
+            BUCKET_NOTIFICATION_CONFIG => state(&bm.notification_config_xml, &bm.notification_config),
+            BUCKET_LIFECYCLE_CONFIG => state(&bm.lifecycle_config_xml, &bm.lifecycle_config),
+            OBJECT_LOCK_CONFIG => state(&bm.object_lock_config_xml, &bm.object_lock_config),
+            BUCKET_VERSIONING_CONFIG => state(&bm.versioning_config_xml, &bm.versioning_config),
+            BUCKET_SSECONFIG => state(&bm.encryption_config_xml, &bm.sse_config),
+            BUCKET_TAGGING_CONFIG => state(&bm.tagging_config_xml, &bm.tagging_config),
+            BUCKET_REPLICATION_CONFIG => state(&bm.replication_config_xml, &bm.replication_config),
+            BUCKET_CORS_CONFIG => state(&bm.cors_config_xml, &bm.cors_config),
+            BUCKET_LOGGING_CONFIG => state(&bm.logging_config_xml, &bm.logging_config),
+            BUCKET_WEBSITE_CONFIG => state(&bm.website_config_xml, &bm.website_config),
+            BUCKET_ACCELERATE_CONFIG => state(&bm.accelerate_config_xml, &bm.accelerate_config),
+            BUCKET_REQUEST_PAYMENT_CONFIG => state(&bm.request_payment_config_xml, &bm.request_payment_config),
+            BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG => state(&bm.public_access_block_config_xml, &bm.public_access_block_config),
+            _ => panic!("unknown XML config: {config}"),
+        }
+    }
+
+    // A persisted payload can change without going through the request decoder.
+    fn replace_xml_raw(bm: &mut BucketMetadata, config: &str, raw: Vec<u8>) {
+        match config {
+            BUCKET_NOTIFICATION_CONFIG => bm.notification_config_xml = raw,
+            BUCKET_LIFECYCLE_CONFIG => bm.lifecycle_config_xml = raw,
+            OBJECT_LOCK_CONFIG => bm.object_lock_config_xml = raw,
+            BUCKET_VERSIONING_CONFIG => bm.versioning_config_xml = raw,
+            BUCKET_SSECONFIG => bm.encryption_config_xml = raw,
+            BUCKET_TAGGING_CONFIG => bm.tagging_config_xml = raw,
+            BUCKET_REPLICATION_CONFIG => bm.replication_config_xml = raw,
+            BUCKET_CORS_CONFIG => bm.cors_config_xml = raw,
+            BUCKET_LOGGING_CONFIG => bm.logging_config_xml = raw,
+            BUCKET_WEBSITE_CONFIG => bm.website_config_xml = raw,
+            BUCKET_ACCELERATE_CONFIG => bm.accelerate_config_xml = raw,
+            BUCKET_REQUEST_PAYMENT_CONFIG => bm.request_payment_config_xml = raw,
+            BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG => bm.public_access_block_config_xml = raw,
+            _ => panic!("unknown XML config: {config}"),
+        }
+    }
+
+    #[test]
+    fn reparse_xml_rejects_stale_views_after_corruption() {
+        let corrupt = b"<not-a-valid-document";
+        for (config, valid) in REPARSE_XML_FIXTURES {
+            let mut bm = BucketMetadata::new("reparse-corrupt");
+            bm.update_config(config, valid.to_vec()).expect("store valid XML");
+            bm.parse_all_configs().expect("parse valid XML");
+            assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: valid setup");
+
+            replace_xml_raw(&mut bm, config, corrupt.to_vec());
+            let before = bm.marshal_msg().expect("marshal corrupt raw metadata");
+            bm.parse_all_configs()
+                .expect("one corrupt config must not fail the metadata load");
+
+            assert_eq!(reparse_xml_state(&bm, config), "unreadable", "{config}: stale view accepted");
+            assert_eq!(bm.xml_config_unreadable_len(config), Some(corrupt.len()), "{config}: missing refusal");
+            assert_eq!(
+                bm.marshal_msg().expect("marshal reparsed metadata"),
+                before,
+                "{config}: persisted bytes changed"
+            );
+        }
+    }
+
+    #[test]
+    fn reparse_xml_clears_stale_views_after_raw_is_removed() {
+        for (config, valid) in REPARSE_XML_FIXTURES {
+            let mut bm = BucketMetadata::new("reparse-removed");
+            bm.update_config(config, valid.to_vec()).expect("store valid XML");
+            bm.parse_all_configs().expect("parse valid XML");
+            assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: valid setup");
+
+            replace_xml_raw(&mut bm, config, Vec::new());
+            let before = bm.marshal_msg().expect("marshal absent raw metadata");
+            bm.parse_all_configs().expect("parse removed XML");
+
+            assert_eq!(reparse_xml_state(&bm, config), "absent", "{config}: stale view survived removal");
+            assert!(!bm.xml_config_unreadable(config), "{config}: absence is not corruption");
+            assert_eq!(
+                bm.marshal_msg().expect("marshal reparsed metadata"),
+                before,
+                "{config}: persisted bytes changed"
+            );
+        }
+    }
+
+    #[test]
+    fn reparse_xml_recovers_after_corrupt_raw_is_repaired() {
+        for (config, valid) in REPARSE_XML_FIXTURES {
+            let mut bm = BucketMetadata::new("reparse-repaired");
+            bm.update_config(config, valid.to_vec()).expect("store valid XML");
+            bm.parse_all_configs().expect("parse valid XML");
+            replace_xml_raw(&mut bm, config, b"<not-a-valid-document".to_vec());
+            bm.parse_all_configs().expect("parse corrupt XML");
+            assert_eq!(reparse_xml_state(&bm, config), "unreadable", "{config}: corrupt setup");
+
+            replace_xml_raw(&mut bm, config, valid.to_vec());
+            let before = bm.marshal_msg().expect("marshal repaired raw metadata");
+            bm.parse_all_configs().expect("parse repaired XML");
+
+            assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: repaired config must become readable");
+            assert!(!bm.xml_config_unreadable(config), "{config}: repaired config is not corrupt");
+            assert_eq!(
+                bm.marshal_msg().expect("marshal reparsed metadata"),
+                before,
+                "{config}: persisted bytes changed"
+            );
+        }
+    }
+
     #[test]
     fn lifecycle_update_config_clears_parsed_config_on_delete() {
         let mut bm = BucketMetadata::new("test-bucket");

crates/ecstore/src/bucket/metadata_sys.rs

@@ -178,6 +178,13 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
     Ok(ObjectLockConfigState::ConfirmedAbsent)
 }
 
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct CachedQuotaMetadataSnapshot {
+    pub(crate) quota: Option<BucketQuota>,
+    pub(crate) bucket_incarnation: Uuid,
+    pub(crate) quota_revision: OffsetDateTime,
+}
+
 /// Convert the persisted serving-layer configuration into the storage-level
 /// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention)
 /// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a
@@ -314,6 +321,33 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
     Ok(lock.remove(bucket).await)
 }
 
+/// Remove a peer's cached metadata only when the bucket is absent under the
+/// namespace read fence. A delayed delete notification must not withdraw the
+/// metadata of a same-name bucket that has already been recreated.
+pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
+    if is_meta_bucketname(bucket) {
+        return Ok(false);
+    }
+    // Lock order: bucket namespace read -> metadata system read (released)
+    // -> metadata publish, matching peer reloads.
+    let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
+    let namespace_guard = namespace_lock
+        .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
+        .await?;
+    let sys = bucket_metadata_sys_of(&api.ctx)?;
+    let sys = sys.read().await.clone();
+    if sys
+        .bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
+        .await?
+    {
+        return Ok(false);
+    }
+    await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
+        Ok(sys.remove(bucket).await)
+    })
+    .await
+}
+
 fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
     let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
         warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -491,6 +525,36 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance:
     sys.get_bucket_incarnation_id_from_disk(bucket).await
 }
 
+/// Directory loss is recoverable only for a committed, non-retired generation.
+/// The caller retains the bucket lifecycle fence through the ensuing reconstruction.
+pub(crate) async fn validate_bucket_reconstruction_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    expected: Uuid,
+) -> Result<()> {
+    let api = bucket_metadata_sys_of(ctx)?.read().await.object_store();
+    let lock = api
+        .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
+        .await?;
+    let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
+    let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?;
+    if !persisted
+        || !metadata.bucket_incarnation_sidecar
+        || expected.is_nil()
+        || metadata.bucket_incarnation_id != expected
+        || !metadata.bucket_creation_committed
+    {
+        return Err(Error::PreconditionFailed);
+    }
+    if crate::bucket::retirement::is_retired(api, bucket, expected).await? {
+        return Err(Error::BucketNotFound(bucket.to_owned()));
+    }
+    if guard.is_lock_lost() {
+        return Err(Error::PreconditionFailed);
+    }
+    Ok(())
+}
+
 /// Validate against disk while retaining an already-held Object Lock fence.
 pub(crate) async fn get_bucket_incarnation_id_for_options_in(
     ctx: &crate::runtime::instance::InstanceContext,
@@ -576,6 +640,27 @@ pub(crate) async fn update_in(
     Box::pin(update_with_sys(require_bucket_metadata_sys_in(ctx)?, bucket, config_file, data)).await
 }
 
+/// Validate a caller's fence after acquiring the lifecycle and metadata
+/// transaction locks, before changing the configuration.
+pub(crate) async fn update_validated_in<F>(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    config_file: &str,
+    data: Vec<u8>,
+    validate: F,
+) -> Result<OffsetDateTime>
+where
+    F: FnOnce() -> Result<()> + Send,
+{
+    Box::pin(async {
+        let sys = require_bucket_metadata_sys_in(ctx)?;
+        let guard = acquire_config_write_guard(sys.clone(), bucket).await?;
+        validate()?;
+        update_under_config_write_guard(sys, &guard, config_file, data, None).await
+    })
+    .await
+}
+
 pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
     delete_with_sys(get_bucket_metadata_sys()?, bucket, config_file).await
 }
@@ -1132,7 +1217,7 @@ async fn acquire_transaction_lock_with_sys(
 /// agree on, so renaming it would leave a mixed-version cluster with two
 /// disjoint keys — and old and new nodes would stop excluding each other on
 /// the very writes that are serialized today.
-fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
+pub(crate) fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
     format!("bucket-targets/{bucket}/transaction.lock")
 }
 
@@ -1271,14 +1356,34 @@ pub(crate) async fn get_object_lock_config_state_in(
 pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
+) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
+    get_object_lock_config_and_incarnation_from_disk_with_guard_in(ctx, bucket, None).await
+}
+
+/// Bucket deletion already owns the namespace/lifecycle fences. Reuse its
+/// live guard rather than recursively acquiring the bucket namespace lock.
+/// The caller must also hold the metadata transaction read lock.
+pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    guard: Option<&rustfs_lock::NamespaceLockGuard>,
 ) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
     let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
 
-    match bucket_meta_sys
-        .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
-        .await?
-    {
+    let authority = match guard {
+        Some(guard) => {
+            bucket_meta_sys
+                .read_authoritative_metadata_from_disk_under_guard(bucket, guard)
+                .await?
+        }
+        None => {
+            bucket_meta_sys
+                .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
+                .await?
+        }
+    };
+    match authority {
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
@@ -1331,13 +1436,60 @@ pub(crate) async fn get_quota_config_and_incarnation_from_disk_in(
     let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
 
-    match bucket_meta_sys
-        .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
-        .await?
+    quota_config_and_incarnation_from_authority(
+        bucket,
+        bucket_meta_sys
+            .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
+            .await?,
+    )
+}
+
+pub(crate) async fn get_quota_config_and_incarnation_from_disk_for_options_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    opts: &ObjectOptions,
+    transaction_guard: &Arc<rustfs_lock::NamespaceLockGuard>,
+) -> Result<(Option<BucketQuota>, Uuid, OffsetDateTime)> {
+    let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() else {
+        return get_quota_config_and_incarnation_from_disk_in(ctx, bucket).await;
+    };
+    let sys = bucket_metadata_sys_of(ctx)?.read().await.clone();
+    let store = sys.object_store();
+    let snapshot_guard = snapshot
+        .metadata_transaction_guard_for(store.id, bucket, opts.expected_bucket_incarnation_id)
+        .filter(|guard| Arc::ptr_eq(guard, transaction_guard))
+        .ok_or_else(|| Error::other("quota snapshot does not hold this bucket's metadata transaction fence"))?;
+    if snapshot_guard.is_released() {
+        return Err(Error::other("quota snapshot metadata transaction fence was released"));
+    }
+
+    // The snapshot carries the caller's lifecycle fence, excluding deletion.
+    // Its transaction guard excludes config writes, so a same-name object
+    // owner need not recursively acquire the bucket namespace here.
+    let authority = sys
+        .read_authoritative_metadata_from_disk_under_guard(bucket, transaction_guard)
+        .await?;
+    if snapshot_guard.is_lock_lost()
+        || snapshot_guard.is_released()
+        || matches!(&authority, BucketMetadataAuthority::Authoritative(metadata)
+            if !snapshot.is_valid_for_destructive_put(store.id, bucket, metadata.bucket_incarnation_id))
     {
+        return Err(Error::other("quota snapshot fence or bucket incarnation changed during the disk read"));
+    }
+    quota_config_and_incarnation_from_authority(bucket, authority)
+}
+
+fn quota_config_and_incarnation_from_authority(
+    bucket: &str,
+    authority: BucketMetadataAuthority,
+) -> Result<(Option<BucketQuota>, Uuid, OffsetDateTime)> {
+    match authority {
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
+            if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
+                return Err(Error::other("persisted bucket quota configuration is invalid"));
+            }
             Ok((
                 metadata.quota_config.clone(),
                 metadata.bucket_incarnation_id,
@@ -1352,6 +1504,15 @@ pub(crate) async fn get_quota_config_and_incarnation_from_disk_in(
     }
 }
 
+pub(crate) async fn get_cached_quota_config_and_incarnation_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+) -> Result<Option<CachedQuotaMetadataSnapshot>> {
+    let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
+    let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
+    bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
+}
+
 pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -2703,29 +2864,30 @@ impl BucketMetadataSys {
     async fn read_authoritative_metadata_from_disk_under_transaction_lock(
         &self,
         bucket: &str,
+    ) -> Result<BucketMetadataAuthority> {
+        let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
+        let namespace_guard = namespace_lock
+            .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await?;
+        self.read_authoritative_metadata_from_disk_under_guard(bucket, &namespace_guard)
+            .await
+    }
+
+    async fn read_authoritative_metadata_from_disk_under_guard(
+        &self,
+        bucket: &str,
+        guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<BucketMetadataAuthority> {
         #[cfg(test)]
         if self.object_lock_disk_read_errors.write().await.remove(bucket) {
             return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
         }
 
-        let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
-        let namespace_guard = namespace_lock
-            .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
-            .await?;
-        match await_bucket_namespace_operation(
-            Some(&namespace_guard),
-            bucket,
-            "bucket metadata snapshot existence check",
-            async {
-                self.object_store()
-                    .get_bucket_info_from_sets_at_read_quorum(
-                        bucket,
-                        &crate::storage_api_contracts::bucket::BucketOptions::default(),
-                    )
-                    .await
-            },
-        )
+        match await_bucket_namespace_operation(Some(guard), bucket, "bucket metadata snapshot existence check", async {
+            self.object_store()
+                .get_bucket_info_from_sets_at_read_quorum(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
+                .await
+        })
         .await
         {
             Ok(_) => {}
@@ -2734,7 +2896,7 @@ impl BucketMetadataSys {
         }
 
         let (metadata, persisted) = await_bucket_namespace_operation(
-            Some(&namespace_guard),
+            Some(guard),
             bucket,
             "bucket metadata authoritative snapshot",
             load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
@@ -2866,6 +3028,27 @@ impl BucketMetadataSys {
         }
     }
 
+    async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result<Option<CachedQuotaMetadataSnapshot>> {
+        let metadata = {
+            let metadata_map = self.metadata_map.read().await;
+            metadata_map.get(bucket).cloned()
+        };
+        let Some(metadata) = metadata else {
+            return Ok(None);
+        };
+        if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() {
+            return Ok(None);
+        }
+        if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
+            return Err(Error::other("persisted bucket quota configuration is invalid"));
+        }
+        Ok(Some(CachedQuotaMetadataSnapshot {
+            quota: metadata.quota_config.clone(),
+            bucket_incarnation: metadata.bucket_incarnation_id,
+            quota_revision: metadata.quota_config_updated_at,
+        }))
+    }
+
     pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
@@ -4121,6 +4304,114 @@ mod tests {
         sys.persist_new_and_set(bm).await.expect("initial metadata should persist");
     }
 
+    #[tokio::test]
+    async fn reparse_xml_rmw_corruption_refuses_cached_and_reloaded_encryption() {
+        use crate::bucket::metadata::{BUCKET_SSECONFIG, is_unreadable_config_error};
+        use std::sync::atomic::{AtomicUsize, Ordering};
+
+        let (dirs, ecstore) = isolated_store_over_temp_disks().await;
+        let sys = BucketMetadataSys::new(ecstore);
+        let bucket = "reparse-encryption-corrupt";
+        let valid = br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
+        let corrupt = b"<ServerSideEncryptionConfiguration";
+        persist_bucket_with_raw_config(&sys, &dirs, bucket, BUCKET_SSECONFIG, valid).await;
+        sys.get_sse_config(bucket).await.expect("valid encryption must be readable");
+
+        let updated = sys
+            .update_config_with(bucket, BUCKET_SSECONFIG, |bm| {
+                assert!(bm.sse_config.is_some(), "rewrite must start with parsed encryption");
+                Ok(corrupt.to_vec())
+            })
+            .await
+            .expect("persist future or corrupt raw encryption without failing the whole bucket");
+
+        let err = sys
+            .get_sse_config(bucket)
+            .await
+            .expect_err("cached old encryption must not hide corrupt raw");
+        assert!(is_unreadable_config_error(&err), "cached refusal must retain its type: {err}");
+        let (cached, _) = sys.get_config(bucket).await.expect("bucket remains readable");
+        assert_eq!(cached.encryption_config_xml, corrupt);
+        assert_eq!(cached.encryption_config_updated_at, updated);
+
+        sys.metadata_map.write().await.clear();
+        let err = sys
+            .get_sse_config(bucket)
+            .await
+            .expect_err("reloaded encryption must still refuse corrupt raw");
+        assert!(is_unreadable_config_error(&err), "reloaded refusal must retain its type: {err}");
+
+        let calls = AtomicUsize::new(0);
+        let err = sys
+            .update_config_with(bucket, BUCKET_SSECONFIG, |_| {
+                calls.fetch_add(1, Ordering::SeqCst);
+                Ok(Vec::new())
+            })
+            .await
+            .expect_err("another rewrite must refuse before it consumes corrupt encryption");
+        assert!(is_unreadable_config_error(&err));
+        assert_eq!(calls.load(Ordering::SeqCst), 0, "refused rewrite must not erase the original raw");
+        let disk = sys
+            .get_config_from_disk(bucket)
+            .await
+            .expect("read unchanged metadata from disk");
+        assert_eq!(disk.encryption_config_xml, corrupt);
+        assert_eq!(disk.encryption_config_updated_at, updated);
+
+        sys.update(bucket, BUCKET_SSECONFIG, valid.to_vec())
+            .await
+            .expect("explicit repair must remain possible");
+        sys.get_sse_config(bucket)
+            .await
+            .expect("repaired cached encryption must be readable");
+        sys.metadata_map.write().await.clear();
+        sys.get_sse_config(bucket).await.expect("repair must survive a disk reload");
+    }
+
+    #[tokio::test]
+    async fn reparse_xml_rmw_removal_clears_cached_and_reloaded_encryption() {
+        use crate::bucket::metadata::BUCKET_SSECONFIG;
+
+        let (dirs, ecstore) = isolated_store_over_temp_disks().await;
+        let sys = BucketMetadataSys::new(ecstore);
+        let bucket = "reparse-encryption-removed";
+        let valid = br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
+        persist_bucket_with_raw_config(&sys, &dirs, bucket, BUCKET_SSECONFIG, valid).await;
+        sys.get_sse_config(bucket).await.expect("valid encryption must be readable");
+
+        let updated = sys
+            .update_config_with(bucket, BUCKET_SSECONFIG, |bm| {
+                assert!(bm.sse_config.is_some(), "rewrite must start with parsed encryption");
+                Ok(Vec::new())
+            })
+            .await
+            .expect("removing encryption must persist");
+
+        assert_eq!(
+            sys.get_sse_config(bucket)
+                .await
+                .expect_err("old cached encryption must be gone"),
+            Error::ConfigNotFound
+        );
+        let (cached, _) = sys.get_config(bucket).await.expect("bucket remains readable");
+        assert!(cached.encryption_config_xml.is_empty());
+        assert_eq!(cached.encryption_config_updated_at, updated);
+
+        sys.metadata_map.write().await.clear();
+        assert_eq!(
+            sys.get_sse_config(bucket)
+                .await
+                .expect_err("removed encryption must stay absent after reload"),
+            Error::ConfigNotFound
+        );
+        let disk = sys
+            .get_config_from_disk(bucket)
+            .await
+            .expect("read removed encryption from disk");
+        assert!(disk.encryption_config_xml.is_empty());
+        assert_eq!(disk.encryption_config_updated_at, updated);
+    }
+
     /// rustfs/backlog#1734: a read-modify-write of a stored config that cannot
     /// be parsed must be refused before `mutate` runs. Otherwise `mutate` sees
     /// the unreadable config as absent, rebuilds it from nothing, and the
@@ -4662,6 +4953,154 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn quota_admission_rejects_stale_cached_no_quota() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system");
+        let bucket = "stale-no-quota-admission";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+        let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata");
+        store
+            .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec())
+            .await
+            .expect("publish quota to authoritative storage");
+        // Model a peer that has not received the newer configuration snapshot.
+        sys.read().await.set(bucket.to_owned(), stale).await;
+        assert!(
+            get_cached_quota_config_and_incarnation_in(&store.ctx, bucket)
+                .await
+                .expect("read stale cache")
+                .expect("cached bucket identity")
+                .quota
+                .is_none()
+        );
+
+        temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async {
+            let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0)
+                .await
+                .err()
+                .expect("persisted quota must reject admission without its snapshot");
+            assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}");
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
+        let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
+        let bucket = "cross-node-negative-cache-refresh";
+        let node_b_sys = node_b.read().await.clone();
+
+        assert_eq!(
+            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
+            Error::BucketNotFound(bucket.to_string()),
+            "the probe must populate node B's missing-bucket cache"
+        );
+
+        for dir in &dirs {
+            std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
+        }
+        let created = BucketMetadata::new(bucket);
+        let created_incarnation = created.bucket_incarnation_id;
+        node_a
+            .read()
+            .await
+            .persist_new_and_set(created)
+            .await
+            .expect("node A should publish the created bucket metadata");
+
+        node_b_sys
+            .reload_from_store(bucket)
+            .await
+            .expect("peer reload should clear the stale negative cache");
+        assert_eq!(
+            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
+            created_incarnation,
+            "node B should resolve the newly created bucket immediately"
+        );
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    #[serial]
+    async fn peer_delete_clears_metadata_for_absent_bucket() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
+        let bucket = "peer-delete-absent-bucket";
+
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("initial bucket creation");
+        let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
+        store
+            .delete_bucket(bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("bucket deletion");
+
+        // Model a peer that still caches the metadata after the physical delete.
+        sys.read().await.set(bucket.to_string(), cached).await;
+        assert!(
+            remove_bucket_metadata_if_absent(store.clone(), bucket)
+                .await
+                .expect("conditional peer delete"),
+            "the peer must evict the deleted bucket's cached metadata"
+        );
+        let sys = sys.read().await;
+        assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
+        assert_eq!(
+            sys.get_bucket_incarnation_id(bucket)
+                .await
+                .expect_err("deleted bucket lookup"),
+            Error::BucketNotFound(bucket.to_string()),
+        );
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    #[serial]
+    async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
+        let bucket = "delayed-peer-delete-recreated-bucket";
+
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("initial bucket creation");
+        let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
+        store
+            .delete_bucket(bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("bucket deletion");
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("bucket recreation");
+        let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
+        assert_ne!(old_incarnation, new_incarnation);
+
+        let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
+            .await
+            .expect("conditional peer delete");
+        assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
+        assert_eq!(
+            sys.read()
+                .await
+                .get(bucket)
+                .await
+                .expect("recreated metadata should remain cached")
+                .bucket_incarnation_id,
+            new_incarnation,
+        );
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
     #[serial]
     async fn stale_config_request_cannot_mutate_a_recreated_bucket() {
@@ -4736,6 +5175,231 @@ mod tests {
             .expect("DeleteBucket should succeed");
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = "quota-same-name-owner";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("quota fixture bucket");
+        let snapshot = store
+            .object_lock_config_snapshot(bucket)
+            .await
+            .expect("guarded Object Lock snapshot");
+        let incarnation = store.bucket_incarnation_id(bucket).await.expect("fixture incarnation");
+        let transaction_guard = snapshot
+            .metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
+            .expect("snapshot's actual transaction guard");
+        let expected = get_quota_config_and_incarnation_from_disk_for_options_in(
+            &store.ctx,
+            bucket,
+            &ObjectOptions::default(),
+            &transaction_guard,
+        )
+        .await
+        .expect("a reader without a snapshot must retain the ordinary namespace path");
+
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("fixture metadata system");
+        let sys = sys.read().await.clone();
+        let mut stale = (*sys.get(bucket).await.expect("cached fixture metadata")).clone();
+        let metadata_path = stale.save_file_path();
+        stale.quota_config = Some(BucketQuota {
+            quota: Some(123),
+            ..Default::default()
+        });
+        sys.set(bucket.to_string(), Arc::new(stale)).await;
+        let namespace = store.new_ns_lock(bucket, bucket).await.expect("same-name object namespace");
+        let _owner = namespace
+            .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await
+            .expect("same-name object write owner");
+        let opts = ObjectOptions {
+            object_lock_config_snapshot: Some(snapshot),
+            expected_bucket_incarnation_id: Some(incarnation),
+            ..Default::default()
+        };
+        assert_eq!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .expect("quota disk read must reuse the held lifecycle and transaction fences"),
+            expected,
+            "the cached quota must not replace disk authority"
+        );
+
+        crate::config::com::save_config_with_opts(
+            store.clone(),
+            &metadata_path,
+            b"corrupt bucket metadata".to_vec(),
+            &ObjectOptions {
+                write_completion: crate::object_api::WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("inject committed metadata corruption");
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .is_err(),
+            "a valid snapshot must not hide corrupt disk metadata"
+        );
+        crate::config::com::delete_config(store.clone(), &metadata_path)
+            .await
+            .expect("remove the persisted metadata while retaining its sidecar");
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .is_err(),
+            "a valid snapshot must not promote fabricated metadata to authority"
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = "quota-snapshot-scope";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("quota fixture bucket");
+        let incarnation = store.bucket_incarnation_id(bucket).await.expect("fixture incarnation");
+        let snapshot = store
+            .object_lock_config_snapshot(bucket)
+            .await
+            .expect("guarded Object Lock snapshot");
+        let transaction_guard = snapshot
+            .metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
+            .expect("snapshot's actual transaction guard");
+        let mut opts = ObjectOptions {
+            object_lock_config_snapshot: Some(snapshot),
+            expected_bucket_incarnation_id: Some(incarnation),
+            ..Default::default()
+        };
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, "another-bucket", &opts, &transaction_guard)
+                .await
+                .is_err()
+        );
+        opts.expected_bucket_incarnation_id = Some(Uuid::new_v4());
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .is_err()
+        );
+        opts.expected_bucket_incarnation_id = Some(incarnation);
+        let other_guard = Arc::new(
+            acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket)
+                .await
+                .expect("distinct real transaction read guard"),
+        );
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &other_guard)
+                .await
+                .is_err(),
+            "the same namespace key does not prove that this is the snapshot's actual guard"
+        );
+        let (_other_dirs, other_store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(other_store.clone(), Vec::new()).await;
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&other_store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .is_err()
+        );
+        opts.object_lock_config_snapshot = Some(Arc::new(crate::object_api::ObjectLockConfigSnapshot::new(
+            ObjectLockConfigState::ConfirmedAbsent,
+        )));
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .is_err(),
+            "an unguarded snapshot must not authorize namespace reuse"
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation() {
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = "quota-snapshot-loss";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("quota fixture bucket");
+        let sys = bucket_metadata_sys_of(&store.ctx).expect("fixture metadata system");
+        let sys = sys.read().await.clone();
+        let mut metadata = (*sys.get(bucket).await.expect("fixture metadata")).clone();
+        let incarnation = metadata.bucket_incarnation_id;
+        let _lifecycle_guard = store
+            .acquire_bucket_lifecycle_read_lock(bucket)
+            .await
+            .expect("real lifecycle owner");
+        let transaction_guard = Arc::new(
+            acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket)
+                .await
+                .expect("real metadata transaction owner"),
+        );
+        let (lifecycle_fence, lost) = crate::object_api::NamespaceLockFence::loss_handle_for_test();
+        let snapshot = crate::object_api::ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence(
+            store.id,
+            bucket,
+            incarnation,
+            metadata.object_lock_config_updated_at,
+            ObjectLockConfigState::ConfirmedAbsent,
+            lifecycle_fence,
+            Arc::clone(&transaction_guard),
+        );
+        let opts = ObjectOptions {
+            object_lock_config_snapshot: Some(Arc::new(snapshot)),
+            expected_bucket_incarnation_id: Some(incarnation),
+            ..Default::default()
+        };
+        let mut read = Box::pin(get_quota_config_and_incarnation_from_disk_for_options_in(
+            &store.ctx,
+            bucket,
+            &opts,
+            &transaction_guard,
+        ));
+        assert!(
+            futures::poll!(read.as_mut()).is_pending(),
+            "the authoritative disk read must be in flight"
+        );
+        lost.store(true, std::sync::atomic::Ordering::Release);
+        assert!(read.await.is_err(), "a lifecycle fence lost during the disk read must reject its result");
+
+        let snapshot = store
+            .object_lock_config_snapshot(bucket)
+            .await
+            .expect("fresh guarded snapshot");
+        let transaction_guard = snapshot
+            .metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
+            .expect("snapshot's actual transaction guard");
+        let opts = ObjectOptions {
+            object_lock_config_snapshot: Some(snapshot),
+            expected_bucket_incarnation_id: Some(incarnation),
+            ..Default::default()
+        };
+        metadata.bucket_incarnation_id = Uuid::new_v4();
+        save_bucket_incarnation(store.clone(), bucket, metadata.bucket_incarnation_id)
+            .await
+            .expect("inject a new disk incarnation");
+        metadata
+            .save_with_store_committed(store.clone())
+            .await
+            .expect("persist its matching metadata blob");
+        assert!(
+            get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
+                .await
+                .is_err(),
+            "a matching sidecar and blob from another incarnation must not reuse the old snapshot"
+        );
+    }
+
     /// Writers on different nodes updating *different* config files of one
     /// bucket must both survive. Each rewrites the whole metadata blob, so
     /// without a lock spanning the read-modify-write the later save carries

crates/ecstore/src/bucket/migration.rs

@@ -273,6 +273,30 @@ where
 
     let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
 
+    try_migrate_bucket_metadata_for_buckets(store, &buckets).await
+}
+
+/// Migrate the already discovered startup candidates without repeating a
+/// majority-only namespace listing. Every migration write retains its quorum.
+pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        > + ObjectOperations<
+            Error = Error,
+            ObjectInfo = ObjectInfo,
+            ObjectOptions = ObjectOptions,
+            FileInfo = FileInfo,
+            ObjectToDelete = ObjectToDelete,
+            DeletedObject = DeletedObject,
+        >,
+{
     if buckets.is_empty() {
         debug!("No migrating bucket metadata found");
         return Ok(());

crates/ecstore/src/bucket/quota/mod.rs

@@ -32,7 +32,10 @@ pub enum QuotaType {
     Hard,
 }
 
-pub(crate) const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
+pub const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
+/// Sharded durable reservations. Older nodes reject this protocol instead of
+/// silently falling back to the single-ledger protocol.
+pub const QUOTA_RESERVATION_PROTOCOL_V2: u32 = 2;
 
 /// Bucket quota configuration. quota_type defaults to Hard when omitted.
 #[derive(Debug, Default, Clone, PartialEq)]
@@ -89,7 +92,10 @@ impl<'de> Deserialize<'de> for BucketQuota {
         D: Deserializer<'de>,
     {
         let wire = BucketQuotaWire::deserialize(deserializer)?;
-        let quota = if wire.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1) {
+        let quota = if matches!(
+            wire.reservation_protocol,
+            Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
+        ) {
             Some(
                 wire.reservation_quota
                     .ok_or_else(|| D::Error::custom("reservation_quota is required for reservation protocol v1"))?,
@@ -129,17 +135,33 @@ impl BucketQuota {
         }
     }
 
+    /// Creates a quota using the sharded durable reservation protocol.
+    /// Deployment must complete the mixed-version rollout before enabling it;
+    /// older nodes fail closed when they see protocol v2.
+    pub fn new_sharded(quota: u64) -> Self {
+        let mut config = Self::new(Some(quota));
+        config.reservation_protocol = Some(QUOTA_RESERVATION_PROTOCOL_V2);
+        config
+    }
+
     pub fn get_quota_limit(&self) -> Option<u64> {
         self.quota
     }
 
     pub fn uses_durable_reservations(&self) -> bool {
-        self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1)
+        matches!(
+            self.reservation_protocol,
+            Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
+        )
+    }
+
+    pub fn uses_sharded_reservations(&self) -> bool {
+        self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V2)
     }
 
     pub fn has_unsupported_reservation_protocol(&self) -> bool {
         self.reservation_protocol
-            .is_some_and(|version| version != QUOTA_RESERVATION_PROTOCOL_V1)
+            .is_some_and(|version| !matches!(version, QUOTA_RESERVATION_PROTOCOL_V1 | QUOTA_RESERVATION_PROTOCOL_V2))
     }
 
     pub fn check_operation_allowed(&self, current_usage: u64, operation_size: u64) -> bool {
@@ -322,9 +344,21 @@ mod tests {
     }
 
     #[test]
-    fn unknown_reservation_protocol_does_not_activate_v1() {
+    fn sharded_quota_roundtrips_with_authoritative_reservation_quota() {
+        let quota = BucketQuota::new_sharded(4096);
+        let json = serde_json::to_value(&quota).expect("sharded quota should serialize");
+        assert_eq!(json["quota"], 0);
+        assert_eq!(json["reservation_protocol"], QUOTA_RESERVATION_PROTOCOL_V2);
+        assert_eq!(json["reservation_quota"], 4096);
+        let restored: BucketQuota = serde_json::from_value(json).expect("sharded quota should parse");
+        assert_eq!(restored.quota, Some(4096));
+        assert!(restored.uses_sharded_reservations());
+    }
+
+    #[test]
+    fn unknown_reservation_protocol_does_not_activate_any_known_protocol() {
         let quota: BucketQuota =
-            serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":2,"reservation_quota":2048}"#)
+            serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":3,"reservation_quota":2048}"#)
                 .expect("future protocol should remain parseable");
 
         assert!(!quota.uses_durable_reservations());

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -65,9 +65,9 @@ use super::replication_target_boundary::{
     SsecPassthroughGate, TargetClient, VersionIdentityCapability, is_replication_target_offline_error,
     replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options,
     replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted,
-    replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version,
-    resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put,
-    version_identity_drifted,
+    replication_put_object_header_size, replication_put_object_options, replication_source_etag_is_comparable,
+    replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present,
+    ssec_passthrough_gate, version_identity_capability_from_put, version_identity_drifted,
 };
 use super::replication_versioning_boundary::ReplicationVersioningStore;
 use super::runtime_boundary as runtime_sources;
@@ -349,15 +349,22 @@ fn replica_etag_verification_enabled() -> bool {
 /// target that stores a transformed payload (e.g. undecoded `aws-chunked`
 /// frames, #6853) returns the ETag of what it actually wrote. Reporting
 /// COMPLETED over such a replica is silent corruption, so a decidable
-/// mismatch fails the replication instead. An SSE-C ciphertext passthrough
-/// transfer is exempt: the wire bytes are ciphertext while the source ETag is
-/// the plaintext MD5, and that path has its own HEAD-back audit.
+/// mismatch fails the replication instead. Encrypted sources are exempt: an
+/// SSE-C ciphertext passthrough sends ciphertext while the source ETag is the
+/// plaintext MD5 (that path has its own HEAD-back audit), and a managed-SSE
+/// (SSE-S3 / SSE-KMS) source stores a keyed fingerprint, or a legacy plaintext
+/// MD5, that the ETag of the target's re-encrypted replica can neither confirm
+/// nor refute. The checksum header on the replication PUT stays the integrity
+/// check for those objects.
 fn verify_single_part_replica(
     object_info: &ObjectInfo,
     response: &RemotePutObjectResponse,
     ciphertext_passthrough: bool,
 ) -> std::result::Result<(), std::io::Error> {
-    if ciphertext_passthrough || !replica_etag_verification_enabled() {
+    if ciphertext_passthrough
+        || !replication_source_etag_is_comparable(&object_info.user_defined)
+        || !replica_etag_verification_enabled()
+    {
         return Ok(());
     }
     if single_part_replica_etag_mismatch(object_info.etag.as_deref(), response.etag.as_deref()) {
@@ -448,6 +455,26 @@ fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
     is_version_id_mismatch(code, raw_status)
 }
 
+/// A version-addressed HEAD rejected because the target cannot address the
+/// source version id (AWS S3: bodiless 400). Narrower than
+/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine
+/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a
+/// completion verdict follows instead of another non-destructive HEAD.
+fn rejects_source_version_id(err: &SdkError<HeadObjectError>) -> bool {
+    is_version_id_format_mismatch(err) && !has_raw_status(err, 403)
+}
+
+/// Whether the target's latest version of `object` is a delete marker. For
+/// targets that mint their own version ids, which cannot be asked about the
+/// source marker id: S3 answers a versionless HEAD on such a key with 404
+/// (405 on some targets) plus `x-amz-delete-marker: true`.
+async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool {
+    match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await {
+        Ok(head) => head.delete_marker == Some(true),
+        Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"),
+    }
+}
+
 async fn mark_replication_target_offline_if_needed(target_client: &Arc<TargetClient>, err: &(impl Display + ?Sized)) {
     if is_replication_target_offline_error(err) {
         ReplicationTargetStore::mark_target_offline(target_client).await;
@@ -1846,7 +1873,17 @@ async fn verify_resync_head_result(
             // only absence proves completion. All other outcomes remain
             // failed (backlog#862 / #799 B13).
             let retryable = if roi.version_purge_status.is_empty() {
-                matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
+                match classify_delete_marker_head_error(&err) {
+                    // A target that mints its own version ids never holds the
+                    // source marker id; verify the marker on the latest version.
+                    DeleteMarkerHeadOutcome::Ambiguous => {
+                        let mints_own = rejects_source_version_id(&err)
+                            || ReplicationTargetStore::version_identity_capability(&target_client.arn)
+                                .version_addressing_unreliable();
+                        !(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await)
+                    }
+                    DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false,
+                }
             } else {
                 !matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
             };
@@ -3567,7 +3604,16 @@ async fn replicate_delete_to_target(
         }
     }
 
-    if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
+    // A target that mints its own version ids never holds the source marker
+    // id, so a version-addressed pre-check cannot succeed there; the marker
+    // is created by the versionless DELETE below.
+    let version_addressable =
+        !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable();
+    if version_addressable
+        && !is_version_purge
+        && dobj.delete_object.delete_marker
+        && dobj.delete_object.delete_marker_version_id.is_some()
+    {
         match head_object_for_worker(
             tgt_client.as_ref(),
             &tgt_client.bucket,
@@ -3576,6 +3622,11 @@ async fn replicate_delete_to_target(
         )
         .await
         {
+            // The verdict is not known yet (e.g. no PUT since restart) and the
+            // target rejected the source id: learn it as the PUT path does.
+            Err(e) if rejects_source_version_id(e.as_ref()) => {
+                ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn);
+            }
             Ok(head) if head.delete_marker == Some(true) => {
                 // Some targets expose an existing marker as a successful HEAD
                 // with the marker bit instead of the usual 405. Treat that as
@@ -5826,6 +5877,57 @@ mod tests {
     use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
     use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError;
 
+    #[test]
+    fn verify_single_part_replica_exempts_managed_sse_sources() {
+        // A new encrypted write stores a keyed fingerprint; a target that
+        // computes its own ETag answers with the MD5 of the plaintext it
+        // received. Both are 32 hex characters, so without the exemption the
+        // audit would fail every SSE-S3 / SSE-KMS replica (rustfs/backlog).
+        let fingerprint = "00112233445566778899aabbccddeeff";
+        let replica_md5 = "ffeeddccbbaa99887766554433221100";
+        let response = RemotePutObjectResponse {
+            version_id: None,
+            etag: Some(replica_md5.to_string()),
+        };
+        for sse in ["AES256", "aws:kms"] {
+            let object_info = ObjectInfo {
+                bucket: "source".to_string(),
+                name: "encrypted".to_string(),
+                etag: Some(fingerprint.to_string()),
+                user_defined: HashMap::from([("x-amz-server-side-encryption".to_string(), sse.to_string())]).into(),
+                ..Default::default()
+            };
+            verify_single_part_replica(&object_info, &response, false)
+                .unwrap_or_else(|err| panic!("{sse}: a managed-SSE source must not be audited against the replica MD5: {err}"));
+        }
+    }
+
+    #[test]
+    fn verify_single_part_replica_still_fails_plaintext_mismatch() {
+        if !replica_etag_verification_enabled() {
+            return;
+        }
+        let object_info = ObjectInfo {
+            bucket: "source".to_string(),
+            name: "plain".to_string(),
+            etag: Some("00112233445566778899aabbccddeeff".to_string()),
+            ..Default::default()
+        };
+        let mismatch = RemotePutObjectResponse {
+            version_id: None,
+            etag: Some("ffeeddccbbaa99887766554433221100".to_string()),
+        };
+        let err = verify_single_part_replica(&object_info, &mismatch, false)
+            .expect_err("a plaintext source whose replica MD5 differs must fail the replication");
+        assert_eq!(err.to_string(), REPLICA_ETAG_MISMATCH_ERROR);
+
+        let matching = RemotePutObjectResponse {
+            version_id: None,
+            etag: object_info.etag.clone(),
+        };
+        verify_single_part_replica(&object_info, &matching, false).expect("matching plaintext MD5s pass the audit");
+    }
+
     #[test]
     fn multipart_read_plan_preserves_legacy_plain_part_ranges() {
         const MIB: usize = 1024 * 1024;
@@ -6675,6 +6777,93 @@ mod tests {
         assert!(seen[0].starts_with("HEAD "), "{seen:?}");
     }
 
+    /// AWS S3 answers a HEAD for a version id it did not mint with a bodiless
+    /// 400. On a target with no verdict yet, that teaches `MintsOwn` and the
+    /// marker is still created with one versionless DELETE.
+    #[tokio::test]
+    async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() {
+        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
+            if line.starts_with("HEAD ") {
+                empty_response("400 Bad Request")
+            } else if line.starts_with("DELETE ") {
+                empty_response("204 No Content")
+            } else {
+                empty_response("500 Unexpected")
+            }
+        });
+        let target = test_target_client(endpoint);
+        register_test_target(&target).await;
+        let dobj = marker_creation_dobj(&target.arn);
+
+        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
+
+        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
+        assert_eq!(
+            ReplicationTargetStore::version_identity_capability(&target.arn),
+            VersionIdentityCapability::MintsOwn
+        );
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 2, "{seen:?}");
+        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
+        assert!(seen[1].starts_with("DELETE "), "{seen:?}");
+        assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
+    }
+
+    /// A bodiless 403 on the version-addressed HEAD may be a genuine
+    /// GetObjectVersion denial on an id-adopting target. It must fail closed:
+    /// no DELETE, and no `MintsOwn` verdict cached for the ARN.
+    #[tokio::test]
+    async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() {
+        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
+            if line.starts_with("HEAD ") {
+                empty_response("403 Forbidden")
+            } else if line.starts_with("DELETE ") {
+                empty_response("204 No Content")
+            } else {
+                empty_response("500 Unexpected")
+            }
+        });
+        let target = test_target_client(endpoint);
+        register_test_target(&target).await;
+        let dobj = marker_creation_dobj(&target.arn);
+
+        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
+
+        assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
+        assert_ne!(
+            ReplicationTargetStore::version_identity_capability(&target.arn),
+            VersionIdentityCapability::MintsOwn
+        );
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}");
+        assert!(seen[0].starts_with("HEAD "), "{seen:?}");
+    }
+
+    /// A target already known to mint its own version ids is never asked for
+    /// the source marker id: the marker is created by the versionless DELETE.
+    #[tokio::test]
+    async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() {
+        let (endpoint, server) = spawn_scripted_target_server(1, |line| {
+            if line.starts_with("DELETE ") {
+                empty_response("204 No Content")
+            } else {
+                empty_response("500 Unexpected")
+            }
+        });
+        let target = test_target_client(endpoint);
+        register_test_target(&target).await;
+        ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn);
+        let dobj = marker_creation_dobj(&target.arn);
+
+        let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
+
+        assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}");
+        assert!(seen[0].starts_with("DELETE "), "{seen:?}");
+        assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}");
+    }
+
     /// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
     /// 405, so the SDK reports no error code. That is the marker having
     /// propagated, not a failure (rustfs/backlog#2479).
@@ -6765,6 +6954,76 @@ mod tests {
         server.join().expect("test HTTP server should finish");
     }
 
+    /// AWS answers `HEAD ?versionId=<source marker id>` with a bodiless 400.
+    /// The marker is verified through the latest version instead: a
+    /// versionless 404 with `x-amz-delete-marker: true` proves it propagated.
+    #[tokio::test]
+    async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() {
+        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
+            if line.contains("versionId=") {
+                empty_response("400 Bad Request")
+            } else {
+                "HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
+                    .to_string()
+            }
+        });
+        let target = test_target_client(endpoint);
+        let roi = delete_marker_roi();
+        let mut st = TargetReplicationResyncStatus::default();
+
+        let head_result =
+            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
+        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
+
+        assert!(err.is_none(), "{err:?}");
+        assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0));
+        let seen = server.join().expect("test HTTP server should finish");
+        assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}");
+    }
+
+    /// A live latest version on a target that rejects the source id means the
+    /// marker has not propagated: the verification stays failed.
+    #[tokio::test]
+    async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() {
+        let (endpoint, server) = spawn_scripted_target_server(2, |line| {
+            if line.contains("versionId=") {
+                empty_response("400 Bad Request")
+            } else {
+                empty_response("200 OK")
+            }
+        });
+        let target = test_target_client(endpoint);
+        let roi = delete_marker_roi();
+        let mut st = TargetReplicationResyncStatus::default();
+
+        let head_result =
+            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
+        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
+
+        assert!(err.is_some(), "a live latest version must not verify the marker");
+        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
+        server.join().expect("test HTTP server should finish");
+    }
+
+    /// A bodiless 403 may be a genuine permission denial: it is not evidence
+    /// that the target mints its own ids, so no latest-version re-check runs.
+    #[tokio::test]
+    async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() {
+        let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden"));
+        let target = test_target_client(endpoint);
+        let roi = delete_marker_roi();
+        let mut st = TargetReplicationResyncStatus::default();
+
+        let head_result =
+            head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
+        let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
+
+        assert!(err.is_some(), "a 403 must not count as a propagated delete marker");
+        assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
+        let seen = server.join().expect("test HTTP server should finish");
+        assert_eq!(seen.len(), 1, "{seen:?}");
+    }
+
     /// Serves exactly `requests` connections, answering each from
     /// `respond(request_line)`, and returns the request lines it saw. Reads
     /// the whole request (headers plus `Content-Length` body, honoring

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -157,6 +157,19 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
     rustfs_replication::is_ssec_encrypted(user_defined)
 }
 
+/// Whether the stored ETag of a replication source can be compared with the
+/// ETag a target returns for the bytes it received. Only an unencrypted source
+/// stores the MD5 of the bytes that go on the wire. A managed-SSE (SSE-S3 /
+/// SSE-KMS) or SSE-C source stores a keyed fingerprint (new writes) or a legacy
+/// plaintext MD5, and the target re-encrypts what it receives, so neither
+/// equality nor inequality of the two ETags proves anything about the replica.
+pub(crate) fn replication_source_etag_is_comparable(user_defined: &HashMap<String, String>) -> bool {
+    matches!(
+        classify_replication_source_encryption(user_defined),
+        ReplicationSourceEncryption::Plaintext
+    )
+}
+
 /// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
 /// judgment owned by `rustfs-replication`: extract the echoed
 /// customer-algorithm header and let the crate-owned policy decide.
@@ -578,6 +591,22 @@ mod tests {
     use time::Duration;
     use uuid::Uuid;
 
+    #[test]
+    fn replication_source_etag_is_comparable_only_for_plaintext_sources() {
+        assert!(replication_source_etag_is_comparable(&HashMap::new()));
+        for (key, value) in [
+            (metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"),
+            (metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"),
+            (SSEC_ALGORITHM_HEADER, "AES256"),
+        ] {
+            let metadata = HashMap::from([(key.to_string(), value.to_string())]);
+            assert!(
+                !replication_source_etag_is_comparable(&metadata),
+                "{key}={value} must not expose its stored ETag to the replica audit"
+            );
+        }
+    }
+
     /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the
     /// SSE intent as lowercase `x-amz-server-side-encryption`
     /// (`encryption_material_to_metadata`; header-derived keys come from a

crates/ecstore/src/cache_value/metacache_set.rs

@@ -12,9 +12,11 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+#[cfg(test)]
+use crate::disk::DiskIoClass;
 use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
 use crate::disk::error::DiskError;
-use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
+use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions, current_disk_io_context, with_optional_disk_io_context};
 use futures::future::join_all;
 use metrics::counter;
 use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
@@ -245,6 +247,8 @@ pub struct ListPathRawOptions {
     pub(crate) test_fallback_reader_behaviors: Vec<TestReaderBehavior>,
     #[cfg(test)]
     pub(crate) peek_timeout: Option<Duration>,
+    #[cfg(test)]
+    pub(crate) test_disk_io_context_observed: Option<Arc<AtomicBool>>,
     // pub agreed: Option<Arc<dyn Fn(MetaCacheEntry) + Send + Sync>>,
     // pub partial: Option<Arc<dyn Fn(MetaCacheEntries, &[Option<Error>]) + Send + Sync>>,
     // pub finished: Option<Arc<dyn Fn(&[Option<Error>]) + Send + Sync>>,
@@ -276,6 +280,8 @@ impl Clone for ListPathRawOptions {
             test_fallback_reader_behaviors: self.test_fallback_reader_behaviors.clone(),
             #[cfg(test)]
             peek_timeout: self.peek_timeout,
+            #[cfg(test)]
+            test_disk_io_context_observed: self.test_disk_io_context_observed.clone(),
             ..Default::default()
         }
     }
@@ -344,6 +350,7 @@ async fn list_path_raw_inner(
     let log_path = opts.path.clone();
 
     let mut jobs = JoinSet::new();
+    let disk_io_context = current_disk_io_context();
     let mut readers = Vec::with_capacity(opts.disks.len());
     let fds = Arc::new(TokioMutex::new(opts.fallback_disks.iter().flatten().cloned().collect::<VecDeque<_>>()));
     #[cfg(test)]
@@ -364,11 +371,19 @@ async fn list_path_raw_inner(
         let test_fallbacks_clone = test_fallbacks.clone();
         let cancel_rx_clone = cancel_rx.clone();
         let producer_errs_clone = producer_errs.clone();
+        let disk_io_context_clone = disk_io_context.clone();
         let (rd, wr) = tokio::io::duplex(64);
         readers.push(MetacacheReader::new(rd));
-        jobs.spawn(async move {
+        jobs.spawn(with_optional_disk_io_context(disk_io_context_clone, async move {
             let mut wr = PublishedBytesWriter::new(wr);
             #[cfg(test)]
+            if let Some(observer) = &opts_clone.test_disk_io_context_observed {
+                observer.store(
+                    crate::disk::current_disk_io_context().is_some_and(|context| context.class == DiskIoClass::Background),
+                    std::sync::atomic::Ordering::Release,
+                );
+            }
+            #[cfg(test)]
             let test_primary_error = if let Some(behavior) = opts_clone.test_reader_behaviors.get(disk_idx).cloned() {
                 match behavior {
                     TestReaderBehavior::Eof => return Ok(()),
@@ -652,7 +667,7 @@ async fn list_path_raw_inner(
 
             // warn!("list_path_raw: while need_fallback done");
             Ok(())
-        });
+        }));
     }
 
     let revjob_rx = rx.clone();
@@ -1168,6 +1183,68 @@ mod tests {
         assert_eq!(err, DiskError::ErasureReadQuorum);
     }
 
+    #[tokio::test]
+    async fn list_path_raw_joinset_producers_inherit_scanner_disk_io_class() {
+        let observed = Arc::new(AtomicBool::new(false));
+        crate::disk::with_background_disk_io(
+            CancellationToken::new(),
+            list_path_raw(
+                CancellationToken::new(),
+                ListPathRawOptions {
+                    disks: vec![None],
+                    min_disks: 1,
+                    test_reader_behaviors: vec![TestReaderBehavior::Eof],
+                    test_disk_io_context_observed: Some(observed.clone()),
+                    ..Default::default()
+                },
+            ),
+        )
+        .await
+        .expect("empty test producer should complete the listing");
+
+        assert!(
+            observed.load(std::sync::atomic::Ordering::Acquire),
+            "the per-disk JoinSet producer must retain the scanner background class"
+        );
+    }
+
+    #[tokio::test]
+    async fn scanner_joinset_walk_uses_the_local_disk_background_gate() {
+        let root = tempfile::tempdir().expect("local disk root should be created");
+        let endpoint = crate::disk::endpoint::Endpoint::try_from(root.path().to_str().expect("disk path should be UTF-8"))
+            .expect("endpoint should parse");
+        let local = Arc::new(
+            crate::disk::local::LocalDisk::new(&endpoint, false)
+                .await
+                .expect("local disk should open"),
+        );
+        let wrapper = crate::disk::disk_store::LocalDiskWrapper::new(local, false);
+        wrapper.make_volume("bucket").await.expect("bucket volume should be created");
+        let disk = Arc::new(crate::disk::Disk::Local(Box::new(wrapper.clone())));
+        let cancellation = CancellationToken::new();
+
+        crate::disk::with_background_disk_io(
+            cancellation.clone(),
+            list_path_raw(
+                cancellation,
+                ListPathRawOptions {
+                    disks: vec![Some(disk)],
+                    bucket: "bucket".to_string(),
+                    min_disks: 1,
+                    recursive: true,
+                    ..Default::default()
+                },
+            ),
+        )
+        .await
+        .expect("scanner metacache walk should complete");
+
+        assert!(
+            wrapper.background_io_admissions_for_tests() > 0,
+            "the LocalDiskWrapper walk should acquire background admission inside its JoinSet producer"
+        );
+    }
+
     #[tokio::test]
     async fn list_path_raw_returns_typed_quorum_error_for_multiple_drive_failures() {
         let err = list_path_raw(

crates/ecstore/src/cluster/rpc/client.rs

@@ -24,10 +24,11 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
 #[cfg(test)]
 use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
 use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
+#[cfg(test)]
 use crate::runtime::sources as runtime_sources;
 use http::{Request as HttpRequest, Response as HttpResponse, Uri};
 use rustfs_protos::{
-    ChannelClass, create_new_channel, get_channel_for_class,
+    ChannelClass, get_channel_for_class,
     proto_gen::node_service::{
         heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
         scanner_control_service_client::ScannerControlServiceClient,
@@ -53,7 +54,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
 /// 3. Subsequent calls will attempt fresh connections
 /// 4. If node is still down, connection will fail fast (3s timeout)
 pub async fn node_service_time_out_client(
-    addr: &String,
+    addr: &str,
     interceptor: TonicInterceptor,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     // Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -66,12 +67,9 @@ pub(crate) async fn scanner_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match runtime_sources::cached_node_channel(addr).await {
-        Some(channel) => channel,
-        None => create_new_channel(addr)
-            .await
-            .map_err(|err| crate::error::Error::other(err.to_string()))?,
-    };
+    let channel = get_channel_for_class(addr, ChannelClass::Control)
+        .await
+        .map_err(|err| crate::error::Error::other(err.to_string()))?;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
     Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -84,10 +82,7 @@ pub async fn heal_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match runtime_sources::cached_node_channel(addr).await {
-        Some(channel) => channel,
-        None => create_new_channel(addr).await?,
-    };
+    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
     let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -100,10 +95,7 @@ pub async fn tier_mutation_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match runtime_sources::cached_node_channel(addr).await {
-        Some(channel) => channel,
-        None => create_new_channel(addr).await?,
-    };
+    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
     let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -118,22 +110,12 @@ pub async fn tier_mutation_control_time_out_client(
 /// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
 /// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
 pub async fn node_service_time_out_client_for_class(
-    addr: &String,
+    addr: &str,
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = match class {
-        ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
-            Some(channel) => {
-                debug!("Using cached gRPC channel for: {}", addr);
-                channel
-            }
-            // No cached connection, create new one.
-            None => create_new_channel(addr).await?,
-        },
-        ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
-    };
+    let channel = get_channel_for_class(addr, class).await?;
 
     let max_message_size = rustfs_protos::internode_rpc_max_message_size();
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -142,8 +124,24 @@ pub async fn node_service_time_out_client_for_class(
         .max_encoding_message_size(max_message_size))
 }
 
+pub(crate) async fn node_service_delete_recovery_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+    timeout: std::time::Duration,
+) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
+    let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
+    let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
+        .await
+        .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
+    let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
+    let max_message_size = rustfs_protos::internode_rpc_max_message_size();
+    Ok(NodeServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
 pub async fn node_service_time_out_client_no_auth(
-    addr: &String,
+    addr: &str,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
 }
@@ -355,6 +353,13 @@ fn apply_peer_replay_response(
     }
 }
 
+fn rename_data_grpc_stage(path: &str) -> bool {
+    matches!(
+        path,
+        "/node_service.NodeService/RenameData" | "/node_service.NodeService/RenameDataAtIncarnation"
+    )
+}
+
 impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
 where
     S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
@@ -372,6 +377,12 @@ where
     }
 
     fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
+        let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path());
+        let request_scope_started = if observe_rename_data {
+            rustfs_io_metrics::put_stage_timer()
+        } else {
+            None
+        };
         let authenticated = self.audience.as_ref().is_some_and(|_| {
             request
                 .headers()
@@ -406,11 +417,30 @@ where
                 }
             }
         }
+        rustfs_io_metrics::record_put_object_stage_duration_from(
+            rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE,
+            request_scope_started,
+        );
 
         let audience = self.audience.clone();
         let future = self.inner.call(request);
         Box::pin(async move {
-            let response = future.await?;
+            let response = if observe_rename_data {
+                rustfs_io_metrics::observe_put_stage_future(
+                    future,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY,
+                )
+                .await?
+            } else {
+                future.await?
+            };
+            let replay_response_started = if observe_rename_data {
+                rustfs_io_metrics::put_stage_timer()
+            } else {
+                None
+            };
             if let (Some(audience), Some(challenge)) = (audience, challenge) {
                 let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
                 if let Err(error) = &response_state
@@ -428,6 +458,10 @@ where
                 }
                 apply_peer_replay_response(audience, sent_state, response_state);
             }
+            rustfs_io_metrics::record_put_object_stage_duration_from(
+                rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE,
+                replay_response_started,
+            );
             Ok(response)
         })
     }
@@ -546,6 +580,126 @@ mod tests {
     use tracing_opentelemetry::OpenTelemetrySpanExt;
     use tracing_subscriber::{Registry, layer::SubscriberExt};
 
+    #[tokio::test]
+    async fn delete_recovery_channel_uses_its_scoped_deadline() {
+        use bytes::Bytes;
+        use http_body_util::StreamBody;
+        use hyper::{body::Frame, service::service_fn};
+        use hyper_util::rt::{TokioExecutor, TokioIo};
+        use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
+        use tonic_prost::prost::Message;
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("bind recovery peer");
+        let addr = format!("http://{}", listener.local_addr().expect("peer address"));
+        let server = tokio::spawn(async move {
+            loop {
+                let (stream, _) = listener.accept().await.expect("accept recovery connection");
+                tokio::spawn(async move {
+                    let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
+                        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                        let payload = RecoverDeleteTransactionResponse {
+                            success: true,
+                            protocol_version: 1,
+                            result: Bytes::new(),
+                            error: None,
+                        }
+                        .encode_to_vec();
+                        let mut data = vec![0];
+                        data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
+                        data.extend_from_slice(&payload);
+                        let mut trailers = http::HeaderMap::new();
+                        trailers.insert("grpc-status", "0".parse().expect("status"));
+                        let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
+                            vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
+                        let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
+                        response
+                            .headers_mut()
+                            .insert("content-type", "application/grpc".parse().expect("content type"));
+                        Ok::<_, std::convert::Infallible>(response)
+                    });
+                    let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
+                        .serve_connection(TokioIo::new(stream), service)
+                        .await;
+                });
+            }
+        });
+        let request = || RecoverDeleteTransactionRequest {
+            disk: "disk".into(),
+            volume: "bucket".into(),
+            protocol_version: 1,
+            operation: Bytes::new(),
+        };
+        // Seed an existing short-lived control channel to reproduce the inherited deadline.
+        let control = tonic::transport::Endpoint::from_shared(addr.clone())
+            .expect("test endpoint")
+            .timeout(std::time::Duration::from_millis(50))
+            .connect()
+            .await
+            .expect("control connect");
+        rustfs_common::cache_connection(addr.clone(), control).await;
+        let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
+            .await
+            .expect("control client");
+        let error = foreground
+            .recover_delete_transaction(request())
+            .await
+            .expect_err("short control deadline");
+        assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
+        // The 2-second test override bypasses the production range parser, which rejects 2.
+        let mut recovery = node_service_delete_recovery_client(
+            &addr,
+            TonicInterceptor::NoOp(NoOpInterceptor),
+            std::time::Duration::from_secs(2),
+        )
+        .await
+        .expect("scoped recovery client");
+        assert!(
+            recovery
+                .recover_delete_transaction(request())
+                .await
+                .expect("scoped deadline must survive control deadline")
+                .into_inner()
+                .success
+        );
+        server.abort();
+        rustfs_protos::evict_failed_connection(&addr).await;
+    }
+
+    #[tokio::test]
+    async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
+        let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
+        let addr = format!("https://{}", listener.local_addr().expect("local address"));
+        drop(listener);
+        for factory in 0..4 {
+            let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
+            runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
+            let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
+            let _result = match factory {
+                0 => scanner_control_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+                1 => heal_control_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+                2 => tier_mutation_control_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+                _ => node_service_time_out_client(&addr, interceptor)
+                    .await
+                    .map(|_| ())
+                    .map_err(|e| e.to_string()),
+            };
+            assert!(
+                !runtime_sources::test_node_channel_is_cached(&addr).await,
+                "factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
+            );
+        }
+    }
+
     #[derive(Clone)]
     struct EpochProofService {
         audience: String,

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -289,13 +289,9 @@ async fn client(
     >,
     NetworkPeerProbeError,
 > {
-    node_service_time_out_client_for_class(
-        &address.to_owned(),
-        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
-        class,
-    )
-    .await
-    .map_err(|_| NetworkPeerProbeError::Unreachable)
+    node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
+        .await
+        .map_err(|_| NetworkPeerProbeError::Unreachable)
 }
 
 fn ping_request(payload: &[u8]) -> PingRequest {
@@ -329,6 +325,24 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
     match status.code() {
         Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
         Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
+        Code::Cancelled | Code::Internal => {
+            let mut source = std::error::Error::source(&status);
+            while let Some(error) = source {
+                if error
+                    .downcast_ref::<h2::Error>()
+                    .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
+                {
+                    return NetworkPeerProbeError::Unreachable;
+                }
+                source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
+                    // io::Error::source skips its immediate custom error.
+                    error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
+                } else {
+                    error.source()
+                };
+            }
+            NetworkPeerProbeError::ProtocolFailure
+        }
         _ => NetworkPeerProbeError::ProtocolFailure,
     }
 }
@@ -339,6 +353,35 @@ mod tests {
     use crate::layout::endpoint::Endpoint;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
+    #[test]
+    fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
+        for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
+            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
+            assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
+            let code = status.code();
+            assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
+            let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
+            wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
+            assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
+        }
+        for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
+            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
+            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
+        }
+        for status in [
+            tonic::Status::cancelled("peer cancelled the application request"),
+            tonic::Status::internal("peer rejected the probe"),
+            tonic::Status::unauthenticated("authentication rejected"),
+            tonic::Status::permission_denied("authorization rejected"),
+        ] {
+            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
+        }
+        assert_eq!(
+            map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
+            NetworkPeerProbeError::TimedOut
+        );
+    }
+
     fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
         NetworkPeerProbeClient::from_endpoint_pools(&topology())
             .with_diagnostic_pacing(Instant::now(), duration)

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -640,6 +640,7 @@ pub struct PeerLiveEventsBatch {
 pub struct ScannerPublicationLease {
     pub token: Uuid,
     pub movement_generation: u64,
+    pub observational_only: bool,
     /// Stable storage owner identity. This is distinct from the activity
     /// session and is bound into both acquire and release proofs.
     pub owner_id: String,
@@ -658,6 +659,7 @@ fn validate_scanner_publication_lease_response_fields(
     response: &ScannerPublicationLeaseResponse,
     expected_session_id: &str,
     expected_generation: u64,
+    expected_observational_only: bool,
 ) -> Result<(Uuid, String)> {
     if !response.success {
         return Err(Error::other(
@@ -674,6 +676,9 @@ fn validate_scanner_publication_lease_response_fields(
     if response.session_id != expected_session_id {
         return Err(Error::other("peer returned a different scanner publication lease session"));
     }
+    if response.observational_only != expected_observational_only {
+        return Err(Error::other("peer returned a different scanner publication lease purpose"));
+    }
     let owner_id = Uuid::parse_str(&response.owner_id)
         .ok()
         .filter(|owner_id| !owner_id.is_nil())
@@ -2418,6 +2423,7 @@ impl PeerRestClient {
         &self,
         expected_session_id: &str,
         expected_generation: u64,
+        observational_only: bool,
     ) -> Result<ScannerPublicationLease> {
         let request_started = std::time::Instant::now();
         self.finalize_result(
@@ -2434,6 +2440,7 @@ impl PeerRestClient {
                     ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
                     expected_session_id: expected_session_id.to_string(),
                     token: Bytes::new(),
+                    observational_only,
                 });
                 let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
                     .map_err(|_| Error::other("scanner publication lease request is too large to authenticate"))?;
@@ -2444,11 +2451,16 @@ impl PeerRestClient {
                         .map_err(|_| Error::other("scanner publication lease response is too large to authenticate"))?;
                 verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
                     .map_err(|_| Error::other("peer returned an invalid scanner publication lease proof"))?;
-                let (token, owner_id) =
-                    validate_scanner_publication_lease_response_fields(&response, expected_session_id, expected_generation)?;
+                let (token, owner_id) = validate_scanner_publication_lease_response_fields(
+                    &response,
+                    expected_session_id,
+                    expected_generation,
+                    observational_only,
+                )?;
                 Ok(ScannerPublicationLease {
                     token,
                     movement_generation: response.movement_generation,
+                    observational_only,
                     owner_id,
                     session_id: response.session_id,
                     expires_at: scanner_publication_lease_deadline(
@@ -2483,6 +2495,7 @@ impl PeerRestClient {
                     ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
                     expected_session_id: lease.session_id.clone(),
                     token: lease.token.as_bytes().to_vec().into(),
+                    observational_only: lease.observational_only,
                 });
                 let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
                     .map_err(|_| Error::other("scanner publication lease validation request is too large to authenticate"))?;
@@ -2494,8 +2507,12 @@ impl PeerRestClient {
                     )?;
                 verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
                     .map_err(|_| Error::other("peer returned an invalid scanner publication lease validation proof"))?;
-                let (token, owner_id) =
-                    validate_scanner_publication_lease_response_fields(&response, &lease.session_id, lease.movement_generation)?;
+                let (token, owner_id) = validate_scanner_publication_lease_response_fields(
+                    &response,
+                    &lease.session_id,
+                    lease.movement_generation,
+                    lease.observational_only,
+                )?;
                 if token != lease.token {
                     return Err(Error::other("peer returned a different scanner publication lease token"));
                 }
@@ -3001,15 +3018,16 @@ mod tests {
             response_proof: Bytes::new(),
             owner_id: Uuid::new_v4().to_string(),
             session_id: "session-a".to_string(),
+            observational_only: false,
         };
 
-        assert!(validate_scanner_publication_lease_response_fields(&response, "session-a", 7).is_ok());
+        assert!(validate_scanner_publication_lease_response_fields(&response, "session-a", 7, false).is_ok());
 
         let stale_generation = ScannerPublicationLeaseResponse {
             movement_generation: 6,
             ..response.clone()
         };
-        let error = validate_scanner_publication_lease_response_fields(&stale_generation, "session-a", 7)
+        let error = validate_scanner_publication_lease_response_fields(&stale_generation, "session-a", 7, false)
             .expect_err("a response from an older movement generation must be rejected");
         assert!(error.to_string().contains("different scanner publication lease generation"));
 
@@ -3017,9 +3035,21 @@ mod tests {
             session_id: "session-b".to_string(),
             ..response
         };
-        let error = validate_scanner_publication_lease_response_fields(&stale_session, "session-a", 7)
+        let error = validate_scanner_publication_lease_response_fields(&stale_session, "session-a", 7, false)
             .expect_err("a response from an older scanner session must be rejected");
         assert!(error.to_string().contains("different scanner publication lease session"));
+
+        let observational = ScannerPublicationLeaseResponse {
+            observational_only: true,
+            ..stale_session
+        };
+        assert!(
+            validate_scanner_publication_lease_response_fields(&observational, "session-b", 7, true).is_ok(),
+            "the signed purpose echo should validate an observation lease"
+        );
+        let error = validate_scanner_publication_lease_response_fields(&observational, "session-b", 7, false)
+            .expect_err("a peer must not upgrade an observational lease to authoritative");
+        assert!(error.to_string().contains("different scanner publication lease purpose"));
     }
 
     #[test]

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -14,7 +14,8 @@
 
 use crate::cluster::rpc::client::{
     AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
-    node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
+    node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
+    node_service_time_out_client_no_auth,
 };
 use crate::cluster::rpc::internode_data_transport::{
     InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -22,10 +23,11 @@ use crate::cluster::rpc::internode_data_transport::{
 };
 use crate::disk::error::{Error, Result};
 use crate::disk::{
-    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
-    DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
-    ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
-    batch_read_version_one_by_one,
+    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
+    DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
+    DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
+    DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
+    RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
     disk_store::{
         DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
         get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -59,10 +61,10 @@ use rustfs_protos::proto_gen::node_service::{
     CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
     DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
     MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
-    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
-    SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
-    SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
-    node_service_client::NodeServiceClient,
+    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
+    RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
+    SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
+    WriteMetadataRequest, node_service_client::NodeServiceClient,
 };
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
@@ -142,6 +144,45 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
         .collect()
 }
 
+fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
+    if codes.len() != expected_len {
+        return Err(Error::other("malformed recoverable batch delete response length"));
+    }
+    codes
+        .into_iter()
+        .map(|code| match code {
+            None => Ok(None),
+            Some(code) => DiskError::from_u32(code)
+                .map(Some)
+                .ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
+        })
+        .collect()
+}
+
+fn validate_delete_recovery_inspection(
+    volume: &str,
+    object: &str,
+    transaction: Uuid,
+    inspection: &DeleteRecoveryInspection,
+) -> Result<()> {
+    if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+        || inspection.id.object != object
+        || inspection.id.transaction != transaction
+        || transaction.is_nil()
+    {
+        return Err(Error::other("delete recovery inspection does not match request"));
+    }
+    if let Some(journal) = &inspection.journal
+        && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || journal.volume != volume
+            || journal.object != object
+            || journal.transaction != transaction)
+    {
+        return Err(Error::other("delete recovery journal does not match request"));
+    }
+    Ok(())
+}
+
 fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
     if !response.success {
         return Err(response.error.unwrap_or_default().into());
@@ -858,6 +899,75 @@ fn spawn_control_channel_prewarm(addr: String) {
 }
 
 impl RemoteDisk {
+    async fn recover_delete_transaction(
+        &self,
+        volume: &str,
+        operation: DeleteRecoveryRpcOperation,
+    ) -> Result<DeleteRecoveryRpcReply> {
+        let payload = rmp_serde::to_vec_named(&operation)?;
+        if payload.len() > operation.max_wire_payload_bytes() {
+            return Err(Error::other("delete recovery operation exceeds payload limit"));
+        }
+        let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
+            Some(crate::disk::get_delete_recovery_timeout()?)
+        } else {
+            None
+        };
+        self.execute_with_timeout(
+            || async {
+                let mut client = if let Some(timeout) = settle_timeout {
+                    if let Some(error) = self.offline_bypass_error() {
+                        return Err(error);
+                    }
+                    node_service_delete_recovery_client(
+                        &self.addr,
+                        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
+                        timeout,
+                    )
+                    .await?
+                } else {
+                    self.get_client().await?
+                };
+                let mut request = Request::new(RecoverDeleteTransactionRequest {
+                    disk: self.endpoint.to_string(),
+                    volume: volume.to_owned(),
+                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+                    operation: payload.into(),
+                });
+                let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
+                    .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
+                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
+                // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
+                let response = client
+                    .recover_delete_transaction(request)
+                    .await
+                    .map_err(|status| {
+                        if status.code() == Code::Unimplemented {
+                            DiskError::UnsupportedDisk
+                        } else {
+                            status.into()
+                        }
+                    })?
+                    .into_inner();
+                if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
+                    return Err(DiskError::UnsupportedDisk);
+                }
+                if !response.success {
+                    return Err(response
+                        .error
+                        .ok_or_else(|| Error::other("missing delete recovery error"))?
+                        .into());
+                }
+                if response.error.is_some() {
+                    return Err(Error::other("malformed delete recovery success"));
+                }
+                rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
+            },
+            settle_timeout.unwrap_or_else(get_max_timeout_duration),
+        )
+        .await
+    }
+
     async fn rename_file_with_durability(
         &self,
         src_volume: &str,
@@ -1920,6 +2030,12 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
 }
 
+// Optional fields are omitted by serde; named maps keep rollback flags and
+// transaction UUIDs independent of field order across peer versions.
+fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
+    Ok(rmp_serde::to_vec_named(value)?)
+}
+
 fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
 }
@@ -2163,6 +2279,7 @@ impl RemoteDisk {
         self.execute_with_timeout_for_op(
             "rename_data",
             || async {
+                let prepare_started = rustfs_io_metrics::put_stage_timer();
                 let file_info = compat_json(fi)?;
                 let file_info_bin = encode_file_info_msgpack(fi)?;
                 let mut client = self.get_client().await?;
@@ -2190,24 +2307,67 @@ impl RemoteDisk {
                 } else {
                     attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
                 }
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
+                    prepare_started,
+                );
 
-                let response = if incarnation_bound {
-                    // Older peers return Unimplemented before mutation; never downgrade.
-                    client.rename_data_at_incarnation(request).await?
-                } else {
-                    client.rename_data(request).await?
-                }
-                .into_inner();
+                let rpc_started = rustfs_io_metrics::put_stage_timer();
+                let response = rustfs_io_metrics::observe_put_stage_future(
+                    async {
+                        if incarnation_bound {
+                            // Older peers return Unimplemented before mutation; never downgrade.
+                            client.rename_data_at_incarnation(request).await
+                        } else {
+                            client.rename_data(request).await
+                        }
+                    },
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
+                )
+                .await?;
+                let response_process_started = rustfs_io_metrics::put_stage_timer();
+                let response_headers_started = rustfs_io_metrics::put_stage_timer();
+                let _ = response.metadata().len();
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS,
+                    response_headers_started,
+                );
+                let into_inner_started = rustfs_io_metrics::put_stage_timer();
+                let response = response.into_inner();
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER,
+                    into_inner_started,
+                );
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC,
+                    rpc_started,
+                );
 
+                let success_check_started = rustfs_io_metrics::put_stage_timer();
                 if !response.success {
                     return Err(response.error.unwrap_or_default().into());
                 }
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK,
+                    success_check_started,
+                );
 
+                let decode_started = rustfs_io_metrics::put_stage_timer();
                 let rename_data_resp = decode_msgpack_or_json::<RenameDataResp>(
                     &response.rename_data_resp_bin,
                     &response.rename_data_resp,
                     "RenameDataResp",
                 )?;
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE,
+                    decode_started,
+                );
+                rustfs_io_metrics::record_put_object_stage_duration_from(
+                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS,
+                    response_process_started,
+                );
 
                 Ok(rename_data_resp)
             },
@@ -2566,7 +2726,7 @@ impl DiskAPI for RemoteDisk {
                 // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
                 // JSON + msgpack until its fallback counter has read zero across a release window.
                 let file_info_bin = encode_file_info_msgpack(&fi)?;
-                let opts_bin = encode_msgpack(&opts)?;
+                let opts_bin = encode_delete_options_msgpack(&opts)?;
                 let conditional_marker = opts.expected_delete_marker.is_some();
                 let file_info = serde_json::to_string(&fi)?;
                 let opts = serde_json::to_string(&opts)?;
@@ -2638,7 +2798,7 @@ impl DiskAPI for RemoteDisk {
 
         // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
         // msgpack until its fallback counter has read zero across a release window.
-        let opts_bin = match encode_msgpack(&opts) {
+        let opts_bin = match encode_delete_options_msgpack(&opts) {
             Ok(opts_bin) => opts_bin,
             Err(err) => {
                 let mut errors = Vec::with_capacity(versions.len());
@@ -3617,6 +3777,138 @@ impl DiskAPI for RemoteDisk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> Result<()> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::ApplyVersion {
+                    object: object.to_owned(),
+                    file_info: Box::new(file_info),
+                    force_del_marker,
+                    opts,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
+            _ => Err(Error::other("unexpected recoverable delete reply")),
+        }
+    }
+
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        let expected_len = versions.len();
+        let result = match self
+            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
+            .await
+        {
+            Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
+            Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
+            Err(error) => Err(error),
+        };
+        match result {
+            Ok(errors) => errors,
+            Err(error) => vec![Some(error); expected_len],
+        }
+    }
+
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Discover {
+                    object: object.map(str::to_owned),
+                    cursor,
+                    limit,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Discovered(page) => {
+                if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
+                    return Err(Error::other("delete recovery discovery escaped requested object"));
+                }
+                if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
+                    || page
+                        .transactions
+                        .iter()
+                        .any(|id| id.transaction.is_nil() || id.object.is_empty())
+                {
+                    return Err(Error::other("malformed delete discovery page"));
+                }
+                Ok(page)
+            }
+            _ => Err(Error::other("unexpected delete discovery reply")),
+        }
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Inspect {
+                    object: object.to_owned(),
+                    transaction,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Inspected(inspection) => {
+                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
+                Ok(inspection)
+            }
+            _ => Err(Error::other("unexpected delete inspection reply")),
+        }
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Settle {
+                    object: object.to_owned(),
+                    transaction,
+                    expected_fingerprint,
+                    action,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Settled(inspection) => {
+                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
+                Ok(inspection)
+            }
+            _ => Err(Error::other("unexpected delete settlement reply")),
+        }
+    }
+
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
         self.execute_with_timeout(
             || async {
@@ -4030,6 +4322,45 @@ mod tests {
 
     static INIT: Once = Once::new();
 
+    #[test]
+    fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
+        let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
+            .expect("valid per-item errors");
+        assert!(errors[0].is_none());
+        assert_eq!(errors[1], Some(DiskError::FileNotFound));
+        assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
+        assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
+        assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
+    }
+
+    #[test]
+    fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
+        let transaction = Uuid::new_v4();
+        let inspection = DeleteRecoveryInspection {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            id: crate::disk::DeleteTransactionId {
+                object: "object".into(),
+                transaction,
+            },
+            state: crate::disk::DeleteRecoveryState::Absent,
+            journal: None,
+            current_digest: None,
+            current_versions: Vec::new(),
+            backup_digest: None,
+            rollback_data_complete: false,
+            commit_data_safe: false,
+            explicit_delete_safe: false,
+            fingerprint: [0; 32],
+        };
+        validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
+        assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
+        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
+        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
+        let mut incompatible = inspection;
+        incompatible.protocol_version += 1;
+        assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
+    }
+
     #[test]
     fn compare_and_update_wire_outcome_fails_closed() {
         assert_eq!(
@@ -4063,32 +4394,19 @@ mod tests {
     }
 
     #[test]
-    fn retired_marker_options_preserve_legacy_positional_wire_shape() {
+    fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
         #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
         struct LegacyDeleteOptions {
             recursive: bool,
             immediate: bool,
             undo_write: bool,
+            #[serde(default)]
             undo_delete: bool,
             old_data_dir: Option<Uuid>,
+            #[serde(default, skip_serializing_if = "Option::is_none")]
+            expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
         }
-        let legacy = LegacyDeleteOptions {
-            recursive: false,
-            immediate: false,
-            undo_write: false,
-            undo_delete: false,
-            old_data_dir: None,
-        };
-        let original = encode_msgpack(&legacy).unwrap();
-        let current = encode_msgpack(&DeleteOptions::default()).unwrap();
-        assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
-        assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
-        assert!(
-            rmp_serde::from_slice::<DeleteOptions>(&original)
-                .unwrap()
-                .expected_delete_marker
-                .is_none()
-        );
+        let transaction = Uuid::new_v4();
         let mut marker = FileInfo {
             deleted: true,
             version_id: Some(Uuid::new_v4()),
@@ -4097,12 +4415,69 @@ mod tests {
         };
         marker.set_delete_marker_incarnation(Uuid::new_v4());
         let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
+        for expected_delete_marker in [None, Some(marker)] {
+            for (undo_write, undo_delete) in [(true, false), (false, true)] {
+                let legacy = LegacyDeleteOptions {
+                    recursive: false,
+                    immediate: false,
+                    undo_write,
+                    undo_delete,
+                    old_data_dir: Some(transaction),
+                    expected_delete_marker: expected_delete_marker.clone(),
+                };
+                // Older senders use positional arrays. Their unchanged field prefix and
+                // missing optional fields must also decode on the named-map sender's peer.
+                for (encoding, payload) in [
+                    ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
+                    ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
+                ] {
+                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
+                    assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
+                    assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
+                    assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
+                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
+                    assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
+                    assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
+                }
+                for incarnation in [None, Some(Uuid::new_v4())] {
+                    let current = DeleteOptions {
+                        undo_write,
+                        undo_delete,
+                        old_data_dir: Some(transaction),
+                        expected_delete_marker: expected_delete_marker.clone(),
+                        delete_recovery_bucket_incarnation: incarnation,
+                        recoverable_delete: true,
+                        ..Default::default()
+                    };
+                    let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
+                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
+                    assert_eq!(decoded.undo_write, undo_write);
+                    assert_eq!(decoded.undo_delete, undo_delete);
+                    assert_eq!(decoded.old_data_dir, Some(transaction));
+                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
+                    assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
+                    assert!(decoded.recoverable_delete);
+                    let decoded: LegacyDeleteOptions =
+                        rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
+                    assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
         let options = DeleteOptions {
-            expected_delete_marker: Some(marker.clone()),
+            undo_write: true,
+            old_data_dir: Some(Uuid::new_v4()),
             ..Default::default()
         };
-        let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
-        assert_eq!(decoded.expected_delete_marker, Some(marker));
+        let json = serde_json::to_string(&options).expect("valid compatibility JSON");
+        assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
+        let decoded: DeleteOptions =
+            decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
+        assert!(decoded.undo_write);
+        assert_eq!(decoded.old_data_dir, options.old_data_dir);
     }
 
     #[test]
@@ -4785,6 +5160,8 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![0x14, 0x35]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)),
+            old_current_source_checked: false,
+            old_current_source: None,
         };
         let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode");
         let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test();
@@ -5187,6 +5564,8 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![1_u8; 32]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)),
+            old_current_source_checked: false,
+            old_current_source: None,
         };
         let json = serde_json::to_vec(&response).expect("rename data response json should encode");
         let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode");

crates/ecstore/src/core/pools.rs

@@ -3691,6 +3691,55 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error {
     }
 }
 
+fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error {
+    match err {
+        rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable {
+            mode: "read",
+            bucket: RUSTFS_META_BUCKET.to_string(),
+            object: object.to_string(),
+            required,
+            achieved,
+        },
+        other => Error::Lock(other),
+    }
+}
+
+/// Read-only admission to an already durable rebalance run. Keeping this
+/// separate from the activation writer prevents admission from authorizing a
+/// pool or rebalance metadata write under shared namespace locks.
+pub(crate) struct PoolRebalanceWorkerFence {
+    pool_meta_guard: rustfs_lock::NamespaceLockGuard,
+    rebalance_meta_guard: rustfs_lock::NamespaceLockGuard,
+    fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
+}
+
+impl PoolRebalanceWorkerFence {
+    pub(crate) fn set_fleet_proof(
+        &mut self,
+        fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
+    ) {
+        self.fleet_proof = fleet_proof;
+    }
+
+    pub(crate) fn ensure_held(&self) -> Result<()> {
+        if self.pool_meta_guard.is_lock_lost()
+            || self.pool_meta_guard.is_released()
+            || self.rebalance_meta_guard.is_lock_lost()
+            || self.rebalance_meta_guard.is_released()
+        {
+            return Err(Error::other("worker admission lock lost before rebalance activation"));
+        }
+        if self
+            .fleet_proof
+            .as_ref()
+            .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
+        {
+            return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
+        }
+        Ok(())
+    }
+}
+
 pub(crate) struct PoolRebalanceActivationFence {
     pool_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
     rebalance_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
@@ -3762,6 +3811,35 @@ tokio::task_local! {
     pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc<tokio::sync::Notify>;
 }
 
+pub(crate) async fn acquire_pool_rebalance_worker_locks<S>(pool: Arc<S>) -> Result<PoolRebalanceWorkerFence>
+where
+    S: crate::storage_api_contracts::namespace::NamespaceLocking<
+            Error = Error,
+            NamespaceLock = rustfs_lock::NamespaceLockWrapper,
+        >,
+{
+    // Match activation and entry admission: rebalance.bin -> pool.bin. Joining
+    // the same durable run must coexist with entries retaining their run fence.
+    let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?;
+    #[cfg(test)]
+    let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one());
+    let rebalance_meta_guard = rebalance_meta_lock
+        .get_read_lock(get_lock_acquire_timeout())
+        .await
+        .map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?;
+    let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
+    let pool_meta_guard = pool_meta_lock
+        .get_read_lock(get_lock_acquire_timeout())
+        .await
+        .map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?;
+
+    Ok(PoolRebalanceWorkerFence {
+        pool_meta_guard,
+        rebalance_meta_guard,
+        fleet_proof: None,
+    })
+}
+
 pub(crate) async fn acquire_pool_rebalance_activation_locks<S>(
     pool: Arc<S>,
     fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
@@ -12430,6 +12508,18 @@ impl ECStore {
         Ok(pool_meta)
     }
 
+    pub(crate) async fn load_runtime_pool_meta_under_worker_fence(
+        &self,
+        write_state: &mut PoolMetaWriteState,
+        worker_fence: &PoolRebalanceWorkerFence,
+        operation: &str,
+    ) -> Result<PoolMeta> {
+        worker_fence.ensure_held()?;
+        let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?;
+        worker_fence.ensure_held()?;
+        Ok(pool_meta)
+    }
+
     pub(crate) async fn load_runtime_pool_meta_under_activation_fence(
         &self,
         write_state: &mut PoolMetaWriteState,

crates/ecstore/src/data_usage/mod.rs

@@ -738,7 +738,12 @@ where
         .filter(|etag| !etag.is_empty())
         .map(str::to_owned)
         .ok_or_else(|| Error::other("data usage snapshot has no ETag"))?;
-    let mut data_usage_info = parse_usage_snapshot(&reader.read_all().await?)?;
+    let mut data_usage_info = parse_usage_snapshot(
+        &reader
+            .read_all()
+            .await
+            .map_err(|err| map_data_usage_metadata_read_error(err, object))?,
+    )?;
     populate_backward_compatible_usage_maps(&mut data_usage_info);
     validate_complete_usage_snapshot(&mut data_usage_info);
     Ok(Some((data_usage_info, revision)))
@@ -2997,6 +3002,61 @@ mod tests {
         }
     }
 
+    #[derive(Debug)]
+    struct UsageSnapshotReadFailureStore;
+
+    struct UsageSnapshotFailureReader;
+
+    impl tokio::io::AsyncRead for UsageSnapshotFailureReader {
+        fn poll_read(
+            self: std::pin::Pin<&mut Self>,
+            _cx: &mut std::task::Context<'_>,
+            _buf: &mut tokio::io::ReadBuf<'_>,
+        ) -> std::task::Poll<std::io::Result<()>> {
+            std::task::Poll::Ready(Err(std::io::Error::other(Error::VolumeNotFound)))
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl crate::storage_api_contracts::object::ObjectIO for UsageSnapshotReadFailureStore {
+        type Error = Error;
+        type RangeSpec = crate::storage_api_contracts::range::HTTPRangeSpec;
+        type HeaderMap = http::HeaderMap;
+        type ObjectOptions = ObjectOptions;
+        type ObjectInfo = ObjectInfo;
+        type GetObjectReader = crate::object_api::GetObjectReader;
+        type PutObjectReader = PutObjReader;
+
+        async fn get_object_reader(
+            &self,
+            _bucket: &str,
+            _object: &str,
+            _range: Option<Self::RangeSpec>,
+            _headers: Self::HeaderMap,
+            _opts: &Self::ObjectOptions,
+        ) -> Result<Self::GetObjectReader, Self::Error> {
+            Ok(crate::object_api::GetObjectReader {
+                stream: Box::new(UsageSnapshotFailureReader),
+                object_info: ObjectInfo {
+                    etag: Some("usage-revision".to_string()),
+                    ..Default::default()
+                },
+                buffered_body: None,
+                body_source: Default::default(),
+            })
+        }
+
+        async fn put_object(
+            &self,
+            _bucket: &str,
+            _object: &str,
+            _data: &mut Self::PutObjectReader,
+            _opts: &Self::ObjectOptions,
+        ) -> Result<Self::ObjectInfo, Self::Error> {
+            unimplemented!("the read-failure fixture does not write")
+        }
+    }
+
     /// Minimal ObjectIO backing `load_data_usage_cache` tests: records the keys
     /// read and fails the first N reads with a transient (non-absence) error.
     #[derive(Debug, Default)]
@@ -3161,6 +3221,20 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn data_usage_removal_maps_missing_system_volume_during_snapshot_stream_read() {
+        let store = UsageSnapshotReadFailureStore;
+
+        let error = load_data_usage_for_bucket_removal(&store, "bucket-metadata/.usage.json")
+            .await
+            .expect_err("a system volume disappearing during body read must remain a retryable storage error");
+
+        assert_eq!(
+            error,
+            Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string())
+        );
+    }
+
     #[tokio::test]
     async fn load_data_usage_cache_retries_a_transient_failure() {
         let name = "usage-cache";

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -15,7 +15,9 @@
 use crate::cluster::rpc::{
     ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
 };
-use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
+use crate::data_usage::{
+    DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
+};
 use crate::error::{Error, Result};
 use crate::{
     disk::endpoint::{Endpoint, EndpointType},
@@ -42,7 +44,11 @@ use std::{
 use time::OffsetDateTime;
 use tokio::time::timeout;
 use tonic::Request;
-use tracing::warn;
+use tracing::{error, trace, warn};
+
+const EVENT_SERVER_PING: &str = "server_ping";
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics";
 
 use shadow_rs::shadow;
 
@@ -157,11 +163,22 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
 
         let response: PingResponse = client.ping(request).await?.into_inner();
 
-        let ping_response_body = flatbuffers::root::<PingBody>(&response.body);
-        if let Err(e) = ping_response_body {
-            eprintln!("{e}");
+        if flatbuffers::root::<PingBody>(&response.body).is_err() {
+            error!(
+                event = EVENT_SERVER_PING,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
+                result = "invalid_response",
+                "Server ping response decoding failed"
+            );
         } else {
-            println!("ping_resp:body(flatbuffer): {ping_response_body:?}");
+            trace!(
+                event = EVENT_SERVER_PING,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
+                result = "success",
+                "Server ping completed"
+            );
         }
 
         Ok(())
@@ -294,7 +311,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
         apply_data_usage_result(
-            load_data_usage_from_backend_cached(store.clone()).await,
+            load_server_info_usage_snapshot(store.clone()).await,
             &mut buckets,
             &mut objects,
             &mut versions,
@@ -967,6 +984,93 @@ mod tests {
         assert_eq!(usage.size, 6);
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn server_info_reads_complete_observation_without_promoting_authority() {
+        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
+        use crate::config::com::{read_config, save_config};
+        use crate::data_usage::{
+            DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
+            load_data_usage_from_backend_cached,
+        };
+        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
+        use std::time::{Duration, SystemTime};
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let primary = DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
+            scanner_epoch: Some(1),
+            usage_snapshot_bootstrap_pending: true,
+            usage_snapshot_converged: Some(false),
+            ..Default::default()
+        };
+        let observed = DataUsageInfo {
+            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
+            scanner_epoch: Some(1),
+            scanner_cycle: Some(1),
+            buckets_count: 1,
+            objects_total_count: 3,
+            versions_total_count: 4,
+            delete_markers_total_count: 1,
+            objects_total_size: 6,
+            buckets_usage: [(
+                "bucket".to_string(),
+                BucketUsageInfo {
+                    objects_count: 3,
+                    versions_count: 4,
+                    delete_markers_count: 1,
+                    size: 6,
+                    ..Default::default()
+                },
+            )]
+            .into_iter()
+            .collect(),
+            usage_snapshot_complete: true,
+            usage_snapshot_converged: Some(false),
+            usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
+            ..Default::default()
+        };
+        let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
+        save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
+            .await
+            .expect("bootstrap should persist");
+        save_config(
+            store.clone(),
+            DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
+            serde_json::to_vec(&observed).expect("observation should encode"),
+        )
+        .await
+        .expect("observation should persist");
+        invalidate_data_usage_snapshot_cache().await;
+
+        let displayed = super::load_server_info_usage_snapshot(store.clone())
+            .await
+            .expect("server info should load the complete observation");
+        assert_eq!(displayed.usage_snapshot_converged, Some(false));
+        let mut buckets = rustfs_madmin::Buckets::default();
+        let mut objects = rustfs_madmin::Objects::default();
+        let mut versions = rustfs_madmin::Versions::default();
+        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
+        let mut usage = rustfs_madmin::Usage::default();
+        apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
+        assert!(objects.error.is_none());
+        assert_eq!(
+            (buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
+            (1, 3, 4, 1, 6)
+        );
+        let quota = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("quota should retain its bootstrap state");
+        assert!(!quota.is_complete_bucket_usage_snapshot());
+        assert_eq!(
+            read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
+                .await
+                .expect("primary should remain readable"),
+            primary_bytes
+        );
+        invalidate_data_usage_snapshot_cache().await;
+    }
+
     #[test]
     fn incomplete_data_usage_is_unavailable_in_server_info() {
         let mut buckets = rustfs_madmin::Buckets::default();

crates/ecstore/src/diagnostics/get.rs

@@ -182,6 +182,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
 pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
 pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
 pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
+pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
 pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
 pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";

crates/ecstore/src/diagnostics/object_lock.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use rustfs_io_metrics::record_object_lock_acquire_failure;
 use rustfs_lock::{LockError, NamespaceLockGuard, NamespaceLockWrapper};
 use std::time::{Duration, Instant};
 use tracing::Span;
@@ -21,6 +22,26 @@ const EVENT_OBJECT_NAMESPACE_LOCK: &str = "object_namespace_lock";
 const LOG_COMPONENT_ECSTORE: &str = "ecstore";
 const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
 
+pub(crate) fn record_lock_acquire_failure(op: &'static str, bucket: &str, object: &str, mode: &'static str, error: &LockError) {
+    let resource_class = if bucket == crate::disk::RUSTFS_META_MULTIPART_BUCKET {
+        "multipart_metadata"
+    } else if bucket == crate::disk::RUSTFS_META_BUCKET || bucket == ".minio.sys" {
+        "system_metadata"
+    } else if object.is_empty() || object == bucket {
+        "bucket_namespace"
+    } else {
+        "object_namespace"
+    };
+    let reason = match error {
+        LockError::Timeout { .. } => "timeout",
+        LockError::AlreadyLocked { .. } => "contention",
+        LockError::QuorumNotReached { .. } | LockError::InsufficientNodes { .. } => "quorum",
+        LockError::Network { .. } | LockError::Internal { .. } => "transport",
+        _ => "other",
+    };
+    record_object_lock_acquire_failure(op, mode, resource_class, reason);
+}
+
 /// Diagnostic lifetime only; it never owns or changes a namespace lock.
 /// Keep it alongside the real guard when ownership moves to a commit tail.
 #[derive(Default)]

crates/ecstore/src/disk/error_reduce.rs

@@ -131,6 +131,15 @@ pub fn build_write_quorum_failure_summary(
 
 fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
     let Some(dominant_error) = dominant_error else {
+        // The reducer also returns None when every failure is ignored for
+        // quorum voting. That case contains no successful nil entry.
+        if !errors.is_empty() && errors.iter().all(Option::is_some) {
+            return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
+                "disk_not_found"
+            } else {
+                "all_ignored"
+            };
+        }
         return "nil_dominated";
     };
 
@@ -387,9 +396,9 @@ mod tests {
     }
 
     #[test]
-    fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
-        // All disks offline (all DiskNotFound). The dominant error must be None and
-        // the label must be "nil_dominated" rather than falling back to "other_error".
+    fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
+        // Ignoring unavailable disks for quorum reduction must not imply a
+        // successful nil entry in the diagnostic label.
         let errors = vec![
             Some(Error::DiskNotFound),
             Some(Error::DiskNotFound),
@@ -397,7 +406,21 @@ mod tests {
         ];
         let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
         assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.dominant_error_label, "nil_dominated");
+        assert_eq!(summary.achieved, 0);
+        assert_eq!(summary.dominant_error_label, "disk_not_found");
+    }
+
+    #[test]
+    fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
+        let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
+        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
+        assert_eq!(summary.dominant_error, None);
+        assert_eq!(summary.achieved, 0);
+        assert_eq!(summary.dominant_error_label, "all_ignored");
+        assert_eq!(
+            reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
+            Some(Error::ErasureWriteQuorum)
+        );
     }
 
     #[test]

crates/ecstore/src/disk/fs.rs

@@ -223,12 +223,6 @@ pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()
     std::fs::rename(from, to)
 }
 
-#[cfg(any(not(windows), test))]
-#[tracing::instrument(level = "debug", skip_all)]
-pub async fn read_file(path: impl AsRef<Path>) -> io::Result<Vec<u8>> {
-    fs::read(path.as_ref()).await
-}
-
 // Bucket existence cache - reduces statx syscalls for repeated bucket checks
 
 /// Cache for bucket directory existence checks.
@@ -614,28 +608,6 @@ mod tests {
         assert_eq!(content, "test content");
     }
 
-    #[tokio::test]
-    async fn test_read_file() {
-        let temp_dir = TempDir::new().unwrap();
-        let file_path = temp_dir.path().join("test_read.txt");
-
-        let test_content = b"This is test content for reading";
-        tokio::fs::write(&file_path, test_content).await.unwrap();
-
-        // Read file
-        let read_content = read_file(&file_path).await.unwrap();
-        assert_eq!(read_content, test_content);
-    }
-
-    #[tokio::test]
-    async fn test_read_file_nonexistent() {
-        let temp_dir = TempDir::new().unwrap();
-        let file_path = temp_dir.path().join("nonexistent.txt");
-
-        // Should fail for non-existent file
-        assert!(read_file(&file_path).await.is_err());
-    }
-
     #[tokio::test]
     async fn test_same_file() {
         let temp_dir = TempDir::new().unwrap();

crates/ecstore/src/disk/io_admission.rs

@@ -0,0 +1,706 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Per-local-disk foreground-priority admission for scanner I/O.
+
+use std::collections::VecDeque;
+use std::future::Future;
+use std::io;
+use std::pin::Pin;
+use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+use std::sync::{Arc, Mutex};
+use std::task::{Context, Poll};
+use std::time::Duration;
+
+use metrics::{counter, gauge, histogram};
+use rustfs_rio::ChunkReader;
+use tokio::io::{AsyncRead, ReadBuf};
+use tokio::sync::Notify;
+use tokio::time::Instant;
+use tokio_util::sync::CancellationToken;
+
+const BACKGROUND_FOREGROUND_BURST: u32 = 8;
+const BACKGROUND_MAX_WAIT: Duration = Duration::from_millis(100);
+
+tokio::task_local! {
+    static DISK_IO_CONTEXT: DiskIoContext;
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(crate) enum DiskIoClass {
+    Foreground,
+    Background,
+}
+
+#[derive(Clone, Debug)]
+pub(crate) struct DiskIoContext {
+    pub(crate) class: DiskIoClass,
+    pub(crate) cancellation: CancellationToken,
+    pub(crate) admission: Option<Arc<DiskIoAdmission>>,
+    pub(crate) permit: Option<DiskIoPermit>,
+}
+
+/// Mark disk I/O issued by `future` as background work for local-disk admission.
+#[doc(hidden)]
+pub async fn with_background_disk_io<F: Future>(cancellation: CancellationToken, future: F) -> F::Output {
+    with_disk_io_context(
+        DiskIoContext {
+            class: DiskIoClass::Background,
+            cancellation,
+            admission: None,
+            permit: None,
+        },
+        future,
+    )
+    .await
+}
+
+pub(crate) async fn with_disk_io_context<F: Future>(context: DiskIoContext, future: F) -> F::Output {
+    DISK_IO_CONTEXT.scope(context, future).await
+}
+
+pub(crate) async fn with_disk_io_permit<F: Future>(permit: Option<DiskIoPermit>, future: F) -> F::Output {
+    let Some(permit) = permit else {
+        return future.await;
+    };
+    let mut context = current_disk_io_context().unwrap_or_else(|| DiskIoContext {
+        class: DiskIoClass::Foreground,
+        cancellation: CancellationToken::new(),
+        admission: None,
+        permit: None,
+    });
+    context.admission = Some(permit.admission());
+    with_disk_io_context(
+        DiskIoContext {
+            permit: Some(permit),
+            ..context
+        },
+        future,
+    )
+    .await
+}
+
+pub(crate) async fn with_optional_disk_io_context<F: Future>(context: Option<DiskIoContext>, future: F) -> F::Output {
+    match context {
+        Some(context) => with_disk_io_context(context, future).await,
+        None => future.await,
+    }
+}
+
+pub(crate) fn current_disk_io_context() -> Option<DiskIoContext> {
+    DISK_IO_CONTEXT.try_with(Clone::clone).ok()
+}
+
+/// Acquire one low-level permit only when the current walker explicitly owns a gate.
+/// Ordinary wrapper calls use their outer operation permit instead.
+pub(crate) async fn acquire_context_disk_io_permit() -> io::Result<Option<DiskIoPermit>> {
+    let Some(context) = current_disk_io_context() else {
+        return Ok(None);
+    };
+    let Some(admission) = context.admission else {
+        return Ok(None);
+    };
+    if let Some(permit) = context.permit
+        && permit.is_for(&admission)
+    {
+        return Ok(Some(permit));
+    }
+    match context.class {
+        DiskIoClass::Foreground => Ok(Some(admission.foreground())),
+        DiskIoClass::Background => Ok(Some(admission.background(&context.cancellation).await?)),
+    }
+}
+
+#[derive(Debug, Default)]
+struct AdmissionState {
+    foreground_completions: u32,
+    next_background_ticket: u64,
+    background_queue: VecDeque<u64>,
+    background_front_since: Option<Instant>,
+}
+
+/// One gate is shared by all handles to a local disk, including reconnects.
+/// Foreground I/O never waits on this gate. Background I/O runs exclusively
+/// while the disk is idle, or after a bounded foreground burst/wait.
+#[derive(Debug)]
+pub(crate) struct DiskIoAdmission {
+    state: Mutex<AdmissionState>,
+    changed: Notify,
+    disk: String,
+    foreground_active: AtomicUsize,
+    background_active: AtomicBool,
+    background_waiters: AtomicUsize,
+    #[cfg(test)]
+    background_admissions: AtomicUsize,
+}
+
+impl DiskIoAdmission {
+    pub(crate) fn new(disk: impl Into<String>) -> Arc<Self> {
+        Arc::new(Self {
+            state: Mutex::new(AdmissionState::default()),
+            changed: Notify::new(),
+            disk: disk.into(),
+            foreground_active: AtomicUsize::new(0),
+            background_active: AtomicBool::new(false),
+            background_waiters: AtomicUsize::new(0),
+            #[cfg(test)]
+            background_admissions: AtomicUsize::new(0),
+        })
+    }
+
+    #[cfg(test)]
+    pub(crate) fn background_admissions_for_tests(&self) -> usize {
+        self.background_admissions.load(Ordering::Relaxed)
+    }
+
+    #[cfg(test)]
+    pub(crate) fn background_active_for_tests(&self) -> bool {
+        self.background_active.load(Ordering::Acquire)
+    }
+
+    #[cfg(test)]
+    fn foreground_active_for_tests(&self) -> usize {
+        self.foreground_active.load(Ordering::Acquire)
+    }
+
+    pub(crate) fn foreground(self: &Arc<Self>) -> DiskIoPermit {
+        self.foreground_active.fetch_add(1, Ordering::AcqRel);
+        if self.background_waiters.load(Ordering::Acquire) > 0 {
+            self.record_state();
+        }
+        DiskIoPermit {
+            lease: Arc::new(DiskIoPermitLease {
+                admission: self.clone(),
+                class: DiskIoClass::Foreground,
+            }),
+        }
+    }
+
+    pub(crate) async fn background(self: &Arc<Self>, cancellation: &CancellationToken) -> std::io::Result<DiskIoPermit> {
+        let started_at = Instant::now();
+        let ticket = {
+            let mut state = self.state.lock().unwrap_or_else(|error| error.into_inner());
+            let ticket = state.next_background_ticket;
+            state.next_background_ticket = state.next_background_ticket.wrapping_add(1);
+            if state.background_queue.is_empty() && !self.background_active.load(Ordering::Acquire) {
+                state.background_front_since = Some(Instant::now());
+            }
+            state.background_queue.push_back(ticket);
+            self.background_waiters.fetch_add(1, Ordering::Release);
+            ticket
+        };
+        self.record_state();
+
+        let mut waiter = BackgroundWaiter {
+            admission: self.clone(),
+            ticket,
+            waiting: true,
+        };
+        loop {
+            let notified = self.changed.notified();
+            tokio::pin!(notified);
+            notified.as_mut().enable();
+
+            let granted = {
+                let mut state = self.state.lock().unwrap_or_else(|error| error.into_inner());
+                let is_front = state.background_queue.front() == Some(&ticket);
+                let bounded_wait_elapsed = is_front
+                    && state
+                        .background_front_since
+                        .is_some_and(|front_since| front_since.elapsed() >= BACKGROUND_MAX_WAIT);
+                if is_front
+                    && !self.background_active.load(Ordering::Acquire)
+                    && (self.foreground_active.load(Ordering::Acquire) == 0
+                        || state.foreground_completions >= BACKGROUND_FOREGROUND_BURST
+                        || bounded_wait_elapsed)
+                {
+                    self.background_active.store(true, Ordering::Release);
+                    state.background_queue.pop_front();
+                    state.background_front_since = None;
+                    self.background_waiters.fetch_sub(1, Ordering::AcqRel);
+                    state.foreground_completions = 0;
+                    true
+                } else {
+                    false
+                }
+            };
+
+            if granted {
+                waiter.waiting = false;
+                #[cfg(test)]
+                self.background_admissions.fetch_add(1, Ordering::Relaxed);
+                histogram!("rustfs_disk_io_admission_wait_seconds", "disk" => self.disk.clone(), "class" => "background")
+                    .record(started_at.elapsed().as_secs_f64());
+                counter!("rustfs_disk_io_admission_total", "disk" => self.disk.clone(), "class" => "background").increment(1);
+                self.record_state();
+                return Ok(DiskIoPermit {
+                    lease: Arc::new(DiskIoPermitLease {
+                        admission: self.clone(),
+                        class: DiskIoClass::Background,
+                    }),
+                });
+            }
+
+            let remaining = {
+                let state = self.state.lock().unwrap_or_else(|error| error.into_inner());
+                if state.background_queue.front() == Some(&ticket) {
+                    state
+                        .background_front_since
+                        .map(|front_since| BACKGROUND_MAX_WAIT.saturating_sub(front_since.elapsed()))
+                } else {
+                    None
+                }
+            };
+            if let Some(remaining) = remaining.filter(|remaining| !remaining.is_zero()) {
+                tokio::select! {
+                    _ = cancellation.cancelled() => {
+                        return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "background disk I/O admission cancelled"));
+                    }
+                    _ = tokio::time::sleep(remaining) => {
+                        self.changed.notify_waiters();
+                    }
+                    _ = &mut notified => {}
+                }
+            } else {
+                tokio::select! {
+                    _ = cancellation.cancelled() => {
+                        return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "background disk I/O admission cancelled"));
+                    }
+                    _ = &mut notified => {}
+                }
+            }
+        }
+    }
+
+    fn record_state(&self) {
+        let (foreground_active, background_waiters) = {
+            let state = self.state.lock().unwrap_or_else(|error| error.into_inner());
+            (
+                if state.background_queue.is_empty() {
+                    0
+                } else {
+                    self.foreground_active.load(Ordering::Acquire)
+                },
+                state.background_queue.len(),
+            )
+        };
+        let foreground_active = u32::try_from(foreground_active).unwrap_or(u32::MAX);
+        let background_waiters = u32::try_from(background_waiters).unwrap_or(u32::MAX);
+        gauge!("rustfs_disk_io_admission_active", "disk" => self.disk.clone(), "class" => "foreground")
+            .set(f64::from(foreground_active));
+        gauge!("rustfs_disk_io_admission_active", "disk" => self.disk.clone(), "class" => "background").set(
+            if self.background_active.load(Ordering::Acquire) {
+                1.0
+            } else {
+                0.0
+            },
+        );
+        gauge!("rustfs_disk_io_admission_waiters", "disk" => self.disk.clone(), "class" => "background")
+            .set(f64::from(background_waiters));
+    }
+}
+
+struct BackgroundWaiter {
+    admission: Arc<DiskIoAdmission>,
+    ticket: u64,
+    waiting: bool,
+}
+
+impl Drop for BackgroundWaiter {
+    fn drop(&mut self) {
+        if !self.waiting {
+            return;
+        }
+        let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
+        if let Some(index) = state.background_queue.iter().position(|ticket| *ticket == self.ticket) {
+            state.background_queue.remove(index);
+            if index == 0 {
+                state.background_front_since =
+                    if state.background_queue.is_empty() || self.admission.background_active.load(Ordering::Acquire) {
+                        None
+                    } else {
+                        Some(Instant::now())
+                    };
+            }
+            self.admission.background_waiters.fetch_sub(1, Ordering::AcqRel);
+        }
+        drop(state);
+        self.admission.record_state();
+        self.admission.changed.notify_waiters();
+    }
+}
+
+#[derive(Debug, Clone)]
+pub(crate) struct DiskIoPermit {
+    lease: Arc<DiskIoPermitLease>,
+}
+
+impl DiskIoPermit {
+    pub(crate) fn is_for(&self, admission: &Arc<DiskIoAdmission>) -> bool {
+        Arc::ptr_eq(&self.lease.admission, admission)
+    }
+
+    fn admission(&self) -> Arc<DiskIoAdmission> {
+        self.lease.admission.clone()
+    }
+}
+
+#[derive(Debug)]
+struct DiskIoPermitLease {
+    admission: Arc<DiskIoAdmission>,
+    class: DiskIoClass,
+}
+
+pub(crate) fn spawn_blocking_with_disk_io_permit<T, F>(
+    io_permit: Option<DiskIoPermit>,
+    operation: F,
+) -> tokio::task::JoinHandle<T>
+where
+    T: Send + 'static,
+    F: FnOnce() -> T + Send + 'static,
+{
+    tokio::task::spawn_blocking(move || {
+        let _io_permit = io_permit;
+        operation()
+    })
+}
+
+type AdmissionWait = Pin<Box<dyn Future<Output = io::Result<DiskIoPermit>> + Send>>;
+
+impl Drop for DiskIoPermitLease {
+    fn drop(&mut self) {
+        match self.class {
+            DiskIoClass::Foreground => {
+                self.admission.foreground_active.fetch_sub(1, Ordering::AcqRel);
+                let record_state = self.admission.background_waiters.load(Ordering::Acquire) > 0;
+                if record_state {
+                    let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
+                    if !state.background_queue.is_empty() {
+                        state.foreground_completions = state.foreground_completions.saturating_add(1);
+                    }
+                }
+                if record_state {
+                    self.admission.record_state();
+                    self.admission.changed.notify_waiters();
+                }
+            }
+            DiskIoClass::Background => {
+                let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
+                self.admission.background_active.store(false, Ordering::Release);
+                state.background_front_since = if state.background_queue.is_empty() {
+                    None
+                } else {
+                    Some(Instant::now())
+                };
+                drop(state);
+                self.admission.record_state();
+                self.admission.changed.notify_waiters();
+            }
+        }
+    }
+}
+
+pub(crate) struct AdmissionReader<R> {
+    inner: R,
+    admission: Arc<DiskIoAdmission>,
+    context: DiskIoContext,
+    wait: Mutex<Option<AdmissionWait>>,
+    permit: Option<DiskIoPermit>,
+}
+
+impl<R> AdmissionReader<R> {
+    pub(crate) fn new(inner: R, admission: Arc<DiskIoAdmission>, context: DiskIoContext) -> Self {
+        Self {
+            inner,
+            admission,
+            context,
+            wait: Mutex::new(None),
+            permit: None,
+        }
+    }
+
+    fn poll_permit(&mut self, cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+        if self.permit.is_some() {
+            return Poll::Ready(Ok(()));
+        }
+
+        if self.context.class == DiskIoClass::Foreground {
+            self.permit = Some(self.admission.foreground());
+            return Poll::Ready(Ok(()));
+        }
+
+        let mut wait = self.wait.lock().unwrap_or_else(|error| error.into_inner());
+        if wait.is_none() {
+            let admission = self.admission.clone();
+            let cancellation = self.context.cancellation.clone();
+            *wait = Some(Box::pin(async move { admission.background(&cancellation).await }));
+        }
+        let Some(wait_future) = wait.as_mut() else {
+            return Poll::Ready(Err(io::Error::other("background disk I/O admission waiter is missing")));
+        };
+        let result = wait_future.as_mut().poll(cx);
+        match result {
+            Poll::Ready(Ok(permit)) => {
+                wait.take();
+                drop(wait);
+                self.permit = Some(permit);
+                Poll::Ready(Ok(()))
+            }
+            Poll::Ready(Err(error)) => {
+                wait.take();
+                Poll::Ready(Err(error))
+            }
+            Poll::Pending => Poll::Pending,
+        }
+    }
+
+    fn finish_poll<T>(&mut self, result: Poll<io::Result<T>>) -> Poll<io::Result<T>> {
+        if result.is_ready() {
+            self.permit = None;
+        }
+        result
+    }
+}
+
+impl<R: AsyncRead + Unpin> AsyncRead for AdmissionReader<R> {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
+        match self.poll_permit(cx) {
+            Poll::Ready(Ok(())) => {}
+            Poll::Ready(Err(error)) => return Poll::Ready(Err(error)),
+            Poll::Pending => return Poll::Pending,
+        }
+        let result = Pin::new(&mut self.inner).poll_read(cx, buf);
+        self.finish_poll(result)
+    }
+}
+
+impl<R: ChunkReader + Unpin> ChunkReader for AdmissionReader<R> {
+    fn poll_read_chunk(mut self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<bytes::Bytes>>> {
+        match self.poll_permit(cx) {
+            Poll::Ready(Ok(())) => {}
+            Poll::Ready(Err(error)) => return Poll::Ready(Err(error)),
+            Poll::Pending => return Poll::Pending,
+        }
+        let result = Pin::new(&mut self.inner).poll_read_chunk(cx, max);
+        self.finish_poll(result)
+    }
+}
+
+pub(crate) struct BoxedChunkReader(pub(crate) rustfs_rio::ChunkReaderBox);
+
+impl AsyncRead for BoxedChunkReader {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
+        Pin::new(&mut self.0).poll_read(cx, buf)
+    }
+}
+
+impl ChunkReader for BoxedChunkReader {
+    fn poll_read_chunk(self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<bytes::Bytes>>> {
+        Pin::new(self.get_mut().0.as_mut()).poll_read_chunk(cx, max)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+
+    struct BlockingWorker(tokio::task::JoinHandle<()>);
+
+    async fn wait_for_background_waiters(admission: &DiskIoAdmission, expected: usize) {
+        tokio::time::timeout(Duration::from_secs(1), async {
+            loop {
+                let waiters = admission
+                    .state
+                    .lock()
+                    .unwrap_or_else(|error| error.into_inner())
+                    .background_queue
+                    .len();
+                if waiters == expected {
+                    return;
+                }
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("background admission waiter should reach expected count");
+    }
+
+    async fn wait_for_foreground_active(admission: &DiskIoAdmission, expected: usize) {
+        tokio::time::timeout(Duration::from_secs(1), async {
+            loop {
+                let active = admission.foreground_active.load(Ordering::Acquire);
+                if active == expected {
+                    return;
+                }
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("foreground disk read should retain its permit while pending");
+    }
+
+    #[tokio::test]
+    async fn foreground_bypasses_background_and_background_progress_is_bounded() {
+        let admission = DiskIoAdmission::new("disk-a");
+        let foreground = admission.foreground();
+        let background_admission = admission.clone();
+        let cancellation = CancellationToken::new();
+        let background = tokio::spawn(async move { background_admission.background(&cancellation).await });
+
+        wait_for_background_waiters(&admission, 1).await;
+        for _ in 0..BACKGROUND_FOREGROUND_BURST {
+            drop(admission.foreground());
+        }
+
+        let background_permit = tokio::time::timeout(Duration::from_secs(1), background)
+            .await
+            .expect("background admission should progress after the foreground burst")
+            .expect("background waiter task should complete")
+            .expect("background admission should not be cancelled");
+        assert_eq!(
+            admission.foreground_active.load(Ordering::Acquire),
+            1,
+            "foreground operations remain admitted while one background operation runs"
+        );
+        drop(background_permit);
+        drop(foreground);
+    }
+
+    #[tokio::test]
+    async fn cancellation_removes_a_queued_background_waiter() {
+        let admission = DiskIoAdmission::new("disk-a");
+        let background = admission
+            .background(&CancellationToken::new())
+            .await
+            .expect("first background operation should be admitted");
+        let cancellation = CancellationToken::new();
+        let waiter_admission = admission.clone();
+        let waiter_cancellation = cancellation.clone();
+        let waiter = tokio::spawn(async move { waiter_admission.background(&waiter_cancellation).await });
+        wait_for_background_waiters(&admission, 1).await;
+
+        cancellation.cancel();
+        let error = tokio::time::timeout(Duration::from_secs(1), waiter)
+            .await
+            .expect("cancelled background waiter should stop")
+            .expect("background waiter task should complete")
+            .expect_err("cancelled admission must not start disk I/O");
+        assert_eq!(error.kind(), io::ErrorKind::Interrupted);
+        assert_eq!(
+            admission
+                .state
+                .lock()
+                .unwrap_or_else(|error| error.into_inner())
+                .background_queue
+                .len(),
+            0
+        );
+        drop(background);
+    }
+
+    #[tokio::test]
+    async fn background_admission_is_isolated_per_disk() {
+        let disk_a = DiskIoAdmission::new("disk-a");
+        let disk_b = DiskIoAdmission::new("disk-b");
+        let _disk_a_permit = disk_a
+            .background(&CancellationToken::new())
+            .await
+            .expect("disk A background operation should be admitted");
+        let _disk_b_permit = tokio::time::timeout(Duration::from_secs(1), disk_b.background(&CancellationToken::new()))
+            .await
+            .expect("disk A must not block disk B")
+            .expect("disk B background operation should be admitted");
+    }
+
+    #[tokio::test]
+    async fn streamed_read_holds_admission_until_disk_read_resolves() {
+        let admission = DiskIoAdmission::new("disk-a");
+        let (mut writer, reader) = tokio::io::duplex(8);
+        let mut reader = AdmissionReader::new(
+            reader,
+            admission.clone(),
+            DiskIoContext {
+                class: DiskIoClass::Foreground,
+                cancellation: CancellationToken::new(),
+                admission: None,
+                permit: None,
+            },
+        );
+        let read = tokio::spawn(async move {
+            let mut bytes = [0; 4];
+            reader
+                .read_exact(&mut bytes)
+                .await
+                .expect("streamed disk read should complete");
+            bytes
+        });
+
+        wait_for_foreground_active(&admission, 1).await;
+        writer.write_all(b"rust").await.expect("test bytes should be written");
+        assert_eq!(read.await.expect("reader task should finish"), *b"rust");
+        wait_for_foreground_active(&admission, 0).await;
+    }
+
+    #[tokio::test]
+    async fn blocking_worker_keeps_permit_after_caller_timeout() {
+        let admission = DiskIoAdmission::new("disk-a");
+        let (started_tx, started_rx) = tokio::sync::oneshot::channel();
+        let worker = spawn_blocking_with_disk_io_permit(Some(admission.foreground()), move || {
+            started_tx.send(()).expect("worker start should be observed");
+            std::thread::sleep(Duration::from_millis(100));
+        });
+
+        started_rx.await.expect("blocking worker should start");
+        assert!(tokio::time::timeout(Duration::from_millis(10), worker).await.is_err());
+        assert_eq!(admission.foreground_active_for_tests(), 1);
+        wait_for_foreground_active(&admission, 0).await;
+    }
+
+    #[tokio::test]
+    async fn cloned_permit_remains_active_until_all_workers_finish() {
+        let admission = DiskIoAdmission::new("disk-a");
+        let permit = admission.foreground();
+        let worker_permit = permit.clone();
+        drop(permit);
+        assert_eq!(admission.foreground_active_for_tests(), 1);
+
+        drop(worker_permit);
+        assert_eq!(admission.foreground_active_for_tests(), 0);
+    }
+
+    #[tokio::test]
+    async fn scoped_foreground_permit_is_inherited_by_blocking_worker() {
+        let admission = DiskIoAdmission::new("disk-a");
+        let permit = admission.foreground();
+        let worker = with_disk_io_permit(Some(permit), async {
+            let worker_permit = acquire_context_disk_io_permit()
+                .await
+                .expect("context permit should be available")
+                .expect("scoped permit should be inherited");
+            let (started_tx, started_rx) = tokio::sync::oneshot::channel();
+            let worker = spawn_blocking_with_disk_io_permit(Some(worker_permit), move || {
+                started_tx.send(()).expect("worker start should be observed");
+                std::thread::sleep(Duration::from_millis(100));
+            });
+            started_rx.await.expect("blocking worker should start");
+            BlockingWorker(worker)
+        })
+        .await;
+
+        assert_eq!(admission.foreground_active_for_tests(), 1);
+        assert!(tokio::time::timeout(Duration::from_millis(10), worker.0).await.is_err());
+        wait_for_foreground_active(&admission, 0).await;
+    }
+}

crates/ecstore/src/disk/local/commit.rs

@@ -47,6 +47,33 @@ use tokio::fs;
 use tracing::{info, warn};
 use uuid::Uuid;
 
+fn transitioned_old_current_source(
+    dst_meta_existed: bool,
+    dst_meta_unparsable: bool,
+    xlmeta: &FileMeta,
+    volume: &str,
+    path: &str,
+    version_id: Uuid,
+) -> (bool, Option<FileInfo>) {
+    if dst_meta_unparsable {
+        return (false, None);
+    }
+    if !dst_meta_existed {
+        return (true, None);
+    }
+    let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else {
+        return (false, None);
+    };
+    if old_current.version_id.unwrap_or_default() != version_id {
+        return (true, None);
+    }
+    if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE {
+        (true, Some(old_current))
+    } else {
+        (true, None)
+    }
+}
+
 /// Hold later repair publications after admitting one baseline object. The
 /// fixture arms this on one replacement disk before rejoining the cluster.
 #[cfg(feature = "e2e-test-hooks")]
@@ -214,7 +241,7 @@ pub(super) async fn lock_rename_commit_directories(
     publication_root: &os::PublicationRoot,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<os::RenameCommitGuard> {
-    #[cfg(windows)]
+    #[cfg(any(windows, target_os = "linux"))]
     let result = {
         let source_parent = source_parent.to_path_buf();
         let destination_parent = destination_parent.to_path_buf();
@@ -222,7 +249,7 @@ pub(super) async fn lock_rename_commit_directories(
         let publication_root = publication_root.clone();
         os::run_blocking_namespace_operation(mutation_lease, move || {
             let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root);
-            #[cfg(test)]
+            #[cfg(all(test, windows))]
             if result.is_ok() {
                 run_destination_commit_directory_preparation(&destination_parent);
             }
@@ -230,7 +257,7 @@ pub(super) async fn lock_rename_commit_directories(
         })
         .await
     };
-    #[cfg(not(windows))]
+    #[cfg(not(any(windows, target_os = "linux")))]
     let result = {
         let _ = mutation_lease;
         os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root)
@@ -244,12 +271,12 @@ pub(super) async fn lock_rename_commit_directories(
     result.map_err(to_file_error).map_err(DiskError::from)
 }
 
-async fn read_rename_destination_metadata(
+async fn prepare_rename_destination_metadata(
     file_path: &Path,
     rename_commit_guard: &os::RenameCommitGuard,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<Option<Bytes>> {
-    #[cfg(windows)]
+    #[cfg(any(windows, target_os = "linux"))]
     let result = {
         let file_path = file_path.to_path_buf();
         let rename_commit_guard = rename_commit_guard.clone();
@@ -258,10 +285,10 @@ async fn read_rename_destination_metadata(
         })
         .await
     };
-    #[cfg(not(windows))]
+    #[cfg(not(any(windows, target_os = "linux")))]
     let _ = (rename_commit_guard, mutation_lease);
-    #[cfg(not(windows))]
-    let result = match super::super::fs::read_file(file_path).await {
+    #[cfg(not(any(windows, target_os = "linux")))]
+    let result = match fs::read(file_path).await {
         Ok(data) => Ok(Some(data)),
         Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
         Err(err) => Err(err),
@@ -398,7 +425,7 @@ impl LocalDisk {
 
         // xl.meta path
         let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?;
-        let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
+        let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
 
         // data_dir path
         let has_data_dir_path = {
@@ -453,8 +480,7 @@ impl LocalDisk {
             fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?;
         }
         // Acquire the common trees before reading destination metadata. On
-        // Windows this pins the object directory identity across metadata
-        // preparation, data publication, rollback backup, and final commit.
+        // Windows this pins the object directory identity across publication.
         let rename_commit_guard = lock_rename_commit_directories(
             src_file_parent,
             dst_file_parent,
@@ -472,9 +498,12 @@ impl LocalDisk {
             source_volume_dir: &src_volume_dir,
             data_paths: has_data_dir_path.as_ref(),
         };
-        let has_dst_buf =
-            read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
-                .await?;
+        let has_dst_buf = prepare_rename_destination_metadata(
+            commit.destination_metadata,
+            &commit.directory_guard,
+            commit.mutation_lease.clone(),
+        )
+        .await?;
 
         if no_inline {
             // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -508,6 +537,14 @@ impl LocalDisk {
             }
 
             let version_id = fi.version_id.unwrap_or_default();
+            let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
+                has_dst_buf.is_some(),
+                dst_meta_unparsable,
+                &xlmeta,
+                dst_volume,
+                dst_path,
+                version_id,
+            );
             let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
             let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
             let rollback_data_dir = has_old_data_dir.or_else(|| {
@@ -798,7 +835,9 @@ impl LocalDisk {
                 && let Some(parent) = dst_file_path.parent()
             {
                 let fsync_started = rustfs_io_metrics::put_stage_timer();
-                if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
+                if let Err(err) =
+                    os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await
+                {
                     rustfs_io_metrics::record_put_object_stage_duration_from(
                         rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
                         fsync_started,
@@ -897,6 +936,8 @@ impl LocalDisk {
                 cleanup_data_dir: has_old_data_dir,
                 sign: version_signature,
                 old_current_size,
+                old_current_source_checked,
+                old_current_source,
             })
         } else {
             // Inline metadata preparation is blocking. The transaction lease is
@@ -910,6 +951,8 @@ impl LocalDisk {
                 None
             };
             let dst_path_for_failpoint = dst_path.to_string();
+            let dst_volume_for_source = dst_volume.to_string();
+            let dst_path_for_source = dst_path.to_string();
             #[cfg(windows)]
             let source_parent = src_file_parent.to_path_buf();
             let rename_commit_guard_for_preparation = commit.directory_guard.clone();
@@ -954,6 +997,14 @@ impl LocalDisk {
                 };
 
                 let version_id = fi.version_id.unwrap_or_default();
+                let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
+                    has_dst_buf.is_some(),
+                    dst_meta_unparsable,
+                    &xlmeta,
+                    &dst_volume_for_source,
+                    &dst_path_for_source,
+                    version_id,
+                );
                 let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
                 let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
                 let rollback_data_dir = old_data_dir.or_else(|| {
@@ -1003,6 +1054,8 @@ impl LocalDisk {
                     old_data_dir,
                     version_signature,
                     old_current_size,
+                    old_current_source_checked,
+                    old_current_source,
                     staged_rollback_path,
                     has_dst_buf.is_none(),
                     prepared_metadata_source,
@@ -1022,6 +1075,8 @@ impl LocalDisk {
                 cleanup_data_dir,
                 version_signature,
                 old_current_size,
+                old_current_source_checked,
+                old_current_source,
                 mut local_rollback_path,
                 destination_was_absent,
                 prepared_metadata_source,
@@ -1147,10 +1202,11 @@ impl LocalDisk {
                     && let Some(dst_parent) = dst_file_path.parent()
                 {
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
-                    if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
+                    if let Err(err) = os::fsync_commit_directory(
                         dst_parent,
+                        &commit.directory_guard,
                         commit.mutation_lease.clone(),
-                        admission,
+                        Some(admission),
                     )
                     .await
                     {
@@ -1262,6 +1318,8 @@ impl LocalDisk {
                 cleanup_data_dir,
                 sign: version_signature,
                 old_current_size,
+                old_current_source_checked,
+                old_current_source,
             })
         }
     }