feat(capture): add explicit schema-bound checkpoint capabilities

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -45,13 +45,11 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
 - `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
 - `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
 
 ### S3 object actions, copy, multipart, and upload policy validation
 
 - `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
 - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
 - `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
 - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -112,7 +110,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
 - `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
-- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
 
 ### Serde deserialization and input validation
 
@@ -146,13 +143,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
 - Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=419d60fc0a4451248ae91c2dd185dcb261f4bb81a9f26ba7f357a6c74d650892
-sha256-linux=96dffebaebb2c100d71490beabac5faf2a4d9b4ab7cbab08e4ec359ee23b77cd
+sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
+sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22

.config/e2e-nightly-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
-sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926
+sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
+sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=ce16185f87c6cd30aa0891ebd22be3352aedc10040d48cc5b00652d72f7be3b5
+sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10

.config/make/tests.mak

@@ -52,9 +52,6 @@ script-tests: ## Run shell script tests
 	./scripts/check_embedded_secrets.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test

.config/nextest.toml

@@ -88,11 +88,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
-# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
-[[profile.default.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
-setup = 'ecstore-base-stack'
-
 [[profile.default.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
@@ -344,11 +339,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
-# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
-[[profile.ci.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
-setup = 'ecstore-base-stack'
-
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
 setup = 'ecstore-base-stack'
@@ -605,7 +595,6 @@ default-filter = """
         test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
-        | test(/^conditional_delete_objects_test::/)
         | test(/^reliant::tiering::/)
         | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)

.docker/observability/grafana/dashboards/rustfs.json

@@ -1748,7 +1748,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1848,7 +1848,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"

.docker/observability/prometheus.yml

@@ -61,7 +61,6 @@ scrape_configs:
 otlp:
   promote_resource_attributes:
     - service.instance.id
-    - process.pid
     - service.name
     - service.namespace
     - cloud.availability_zone

.docker/observability/tests/bucket-usage.test.yml

@@ -1,48 +0,0 @@
-evaluation_interval: 1m
-tests:
-- name: bucket usage keeps clusters separate and deduplicates reporting nodes
-  interval: 1m
-  input_series:
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
-    values: '4096 1024'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
-    values: '4096 1024'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
-    values: '65536 65536'
-  - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
-    values: '0 0'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
-    values: '3 1'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
-    values: '3 1'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
-    values: '70 70'
-  - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
-    values: '0 0'
-  promql_expr_test:
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1024
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-  - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
-    eval_time: 1m
-    exp_samples:
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
-      value: 1024
-    - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
-      value: 0
-    - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
-      value: 65536
-  - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
-    eval_time: 1m
-    exp_samples: []

.github/actions/setup-docker-compose/action.yml

@@ -1,39 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Set up Docker Compose
-description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
-
-runs:
-  using: composite
-  steps:
-    - name: Install verified Docker Compose CLI plugin
-      shell: bash
-      run: |
-        set -euo pipefail
-        [[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
-        plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
-        mkdir -p "$plugin_dir"
-        compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
-        compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
-        trap 'rm -f "$compose_download" "$compose_staged"' EXIT
-        curl --fail --location --retry 3 \
-          https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
-          --output "$compose_download"
-        printf '%s  %s\n' \
-          dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
-          "$compose_download" | sha256sum --check --strict
-        install -m 0755 "$compose_download" "$compose_staged"
-        mv -f "$compose_staged" "$plugin_dir/docker-compose"
-        [[ $(docker compose version --short) == 2.40.3 ]]

.github/actions/setup/action.yml

@@ -36,16 +36,6 @@ inputs:
     description: "Install cross-compilation tools"
     required: false
     default: "false"
-  zig-version:
-    description: >-
-      Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
-      baseline semantics cannot drift with an untested zig release.
-    required: false
-    default: "0.17.0"
-  cargo-zigbuild-version:
-    description: "cargo-zigbuild version installed for cross builds."
-    required: false
-    default: "0.23.4"
   target:
     description: "Target architecture to add"
     required: false
@@ -109,14 +99,10 @@ runs:
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
       uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
-      with:
-        version: ${{ inputs.zig-version }}
 
     - name: Install cargo-zigbuild
       if: inputs.install-cross-tools == 'true'
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
-      with:
-        tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
 
     - name: Install cargo-nextest
       if: inputs.install-test-tools == 'true'
@@ -140,5 +126,5 @@ runs:
         # normalises, so cargo fingerprints stay valid.
         cache-all-crates: false
         cache-on-failure: true
-        shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
+        shared-key: ${{ inputs.cache-shared-key }}
         save-if: ${{ inputs.cache-save-if }}

.github/workflows/build.yml

@@ -213,12 +213,12 @@ jobs:
           fi
 
           all='{"include":[
-            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
-            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
+            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
+            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
           ]}'
 
           if [[ "${selected}" == "all" ]]; then
@@ -246,8 +246,7 @@ jobs:
           echo "Selected platforms: ${selected}"
 
   build-rustfs:
-    # Acceptance workflows select artifacts by this stable platform name.
-    name: Build RustFS (${{ matrix.target_id }})
+    name: Build RustFS
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
@@ -274,7 +273,7 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
+          cache-shared-key: build-${{ matrix.target }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -411,53 +410,35 @@ jobs:
         shell: bash
         env:
           BUILD_TARGET: ${{ matrix.target }}
-          BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
         run: |
           # Force rebuild by touching build.rs
           touch rustfs/build.rs
 
+          BINARY_NAMES=(rustfs)
+          if [[ "${{ matrix.platform }}" == "linux" ]]; then
+            BINARY_NAMES+=(rustfs-cli)
+          fi
           FEATURE_ARGS=()
           if [[ -n "${{ matrix.features }}" ]]; then
             FEATURE_ARGS=(--features "${{ matrix.features }}")
           fi
 
-          # gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
-          # cargo-zigbuild) so release binaries link no symbols newer than it
-          # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
-          TARGET_ARG="$BUILD_TARGET"
-          if [[ -n "${{ matrix.glibc }}" ]]; then
-            TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
-          fi
-
-          PROFILE_ARGS=()
-          if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
-            # The in-process profiler resolves names from the final executable.
-            # Retain function symbols without enabling DWARF source information.
-            PROFILE_ARGS=(--config 'profile.release.strip="none"')
-          fi
-          if [[ "${{ matrix.cross }}" == "true" ]]; then
-            cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
-          else
-            cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
-          fi
-
-          # All release targets use same-architecture runners, including GNU
-          # builds that use Zig only to select the glibc ABI floor.
-          bin="target/${{ matrix.target }}/release/rustfs"
-          if [[ -f "${bin}.exe" ]]; then
-            bin="${bin}.exe"
-          fi
-          version_output=$("$bin" --version)
-          if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
-             ! grep -Fxq "build os     : ${BUILD_OS}" <<< "$version_output" || \
-             ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
-            echo "release binary reports an unexpected compile target" >&2
-            exit 1
-          fi
-          if [[ -n "${{ matrix.glibc }}" ]]; then
-            python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
-              "target/${{ matrix.target }}/release/rustfs"
-          fi
+          # Build one binary at a time so release LTO links cannot exhaust a
+          # self-hosted runner by running concurrently.
+          for binary in "${BINARY_NAMES[@]}"; do
+            PROFILE_ARGS=()
+            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+              # The in-process profiler resolves names from the final executable.
+              # Retain function symbols without enabling DWARF source information.
+              PROFILE_ARGS=(--config 'profile.release.strip="none"')
+            fi
+            if [[ "${{ matrix.cross }}" == "true" ]]; then
+              # All cross targets in the matrix are Linux; zigbuild handles them.
+              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
+            else
+              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
+            fi
+          done
 
       - name: Verify macOS binary has no dynamic liblzma dependency
         if: matrix.platform == 'macos'
@@ -573,6 +554,9 @@ jobs:
           fi
 
           BINARY_FILES=("$BINARY_NAME")
+          if [[ "${{ matrix.platform }}" == "linux" ]]; then
+            BINARY_FILES+=("rustfs-cli")
+          fi
 
           if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
             BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -687,7 +671,7 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged Linux binary
+      - name: Verify packaged Linux binaries
         if: matrix.platform == 'linux'
         shell: bash
         run: |
@@ -695,7 +679,7 @@ jobs:
 
           package_file="${{ steps.package.outputs.package_file }}"
           archive_members="$(unzip -Z1 "$package_file" | sort)"
-          expected_members='rustfs'
+          expected_members=$'rustfs\nrustfs-cli'
           target="${{ matrix.target }}"
           features="${{ matrix.features }}"
           if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -707,14 +691,13 @@ jobs:
             exit 1
           fi
 
-          # musl targets build natively; the gnu legs are same-arch
-          # cargo-zigbuild builds pinned to the release glibc baseline. Either
-          # way the binaries run on their runner, so executing them here
+          # Every Linux target builds natively, so executing the binaries here
           # proves each artifact matches its runner architecture.
-          binary_dir="$(mktemp -d)"
-          trap 'rm -rf "$binary_dir"' EXIT
-          unzip -q "$package_file" rustfs -d "$binary_dir"
-          "$binary_dir/rustfs" --version
+          cli_dir="$(mktemp -d)"
+          trap 'rm -rf "$cli_dir"' EXIT
+          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
+          "$cli_dir/rustfs" --version
+          "$cli_dir/rustfs-cli" --help >/dev/null
 
       - name: Verify packaged console
         if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'

.github/workflows/cache-warm.yml

@@ -99,7 +99,7 @@ env:
   RUST_BACKTRACE: 1
 
 jobs:
-  # Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
+  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
   # e2e-tests, e2e-full.
   warm-ci-dev:
     name: Warm ci-dev

.github/workflows/ci.yml

@@ -168,10 +168,13 @@ jobs:
             echo "started_at=$(date --utc --iso-8601=seconds)"
           } > artifacts/test-and-lint/run-metadata.txt
 
-      # Keep full Clippy validation on main, merge groups, and scheduled/manual
-      # runs. PRs retain the complete default test suite and compile-free checks.
+      # Clippy runs before the test pass: lint failures are the most common
+      # CI-only breakage and should surface in minutes, not after 20+ minutes
+      # of tests.
+      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
+      # experiment, since --all-targets is check-only for workspace members and
+      # never links the ~100 test binaries the limit exists to throttle.
       - name: Run clippy lints
-        if: github.event_name != 'pull_request'
         run: |
           ./scripts/ci/resource_sampler.sh start clippy
           trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -280,49 +283,6 @@ jobs:
       - name: Verify rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
 
-      # The workspace already compiled these ignored tests. Keep the same Cargo
-      # package selection to reuse their binaries, then run them serially in
-      # separate processes because they share fixed ports and global state.
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract).
-      - name: Run ignored ILM integration tests serially
-        if: needs.classify-changes.outputs.mode == 'full'
-        env:
-          # Match the preceding workspace build and reuse its feature graph.
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
-        run: |
-          mkdir -p artifacts/ilm-integration
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
-            cargo nextest run --profile default -j1 --run-ignored ignored-only \
-            --all --exclude e2e_test \
-            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/ilm-integration/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-          } > artifacts/ilm-integration/diagnostics.txt
-          exit "${status}"
-
-      - name: Upload ILM test diagnostics
-        if: always() && needs.classify-changes.outputs.mode == 'full'
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
-          path: artifacts/ilm-integration
-          retention-days: 3
-
   # This gate builds into a fresh target directory to isolate the E2E root.
   # Give its cold build a separate budget from workspace tests and migration proofs.
   offline-enrollment-e2e:
@@ -354,11 +314,83 @@ jobs:
       - name: Check offline enrollment E2E root boundary
         run: ./scripts/check_offline_enrollment_e2e.sh
 
+  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
+  # drive the object layer through process-global singletons (the GLOBAL_ENV
+  # ECStore, the global tier-config manager, background-expiry workers) and bind
+  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
+  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
+  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
+  # `-j1` so no two of them share global state or race for a port at once.
+  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
+  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
+  # See rustfs/backlog#1148 (ilm-1) and #1155.
+  test-ilm-integration-serial:
+    name: ILM Integration (serial)
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    needs: [ quick-checks, classify-changes ]
+    runs-on: sm-standard-4
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-dev
+          cache-save-if: 'false'
+          install-build-packaging-tools: 'false'
+
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract).
+      - name: Run ignored ILM integration tests serially
+        env:
+          # Match the measured Test and Lint link budget. The default exposed
+          # all 14 pod CPUs and a cold cache spent the full 80m compiling
+          # without starting one ILM test (main run 32982910990).
+          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+        run: |
+          mkdir -p artifacts/ilm-integration
+          set +e
+          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
+            cargo nextest run -j1 --run-ignored ignored-only \
+            -p rustfs-scanner -p rustfs \
+            -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
+            --status-level all --final-status-level all \
+            2>&1 | tee artifacts/ilm-integration/nextest.log
+          status=${PIPESTATUS[0]}
+          {
+            echo "exit_status=${status}"
+            echo "finished_at=$(date --utc --iso-8601=seconds)"
+            echo
+            echo "Remaining test-related processes:"
+            pgrep -af 'cargo|nextest|target/.*/deps/' || true
+          } > artifacts/ilm-integration/diagnostics.txt
+          exit "${status}"
+
+      - name: Upload ILM test diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
+          path: |
+            artifacts/ilm-integration
+            target/nextest/ci/junit.xml
+
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
-    # Alternate feature configurations run on main, merge groups, and scheduled
-    # or manual validation; the default workspace remains the PR gate.
-    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -474,7 +506,7 @@ jobs:
 
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
-    if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
+    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
     needs: [ quick-checks, classify-changes ]
     runs-on: sm-standard-4
     timeout-minutes: 90
@@ -1232,6 +1264,7 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
+      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - connect-short-credential-boundary
       - test-and-lint-protocols
@@ -1265,6 +1298,7 @@ jobs:
       - quick-checks
       - test-and-lint
       - offline-enrollment-e2e
+      - test-ilm-integration-serial
       - test-and-lint-rio-v2
       - test-and-lint-protocols
       - build-rustfs-debug-binary

.github/workflows/connect-diagnostic-schedule-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -102,7 +94,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -145,7 +137,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
+                  assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
                   assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -170,7 +162,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule

.github/workflows/connect-health-service-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -135,7 +127,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()
@@ -207,7 +199,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=health-service

.github/workflows/connect-performance-drive-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive

.github/workflows/connect-performance-network-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=performance-network

.github/workflows/connect-profile-cpu-acceptance.yml

@@ -77,14 +77,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -126,7 +118,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.jobs[] | select(.name == $name)] as $matches
@@ -181,7 +173,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
             CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \

.github/workflows/connect-profile-memory-acceptance.yml

@@ -57,14 +57,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -135,7 +127,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -180,7 +172,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -206,7 +198,7 @@ jobs:
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
           (cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory

.github/workflows/connect-profile-threads-acceptance.yml

@@ -81,14 +81,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -156,7 +148,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -209,7 +201,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads

.github/workflows/connect-top-api-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-api

.github/workflows/connect-top-disk-acceptance.yml

@@ -68,14 +68,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -146,7 +138,7 @@ jobs:
           ' <<<"$run" >/dev/null
           jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e '
-            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
             | ($matches | length) == 1 and $matches[0].conclusion == "success"
           ' <<<"$jobs" >/dev/null
           artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -191,7 +183,7 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
                   assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
@@ -221,7 +213,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-disk

.github/workflows/connect-top-locks-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-locks

.github/workflows/connect-top-rpc-acceptance.yml

@@ -73,14 +73,6 @@ jobs:
           ref: ${{ inputs.connect_sha }}
           token: ${{ secrets.PF_TESTING_GH_TOKEN }}
 
-      - name: Set up Docker Compose
-        uses: ./.github/actions/setup-docker-compose
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.14"
-
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
@@ -115,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -162,7 +154,7 @@ jobs:
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
           make -C connect-harness e2e-connected-dispatch-check
-          TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
             CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc

.github/workflows/e2e-upgrade.yml

@@ -46,6 +46,9 @@ concurrency:
 env:
   CARGO_TERM_COLOR: always
   RUST_BACKTRACE: 1
+  UPGRADE_SOURCE_VERSION: 1.0.0-rc.5
+  UPGRADE_SOURCE_ASSET: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip
+  UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
 
 jobs:
   build:
@@ -68,6 +71,14 @@ jobs:
           install-build-packaging-tools: "false"
           install-test-tools: "false"
 
+      - name: Download pinned previous release
+        run: |
+          mkdir -p target/debug
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
+          curl --fail --location --retry 3 --output "$archive" \
+            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
+
       - name: Build current RustFS binary
         run: python3 scripts/e2e_binary.py build
 
@@ -78,31 +89,49 @@ jobs:
           path: |
             target/debug/rustfs
             target/debug/rustfs.e2e.json
+            target/debug/${{ env.UPGRADE_SOURCE_ASSET }}
           if-no-files-found: error
           retention-days: 3
           compression-level: 0
           overwrite: true
 
   upgrade:
-    name: Upgrade compatibility (${{ matrix.source.version }})
+    name: ${{ matrix.name }}
     needs: build
-    runs-on: sm-standard-2
-    timeout-minutes: 60
     strategy:
       fail-fast: false
       matrix:
-        source:
-          - version: 1.0.0-rc.5
-            asset: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip
-            sha256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
-          - version: 1.0.1
-            asset: rustfs-linux-x86_64-gnu-v1.0.1.zip
-            sha256: 9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5
+        include:
+          # The two `_from_rc2_` tests keep their names: they assert
+          # release-independent object contracts and pass unchanged against the
+          # newer pinned source, so renaming them would only churn history and
+          # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
+          # single source of truth for which release they actually run against.
+          - name: Direct upgrade from the previous release
+            test: direct_upgrade_from_rc2_preserves_object_contracts
+            artifact: direct-upgrade
+          - name: Mixed-version rolling upgrade from the previous release
+            test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
+            artifact: mixed-version-upgrade
+          - name: Bucket configuration survives the upgrade
+            test: direct_upgrade_from_previous_release_preserves_bucket_configuration
+            artifact: bucket-config-upgrade
+          - name: Rollback reads current bucket metadata
+            test: rollback_to_previous_release_reads_current_bucket_metadata
+            artifact: bucket-config-rollback
+          - name: ODM configuration recovery after rc.5 rollback
+            test: rc5_rollback_requires_restoring_odm_configuration
+            artifact: odm-config-rollback
+          - name: Multipart layouts survive the rc.5 upgrade
+            test: direct_upgrade_from_rc5_preserves_multipart_layouts
+            artifact: multipart-layout-upgrade
+          - name: rc.5 multipart replication baseline
+            test: rc5_baseline_replicates_multipart_layouts
+            artifact: multipart-layout-baseline
+    runs-on: sm-standard-2
+    timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      UPGRADE_SOURCE_VERSION: ${{ matrix.source.version }}
-      UPGRADE_SOURCE_ASSET: ${{ matrix.source.asset }}
-      UPGRADE_SOURCE_SHA256: ${{ matrix.source.sha256 }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -113,7 +142,8 @@ jobs:
         uses: ./.github/actions/setup
         with:
           cache-shared-key: e2e-upgrade-tests
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          # One main-branch consumer warms the shared test-harness dependencies.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
           install-build-packaging-tools: "false"
           install-test-tools: "false"
 
@@ -133,8 +163,6 @@ jobs:
           set -euo pipefail
           mkdir -p "$SOURCE_DIR"
           archive="target/debug/$UPGRADE_SOURCE_ASSET"
-          curl --fail --location --retry 3 --output "$archive" \
-            "https://github.com/rustfs/rustfs/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
           echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
           unzip -q "$archive" -d "$SOURCE_DIR"
           chmod +x "$SOURCE_DIR/rustfs"
@@ -142,53 +170,22 @@ jobs:
           echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
           echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
 
-      # Reuse one compiled harness, while keeping each case in its own process.
-      # Continue after a failure to retain the former matrix's complete verdict.
-      - name: Run upgrade compatibility tests
-        shell: bash
+      - name: Run upgrade compatibility test
         env:
+          RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           NO_PROXY: 127.0.0.1,localhost,::1
         run: |
           export no_proxy="$NO_PROXY"
-          cases=(
-            direct-upgrade:direct_upgrade_from_previous_release_preserves_object_contracts
-            mixed-version-upgrade:rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
-            bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
-            bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
-            multipart-layout-upgrade:direct_upgrade_from_previous_release_preserves_multipart_layouts
-            multipart-layout-baseline:previous_release_baseline_replicates_multipart_layouts
-          )
-          if [[ "$UPGRADE_SOURCE_VERSION" == "1.0.0-rc.5" ]]; then
-            cases+=(odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration)
-          else
-            cases+=(legacy-delete-reservation:stable_upgrade_deletes_foreign_reservations_without_losing_live_objects)
-          fi
-          status=0
-          for entry in "${cases[@]}"; do
-            artifact="${entry%%:*}"
-            test="${entry#*:}"
-            export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
-            export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
-            mkdir -p "$RUSTFS_E2E_LOG_DIR"
-            echo "::group::$test"
-            if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
-              "upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
-              2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
-              echo "PASS: $test"
-            else
-              echo "::error::Upgrade compatibility failed: $test"
-              status=1
-            fi
-            echo "::endgroup::"
-          done
-          exit "$status"
+          python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
+            "upgrade_compatibility_test::${{ matrix.test }}" \
+            -- --ignored --exact --nocapture
 
       - name: Upload scanner/heal G09 evidence
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: upgrade-g09-evidence-${{ matrix.source.version }}-${{ github.run_number }}
-          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
+          name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
           if-no-files-found: ignore
           retention-days: 14
 
@@ -196,7 +193,7 @@ jobs:
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: upgrade-server-logs-${{ matrix.source.version }}-${{ github.run_number }}
+          name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
           path: ${{ runner.temp }}/rustfs-upgrade-logs
           if-no-files-found: warn
           retention-days: 14

.github/workflows/gateway-integration-sync.yml

@@ -6,20 +6,29 @@ name: Gateway Integration Sync
 # Merges `main` into the long-lived `gateway/integration` branch every weekday
 # so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736)
 # never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit
-# pushed straight to the branch. A conflicting merge is aborted and fails the
-# run after its summary records the source and target SHAs, run URL, and exact
-# conflicting paths. It creates no sync branch or pull request. Resolve the
-# merge in an isolated worktree based on the latest target, preserve the
-# two-parent merge commit, and push only after the required checks pass and the
-# target still matches the merge's first parent. If the target moves, restart
-# from its new head.
+# pushed straight to the branch. A conflicting merge is aborted and surfaced as
+# a pull request from `sync/<source>-<date>`, whose head is the un-merged
+# source commit, so the conflict is resolved the same day, in review, with no
+# other change riding along. The run itself stays green in that case: the pull
+# request is the signal. Any other failure (push rejected, pull request not
+# created) fails the run and, on the schedule, files the shared failure issue.
 #
-# Write scope: only `gateway/integration` is ever pushed. The push uses a
-# one-hour installation token of the organisation's GitHub App (contents:
-# write) because that app is the only non-admin bypass actor of the
+# Resolving a conflict pull request: merge `origin/gateway/integration` into
+# the sync branch, resolve, push, get the review. Land it with a fast-forward
+# push of the resolved merge commit to `gateway/integration`, not with the
+# squash button: a squash drops the merge ancestry and the same hunks conflict
+# again on the next run.
+#
+# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and
+# both push targets below are literal refs. The pushes use a one-hour
+# installation token of the organisation's own GitHub App (contents: write)
+# because that app is the only non-admin bypass actor of the
 # `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor,
-# so it cannot push through the required reviews and checks. GITHUB_TOKEN is
-# read-only here.
+# so it cannot push through the required reviews and checks. The conflict pull
+# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the
+# weekly flake.lock pull requests, because this repository keeps "Allow GitHub
+# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is
+# therefore read-only here.
 #
 # `schedule` fires only from the default branch, so the cron is live once this
 # file exists on `main`. `workflow_dispatch` runs from any ref that carries the
@@ -85,7 +94,6 @@ jobs:
         id: merge
         env:
           SOURCE_REF: ${{ inputs.source_ref || 'main' }}
-          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
         run: |
           set -euo pipefail
           if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \
@@ -111,25 +119,13 @@ jobs:
             fi
             exit 0
           fi
-          conflicts=$(git diff --name-only --diff-filter=U)
-          if [[ -z "$conflicts" ]]; then
-            git merge --abort || true
+          git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt"
+          git merge --abort
+          if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then
             echo "::error::git merge failed without reporting conflicting paths; this is not a conflict"
             exit 1
           fi
-          {
-            echo "## Gateway integration sync stopped on conflicts"
-            echo
-            echo "- Run: ${RUN_URL}"
-            echo "- Source: \`${SOURCE_REF}\` at ${source_sha}"
-            echo "- Target: \`${TARGET_BRANCH}\` at ${target_sha}"
-            echo
-            echo "Conflicting files:"
-            printf '%s\n' "$conflicts" | sed 's/^/- /'
-          } >> "$GITHUB_STEP_SUMMARY"
-          git merge --abort
-          echo "::error::${SOURCE_REF} cannot be merged cleanly into ${TARGET_BRANCH}; no branch or pull request was created"
-          exit 1
+          echo "result=conflict" >> "$GITHUB_OUTPUT"
 
       - name: Push the merge commit
         if: steps.merge.outputs.result == 'merged'
@@ -141,12 +137,50 @@ jobs:
           git push origin HEAD:refs/heads/gateway/integration
           echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY"
 
+      - name: Open the conflict pull request
+        if: steps.merge.outputs.result == 'conflict'
+        env:
+          GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }}
+          SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
+          SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
+          TARGET_SHA: ${{ steps.merge.outputs.target_sha }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          date_utc=$(date -u +%Y-%m-%d)
+          sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}"
+          git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}"
+          {
+            echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged."
+            echo
+            echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution."
+            echo
+            echo "- Run: ${RUN_URL}"
+            echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}"
+            echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}"
+            echo
+            echo "## Conflicting files"
+            echo
+            echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:"
+            echo
+            sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt"
+          } > "$RUNNER_TEMP/pr-body.md"
+          title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)"
+          existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty')
+          if [[ -n "$existing" ]]; then
+            gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md"
+            pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url)
+          else
+            pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md")
+          fi
+          echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY"
+
   alert-on-failure:
     name: Alert on scheduled failure
     needs: [sync]
     # Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that
-    # could not push or encountered merge conflicts must not stay visible only
-    # in the Actions tab.
+    # could not push or could not open the conflict pull request must not stay
+    # visible only in the Actions tab.
     if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
     runs-on: ubuntu-latest
     timeout-minutes: 10

.github/workflows/nightly-gnu.yml

@@ -80,24 +80,13 @@ jobs:
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
+          cache-shared-key: build-x86_64-unknown-linux-gnu
           cache-save-if: 'false'
-          install-cross-tools: 'true'
-          target: x86_64-unknown-linux-gnu
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
       - name: Build RustFS
-        # Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
-        # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
-        run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
-
-      - name: Verify glibc symbol baseline
-        run: |
-          python3 scripts/release/check_glibc.py 2.31 \
-            target/x86_64-unknown-linux-gnu/release/rustfs \
-            target/x86_64-unknown-linux-gnu/release/rustfs-cli
-          target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
+        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
 
       - name: Build DEB package
         id: deb

.github/workflows/oidc-keycloak.yml

@@ -20,9 +20,8 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc/**"
+      - "crates/iam/src/oidc.rs"
       - "rustfs/src/admin/handlers/oidc.rs"
-      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -32,9 +31,8 @@ on:
       - ".github/workflows/oidc-keycloak.yml"
       - "crates/config/src/constants/oidc.rs"
       - "crates/iam/src/federation/**"
-      - "crates/iam/src/oidc/**"
+      - "crates/iam/src/oidc.rs"
       - "rustfs/src/admin/handlers/oidc.rs"
-      - "rustfs/src/admin/service/oidc_config.rs"
       - "rustfs/src/admin/handlers/sts.rs"
       - "scripts/test/oidc_keycloak_live.sh"
       - "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"

.github/workflows/windows-filesystem.yml

@@ -21,7 +21,6 @@ on:
       - "crates/ecstore/src/disk/**"
       - "crates/ecstore/src/store/init_format.rs"
       - "crates/ecstore/Cargo.toml"
-      - "crates/utils/src/dunce.rs"
       - "Cargo.toml"
       - "Cargo.lock"
       - ".github/actions/setup/**"
@@ -32,7 +31,6 @@ on:
       - "crates/ecstore/src/disk/**"
       - "crates/ecstore/src/store/init_format.rs"
       - "crates/ecstore/Cargo.toml"
-      - "crates/utils/src/dunce.rs"
       - "Cargo.toml"
       - "Cargo.lock"
       - ".github/actions/setup/**"
@@ -79,10 +77,6 @@ jobs:
         shell: pwsh
         run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
 
-      - name: Test extended-length path conversion
-        shell: pwsh
-        run: cargo test -p rustfs-utils --no-default-features --features path --lib extended_length_path -- --nocapture
-
       - name: Test Windows handle guards
         shell: pwsh
         run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture

.gitignore

@@ -90,4 +90,3 @@ worktrees/*
 # insta scratch files; the accepted .snap files ARE the assertions and are committed
 *.snap.new
 *.pending-snap
-.mimosa

.vscode/launch.json

@@ -7,9 +7,7 @@
             "name": "Debug(only) executable 'rustfs'",
             "env": {
               "RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
-              "RUSTFS_SKIP_BACKGROUND_TASK": "on",
-              // Local test volumes share one physical disk.
-              "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
+              "RUSTFS_SKIP_BACKGROUND_TASK": "on"
               //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
               // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
               // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -46,7 +44,6 @@
             "env": {
                 "RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
                 "RUSTFS_SKIP_BACKGROUND_TASK": "on",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 //"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
                 // "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
                 // "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token" 
@@ -120,8 +117,7 @@
             "env": {
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
-                "RUSTFS_VOLUMES": "./target/volume/test{1...4}",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                //"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
                 // "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -139,31 +135,42 @@
         {
             "type": "lldb",
             "request": "launch",
-            "name": "Debug test_transition_and_restore_flows",
+            "name": "Debug test_lifecycle_transition_basic",
             "cargo": {
                 "args": [
                     "test",
-                    "--no-run",
                     "-p",
                     "rustfs-scanner",
                     "--test",
                     "lifecycle_integration_test",
+                    "serial_tests::test_lifecycle_transition_basic",
                     "-j",
                     "1"
-                ],
-                "filter": {
-                    "name": "lifecycle_integration_test",
-                    "kind": "test"
-                }
+                ]
             },
-            "args": [
-                "serial_tests::test_transition_and_restore_flows",
-                "--exact",
-                "--ignored",
-                "--test-threads=1"
-            ],
+            "args": [],
             "cwd": "${workspaceFolder}"
         },
+        {
+            "name": "Debug executable target/debug/test",
+            "type":  "lldb",
+            "request": "launch",
+            "program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
+            "args": [
+              "--skip",
+              "test_lifecycle_expiry_basic",
+              "--skip",
+              "test_lifecycle_expiry_deletemarker",
+              //"--skip",
+              //"test_lifecycle_transition_basic",
+            ],
+            "cwd": "${workspaceFolder}",
+            //"stopAtEntry": false,
+            //"preLaunchTask": "cargo build",
+            "sourceLanguages": [
+                "rust"
+            ],
+        },
         {
             "name": "Debug executable target/debug/rustfs with sse kms",
             "type":  "lldb",
@@ -278,7 +285,7 @@
                 "RUSTFS_COMPRESSION_ENABLED": "true",
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },
@@ -300,7 +307,6 @@
                 "RUSTFS_ACCESS_KEY": "rustfsadmin",
                 "RUSTFS_SECRET_KEY": "rustfsadmin",
                 "RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
                 "RUSTFS_REGION": "us-east-1",
                 "RUSTFS_ADDRESS": ":9000",
                 "RUSTFS_CONSOLE_ENABLE": "true",
@@ -309,7 +315,7 @@
 
                 "RUSTFS_KMS_ENABLE": "true",
                 "RUSTFS_KMS_BACKEND": "local",
-                "RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
+                "RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
                 "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
             },

CHANGELOG.md

@@ -25,7 +25,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
 
 ### Fixed
-- **Replication status writebacks no longer invalidate scanner cycles**: every replication status writeback (`persist_replication_state_if_current` in `Update` mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path, advanced the dirty-usage generation and woke the scanner, so a site-replication resync or any steadily replicated bucket ended every cycle `superseded`, the authoritative usage baseline was never certified, and the console reported usage freshness as stale for as long as replication ran (#8375). Those writebacks are now status class: they do not touch the dirty-usage generation, the activity snapshot, the dirty-usage wake or the durable dirty journal, and only the process-local producer coverage is recorded. They need no rescan queue, because a deployment with an active replication rule already walks every bucket on every cycle and that walk refreshes `replication_stats`. Tier-expiration cleanup, which removes data, stays on the namespace path. No persisted format, protocol, admin API or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`.
 - **SSE-C TLS requirement now covers copy-source keys**: with `RUSTFS_SSE_C_REQUIRE_TLS=true`, a `CopyObject` or `UploadPartCopy` that carried only the `x-amz-copy-source-server-side-encryption-customer-*` headers (an SSE-C source copied to a destination that is not SSE-C) was accepted on a plaintext transport and was not counted in `rustfs_ssec_plaintext_requests_total`, although those headers carry the source object's customer key. They now count as SSE-C headers: such requests are counted, and refused with `400 InvalidRequest` when the switch is on. Requests over TLS are unaffected.
 - **SSE-KMS encryption context could replace the location entry**: data keys are wrapped under an encryption-context entry, keyed by the bucket name, that ties them to the object's location. A client `x-amz-server-side-encryption-context` entry under the same key replaced it, and because the client context is stored with the object, the replacement also applied on every read. SSE-KMS writes now refuse such a context with `400 InvalidArgument`; other keys, including the names of other buckets, are unaffected. Objects stored with such an entry by an earlier release remain readable. See `docs/operations/kms-backend-security.md`.
 - **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
@@ -86,7 +85,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
 
 ### Changed
-- **Scanner bucket walkers are no longer tied to the disk count**: a new scanner key `walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through `16`, default `1`) spawns that many bucket walkers per local disk of an erasure set, so a single-disk deployment can walk several buckets at once instead of one bucket, one object at a time. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk, and remote disks keep one walker. The walker count is what `max_concurrent_disk_scans` caps and the disk count stays the floor; the `disk_concurrency_budget` log line now reports `walkers_per_disk`, `walkers`, and the `reason` for the effective limit. The default keeps today's I/O profile because a local disk cannot yet report whether it is rotational; the design intends a higher default on flash once it can (`docs/architecture/scanner-per-bucket-convergence.md`). As groundwork for pacing, the raw-enumeration page writer inserts into its building page in place and commits pages without copying them, and a regression test pins the cost of recording 20,000 entries in one prefix so the quadratic bookkeeping removed by [#8114](https://github.com/rustfs/rustfs/pull/8114) (rustfs#8113) cannot return. No persisted format, protocol or admin API changes beyond the additional config key, which older releases preserve and ignore.
 - **Encryption and KMS work merged since `1.0.0-rc.5`** (entries were missing from this section):
   - **Persisted KMS configuration secrets** are sealed field-by-field with `RUSTFS_KMS_CONFIG_SECRET`. **When the variable is unset the secrets are persisted in cleartext and the server only warns** (`persisted KMS configuration carries cleartext secrets`); it never refuses the write. Set it, identically, on every node, and re-save the configuration to seal an existing one.
   - **New v2 ciphertext frame format** with per-frame index binding and final-frame authentication. Its write switch `RUSTFS_ENCRYPTION_FRAME_V2` is **off by default**: v2 frames are unreadable by nodes without v2 read support, and encrypted ciphertext travels verbatim through transition, decommission and SSE-C replication passthrough, so turn it on only after every node — and every RustFS warm/replication target that receives raw ciphertext — runs a release with v2 read support. Reading v2 objects needs no switch.

Cargo.toml

@@ -18,6 +18,7 @@ members = [
     "crates/audit", # Audit target management system with multi-target fan-out
     "crates/checksums", # client checksums
     "crates/common", # Shared utilities and data structures
+    "crates/capture-authority", # Durable capture authority storage
     "crates/data-usage", # Shared data usage models and algorithms
     "crates/config", # Configuration management
     "crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
@@ -74,7 +75,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.99.0"
-version = "1.0.2"
+version = "1.0.1"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -91,56 +92,56 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.2" }
-rustfs-heal = { path = "crates/heal", version = "1.0.2" }
-rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
-rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
-rustfs-audit = { path = "crates/audit", version = "1.0.2" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
-rustfs-common = { path = "crates/common", version = "1.0.2" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
-rustfs-config = { path = "./crates/config", version = "1.0.2" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
-rustfs-iam = { path = "crates/iam", version = "1.0.2" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
-rustfs-license = { path = "crates/license", version = "1.0.2" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
-rustfs-kms = { path = "crates/kms", version = "1.0.2" }
-rustfs-lock = { path = "crates/lock", version = "1.0.2" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
-rustfs-notify = { path = "crates/notify", version = "1.0.2" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
-rustfs-obs = { path = "crates/obs", version = "1.0.2" }
-rustfs-policy = { path = "crates/policy", version = "1.0.2" }
-rustfs-protos = { path = "crates/protos", version = "1.0.2" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
-rustfs-replication = { path = "crates/replication", version = "1.0.2" }
-rustfs-rio = { path = "crates/rio", version = "1.0.2" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
-rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
-rustfs-signer = { path = "crates/signer", version = "1.0.2" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
-rustfs-targets = { path = "crates/targets", version = "1.0.2" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
-rustfs-utils = { path = "crates/utils", version = "1.0.2" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
+rustfs = { path = "./rustfs", version = "1.0.1" }
+rustfs-heal = { path = "crates/heal", version = "1.0.1" }
+rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" }
+rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" }
+rustfs-audit = { path = "crates/audit", version = "1.0.1" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.1" }
+rustfs-common = { path = "crates/common", version = "1.0.1" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" }
+rustfs-config = { path = "./crates/config", version = "1.0.1" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.1" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.1" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" }
+rustfs-iam = { path = "crates/iam", version = "1.0.1" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.1" }
+rustfs-license = { path = "crates/license", version = "1.0.1" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" }
+rustfs-kms = { path = "crates/kms", version = "1.0.1" }
+rustfs-lock = { path = "crates/lock", version = "1.0.1" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.1" }
+rustfs-notify = { path = "crates/notify", version = "1.0.1" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.1" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" }
+rustfs-obs = { path = "crates/obs", version = "1.0.1" }
+rustfs-policy = { path = "crates/policy", version = "1.0.1" }
+rustfs-protos = { path = "crates/protos", version = "1.0.1" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.1" }
+rustfs-replication = { path = "crates/replication", version = "1.0.1" }
+rustfs-rio = { path = "crates/rio", version = "1.0.1" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" }
+rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.1" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" }
+rustfs-signer = { path = "crates/signer", version = "1.0.1" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" }
+rustfs-targets = { path = "crates/targets", version = "1.0.1" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" }
+rustfs-utils = { path = "crates/utils", version = "1.0.1" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -155,10 +156,9 @@ futures-core = "0.3.34"
 futures-lite = "2.6.1"
 futures-util = "0.3.34"
 pollster = "1.0.1"
-pulsar = { default-features = false, version = "=6.9.0" }
-lapin = { default-features = false, version = "4.12.2" }
+pulsar = { default-features = false, version = "6.9.0" }
+lapin = { default-features = false, version = "4.12.0" }
 hyper = { version = "1.12.0" }
-h2 = { version = "0.4.20" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
 hyper-util = { version = "0.1.21" }
 http = "1.5.0"
@@ -169,11 +169,11 @@ minlz = "1.2.3"
 reqwest = "0.13.5"
 rustfs-kafka-async = { version = "1.4.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.53.3" }
+tokio = { version = "1.53.2" }
 tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
-tokio-util = { version = "0.7.20" }
+tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
@@ -181,7 +181,7 @@ tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
 h3 = "0.0.8"
 h3-quinn = "0.0.10"
-quinn = { version = "0.11.12", default-features = false, features = [
+quinn = { version = "0.11.11", default-features = false, features = [
       "futures-io",
       "runtime-tokio",
       "rustls-aws-lc-rs",
@@ -201,7 +201,7 @@ rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
 serde_ignored = { version = "0.1" }
 serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
-serde_json = { version = "1.0.154" }
+serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -220,12 +220,12 @@ openidconnect = { default-features = false, version = "4.0" }
 pbkdf2 = "0.13.0"
 p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] }
 rsa = { version = "=0.10.0-rc.18" }
-rustls = { default-features = false, version = "0.23.46" }
+rustls = { default-features = false, version = "0.23.45" }
 rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
 x509-parser = "0.18.1"
 sha1 = "0.11.0"
-sha2 = "0.11.1"
+sha2 = "0.11.0"
 subtle = "2.6"
 zeroize = { version = "1.9.1" }
 proptest = "1"
@@ -246,8 +246,8 @@ anyhow = "1.0.104"
 arc-swap = "1.9.2"
 rustfs-tokio-tar = { version = "0.7.1" }
 # Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
-tar-codec = "0.0.15"
-tar-framing = "0.0.15"
+tar-codec = "0.0.14"
+tar-framing = "0.0.14"
 atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
@@ -269,7 +269,7 @@ crossbeam-queue = "0.3.14"
 crossbeam-channel = "0.5.17"
 crossbeam-deque = "0.8.8"
 crossbeam-utils = "0.8.23"
-datafusion = { default-features = false, version = "55.2.0" }
+datafusion = { default-features = false, version = "55.1.0" }
 derive_builder = "0.20.2"
 ed25519-dalek = "3.0.0"
 enumset = "1.1.14"
@@ -315,19 +315,21 @@ ratelimit = "2.0.0"
 rayon = "1.12.0"
 rustfs-erasure-codec = { version = "9.0.0" }
 reed-solomon-simd = "3.1.0"
+openraft = { version = "=0.9.25", default-features = false, features = ["serde"] }
+redb = "=4.3.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
 redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
-rust-embed = { version = "8.13.0" }
+rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
-smallvec = { version = "1.16.3" }
+smallvec = { version = "1.16.2" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
@@ -341,12 +343,12 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.34.0" }
+tracing-opentelemetry = { version = "0.34" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.28.0" }
+uuid = { version = "1.27.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
@@ -354,26 +356,26 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "9.0.3"
-zstd = "0.14.2"
+zip = "8.6.0"
+zstd = "0.14.0"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.4", default-features = false }
-opentelemetry = { version = "0.33.1" }
-opentelemetry-appender-tracing = { version = "0.33.1" }
-opentelemetry-otlp = { version = "0.33.1" }
-opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.33.1" }
-opentelemetry-semantic-conventions = { version = "0.33.1" }
-opentelemetry-stdout = { version = "0.33.1" }
-pyroscope = { version = "2.1.2" }
+dial9 = { version = "0.5.3", default-features = false }
+opentelemetry = { version = "0.33.0" }
+opentelemetry-appender-tracing = { version = "0.33.0" }
+opentelemetry-otlp = { version = "0.33.0" }
+opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.0" }
+opentelemetry-semantic-conventions = { version = "0.33.0" }
+opentelemetry-stdout = { version = "0.33.0" }
+pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.2.1" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.64.1" }
 russh-sftp = "3.0.1"
@@ -384,7 +386,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.28.6", default-features = false }
+hotpath = { version = "0.28.5", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.49" }
 

README.md

@@ -141,7 +141,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.2
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -220,7 +220,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
 ./docker-buildx.sh --push
 
 # Build specific version
-./docker-buildx.sh --release 1.0.2 --push
+./docker-buildx.sh --release 1.0.1 --push
 
 # Build for custom registry
 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -238,7 +238,7 @@ You can also use Make targets for convenience:
 ```bash
 make docker-buildx                    # Build locally
 make docker-buildx-push               # Build and push
-make docker-buildx-version VERSION=1.0.2  # Build specific version
+make docker-buildx-version VERSION=1.0.1  # Build specific version
 make help-docker                      # Show all Docker-related commands
 ```
 

README_ZH.md

@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.2
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
 ./docker-buildx.sh --push
 
 # 构建指定版本
-./docker-buildx.sh --release 1.0.2 --push
+./docker-buildx.sh --release 1.0.1 --push
 
 # 构建并推送到自定义仓库
 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
 ```bash
 make docker-buildx                    # 本地构建
 make docker-buildx-push               # 构建并推送
-make docker-buildx-version VERSION=1.0.2  # 构建指定版本
+make docker-buildx-version VERSION=1.0.1  # 构建指定版本
 make help-docker                      # 显示所有 Docker 相关命令
 ```
 

_typos.toml

@@ -51,8 +51,6 @@ unparseable = "unparseable"
 # Disaster-recovery objectives: recovery time and recovery point.
 RTO = "RTO"
 rto = "rto"
-# rustix OFlags variant (POSIX O_WRONLY equivalent).
-WRONLY = "WRONLY"
 
 [files]
 extend-exclude = []

crates/capture-authority/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "rustfs-capture-authority"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+rust-version.workspace = true
+repository.workspace = true
+description = "Durable local storage for capture authority"
+
+[dependencies]
+openraft = { workspace = true }
+redb = { workspace = true }
+sha2 = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+thiserror = { workspace = true }
+tokio = { workspace = true }
+url = { workspace = true }
+
+[dev-dependencies]
+serde_json = { workspace = true }
+tempfile = { workspace = true }
+
+[lints]
+workspace = true

crates/capture-authority/src/application_tests.rs

@@ -0,0 +1,531 @@
+use super::*;
+use crate::application::{APPLY_RESULTS, EVENTS, OPERATIONS, VERSIONS};
+use crate::tests::CrashBackend;
+use redb::ReadableTableMetadata;
+use std::{
+    io::{BufRead, BufReader, Write},
+    process::{Command, Stdio},
+    sync::mpsc,
+    time::Duration,
+};
+fn identity() -> NodeIdentity {
+    NodeIdentity {
+        cluster_id: [7; 16],
+        node_id: 0,
+    }
+}
+fn binding() -> CaptureBinding {
+    CaptureBinding {
+        revision: 9,
+        binding_id: [8; 16],
+        target_id: [9; 16],
+    }
+}
+fn command() -> DecideCreated {
+    DecideCreated {
+        operation_id: [1; 16],
+        object: ObjectIdentity {
+            bucket_incarnation: [2; 16],
+            key: vec![0, 255, 3],
+        },
+        expected_head: None,
+        prepared: PreparedIdentity {
+            version_id: [3; 16],
+            preparation_id: [4; 16],
+            content_digest: [5; 32],
+            content_length: 123,
+        },
+        binding_revision: 9,
+    }
+}
+fn id(index: u64) -> LogId {
+    LogId {
+        term: 8,
+        leader_node: 0,
+        index,
+    }
+}
+fn append(store: &mut Store, index: u64, c: &DecideCreated) {
+    store
+        .append(&[LogEntry {
+            id: id(index),
+            payload: c.encode().expect("encode command"),
+        }])
+        .expect("append command");
+}
+fn apply(store: &mut Store, index: u64, c: &DecideCreated) -> CreatedResult {
+    append(store, index, c);
+    store.save_committed(id(index)).expect("commit log");
+    store.apply_next_created(id(index)).expect("apply command")
+}
+fn success(c: &DecideCreated, index: u64) -> CreatedResult {
+    CreatedResult::Created {
+        version_id: c.prepared.version_id,
+        event_id: c.operation_id,
+        decision_log_id: id(index),
+    }
+}
+fn complete(store: &Store) {
+    let c = command();
+    assert_eq!(store.capture_binding().expect("binding"), Some(binding()));
+    assert_eq!(
+        store.created_head(&c.object).expect("head"),
+        Some(c.prepared.version_id),
+        "U11 acknowledged head lost"
+    );
+    assert_eq!(
+        store.created_version(&c.object, c.prepared.version_id).expect("version"),
+        Some(CreatedVersion {
+            prepared: c.prepared.clone(),
+            decision_log_id: id(0),
+            binding_revision: 9
+        })
+    );
+    assert_eq!(
+        store.created_event(c.operation_id).expect("event"),
+        Some(CreatedEvent {
+            object: c.object.clone(),
+            prepared: c.prepared.clone(),
+            decision_log_id: id(0),
+            binding: binding()
+        }),
+        "U11 complete event missing"
+    );
+    assert_eq!(
+        store.created_operation(c.operation_id).expect("operation"),
+        Some(CreatedOperation {
+            canonical_command: c.encode().expect("encode"),
+            result: success(&c, 0)
+        })
+    );
+    assert_eq!(
+        store.created_apply_result(0).expect("result"),
+        Some(CreatedApplyResult {
+            log_id: id(0),
+            result: success(&c, 0)
+        })
+    );
+    assert_eq!(store.last_applied().expect("last applied"), Some(id(0)));
+}
+fn fixture() -> (tempfile::TempDir, Store) {
+    let dir = tempfile::tempdir().expect("tempdir");
+    let mut store = Store::initialize(dir.path().join("db"), identity()).expect("initialize");
+    store.pin_capture_binding(binding()).expect("pin");
+    (dir, store)
+}
+fn reopen(path: &std::path::Path) -> Store {
+    let result = Store::open(path, identity());
+    assert!(result.is_ok(), "U11 complete reopen: {:?}", result.as_ref().err());
+    result.expect("asserted successful reopen")
+}
+#[test]
+fn created_codec_is_canonical_and_strict() {
+    let c = command();
+    let b = c.encode().expect("encode");
+    let mut golden = b"CRTD\x01".to_vec();
+    golden.extend([1; 16]);
+    golden.extend([2; 16]);
+    golden.extend(3u32.to_be_bytes());
+    golden.extend([0, 255, 3]);
+    golden.push(0);
+    golden.extend([3; 16]);
+    golden.extend([4; 16]);
+    golden.extend([5; 32]);
+    golden.extend(123u64.to_be_bytes());
+    golden.extend(9u64.to_be_bytes());
+    assert_eq!(b, golden);
+    assert_eq!(DecideCreated::decode(&b).expect("decode"), c);
+    let mut variants = Vec::new();
+    let mut v = c.clone();
+    v.operation_id = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.object.bucket_incarnation = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.object.key.push(4);
+    variants.push(v);
+    let mut v = c.clone();
+    v.expected_head = Some([6; 16]);
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.version_id = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.preparation_id = [6; 16];
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.content_digest = [6; 32];
+    variants.push(v);
+    let mut v = c.clone();
+    v.prepared.content_length += 1;
+    variants.push(v);
+    let mut v = c.clone();
+    v.binding_revision += 1;
+    variants.push(v);
+    for v in variants {
+        assert_ne!(v.encode().expect("variant"), b)
+    }
+    for n in 0..b.len() {
+        assert!(DecideCreated::decode(&b[..n]).is_err())
+    }
+    let mut bad = b.clone();
+    bad[4] = 2;
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = b.clone();
+    bad[44] = 2;
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = b.clone();
+    bad.push(0);
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = b;
+    bad[37..41].copy_from_slice(&4097u32.to_be_bytes());
+    assert!(DecideCreated::decode(&bad).is_err());
+    let mut bad = c.clone();
+    bad.operation_id = [0; 16];
+    assert!(bad.encode().is_err());
+    bad = c;
+    bad.object.key = vec![0; MAX_OBJECT_KEY_LENGTH + 1];
+    assert!(bad.encode().is_err());
+}
+#[test]
+fn created_committed_order_and_identity_are_enforced() {
+    let (_dir, mut s) = fixture();
+    let c = command();
+    append(&mut s, 0, &c);
+    append(&mut s, 1, &c);
+    assert!(s.apply_next_created(id(0)).is_err());
+    assert_eq!(s.last_applied().expect("no apply"), None);
+    s.save_committed(id(1)).expect("committed");
+    assert!(matches!(s.apply_next_created(id(1)), Err(StoreError::InvalidApplySequence)));
+    let mut wrong = id(0);
+    wrong.term += 1;
+    assert!(s.apply_next_created(wrong).is_err());
+    wrong = id(0);
+    wrong.leader_node += 1;
+    assert!(s.apply_next_created(wrong).is_err());
+    assert_eq!(s.apply_next_created(id(0)).expect("first"), success(&c, 0));
+    complete(&s);
+    assert_eq!(s.apply_next_created(id(0)).expect("retry"), success(&c, 0));
+    complete(&s);
+    assert!(s.apply_next_created(wrong).is_err());
+    assert_eq!(s.apply_next_created(id(1)).expect("duplicate log"), success(&c, 0));
+    assert_eq!(s.last_applied().expect("last"), Some(id(1)));
+    assert_eq!(
+        s.created_apply_result(1).expect("result"),
+        Some(CreatedApplyResult {
+            log_id: id(1),
+            result: success(&c, 0)
+        })
+    );
+    let tx = s.database.begin_read().expect("read");
+    assert_eq!(tx.open_table(EVENTS).expect("events").len().expect("len"), 1);
+}
+#[test]
+fn created_head_version_event_and_dedup_reopen_together() {
+    let (dir, mut s) = fixture();
+    let c = command();
+    assert_eq!(apply(&mut s, 0, &c), success(&c, 0));
+    drop(s);
+    let mut s = reopen(&dir.path().join("db"));
+    complete(&s);
+    assert_eq!(apply(&mut s, 1, &c), success(&c, 0));
+    let mut changed = c.clone();
+    changed.prepared.content_digest = [6; 32];
+    assert_eq!(
+        apply(&mut s, 2, &changed),
+        CreatedResult::OperationConflict,
+        "U11 full canonical operation conflict"
+    );
+    assert_eq!(
+        s.created_operation(c.operation_id)
+            .expect("original")
+            .expect("operation")
+            .result,
+        success(&c, 0)
+    );
+    let mut stale = c.clone();
+    stale.operation_id = [6; 16];
+    stale.prepared.version_id = [6; 16];
+    assert_eq!(
+        apply(&mut s, 3, &stale),
+        CreatedResult::HeadMismatch {
+            actual_head: Some(c.prepared.version_id)
+        }
+    );
+    let mut reuse = c.clone();
+    reuse.operation_id = [7; 16];
+    reuse.expected_head = Some(c.prepared.version_id);
+    assert_eq!(apply(&mut s, 4, &reuse), CreatedResult::VersionConflict);
+    let mut replacement = stale.clone();
+    replacement.operation_id = [10; 16];
+    replacement.expected_head = Some(c.prepared.version_id);
+    assert_eq!(apply(&mut s, 5, &replacement), success(&replacement, 5));
+    assert_eq!(s.created_head(&c.object).expect("head"), Some(replacement.prepared.version_id));
+    assert_eq!(
+        s.created_version(&c.object, c.prepared.version_id)
+            .expect("old version")
+            .expect("version")
+            .decision_log_id,
+        id(0)
+    );
+    assert!(s.created_event(c.operation_id).expect("old event").is_some());
+    assert!(s.created_event(replacement.operation_id).expect("new event").is_some());
+    assert!(s.created_event(stale.operation_id).expect("rejection event").is_none());
+    assert_eq!(
+        apply(&mut s, 6, &stale),
+        CreatedResult::HeadMismatch {
+            actual_head: Some(c.prepared.version_id)
+        }
+    );
+    drop(s);
+    let s = reopen(&dir.path().join("db"));
+    assert_eq!(s.last_applied().expect("last"), Some(id(6)));
+    let tx = s.database.begin_read().expect("read");
+    assert_eq!(tx.open_table(EVENTS).expect("events").len().expect("len"), 2);
+}
+#[test]
+fn created_binding_is_pinned_and_not_caller_selected() {
+    let (dir, mut s) = fixture();
+    let mut bad = command();
+    bad.binding_revision = 10;
+    assert_eq!(apply(&mut s, 0, &bad), CreatedResult::BindingMismatch);
+    assert!(s.created_head(&bad.object).expect("head").is_none());
+    assert!(s.created_event(bad.operation_id).expect("event").is_none());
+    assert_eq!(
+        s.created_operation(bad.operation_id)
+            .expect("operation")
+            .expect("saved")
+            .result,
+        CreatedResult::BindingMismatch
+    );
+    s.pin_capture_binding(binding()).expect("idempotent pin");
+    let mut changed = binding();
+    changed.target_id = [10; 16];
+    assert!(matches!(s.pin_capture_binding(changed), Err(StoreError::BindingConflict)));
+    drop(s);
+    let s = reopen(&dir.path().join("db"));
+    assert_eq!(s.capture_binding().expect("binding"), Some(binding()));
+}
+#[test]
+fn created_schema_compatibility_and_corruption_are_checked() {
+    let dir = tempfile::tempdir().expect("dir");
+    let path = dir.path().join("db");
+    let s = Store::initialize(&path, identity()).expect("schema1");
+    assert_eq!(s.capture_binding().expect("unmigrated"), None);
+    drop(s);
+    let mut s = reopen(&path);
+    assert_eq!(s.last_applied().expect("no apply"), None);
+    append(&mut s, 0, &command());
+    s.save_committed(id(0)).expect("schema2");
+    drop(s);
+    let mut s = reopen(&path);
+    assert_eq!(s.capture_binding().expect("schema2 unchanged"), None);
+    s.pin_capture_binding(binding()).expect("explicit upgrade");
+    drop(s);
+    let mut s = reopen(&path);
+    assert_eq!(s.capture_binding().expect("schema3"), Some(binding()));
+    s.apply_next_created(id(0)).expect("apply");
+    drop(s);
+    let db = Database::open(&path).expect("raw db");
+    let tx = db.begin_write().expect("write corruption");
+    tx.open_table(EVENTS)
+        .expect("events")
+        .remove(command().operation_id.as_slice())
+        .expect("remove reference");
+    tx.commit().expect("commit corruption");
+    drop(db);
+    let result = Store::open(&path, identity());
+    assert!(
+        matches!(result, Err(StoreError::CorruptRecord)),
+        "U11 missing event rejected before expect"
+    );
+    let (dir, mut s) = fixture();
+    assert_eq!(s.read_committed().expect("empty committed"), None);
+    append(&mut s, 0, &command());
+    s.save_committed(id(0)).expect("committed schema3");
+    assert_eq!(s.capture_binding().expect("no downgrade"), Some(binding()));
+    drop(s);
+    let path = dir.path().join("db");
+    let db = Database::open(&path).expect("raw db");
+    let tx = db.begin_write().expect("write");
+    let mut marker = vec![1];
+    marker.extend(encode_id(id(0)));
+    tx.open_table(META)
+        .expect("meta")
+        .insert("last_applied", marker.as_slice())
+        .expect("corrupt applied");
+    tx.commit().expect("commit");
+    drop(db);
+    assert!(
+        matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
+        "U11 last applied missing result rejected"
+    );
+    let (dir, s) = fixture();
+    drop(s);
+    let path = dir.path().join("db");
+    let db = Database::open(&path).expect("raw pinned db");
+    let tx = db.begin_write().expect("missing binding write");
+    tx.open_table(META)
+        .expect("meta")
+        .remove("capture_binding")
+        .expect("remove binding");
+    tx.commit().expect("binding corruption commit");
+    drop(db);
+    assert!(
+        matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
+        "U11 missing pinned binding rejected"
+    );
+
+    let (dir, mut s) = fixture();
+    let c = command();
+    assert_eq!(apply(&mut s, 0, &c), success(&c, 0));
+    assert_eq!(apply(&mut s, 1, &c), success(&c, 0));
+    drop(s);
+    let path = dir.path().join("db");
+    let db = Database::open(&path).expect("raw duplicate db");
+    let tx = db.begin_write().expect("retarget decision write");
+    let mut key = created::object_bytes(&c.object).expect("object key");
+    key.extend(c.prepared.version_id);
+    let mut version = Vec::new();
+    created::put_prepared(&mut version, &c.prepared);
+    version.extend(encode_id(id(1)));
+    version.extend(binding().revision.to_be_bytes());
+    tx.open_table(VERSIONS)
+        .expect("versions")
+        .insert(key.as_slice(), version.as_slice())
+        .expect("retarget version");
+    let mut event = created::object_bytes(&c.object).expect("event object");
+    created::put_prepared(&mut event, &c.prepared);
+    event.extend(encode_id(id(1)));
+    event.extend(created::binding_bytes(&binding()).expect("binding bytes"));
+    tx.open_table(EVENTS)
+        .expect("events")
+        .insert(c.operation_id.as_slice(), event.as_slice())
+        .expect("retarget event");
+    let canonical = c.encode().expect("canonical");
+    let mut operation = u64::try_from(canonical.len()).expect("length").to_be_bytes().to_vec();
+    operation.extend(canonical);
+    operation.extend(created::result_bytes(&success(&c, 1)));
+    tx.open_table(OPERATIONS)
+        .expect("operations")
+        .insert(c.operation_id.as_slice(), operation.as_slice())
+        .expect("retarget original response");
+    for index in 0..=1 {
+        let mut result = encode_id(id(index));
+        result.extend(created::result_bytes(&success(&c, 1)));
+        tx.open_table(APPLY_RESULTS)
+            .expect("results")
+            .insert(index, result.as_slice())
+            .expect("retarget duplicate response");
+    }
+    tx.commit().expect("coherent retarget commit");
+    drop(db);
+    assert!(
+        matches!(Store::open(&path, identity()), Err(StoreError::CorruptRecord)),
+        "U11 original decision cannot retarget duplicate log"
+    );
+}
+#[test]
+fn created_process_child() {
+    if let Some(path) = std::env::var_os("U11_CHILD_DB") {
+        let mut s = Store::initialize(path, identity()).expect("child init");
+        s.pin_capture_binding(binding()).expect("child pin");
+        assert_eq!(apply(&mut s, 0, &command()), success(&command(), 0));
+        complete(&s);
+        println!("U11_ACK");
+        std::io::stdout().flush().expect("flush ACK");
+        loop {
+            std::thread::park()
+        }
+    }
+    let (_dir, mut s) = fixture();
+    assert_eq!(apply(&mut s, 0, &command()), success(&command(), 0));
+    complete(&s);
+}
+#[test]
+fn created_acknowledged_apply_survives_child_kill() {
+    let dir = tempfile::tempdir().expect("dir");
+    let path = dir.path().join("db");
+    let mut child = Command::new(std::env::current_exe().expect("test executable"))
+        .args(["--exact", "application_tests::created_process_child", "--nocapture"])
+        .env("U11_CHILD_DB", &path)
+        .stdout(Stdio::piped())
+        .stderr(Stdio::inherit())
+        .spawn()
+        .expect("spawn");
+    let stdout = child.stdout.take().expect("stdout");
+    let (tx, rx) = mpsc::channel();
+    let reader = std::thread::spawn(move || {
+        for line in BufReader::new(stdout).lines() {
+            let line = line.expect("child line");
+            if line.ends_with("U11_ACK") {
+                let _ = tx.send(());
+                break;
+            }
+        }
+    });
+    let acknowledged = rx.recv_timeout(Duration::from_secs(20));
+    child.kill().expect("kill child");
+    child.wait().expect("wait child");
+    reader.join().expect("reader cleanup");
+    assert!(acknowledged.is_ok(), "U11 bounded child ACK missing");
+    complete(&reopen(&path));
+}
+#[test]
+fn created_power_cut_before_and_after_apply_flush() {
+    let (dir, mut s) = fixture();
+    append(&mut s, 0, &command());
+    s.save_committed(id(0)).expect("committed");
+    drop(s);
+    let path = dir.path().join("db");
+    let backend = CrashBackend::new(&path);
+    let mut s = Store::from_database(backend.database(), identity()).expect("backend");
+    assert_eq!(s.apply_next_created(id(0)).expect("apply ACK"), success(&command(), 0));
+    backend.cut();
+    drop(s);
+    complete(&reopen(&path));
+    for sync in 1..=2 {
+        for after in [false, true] {
+            let (dir, mut s) = fixture();
+            append(&mut s, 0, &command());
+            s.save_committed(id(0)).expect("committed");
+            drop(s);
+            let path = dir.path().join("db");
+            let backend = CrashBackend::new(&path);
+            let mut s = Store::from_database(backend.database(), identity()).expect("backend");
+            let (entered, resume) = backend.pause_sync(sync, after);
+            let writer = std::thread::spawn(move || {
+                let result = s.apply_next_created(id(0));
+                (s, result)
+            });
+            entered.wait();
+            backend.cut();
+            resume.wait();
+            let (s, result) = writer.join().expect("writer");
+            drop(s);
+            let reopened = Store::open(&path, identity());
+            assert!(
+                reopened.is_ok(),
+                "U11 apply cut must reopen whole old/new state sync={sync} after={after}: {:?}",
+                reopened.as_ref().err()
+            );
+            let s = reopened.expect("asserted reopen");
+            if s.last_applied().expect("last").is_some() {
+                complete(&s)
+            } else {
+                assert!(result.is_err());
+                assert_eq!(s.created_head(&command().object).expect("old head"), None);
+                assert_eq!(
+                    s.created_version(&command().object, command().prepared.version_id)
+                        .expect("old version"),
+                    None
+                );
+                assert_eq!(s.created_event(command().operation_id).expect("old event"), None);
+                assert_eq!(s.created_operation(command().operation_id).expect("old op"), None);
+                assert_eq!(s.created_apply_result(0).expect("old result"), None)
+            }
+            eprintln!("U11 cut boundary sync={sync} after={after}");
+        }
+    }
+}

crates/capture-authority/src/committed.rs

@@ -0,0 +1,63 @@
+use crate::{LOGS, LogId, META, Store, StoreError, decode_id, decode_log, encode_id, number};
+use redb::{ReadableDatabase, ReadableTable};
+
+pub(crate) fn validate_committed(
+    meta: &impl ReadableTable<&'static str, &'static [u8]>,
+    logs: &impl ReadableTable<u64, &'static [u8]>,
+) -> Result<Option<LogId>, StoreError> {
+    let schema = number(meta.get("schema")?.ok_or(StoreError::CorruptRecord)?.value())?;
+    let marker = meta.get("committed")?;
+    match (schema, marker) {
+        (1 | 3 | 4 | 5 | 6, None) => {
+            if matches!(schema, 3..=6) && meta.get("last_applied")?.is_none_or(|v| v.value() != [0]) {
+                return Err(StoreError::CorruptRecord);
+            }
+            Ok(None)
+        }
+        (2 | 3 | 4 | 5 | 6, Some(marker)) => {
+            let id = decode_id(marker.value())?;
+            let record = logs.get(id.index)?.ok_or(StoreError::CorruptRecord)?;
+            if decode_log(record.value())?.id != id {
+                return Err(StoreError::CorruptRecord);
+            }
+            Ok(Some(id))
+        }
+        (1 | 2, _) => Err(StoreError::CorruptRecord),
+        _ => Err(StoreError::UnsupportedSchema),
+    }
+}
+
+impl Store {
+    /// Read the exact committed boundary, validating it against the stored log.
+    pub fn read_committed(&self) -> Result<Option<LogId>, StoreError> {
+        let tx = self.database.begin_read()?;
+        validate_committed(&tx.open_table(META)?, &tx.open_table(LOGS)?)
+    }
+
+    /// Persist a monotonic committed boundary supplied by the authority owner.
+    pub fn save_committed(&mut self, id: LogId) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            let logs = tx.open_table(LOGS)?;
+            let previous = validate_committed(&meta, &logs)?;
+            let record = logs.get(id.index)?.ok_or(StoreError::InvalidCommittedLog)?;
+            if decode_log(record.value())?.id != id {
+                return Err(StoreError::InvalidCommittedLog);
+            }
+            if let Some(previous) = previous {
+                if id.index < previous.index {
+                    return Err(StoreError::CommittedRegression);
+                }
+                if id == previous {
+                    return Ok(());
+                }
+            }
+            meta.insert("committed", encode_id(id).as_slice())?;
+            if previous.is_none() && !matches!(number(meta.get("schema")?.ok_or(StoreError::CorruptRecord)?.value())?, 3..=6) {
+                meta.insert("schema", 2u64.to_be_bytes().as_slice())?;
+            }
+        }
+        self.commit(tx)
+    }
+}

crates/capture-authority/src/created.rs

@@ -0,0 +1,247 @@
+use crate::{LogId, StoreError, decode_id, encode_id};
+
+pub const MAX_OBJECT_KEY_LENGTH: usize = 4096;
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ObjectIdentity {
+    pub bucket_incarnation: [u8; 16],
+    pub key: Vec<u8>,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct PreparedIdentity {
+    pub version_id: [u8; 16],
+    pub preparation_id: [u8; 16],
+    pub content_digest: [u8; 32],
+    pub content_length: u64,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct DecideCreated {
+    pub operation_id: [u8; 16],
+    pub object: ObjectIdentity,
+    pub expected_head: Option<[u8; 16]>,
+    pub prepared: PreparedIdentity,
+    pub binding_revision: u64,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CaptureBinding {
+    pub revision: u64,
+    pub binding_id: [u8; 16],
+    pub target_id: [u8; 16],
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum CreatedResult {
+    Created {
+        version_id: [u8; 16],
+        event_id: [u8; 16],
+        decision_log_id: LogId,
+    },
+    HeadMismatch {
+        actual_head: Option<[u8; 16]>,
+    },
+    OperationConflict,
+    BindingMismatch,
+    VersionConflict,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedVersion {
+    pub prepared: PreparedIdentity,
+    pub decision_log_id: LogId,
+    pub binding_revision: u64,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedEvent {
+    pub object: ObjectIdentity,
+    pub prepared: PreparedIdentity,
+    pub decision_log_id: LogId,
+    pub binding: CaptureBinding,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedOperation {
+    pub canonical_command: Vec<u8>,
+    pub result: CreatedResult,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedOperationProvenance {
+    pub operation: CreatedOperation,
+    pub first_decision_log_id: LogId,
+}
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CreatedApplyResult {
+    pub log_id: LogId,
+    pub result: CreatedResult,
+}
+
+pub(crate) struct Decoder<'a> {
+    bytes: &'a [u8],
+}
+impl<'a> Decoder<'a> {
+    pub(crate) fn new(bytes: &'a [u8]) -> Self {
+        Self { bytes }
+    }
+    pub(crate) fn take<const N: usize>(&mut self) -> Result<[u8; N], StoreError> {
+        let a = self
+            .bytes
+            .get(..N)
+            .ok_or(StoreError::CorruptRecord)?
+            .try_into()
+            .map_err(|_| StoreError::CorruptRecord)?;
+        self.bytes = &self.bytes[N..];
+        Ok(a)
+    }
+    pub(crate) fn u64(&mut self) -> Result<u64, StoreError> {
+        Ok(u64::from_be_bytes(self.take()?))
+    }
+    pub(crate) fn uuid(&mut self) -> Result<[u8; 16], StoreError> {
+        let id = self.take()?;
+        if id == [0; 16] {
+            return Err(StoreError::CorruptRecord);
+        }
+        Ok(id)
+    }
+    pub(crate) fn option(&mut self) -> Result<Option<[u8; 16]>, StoreError> {
+        match self.take::<1>()?[0] {
+            0 => Ok(None),
+            1 => Ok(Some(self.uuid()?)),
+            _ => Err(StoreError::CorruptRecord),
+        }
+    }
+    pub(crate) fn object(&mut self) -> Result<ObjectIdentity, StoreError> {
+        let bucket_incarnation = self.uuid()?;
+        let len = usize::try_from(u32::from_be_bytes(self.take()?)).map_err(|_| StoreError::CorruptRecord)?;
+        if len > MAX_OBJECT_KEY_LENGTH {
+            return Err(StoreError::CorruptRecord);
+        }
+        let key = self.bytes.get(..len).ok_or(StoreError::CorruptRecord)?.to_vec();
+        self.bytes = &self.bytes[len..];
+        Ok(ObjectIdentity { bucket_incarnation, key })
+    }
+    pub(crate) fn prepared(&mut self) -> Result<PreparedIdentity, StoreError> {
+        Ok(PreparedIdentity {
+            version_id: self.uuid()?,
+            preparation_id: self.uuid()?,
+            content_digest: self.take()?,
+            content_length: self.u64()?,
+        })
+    }
+    pub(crate) fn id(&mut self) -> Result<LogId, StoreError> {
+        decode_id(&self.take::<24>()?)
+    }
+    pub(crate) fn binding(&mut self) -> Result<CaptureBinding, StoreError> {
+        Ok(CaptureBinding {
+            revision: self.u64()?,
+            binding_id: self.uuid()?,
+            target_id: self.uuid()?,
+        })
+    }
+    pub(crate) fn finish(self) -> Result<(), StoreError> {
+        if self.bytes.is_empty() {
+            Ok(())
+        } else {
+            Err(StoreError::CorruptRecord)
+        }
+    }
+    pub(crate) fn result(&mut self) -> Result<CreatedResult, StoreError> {
+        match self.take::<1>()?[0] {
+            0 => Ok(CreatedResult::Created {
+                version_id: self.uuid()?,
+                event_id: self.uuid()?,
+                decision_log_id: self.id()?,
+            }),
+            1 => Ok(CreatedResult::HeadMismatch {
+                actual_head: self.option()?,
+            }),
+            2 => Ok(CreatedResult::OperationConflict),
+            3 => Ok(CreatedResult::BindingMismatch),
+            4 => Ok(CreatedResult::VersionConflict),
+            _ => Err(StoreError::CorruptRecord),
+        }
+    }
+}
+pub(crate) fn put_option(b: &mut Vec<u8>, id: Option<[u8; 16]>) {
+    b.push(u8::from(id.is_some()));
+    if let Some(id) = id {
+        b.extend(id)
+    }
+}
+pub(crate) fn object_bytes(o: &ObjectIdentity) -> Result<Vec<u8>, StoreError> {
+    if o.bucket_incarnation == [0; 16] || o.key.len() > MAX_OBJECT_KEY_LENGTH {
+        return Err(StoreError::CorruptRecord);
+    }
+    let capacity = 20usize.checked_add(o.key.len()).ok_or(StoreError::CorruptRecord)?;
+    let mut b = Vec::with_capacity(capacity);
+    b.extend(o.bucket_incarnation);
+    b.extend(
+        u32::try_from(o.key.len())
+            .map_err(|_| StoreError::CorruptRecord)?
+            .to_be_bytes(),
+    );
+    b.extend(&o.key);
+    Ok(b)
+}
+pub(crate) fn put_prepared(b: &mut Vec<u8>, p: &PreparedIdentity) {
+    b.extend(p.version_id);
+    b.extend(p.preparation_id);
+    b.extend(p.content_digest);
+    b.extend(p.content_length.to_be_bytes())
+}
+pub(crate) fn binding_bytes(v: &CaptureBinding) -> Result<Vec<u8>, StoreError> {
+    let mut b = v.revision.to_be_bytes().to_vec();
+    b.extend(v.binding_id);
+    b.extend(v.target_id);
+    let mut d = Decoder::new(&b);
+    d.binding()?;
+    d.finish()?;
+    Ok(b)
+}
+pub(crate) fn result_bytes(r: &CreatedResult) -> Vec<u8> {
+    let mut b = Vec::new();
+    match r {
+        CreatedResult::Created {
+            version_id,
+            event_id,
+            decision_log_id,
+        } => {
+            b.push(0);
+            b.extend(version_id);
+            b.extend(event_id);
+            b.extend(encode_id(*decision_log_id))
+        }
+        CreatedResult::HeadMismatch { actual_head } => {
+            b.push(1);
+            put_option(&mut b, *actual_head)
+        }
+        CreatedResult::OperationConflict => b.push(2),
+        CreatedResult::BindingMismatch => b.push(3),
+        CreatedResult::VersionConflict => b.push(4),
+    }
+    b
+}
+impl DecideCreated {
+    pub fn encode(&self) -> Result<Vec<u8>, StoreError> {
+        let object = object_bytes(&self.object)?;
+        let capacity = object.len().checked_add(134).ok_or(StoreError::CorruptRecord)?;
+        let mut b = Vec::with_capacity(capacity);
+        b.extend(b"CRTD\x01");
+        b.extend(self.operation_id);
+        b.extend(object);
+        put_option(&mut b, self.expected_head);
+        put_prepared(&mut b, &self.prepared);
+        b.extend(self.binding_revision.to_be_bytes());
+        Self::decode(&b)?;
+        Ok(b)
+    }
+    pub fn decode(bytes: &[u8]) -> Result<Self, StoreError> {
+        let mut d = Decoder::new(bytes);
+        if d.take::<5>()? != *b"CRTD\x01" {
+            return Err(StoreError::CorruptRecord);
+        }
+        let c = Self {
+            operation_id: d.uuid()?,
+            object: d.object()?,
+            expected_head: d.option()?,
+            prepared: d.prepared()?,
+            binding_revision: d.u64()?,
+        };
+        d.finish()?;
+        Ok(c)
+    }
+}

crates/capture-authority/src/native_snapshot.rs

@@ -0,0 +1,118 @@
+//! Owned consistent native application snapshot view.
+use crate::{Store, StoreError, application, snapshot};
+use redb::{ReadTransaction, ReadableDatabase, ReadableTable, ReadableTableMetadata};
+use std::{
+    io::Write,
+    sync::{Arc, Mutex},
+};
+
+#[derive(Debug, thiserror::Error)]
+pub enum NativeSnapshotError {
+    #[error(transparent)]
+    Store(#[from] StoreError),
+    #[error(transparent)]
+    Snapshot(#[from] snapshot::SnapshotError),
+    #[error("native snapshot Store lock poisoned")]
+    Poisoned,
+    #[error("native snapshot worker failed")]
+    Worker(#[source] tokio::task::JoinError),
+}
+
+pub struct NativeSnapshotView {
+    // Drop the transaction before the owner so the Database stays open throughout the view.
+    transaction: ReadTransaction,
+    header: snapshot::Header,
+    counts: [u64; 4],
+    schema: u64,
+    _owner: Arc<Mutex<Store>>,
+}
+impl NativeSnapshotView {
+    pub(crate) fn acquire(owner: Arc<Mutex<Store>>) -> Result<Self, NativeSnapshotError> {
+        let transaction = {
+            let store = owner.lock().map_err(|_| NativeSnapshotError::Poisoned)?;
+            if !store.writable {
+                return Err(StoreError::WriteDisabled.into());
+            }
+            store.database.begin_read().map_err(StoreError::from)?
+        };
+        let (applied, membership) = application::native_application_state(&transaction)?;
+        let (header, schema) = {
+            let meta = transaction.open_table(crate::META).map_err(StoreError::from)?;
+            let identity = meta
+                .get("identity")
+                .map_err(StoreError::from)?
+                .ok_or(StoreError::CorruptRecord)?;
+            let schema = crate::number(
+                meta.get("schema")
+                    .map_err(StoreError::from)?
+                    .ok_or(StoreError::CorruptRecord)?
+                    .value(),
+            )?;
+            (
+                snapshot::Header {
+                    cluster_id: crate::decode_identity(identity.value())?.cluster_id,
+                    applied,
+                    membership,
+                    binding: application::read_binding(&meta)?,
+                },
+                schema,
+            )
+        };
+        let mut counts = [0; 4];
+        for (index, definition) in [
+            application::HEADS,
+            application::VERSIONS,
+            application::EVENTS,
+            application::OPERATIONS,
+        ]
+        .into_iter()
+        .enumerate()
+        {
+            counts[index] = transaction
+                .open_table(definition)
+                .map_err(StoreError::from)?
+                .len()
+                .map_err(StoreError::from)?;
+        }
+        Ok(Self {
+            transaction,
+            header,
+            counts,
+            schema,
+            _owner: owner,
+        })
+    }
+
+    pub fn stream_capability_artifact<W: Write>(
+        self,
+        writer: W,
+        limits: snapshot::Limits,
+    ) -> Result<snapshot::CapabilityArtifact<W>, NativeSnapshotError> {
+        if self.schema != 6 {
+            return Err(StoreError::UnsupportedSchema.into());
+        }
+        let output = snapshot::CapabilityOutput::new(writer, limits.total_bytes);
+        Ok(self.stream(output, limits)?.finish())
+    }
+
+    pub fn stream<W: Write>(self, writer: W, limits: snapshot::Limits) -> Result<W, NativeSnapshotError> {
+        let mut encoder = snapshot::Encoder::new(writer, &self.header, self.counts, limits)?;
+        for (definition, section) in [
+            (application::HEADS, snapshot::Section::Heads),
+            (application::VERSIONS, snapshot::Section::Versions),
+            (application::EVENTS, snapshot::Section::Events),
+            (application::OPERATIONS, snapshot::Section::Operations),
+        ] {
+            let table = self.transaction.open_table(definition).map_err(StoreError::from)?;
+            for row in table.iter().map_err(StoreError::from)? {
+                let (key, value) = row.map_err(StoreError::from)?;
+                encoder.row(snapshot::Row {
+                    section,
+                    key: key.value(),
+                    value: value.value(),
+                })?;
+            }
+        }
+        Ok(encoder.finish()?)
+    }
+}

crates/capture-authority/src/node.rs

@@ -0,0 +1,371 @@
+//! Validated capture peer metadata and its independent CNOD v1 codec.
+use serde::{Deserialize, Deserializer, Serialize, Serializer, ser::SerializeStruct};
+
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct CaptureNode {
+    rpc_uri: String,
+}
+
+#[derive(Debug, thiserror::Error)]
+pub enum CaptureNodeError {
+    #[error("invalid RPC URI")]
+    InvalidRpcUri,
+    #[error("invalid RPC URI")]
+    InvalidRpcUriParse(#[source] url::ParseError),
+    #[error("RPC URI too long")]
+    RpcUriTooLong,
+    #[error("invalid node magic")]
+    InvalidNodeMagic,
+    #[error("unsupported node version")]
+    UnsupportedNodeVersion,
+    #[error("invalid node length")]
+    InvalidNodeLength,
+    #[error("truncated node record")]
+    TruncatedNodeRecord,
+    #[error("invalid node UTF-8")]
+    InvalidNodeUtf8(#[source] std::str::Utf8Error),
+    #[error("trailing node bytes")]
+    TrailingNodeBytes,
+}
+impl CaptureNodeError {
+    pub fn code(&self) -> &'static str {
+        match self {
+            Self::InvalidRpcUri | Self::InvalidRpcUriParse(_) => "invalid_rpc_uri",
+            Self::RpcUriTooLong => "rpc_uri_too_long",
+            Self::InvalidNodeMagic => "invalid_node_magic",
+            Self::UnsupportedNodeVersion => "unsupported_node_version",
+            Self::InvalidNodeLength => "invalid_node_length",
+            Self::TruncatedNodeRecord => "truncated_node_record",
+            Self::InvalidNodeUtf8(_) => "invalid_node_utf8",
+            Self::TrailingNodeBytes => "trailing_node_bytes",
+        }
+    }
+}
+impl CaptureNode {
+    pub fn new(rpc_uri: String) -> Result<Self, CaptureNodeError> {
+        let node = Self { rpc_uri };
+        node.validate()?;
+        Ok(node)
+    }
+
+    pub fn rpc_uri(&self) -> &str {
+        &self.rpc_uri
+    }
+
+    pub fn validate(&self) -> Result<(), CaptureNodeError> {
+        let uri = &self.rpc_uri;
+        if uri.len() > 4096 {
+            return Err(CaptureNodeError::RpcUriTooLong);
+        }
+        if uri.is_empty() || uri.chars().any(|c| c.is_whitespace() || c.is_control() || c == '\\') {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let rest = uri
+            .strip_prefix("http://")
+            .or_else(|| uri.strip_prefix("https://"))
+            .ok_or(CaptureNodeError::InvalidRpcUri)?;
+        if uri.contains('?') || uri.contains('#') {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let (authority, path) = rest.split_once('/').map_or((rest, ""), |(a, p)| (a, p));
+        // An empty suffix after the first slash is the only allowed path.
+        if !path.is_empty() || authority.contains('@') {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let (host, port) = if authority.starts_with('[') {
+            let end = authority.find(']').ok_or(CaptureNodeError::InvalidRpcUri)?;
+            let port = authority
+                .get(end + 1..)
+                .and_then(|suffix| suffix.strip_prefix(':'))
+                .ok_or(CaptureNodeError::InvalidRpcUri)?;
+            (&authority[..=end], port)
+        } else {
+            authority.split_once(':').ok_or(CaptureNodeError::InvalidRpcUri)?
+        };
+        if host.is_empty() || port.is_empty() || !port.bytes().all(|b| b.is_ascii_digit()) {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let port = port.parse::<u16>().map_err(|_| CaptureNodeError::InvalidRpcUri)?;
+        if port == 0 {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        let parsed_url = url::Url::parse(uri).map_err(CaptureNodeError::InvalidRpcUriParse)?;
+        if parsed_url.host().is_none() {
+            return Err(CaptureNodeError::InvalidRpcUri);
+        }
+        Ok(())
+    }
+
+    pub fn encode(&self) -> Result<Vec<u8>, CaptureNodeError> {
+        self.validate()?;
+        let length = u32::try_from(self.rpc_uri.len()).map_err(|_| CaptureNodeError::InvalidNodeLength)?;
+        let mut bytes = b"CNOD\x01".to_vec();
+        bytes.extend_from_slice(&length.to_be_bytes());
+        bytes.extend_from_slice(self.rpc_uri.as_bytes());
+        Ok(bytes)
+    }
+
+    pub fn decode(bytes: &[u8]) -> Result<Self, CaptureNodeError> {
+        if bytes.len() < 9 {
+            return Err(CaptureNodeError::TruncatedNodeRecord);
+        }
+        if &bytes[..4] != b"CNOD" {
+            return Err(CaptureNodeError::InvalidNodeMagic);
+        }
+        if bytes[4] != 1 {
+            return Err(CaptureNodeError::UnsupportedNodeVersion);
+        }
+        let length = u32::from_be_bytes(bytes[5..9].try_into().map_err(|_| CaptureNodeError::InvalidNodeLength)?);
+        if !(1..=4096).contains(&length) {
+            return Err(CaptureNodeError::InvalidNodeLength);
+        }
+        let length = usize::try_from(length).map_err(|_| CaptureNodeError::InvalidNodeLength)?;
+        let end = 9_usize.checked_add(length).ok_or(CaptureNodeError::InvalidNodeLength)?;
+        if bytes.len() < end {
+            return Err(CaptureNodeError::TruncatedNodeRecord);
+        }
+        if bytes.len() != end {
+            return Err(CaptureNodeError::TrailingNodeBytes);
+        }
+        let uri = std::str::from_utf8(&bytes[9..end]).map_err(CaptureNodeError::InvalidNodeUtf8)?;
+        Self::new(uri.to_owned())
+    }
+}
+impl Serialize for CaptureNode {
+    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
+        self.validate().map_err(serde::ser::Error::custom)?;
+        let mut state = serializer.serialize_struct("CaptureNode", 1)?;
+        state.serialize_field("rpc_uri", &self.rpc_uri)?;
+        state.end()
+    }
+}
+impl<'de> Deserialize<'de> for CaptureNode {
+    fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
+        #[derive(Deserialize)]
+        #[serde(deny_unknown_fields)]
+        struct Fields {
+            rpc_uri: String,
+        }
+        let fields = Fields::deserialize(deserializer).map_err(|_| serde::de::Error::custom("invalid capture node fields"))?;
+        Self::new(fields.rpc_uri).map_err(serde::de::Error::custom)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::error::Error;
+
+    fn record(payload: &[u8]) -> Vec<u8> {
+        let mut bytes = b"CNOD\x01".to_vec();
+        bytes.extend_from_slice(&u32::try_from(payload.len()).expect("test payload length").to_be_bytes());
+        bytes.extend_from_slice(payload);
+        bytes
+    }
+    fn reject_uri(uri: &str) {
+        let result = CaptureNode::new(uri.to_owned());
+        assert!(result.is_err(), "URI should reject: {uri:?}");
+        assert_eq!(result.expect_err("rejected URI").code(), "invalid_rpc_uri");
+    }
+    fn reject_record(bytes: &[u8], code: &str) {
+        let result = CaptureNode::decode(bytes);
+        assert!(result.is_err(), "record should reject with {code}");
+        assert_eq!(result.expect_err("rejected record").code(), code);
+    }
+    #[test]
+    fn actual_openraft_node_bound() {
+        fn require_node<T: openraft::Node>() {}
+        require_node::<CaptureNode>();
+        assert_eq!(CaptureNode::default().rpc_uri(), "");
+    }
+    #[test]
+    fn default_and_empty_rejected_at_every_boundary() {
+        let node = CaptureNode::default();
+        assert_eq!(node.rpc_uri(), "");
+        reject_uri("");
+        assert_eq!(node.validate().expect_err("empty validation").code(), "invalid_rpc_uri");
+        assert!(serde_json::to_string(&node).is_err());
+        assert!(serde_json::from_str::<CaptureNode>(r#"{"rpc_uri":""}"#).is_err());
+        assert_eq!(node.encode().expect_err("empty encoding").code(), "invalid_rpc_uri");
+        reject_record(&record(b""), "invalid_node_length");
+    }
+    #[test]
+    fn accepted_uris_preserve_exact_bytes() {
+        for uri in [
+            "http://node:80",
+            "https://node:443",
+            "http://node:1",
+            "https://node:65535/",
+            "http://127.0.0.1:9000",
+            "https://[::1]:443/",
+            "http://[2001:db8::1]:1",
+            "http://NoDe:00080/",
+        ] {
+            let result = CaptureNode::new(uri.to_owned());
+            assert!(result.is_ok(), "valid URI construction: {uri}");
+            let node = result.expect("valid URI");
+            assert_eq!(node.rpc_uri(), uri, "preserve explicit default port and raw URI");
+            assert_eq!(node.encode().expect("encode URI"), record(uri.as_bytes()));
+            assert_eq!(CaptureNode::decode(&record(uri.as_bytes())).expect("decode URI"), node);
+        }
+    }
+    #[test]
+    fn golden_cnod_bytes() {
+        let node = CaptureNode {
+            rpc_uri: "http://node:80".to_owned(),
+        };
+        let golden = b"CNOD\x01\x00\x00\x00\x0ehttp://node:80";
+        assert_eq!(node.encode().ok().as_deref(), Some(golden.as_slice()), "golden CNOD bytes");
+    }
+    #[test]
+    fn valid_cnod_decode() {
+        let golden = b"CNOD\x01\x00\x00\x00\x0ehttp://node:80";
+        let decoded = CaptureNode::decode(golden);
+        assert!(decoded.is_ok(), "golden decode succeeds");
+        assert_eq!(decoded.expect("golden decoded node").rpc_uri(), "http://node:80");
+    }
+    #[test]
+    fn serde_shape_roundtrip_and_strict_fields() {
+        let node = CaptureNode::new("https://node:443/".to_owned()).expect("serde node");
+        let json = r#"{"rpc_uri":"https://node:443/"}"#;
+        assert_eq!(serde_json::to_string(&node).expect("serialize node"), json);
+        assert_eq!(serde_json::from_str::<CaptureNode>(json).expect("deserialize node"), node);
+        for json in [
+            r#"{}"#,
+            r#"{"rpc_uri":1}"#,
+            r#"{"rpc_uri":null}"#,
+            r#"{"rpc_uri":"http://node:1","extra":0}"#,
+            r#"{"rpc_uri":"http://node:1","rpc_uri":"http://node:2"}"#,
+        ] {
+            assert!(serde_json::from_str::<CaptureNode>(json).is_err(), "strict fields: {json}");
+        }
+    }
+    #[test]
+    fn credentials_and_raw_uri_shapes_reject() {
+        for uri in [
+            "http://user@node:1",
+            "http://user:password@node:1",
+            "http://@node:1",
+            "HTTP://node:1",
+            "Https://node:1",
+            "http://node",
+            "http://node:",
+            "http://node:0",
+            "http://node:-1",
+            "http://node:+1",
+            "http://node:65536",
+            "http://node:999999999999999999999",
+            "http://[::1:1",
+            "http://[::1]extra:1",
+            "http://[invalid]:1",
+            "http://::1:1",
+            "http://:1",
+            "/disk/path",
+            "file:///disk/path",
+            "http://node:1/./",
+            "http://node:1/../",
+            "http://node:1/%2e/",
+            "http://node:1/%2E%2e/",
+            "http://node:1/path",
+            "http://node:1\\",
+            " http://node:1",
+            "http://node:1\n",
+            "http://no\tde:1",
+            "http://node:1\u{00a0}",
+            "http://node:1\u{0085}",
+            "http://node:1\0",
+        ] {
+            reject_uri(uri);
+        }
+    }
+    #[test]
+    fn query_and_fragment_markers_reject() {
+        for uri in [
+            "http://node:1?",
+            "http://node:1?x=y",
+            "http://node:1/?",
+            "http://node:1#",
+            "http://node:1#fragment",
+            "http://node:1/#",
+        ] {
+            reject_uri(uri);
+        }
+    }
+    #[test]
+    fn utf8_byte_length_boundaries() {
+        for length in [4095, 4096] {
+            let prefix = "http://node:";
+            let uri = format!("{prefix}{}1", "0".repeat(length - prefix.len() - 1));
+            assert_eq!(uri.len(), length);
+            assert!(url::Url::parse(&uri).is_ok(), "boundary fixture accepted by actual URL parser");
+            let node = CaptureNode::new(uri.clone()).expect("boundary node");
+            assert_eq!(node.rpc_uri(), uri);
+            assert_eq!(
+                CaptureNode::decode(&node.encode().expect("boundary encoding")).expect("boundary decoding"),
+                node
+            );
+        }
+        let uri = format!("http://node:{}1", "0".repeat(4097 - "http://node:".len() - 1));
+        assert_eq!(CaptureNode::new(uri).expect_err("oversize URI").code(), "rpc_uri_too_long");
+        let unicode = format!("http://{}:1", "é".repeat(2044));
+        assert!(unicode.chars().count() < 4096);
+        assert!(unicode.len() > 4096);
+        assert_eq!(CaptureNode::new(unicode).expect_err("UTF-8 byte limit").code(), "rpc_uri_too_long");
+    }
+    #[test]
+    fn cnod_headers_and_lengths_reject_exact_codes() {
+        let valid = record(b"http://node:80");
+        for end in 0..9 {
+            reject_record(&valid[..end], "truncated_node_record");
+        }
+        let mut bad = valid.clone();
+        bad[0] = b'X';
+        reject_record(&bad, "invalid_node_magic");
+        bad = valid.clone();
+        bad[4] = 2;
+        reject_record(&bad, "unsupported_node_version");
+        for length in [0_u32, 4097, u32::MAX] {
+            bad = b"CNOD\x01".to_vec();
+            bad.extend_from_slice(&length.to_be_bytes());
+            reject_record(&bad, "invalid_node_length");
+        }
+        for end in 9..valid.len() {
+            reject_record(&valid[..end], "truncated_node_record");
+        }
+        reject_record(&record(&[0xff]), "invalid_node_utf8");
+        reject_record(&record(&[0xc3, 0x28]), "invalid_node_utf8");
+        reject_record(&record(b"invalid URI"), "invalid_rpc_uri");
+    }
+    #[test]
+    fn trailing_bytes_reject() {
+        for tail in [&b"x"[..], &b"xyz"[..]] {
+            let mut bytes = record(b"http://node:80");
+            bytes.extend_from_slice(tail);
+            reject_record(&bytes, "trailing_node_bytes");
+        }
+    }
+    #[test]
+    fn errors_and_sources_redact_input() {
+        let sentinel = "http://SECRET_USER:SECRET_PASSWORD@node:1";
+        let error = CaptureNode::new(sentinel.to_owned()).expect_err("credentials reject");
+        let mut current: &dyn Error = &error;
+        loop {
+            let rendered = format!("{current} {current:?}");
+            assert!(!rendered.contains("SECRET"));
+            match current.source() {
+                Some(source) => current = source,
+                None => break,
+            }
+        }
+        let json = format!(r#"{{"rpc_uri":"{sentinel}"}}"#);
+        let error = serde_json::from_str::<CaptureNode>(&json).expect_err("serde credentials reject");
+        assert!(!format!("{error} {error:?}").contains("SECRET"));
+        let json = format!(r#"{{"{sentinel}":0}}"#);
+        let error = serde_json::from_str::<CaptureNode>(&json).expect_err("unknown fields reject");
+        assert!(!format!("{error} {error:?}").contains("SECRET"));
+        let error = CaptureNode::new("http://[bad]:1".to_owned()).expect_err("parser rejection");
+        assert!(error.source().is_some(), "retain URL parser source");
+        let error = CaptureNode::decode(&record(&[0xff])).expect_err("UTF-8 rejection");
+        assert!(error.source().is_some(), "retain UTF-8 source");
+    }
+}

crates/capture-authority/src/truncate.rs

@@ -0,0 +1,43 @@
+use crate::{LOGS, META, Store, StoreError, decode_id, decode_log, encode_id};
+use redb::ReadableTable;
+
+impl Store {
+    /// Atomically remove every log entry at or after `from`.
+    pub fn truncate_suffix(&mut self, from: u64) -> Result<(), StoreError> {
+        let tx = self.write_transaction()?;
+        {
+            let mut meta = tx.open_table(META)?;
+            let logs = tx.open_table(LOGS)?;
+            if crate::committed::validate_committed(&meta, &logs)?.is_some_and(|id| from <= id.index) {
+                return Err(StoreError::CommittedLogProtected);
+            }
+            drop(logs);
+            let tail = meta.get("tail")?.map(|v| decode_id(v.value())).transpose()?;
+            if tail.is_none_or(|id| from > id.index) {
+                return Ok(());
+            }
+            let mut logs = tx.open_table(LOGS)?;
+            let new_tail = if from == 0 {
+                None
+            } else {
+                let index = from - 1;
+                let record = logs.get(index)?.ok_or(StoreError::InvalidLogSequence)?;
+                let id = decode_log(record.value())?.id;
+                if id.index != index {
+                    return Err(StoreError::CorruptRecord);
+                }
+                Some(id)
+            };
+            logs.retain(|index, _| index < from)?;
+            match new_tail {
+                Some(id) => {
+                    meta.insert("tail", encode_id(id).as_slice())?;
+                }
+                None => {
+                    meta.remove("tail")?;
+                }
+            }
+        }
+        self.commit(tx)
+    }
+}

crates/common/Cargo.toml

@@ -34,7 +34,6 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
 hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
 
 [dependencies]
-arc-swap.workspace = true
 hotpath.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }

crates/common/src/globals.rs

@@ -12,10 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use arc_swap::ArcSwap;
 use std::collections::HashMap;
+use std::sync::LazyLock;
 use std::sync::atomic::{AtomicU64, Ordering};
-use std::sync::{Arc, LazyLock};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -28,42 +27,6 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 
-#[derive(Clone, Default)]
-pub struct OutboundTlsSnapshot {
-    pub generation: u64,
-    pub root_ca_pem: Option<Vec<u8>>,
-    pub mtls_identity: Option<MtlsIdentityPem>,
-}
-
-static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
-    LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
-
-pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
-    GLOBAL_OUTBOUND_TLS_STATE.load_full()
-}
-
-/// Publish identity, trust and generation together; the individual globals are legacy views.
-pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        if current.generation != 0
-            && current.root_ca_pem == snapshot.root_ca_pem
-            && current.mtls_identity == snapshot.mtls_identity
-        {
-            return Arc::clone(current);
-        }
-        Arc::new(OutboundTlsSnapshot {
-            generation: snapshot.generation.max(current.generation.saturating_add(1)),
-            root_ca_pem: snapshot.root_ca_pem.clone(),
-            mtls_identity: snapshot.mtls_identity.clone(),
-        })
-    });
-    let accepted = get_global_outbound_tls_snapshot();
-    *GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
-    *GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
-    GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
-    accepted.generation
-}
-
 /// Log level to use when reporting cached gRPC connection eviction.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub enum ConnectionEvictionLogLevel {
@@ -122,27 +85,17 @@ pub async fn get_global_addr() -> String {
 /// # Arguments
 /// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
 pub async fn set_global_root_cert(cert: Vec<u8>) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.root_ca_pem = Some(cert.clone());
-        Arc::new(next)
-    });
     *GLOBAL_ROOT_CERT.write().await = Some(cert);
 }
 
 /// Clear the global root CA certificate for outbound gRPC clients.
 pub async fn clear_global_root_cert() {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.root_ca_pem = None;
-        Arc::new(next)
-    });
     *GLOBAL_ROOT_CERT.write().await = None;
 }
 
 /// Get the global root CA certificate for outbound gRPC clients.
 pub async fn get_global_root_cert() -> Option<Vec<u8>> {
-    get_global_outbound_tls_snapshot().root_ca_pem.clone()
+    GLOBAL_ROOT_CERT.read().await.clone()
 }
 
 /// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -152,32 +105,22 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
 /// # Arguments
 /// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
 pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.mtls_identity = identity.clone();
-        Arc::new(next)
-    });
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
 /// Get the global mTLS identity for outbound gRPC clients.
 pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
-    get_global_outbound_tls_snapshot().mtls_identity.clone()
+    GLOBAL_MTLS_IDENTITY.read().await.clone()
 }
 
 /// Set the global outbound TLS generation.
 pub fn set_global_outbound_tls_generation(generation: u64) {
-    GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
-        let mut next = (**current).clone();
-        next.generation = generation;
-        Arc::new(next)
-    });
     GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
 }
 
 /// Get the global outbound TLS generation.
 pub fn get_global_outbound_tls_generation() -> u64 {
-    get_global_outbound_tls_snapshot().generation
+    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
 }
 
 /// Evict a stale/dead connection from the global connection cache.
@@ -251,77 +194,8 @@ pub async fn clear_all_connections() {
 ///
 /// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
 /// When absent, clients proceed with standard server-authenticated TLS.
-#[derive(Clone, PartialEq, Eq)]
+#[derive(Clone, Debug)]
 pub struct MtlsIdentityPem {
     pub cert_pem: Vec<u8>,
     pub key_pem: Vec<u8>,
 }
-
-impl std::fmt::Debug for MtlsIdentityPem {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        f.debug_struct("MtlsIdentityPem")
-            .field("cert_bytes", &self.cert_pem.len())
-            .field("key_pem", &"[REDACTED]")
-            .finish()
-    }
-}
-
-#[cfg(test)]
-mod tls_snapshot_tests {
-    use super::*;
-
-    #[tokio::test]
-    async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
-        let start = get_global_outbound_tls_generation().saturating_add(1);
-        let snapshot = OutboundTlsSnapshot {
-            generation: start,
-            root_ca_pem: Some(vec![1]),
-            mtls_identity: Some(MtlsIdentityPem {
-                cert_pem: vec![1],
-                key_pem: vec![1],
-            }),
-        };
-        let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
-        assert_eq!(
-            publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
-                generation: first.saturating_add(1),
-                ..snapshot
-            })
-            .await,
-            first
-        );
-        let writer = tokio::spawn(async move {
-            for tag in 2_u8..=255 {
-                publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
-                    generation: start.saturating_add(u64::from(tag)),
-                    root_ca_pem: Some(vec![tag]),
-                    mtls_identity: Some(MtlsIdentityPem {
-                        cert_pem: vec![tag],
-                        key_pem: vec![tag],
-                    }),
-                })
-                .await;
-                tokio::task::yield_now().await;
-            }
-        });
-        while !writer.is_finished() {
-            let state = get_global_outbound_tls_snapshot();
-            let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
-            assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
-            assert_eq!(identity.cert_pem, identity.key_pem);
-            tokio::task::yield_now().await;
-        }
-        writer.await.expect("writer completes");
-    }
-
-    #[test]
-    fn mtls_identity_debug_redacts_private_key() {
-        let identity = MtlsIdentityPem {
-            cert_pem: b"public certificate".to_vec(),
-            key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
-        };
-        let debug = format!("{identity:?}");
-        assert!(!debug.contains("PRIVATE-KEY-MARKER"));
-        assert!(debug.contains("[REDACTED]"));
-    }
-}

crates/config/README.md

@@ -119,19 +119,11 @@ when restoring `/rustfs/console` for a custom OEM build.
 - `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
 - `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
 - `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
-- `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (canonical)
-- `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (canonical)
-- `RUSTFS_SCANNER_WALKERS_PER_DISK` (canonical)
 
 Scanner cycle budget controls:
 
 - When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance.
 - An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`.
-Scanner walker controls:
-
-- `RUSTFS_SCANNER_WALKERS_PER_DISK` sets how many bucket walkers each local disk of an erasure set runs; the default `1` walks one bucket per disk at a time, which is the right value for rotational media. On flash, `2` to `4` lets one disk walk several buckets concurrently, which matters most for single-disk deployments with many buckets, where the walk is otherwise fully serial. Values outside `1` through `16` are rejected by `admin config set`; an out-of-range or non-numeric environment value is logged as a scanner runtime config error (`state = "startup_apply_failed"`) and the scanner keeps the default. The walkers of one disk share that disk's scanner admission, so peers still see one scan session per physical disk, and `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` still caps the total.
-- The same value is available as the persistent `scanner.walkers_per_disk` key and is reported under `runtime_config.walkers_per_disk` in the admin scanner status; see [Scanner Runtime Controls](../../docs/operations/scanner-runtime-controls.md).
-
 - A timed-out cycle cancels cooperative scanner work, then fences its leader epoch before releasing the lease. An uncooperative I/O operation is dropped after the bounded shutdown window; its cursor is not claimed to be durable and the scanner reports `recovery-required` when the worker cannot stop cooperatively, the cycle state was not confirmed durable, or epoch fencing cannot be persisted.
 
 ## Mmap read environment aliases

crates/config/src/constants/scanner.rs

@@ -59,9 +59,6 @@ pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans";
 /// Scanner config key capping concurrent scanner disk bucket walks per set.
 pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans";
 
-/// Scanner config key setting how many bucket walkers each local disk runs.
-pub const SCANNER_WALKERS_PER_DISK: &str = "walkers_per_disk";
-
 /// Scanner config key controlling how often object loops yield.
 pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects";
 
@@ -89,7 +86,6 @@ pub const SCANNER_KEYS: &[&str] = &[
     SCANNER_CACHE_SAVE_TIMEOUT,
     SCANNER_MAX_CONCURRENT_SET_SCANS,
     SCANNER_MAX_CONCURRENT_DISK_SCANS,
-    SCANNER_WALKERS_PER_DISK,
     SCANNER_YIELD_EVERY_N_OBJECTS,
     SCANNER_ALERT_EXCESS_VERSIONS,
     SCANNER_ALERT_EXCESS_VERSION_SIZE,
@@ -218,14 +214,6 @@ pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCU
 /// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1`
 pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS";
 
-/// Environment variable that sets how many bucket walkers each local disk runs.
-/// The walkers of one disk share its scanner admission, so a peer still sees one
-/// scan session per physical disk. Remote disks keep one walker. The product of
-/// local disks and this factor is what `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS`
-/// caps; the disk count stays the floor. Valid range is `1` through `16`.
-/// - Example: `export RUSTFS_SCANNER_WALKERS_PER_DISK=4`
-pub const ENV_SCANNER_WALKERS_PER_DISK: &str = "RUSTFS_SCANNER_WALKERS_PER_DISK";
-
 /// Environment variable that controls how often scanner object loops yield to the async runtime.
 /// A value of `0` disables this extra object-count yield.
 /// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32`
@@ -240,18 +228,6 @@ pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 4;
 /// Default disk scan concurrency budget.
 pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4;
 
-/// Default bucket walkers per local disk.
-///
-/// One walker per disk keeps today's I/O profile, which is what rotational
-/// media needs. Flash is meant to default higher once a local disk can report
-/// its media type; `rustfs_utils::os::Info::rotational` is never populated
-/// today, so every deployment starts from one walker.
-pub const DEFAULT_SCANNER_WALKERS_PER_DISK: usize = 1;
-
-/// Upper bound for `walkers_per_disk`; a larger factor only multiplies
-/// contention on one spindle or NVMe queue without more disks to spread it.
-pub const MAX_SCANNER_WALKERS_PER_DISK: usize = 16;
-
 /// Default object interval for cooperative scanner yields.
 pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
 

crates/config/src/observability/mod.rs

@@ -51,11 +51,6 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_
 /// PUT records multiple timers and histograms when attribution is active.
 pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
 
-/// Enables sampled in-process GetObject stage summaries without OTLP histograms.
-pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
-pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
-pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
-
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -164,15 +159,6 @@ mod tests {
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
-        assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED");
-        assert_eq!(
-            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE,
-            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"
-        );
-        assert_eq!(
-            ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
-            "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
-        );
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");

crates/e2e_test/src/common.rs

@@ -1542,15 +1542,6 @@ impl RustFSTestClusterEnvironment {
         self.start_node_from_binary(node_idx, &binary_path).await
     }
 
-    /// Start a process and wait only for TCP, allowing tests to bring up a
-    /// quorum one node at a time before the S3 service can become ready.
-    pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        let binary_path = rustfs_binary_path();
-        let volumes_arg = self.build_volumes_arg();
-        self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
-        self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
-    }
-
     /// Start one stopped cluster node with a specific RustFS binary while
     /// preserving the cluster's volume layout and that node's data directory.
     pub async fn start_node_from_binary(

crates/e2e_test/src/conditional_delete_objects_test.rs

@@ -1,381 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! S3 wire semantics for per-object conditional bulk deletion.
-
-use crate::common::RustFSTestEnvironment;
-use aws_sdk_s3::Client;
-use aws_sdk_s3::error::ProvideErrorMetadata;
-use aws_sdk_s3::operation::delete_objects::DeleteObjectsOutput;
-use aws_sdk_s3::operation::put_object::PutObjectOutput;
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
-use std::collections::BTreeSet;
-use std::error::Error;
-
-type TestResult<T = ()> = Result<T, Box<dyn Error + Send + Sync>>;
-type VersionSnapshot = BTreeSet<(String, bool)>;
-
-async fn put(client: &Client, bucket: &str, key: &str, body: &'static [u8]) -> TestResult<PutObjectOutput> {
-    let result = client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(ByteStream::from_static(body))
-        .send()
-        .await?;
-    assert!(result.e_tag().is_some(), "fixture PUT must provide an ETag for the following conditions");
-    Ok(result)
-}
-
-fn identifier(key: &str, etag: Option<&str>, version: Option<&str>) -> TestResult<ObjectIdentifier> {
-    Ok(ObjectIdentifier::builder()
-        .key(key)
-        .set_e_tag(etag.map(str::to_owned))
-        .set_version_id(version.map(str::to_owned))
-        .build()?)
-}
-
-async fn delete_batch(
-    client: &Client,
-    bucket: &str,
-    objects: Vec<ObjectIdentifier>,
-    quiet: bool,
-) -> TestResult<DeleteObjectsOutput> {
-    Ok(client
-        .delete_objects()
-        .bucket(bucket)
-        .delete(Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?)
-        .send()
-        .await?)
-}
-
-async fn assert_body(client: &Client, bucket: &str, key: &str, expected: &[u8]) -> TestResult {
-    let result = client.get_object().bucket(bucket).key(key).send().await?;
-    assert_eq!(result.body.collect().await?.into_bytes().as_ref(), expected);
-    Ok(())
-}
-
-async fn assert_absent(client: &Client, bucket: &str, key: &str) {
-    let error = client
-        .head_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect_err("deleted key must be absent");
-    assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("NotFound"));
-}
-
-async fn versions(client: &Client, bucket: &str, key: &str) -> TestResult<VersionSnapshot> {
-    let result = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
-    assert!(!result.is_truncated().unwrap_or_default(), "focused version fixture fits a single page");
-    let mut snapshot = BTreeSet::new();
-    for version in result.versions().iter().filter(|version| version.key() == Some(key)) {
-        snapshot.insert((version.version_id().expect("version has an identity").to_owned(), false));
-    }
-    for marker in result.delete_markers().iter().filter(|marker| marker.key() == Some(key)) {
-        snapshot.insert((marker.version_id().expect("delete marker has an identity").to_owned(), true));
-    }
-    Ok(snapshot)
-}
-
-#[tokio::test]
-async fn sdk_conditional_delete_objects_mixed_results_overwrites_duplicates_and_quiet() -> TestResult {
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-    let client = env.create_s3_client();
-    let bucket = "conditional-bulk-mixed";
-    client.create_bucket().bucket(bucket).send().await?;
-
-    let old = put(&client, bucket, "overwritten", b"generation A").await?;
-    let current = put(&client, bucket, "overwritten", b"generation B").await?;
-    let matched = put(&client, bucket, "matched", b"matching object").await?;
-    put(&client, bucket, "exists", b"existence object").await?;
-    put(&client, bucket, "unconditional", b"unconditional object").await?;
-    put(&client, bucket, "empty", b"empty condition must preserve this").await?;
-    put(&client, bucket, "empty-quoted", b"quoted empty condition must preserve this").await?;
-    let weak = put(&client, bucket, "weak", b"weak condition must preserve this").await?;
-    let weak_etag = format!("W/\"{}\"", weak.e_tag().expect("PUT supplies an ETag").trim_matches('"'));
-    assert_ne!(old.e_tag(), current.e_tag(), "overwrite fixture must have distinct ETags");
-    let matched_etag = matched.e_tag().expect("PUT supplies an ETag");
-    let result = delete_batch(
-        &client,
-        bucket,
-        vec![
-            identifier("matched", Some(matched_etag), None)?,
-            identifier("overwritten", old.e_tag(), None)?,
-            identifier("matched", Some("stale-etag"), None)?,
-            identifier("matched", Some(matched_etag), None)?,
-            identifier("unconditional", None, None)?,
-            identifier("exists", Some("*"), None)?,
-            identifier("missing", Some("*"), None)?,
-            identifier("empty", Some(""), None)?,
-            identifier("empty-quoted", Some("\"\""), None)?,
-            identifier("weak", Some(&weak_etag), None)?,
-            identifier("unconditional-missing", None, None)?,
-            identifier("missing-specific", Some("opaque-etag"), None)?,
-        ],
-        false,
-    )
-    .await?;
-    assert_eq!(
-        result.deleted().iter().map(|object| object.key()).collect::<Vec<_>>(),
-        [
-            Some("matched"),
-            Some("matched"),
-            Some("unconditional"),
-            Some("exists"),
-            Some("unconditional-missing")
-        ],
-        "duplicate successful inputs retain their separate response entries"
-    );
-    assert_eq!(
-        result
-            .errors()
-            .iter()
-            .map(|error| (error.key(), error.code()))
-            .collect::<Vec<_>>(),
-        [
-            (Some("overwritten"), Some("PreconditionFailed")),
-            (Some("matched"), Some("PreconditionFailed")),
-            (Some("missing"), Some("NoSuchKey")),
-            (Some("empty"), Some("InvalidArgument")),
-            (Some("empty-quoted"), Some("InvalidArgument")),
-            (Some("weak"), Some("InvalidArgument")),
-            (Some("missing-specific"), Some("NoSuchKey")),
-        ]
-    );
-    assert_body(&client, bucket, "overwritten", b"generation B").await?;
-    assert_body(&client, bucket, "empty", b"empty condition must preserve this").await?;
-    assert_body(&client, bucket, "empty-quoted", b"quoted empty condition must preserve this").await?;
-    assert_body(&client, bucket, "weak", b"weak condition must preserve this").await?;
-    for key in ["matched", "unconditional", "exists"] {
-        assert_absent(&client, bucket, key).await;
-    }
-
-    let quiet = put(&client, bucket, "quiet-match", b"quiet matching object").await?;
-    put(&client, bucket, "quiet-rejected", b"quiet rejected object").await?;
-    let result = delete_batch(
-        &client,
-        bucket,
-        vec![
-            identifier("quiet-match", quiet.e_tag(), None)?,
-            identifier("quiet-match", quiet.e_tag(), None)?,
-            identifier("quiet-rejected", Some("stale-etag"), None)?,
-            identifier("missing", Some("*"), None)?,
-        ],
-        true,
-    )
-    .await?;
-    assert!(result.deleted().is_empty(), "Quiet suppresses successful entries");
-    assert_eq!(
-        result
-            .errors()
-            .iter()
-            .map(|error| (error.key(), error.code()))
-            .collect::<Vec<_>>(),
-        [
-            (Some("quiet-rejected"), Some("PreconditionFailed")),
-            (Some("missing"), Some("NoSuchKey"))
-        ]
-    );
-    assert_absent(&client, bucket, "quiet-match").await;
-    assert_body(&client, bucket, "quiet-rejected", b"quiet rejected object").await?;
-    Ok(())
-}
-
-#[tokio::test]
-async fn sdk_conditional_delete_objects_versioning_current_markers_and_explicit_versions() -> TestResult {
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-    let client = env.create_s3_client();
-    for suspended in [false, true] {
-        let bucket = if suspended {
-            "conditional-bulk-suspended"
-        } else {
-            "conditional-bulk-enabled"
-        };
-        client.create_bucket().bucket(bucket).send().await?;
-        client
-            .put_bucket_versioning()
-            .bucket(bucket)
-            .versioning_configuration(
-                VersioningConfiguration::builder()
-                    .status(BucketVersioningStatus::Enabled)
-                    .build(),
-            )
-            .send()
-            .await?;
-        let historical = put(&client, bucket, "history", b"historical payload").await?;
-        let invalid_historical = put(&client, bucket, "invalid", b"historical rejected payload").await?;
-        if suspended {
-            client
-                .put_bucket_versioning()
-                .bucket(bucket)
-                .versioning_configuration(
-                    VersioningConfiguration::builder()
-                        .status(BucketVersioningStatus::Suspended)
-                        .build(),
-                )
-                .send()
-                .await?;
-        }
-        let current = put(&client, bucket, "history", b"current payload").await?;
-        let invalid = put(&client, bucket, "invalid", b"current rejected payload").await?;
-        let invalid_before = versions(&client, bucket, "invalid").await?;
-        let historical_id = historical.version_id().expect("enabled fixture has a UUID version");
-        let invalid_historical_id = invalid_historical
-            .version_id()
-            .expect("enabled rejected fixture has a UUID version");
-        let result = delete_batch(
-            &client,
-            bucket,
-            vec![
-                identifier("history", current.e_tag(), None)?,
-                identifier("history", None, Some(historical_id))?,
-                identifier("history", current.e_tag(), None)?,
-                identifier("invalid", invalid.e_tag(), Some(invalid_historical_id))?,
-                identifier("invalid", Some("*"), Some("null"))?,
-                identifier("invalid", Some("stale-etag"), None)?,
-            ],
-            false,
-        )
-        .await?;
-        assert_eq!(result.deleted().len(), 3, "both current input entries and the historical delete succeed");
-        assert_eq!(
-            result
-                .errors()
-                .iter()
-                .map(|error| (error.key(), error.code()))
-                .collect::<Vec<_>>(),
-            [
-                (Some("invalid"), Some("InvalidRequest")),
-                (Some("invalid"), Some("InvalidRequest")),
-                (Some("invalid"), Some("PreconditionFailed")),
-            ]
-        );
-        assert_eq!(result.errors()[0].version_id(), Some(invalid_historical_id));
-        assert_eq!(result.errors()[1].version_id(), Some("null"));
-        assert_eq!(versions(&client, bucket, "invalid").await?, invalid_before);
-        assert_body(&client, bucket, "invalid", b"current rejected payload").await?;
-
-        let marker = result.deleted()[0]
-            .delete_marker_version_id()
-            .expect("current conditional delete returns a marker");
-        assert_eq!(result.deleted()[0].delete_marker(), Some(true));
-        assert_eq!(result.deleted()[2].delete_marker_version_id(), Some(marker));
-        assert_eq!(result.deleted()[1].version_id(), Some(historical_id));
-        if suspended {
-            assert_eq!(marker, "null", "a suspended current delete returns the created null marker identity");
-        }
-        let history_after = versions(&client, bucket, "history").await?;
-        assert_eq!(
-            history_after.iter().filter(|(_, marker)| *marker).count(),
-            1,
-            "duplicate actions create one marker"
-        );
-        assert!(
-            !history_after.contains(&(historical_id.to_string(), false)),
-            "distinct historical target is removed"
-        );
-        if suspended {
-            assert_eq!(history_after, BTreeSet::from([("null".to_string(), true)]));
-        } else {
-            assert_eq!(
-                history_after,
-                BTreeSet::from([
-                    (current.version_id().expect("current enabled object has a UUID").to_string(), false),
-                    (marker.to_string(), true),
-                ]),
-                "enabled conditional current deletion retains the current data version behind the marker"
-            );
-        }
-
-        let result = delete_batch(
-            &client,
-            bucket,
-            vec![
-                identifier("history", Some("*"), None)?,
-                identifier("history", current.e_tag(), None)?,
-                identifier("missing", Some("*"), None)?,
-            ],
-            false,
-        )
-        .await?;
-        assert!(
-            result.deleted().is_empty(),
-            "current markers and absent objects cannot satisfy a live-object condition"
-        );
-        assert_eq!(result.errors().len(), 3);
-        assert!(result.errors().iter().all(|error| error.code() == Some("NoSuchKey")));
-        assert_eq!(
-            versions(&client, bucket, "history").await?,
-            history_after,
-            "failed conditions create no marker"
-        );
-        assert!(versions(&client, bucket, "missing").await?.is_empty());
-
-        put(&client, bucket, "exists", b"versioned existence payload").await?;
-        let result = delete_batch(&client, bucket, vec![identifier("exists", Some("*"), None)?], false).await?;
-        assert!(result.errors().is_empty());
-        assert_eq!(result.deleted().len(), 1);
-        assert_eq!(result.deleted()[0].delete_marker(), Some(true));
-        assert_eq!(
-            versions(&client, bucket, "exists")
-                .await?
-                .iter()
-                .filter(|(_, marker)| *marker)
-                .count(),
-            1
-        );
-
-        if !suspended {
-            let directory = put(&client, bucket, "dir/", b"directory null-version payload").await?;
-            assert_eq!(
-                versions(&client, bucket, "dir/").await?,
-                BTreeSet::from([("null".to_string(), false)]),
-                "the directory PUT fixture must identify the synthetic null data version"
-            );
-            let result = delete_batch(
-                &client,
-                bucket,
-                vec![identifier("dir/", directory.e_tag(), None)?, identifier("dir/", None, None)?],
-                false,
-            )
-            .await?;
-            assert!(result.errors().is_empty(), "both distinct directory actions must succeed");
-            assert_eq!(result.deleted().len(), 2);
-            assert_eq!(result.deleted()[0].key(), Some("dir/"));
-            assert_eq!(result.deleted()[1].key(), Some("dir/"));
-            assert_eq!(result.deleted()[0].delete_marker(), Some(true));
-            let marker = result.deleted()[0]
-                .delete_marker_version_id()
-                .expect("conditional current-directory deletion creates a versioned marker");
-            assert!(
-                !uuid::Uuid::parse_str(marker)?.is_nil(),
-                "enabled current-directory deletion creates a UUID marker"
-            );
-            assert_eq!(result.deleted()[1].delete_marker(), None);
-            assert_eq!(result.deleted()[1].delete_marker_version_id(), None);
-            assert_eq!(result.deleted()[1].version_id(), None);
-            assert_eq!(
-                versions(&client, bucket, "dir/").await?,
-                BTreeSet::from([(marker.to_string(), true)]),
-                "current-marker creation and synthetic null-version removal are distinct actions: one marker remains and the null data version is gone"
-            );
-        }
-    }
-    Ok(())
-}

crates/e2e_test/src/distributed/harness.rs

@@ -860,12 +860,10 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T
         if status.is_success() {
             break response;
         }
-        // The init stage answers 503 + Retry-After while the cluster capability
-        // proof is still missing; older binaries surfaced the same condition as
-        // a 500 carrying the proof marker.
-        let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE
-            || (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof"));
-        if waiting_for_readiness && Instant::now() < deadline {
+        if status == StatusCode::INTERNAL_SERVER_ERROR
+            && response.contains("requires a live fleet capability proof")
+            && Instant::now() < deadline
+        {
             sleep(Duration::from_secs(1)).await;
             continue;
         }

crates/e2e_test/src/fake_s3_target/README.md

@@ -6,7 +6,7 @@ This module is the shared failure-injection boundary for replication end-to-end
 
 Supported data operations are HeadBucket, GetBucketVersioning, ListObjectsV2, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets created with `create_bucket` are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
 
-`create_bucket_with_object_lock(name)` creates a versioned bucket whose GetObjectLockConfiguration reports `Enabled`; every other bucket answers `ObjectLockConfigurationNotFoundError`, the code RustFS's replication-check classifies as "not enabled". Four switches model remote-target behaviors the fleet has shown, so the outbound target matrix (`crates/e2e_test/src/replication_target_matrix_test.rs`) can replicate every object shape against each: `assign_own_version_ids(true)` ignores the source version id and mints its own (AWS S3 / Wasabi); `reject_aws_chunked_uploads(true)` refuses any PutObject or UploadPart announcing `aws-chunked` framing (`Content-Encoding: aws-chunked`, an `x-amz-trailer`, or a `STREAMING-*` payload hash) with `InvalidRequest` before the body is read (SeaweedFS 3.97, rustfs#6853); `require_checksum_for_object_lock(true)` rejects a PutObject carrying any `x-amz-object-lock-*` header unless it also carries `Content-MD5`, an `x-amz-checksum-*` header, or `x-amz-sdk-checksum-algorithm` (AWS S3 / MinIO, rustfs#7082). `ignore_source_etags(true)` ignores the `x-rustfs-source-etag` / `x-minio-source-etag` replication headers and reports the ETag of the received bytes (AWS S3 and other generic S3 services; the default adopts the source ETag like RustFS/MinIO). Independently of that switch, a `Content-MD5` header is always verified against the body and a mismatch answers `BadDigest`.
+`create_bucket_with_object_lock(name)` creates a versioned bucket whose GetObjectLockConfiguration reports `Enabled`; every other bucket answers `ObjectLockConfigurationNotFoundError`, the code RustFS's replication-check classifies as "not enabled". Three switches model remote-target behaviors the fleet has shown, so the outbound target matrix (`crates/e2e_test/src/replication_target_matrix_test.rs`) can replicate every object shape against each: `assign_own_version_ids(true)` ignores the source version id and mints its own (AWS S3 / Wasabi); `reject_aws_chunked_uploads(true)` refuses any PutObject or UploadPart announcing `aws-chunked` framing (`Content-Encoding: aws-chunked`, an `x-amz-trailer`, or a `STREAMING-*` payload hash) with `InvalidRequest` before the body is read (SeaweedFS 3.97, rustfs#6853); `require_checksum_for_object_lock(true)` rejects a PutObject carrying any `x-amz-object-lock-*` header unless it also carries `Content-MD5`, an `x-amz-checksum-*` header, or `x-amz-sdk-checksum-algorithm` (AWS S3 / MinIO, rustfs#7082). Independently of that switch, a `Content-MD5` header is always verified against the body and a mismatch answers `BadDigest`.
 
 `create_bucket_with_mode(name, BucketMode::Unversioned)` models a plain migration source: PUT overwrites in place, DELETE removes the key without a delete marker, GetBucketVersioning reports no status, and no `x-amz-version-id` is returned by PUT, GET, HEAD, tagging, or multipart completion. The only `versionId` such a bucket accepts is `null`; any other value is rejected with `InvalidArgument`. The mode is fixed at creation.
 

crates/e2e_test/src/fake_s3_target/mod.rs

@@ -537,10 +537,6 @@ struct StoreState {
     /// PutObject carrying any `x-amz-object-lock-*` header must also carry
     /// `Content-MD5` or an `x-amz-checksum-*` header.
     require_checksum_for_object_lock: bool,
-    /// Models AWS S3 and other generic S3 services: the `x-rustfs-source-etag`
-    /// / `x-minio-source-etag` replication headers are ignored and every
-    /// stored object reports the ETag computed from the bytes it received.
-    ignore_source_etags: bool,
     /// Models Wasabi (rustfs/backlog#2340): a version-addressed DELETE of a
     /// version id the target never had answers 404 `NoSuchVersion` instead of
     /// the idempotent 204 RustFS/MinIO give.
@@ -1056,13 +1052,6 @@ impl FakeS3Target {
         lock(&self.backend.store).require_checksum_for_object_lock = enabled;
     }
 
-    /// AWS-like mode: ignore the replication source-ETag headers and report
-    /// the ETag of the received bytes, as AWS S3 does for SSE-S3 and
-    /// unencrypted objects. The default adopts the source ETag (RustFS/MinIO).
-    pub fn ignore_source_etags(&self, enabled: bool) {
-        lock(&self.backend.store).ignore_source_etags = enabled;
-    }
-
     pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
         lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
     }
@@ -2455,13 +2444,12 @@ impl S3 for FakeBackend {
             .map_err(|_| s3s::s3_error!(ServiceUnavailable, "fake target body limiter closed"))?;
         let headers = req.headers;
         let input = req.input;
-        let (assign_own, drop_unlisted, require_checksum, ignore_source_etags, limits, versioned) = {
+        let (assign_own, drop_unlisted, require_checksum, limits, versioned) = {
             let state = lock(&self.store);
             (
                 state.assign_own_version_ids,
                 state.drop_unlisted_replication_headers,
                 state.require_checksum_for_object_lock,
-                state.ignore_source_etags,
                 state.limits,
                 bucket_versioned(&state, &input.bucket),
             )
@@ -2501,7 +2489,7 @@ impl S3 for FakeBackend {
         } else {
             NULL_VERSION_ID.to_string()
         };
-        let e_tag = match source_etag(&headers)?.filter(|_| !ignore_source_etags) {
+        let e_tag = match source_etag(&headers)? {
             Some(value) => value,
             None => {
                 let (digest, permit) = md5_digest(body.clone(), body_permit).await?;
@@ -3124,10 +3112,7 @@ impl S3 for FakeBackend {
             .collect();
         let (body, digests, _body_permits) = assemble_multipart(assembly_parts, total_len, _body_permits).await?;
         let part_count = requested.len();
-        let ignore_source_etags = lock(&self.store).ignore_source_etags;
-        let e_tag = source_etag(&headers)?
-            .filter(|_| !ignore_source_etags)
-            .unwrap_or_else(|| format!("{}-{part_count}", md5_hex(digests)));
+        let e_tag = source_etag(&headers)?.unwrap_or_else(|| format!("{}-{part_count}", md5_hex(digests)));
         let version = ObjectVersion {
             version_id: upload.version_id.clone(),
             body,

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1032,7 +1032,7 @@ mod tests {
 
         let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
         let heal_url = format!("{}/rustfs/admin/v3/heal/{}?forceStart=true", env.url, bucket);
-        start_root_heal_when_control_ready(&heal_url, heal_body, &env.access_key, &env.secret_key)
+        signed_admin_post(&heal_url, Some(heal_body), &env.access_key, &env.secret_key)
             .await
             .expect("admin deep heal should be accepted");
 
@@ -1191,59 +1191,11 @@ mod tests {
         .into())
     }
 
-    fn scanner_cycle_completion_budget(
-        cycle_end: u64,
-        interval: Option<u64>,
-        now: u64,
-        active: bool,
-    ) -> Result<Duration, &'static str> {
-        if cycle_end > now {
-            return Err("scanner cycle timestamp is in the future");
-        }
-        if active {
-            return Ok(Duration::from_secs(60));
-        }
-        let interval = interval
-            .filter(|interval| *interval != 0)
-            .ok_or("scanner schedule has no positive interval")?;
-        let next_cycle = cycle_end
-            .checked_add(interval)
-            .ok_or("scanner schedule timestamp overflows")?;
-        let budget = next_cycle
-            .saturating_sub(now)
-            .checked_add(60)
-            .ok_or("scanner completion budget overflows")?;
-        Ok(Duration::from_secs(budget))
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
-        assert_eq!(
-            scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
-            Ok(Duration::from_secs(259)),
-        );
-        assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
-        assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
-        assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
-    }
-
-    #[test]
-    fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
-        for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
-            assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
-        }
-    }
-
     async fn wait_for_scanner_cycle_after(
         cluster: &RustFSTestClusterEnvironment,
         previous_cycle_end: u64,
     ) -> Result<u64, Box<dyn Error + Send + Sync>> {
-        let mut deadline = Instant::now() + Duration::from_secs(60);
-        let mut schedule_accounted = false;
+        let deadline = Instant::now() + Duration::from_secs(60);
         loop {
             let mut latest_cycle_end = 0;
             let mut versions_observed = false;
@@ -1285,36 +1237,14 @@ mod tests {
                 let versions_scanned = metrics["versions_scanned"]
                     .as_u64()
                     .ok_or("scanner status is missing its version-coverage counter")?;
-                let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
-                if cycle_end > now {
-                    return Err("scanner cycle timestamp is in the future".into());
-                }
-                let schedule = &status["cycle_schedule"];
-                if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
-                    let active = metrics["current_cycle_active"]
-                        .as_bool()
-                        .ok_or("scanner status is missing its active-cycle flag")?;
-                    if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
-                        let interval = schedule["effective_interval_seconds"].as_u64();
-                        let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
-                        deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
-                        schedule_accounted = true;
-                    }
-                }
-                // Superseded walks complete all sets while withholding quota authority.
-                // Deferred and partial attempts do not certify completed scanning.
-                if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
-                    latest_cycle_end = latest_cycle_end.max(cycle_end);
-                }
+                latest_cycle_end = latest_cycle_end.max(cycle_end);
                 versions_observed |= versions_scanned > 0;
                 observations.push(format!(
-                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
+                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
                     metrics["current_cycle"],
                     metrics["current_cycle_active"],
                     metrics["leader_lock_state"],
                     metrics["last_cycle_result"],
-                    metrics["last_cycle_partial_reason"],
-                    schedule,
                 ));
             }
             // The coordinator records cycle completion, but remote workers
@@ -1625,8 +1555,7 @@ mod tests {
         }
 
         if background_enabled {
-            let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
-            wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
+            wait_for_scanner_cycle_after(&cluster, 0).await?;
         }
 
         cluster.stop_node(1)?;
@@ -1969,16 +1898,11 @@ mod tests {
             let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
             let status: serde_json::Value = serde_json::from_str(&status_body)
                 .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?;
-            let admin_active = background_rejoin_heal_evidence
-                || (status["state"].as_str() == Some("active")
-                    && status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1));
-            if admin_active && (background_enabled || only_admin_heal_is_active(&status)) {
+            if background_enabled || only_admin_heal_is_active(&status) {
                 break status;
             }
             if Instant::now() >= partial_deadline {
-                return Err(
-                    format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(),
-                );
+                return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into());
             }
             sleep(Duration::from_millis(50)).await;
         };
@@ -2638,157 +2562,4 @@ mod tests {
         };
         assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
     }
-    async fn two_node_heal_status(
-        cluster: &RustFSTestClusterEnvironment,
-        bucket: Option<&str>,
-        recreate: bool,
-        scan_mode: u8,
-    ) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
-        let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
-        let options = serde_json::json!({
-            "recursive": true, "dryRun": false, "remove": false,
-            "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
-        });
-        let started =
-            start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
-        let started: serde_json::Value = serde_json::from_str(&started)?;
-        let token = started["clientToken"]
-            .as_str()
-            .ok_or_else(|| format!("missing heal token: {started}"))?;
-        let status_url = format!("{url}?clientToken={token}");
-        let deadline = Instant::now() + Duration::from_secs(240);
-        loop {
-            let body = timeout(
-                Duration::from_secs(20),
-                signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
-            )
-            .await??;
-            let status: serde_json::Value = serde_json::from_str(&body)?;
-            match status["summary"].as_str() {
-                Some("finished" | "stopped") => return Ok(status),
-                Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
-                _ => return Err(format!("heal did not reach a terminal state: {status}").into()),
-            }
-        }
-    }
-
-    async fn verify_two_node_directory_recovery(
-        remove_bucket: bool,
-        scan_mode: u8,
-        bucket_scoped: bool,
-        recreate: bool,
-    ) -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
-        // Isolate explicit administrator recovery from scanner, MRF, and disk healing.
-        for key in [
-            "RUSTFS_HEAL_AUTO_HEAL_ENABLE",
-            "RUSTFS_HEAL_MRF_ENABLE",
-            "RUSTFS_SCANNER_ENABLED",
-            "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
-        ] {
-            cluster.set_env(key, "false");
-        }
-        cluster.start().await?;
-        let source = cluster.create_s3_client(0)?;
-        let target = cluster.create_s3_client(1)?;
-        let bucket = "two-node-directory-recovery";
-        source.create_bucket().bucket(bucket).send().await?;
-        let objects = [
-            ("empty", Vec::new()),
-            ("nested/inline", vec![0x3c; 1024]),
-            ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
-        ];
-        for (key, bytes) in &objects {
-            source
-                .put_object()
-                .bucket(bucket)
-                .key(*key)
-                .body(ByteStream::from(bytes.clone()))
-                .send()
-                .await?;
-        }
-        let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
-        let original = objects
-            .iter()
-            .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
-            .collect::<Result<Vec<_>, _>>()?;
-        for census in &original {
-            assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
-        }
-        assert!(
-            !original[2].present_part_fingerprints.is_empty(),
-            "large fixture must exercise external parts"
-        );
-        let removed = Path::new(&cluster.temp_dir).join("removed");
-        tokio::fs::create_dir(&removed).await?;
-        if remove_bucket {
-            tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
-        } else {
-            for (index, (key, _)) in objects.iter().enumerate() {
-                tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
-            }
-        }
-        let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?;
-        assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}");
-        assert_eq!(status["outcome"]["execution"]["state"], "completed");
-        assert_eq!(status["outcome"]["coverage"], "complete");
-        for ((key, _), before) in objects.iter().zip(&original) {
-            let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
-            assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
-            assert_eq!(
-                after.present_part_fingerprints, before.present_part_fingerprints,
-                "external shard bytes must match: {key}"
-            );
-            assert_eq!(
-                after.inline_data_fingerprint, before.inline_data_fingerprint,
-                "inline shard bytes must match: {key}"
-            );
-        }
-        cluster.stop_node(0)?;
-        for (key, expected) in &objects {
-            let actual = timeout(Duration::from_secs(30), async {
-                target
-                    .get_object()
-                    .bucket(bucket)
-                    .key(*key)
-                    .send()
-                    .await?
-                    .body
-                    .collect()
-                    .await
-                    .map(|body| body.into_bytes())
-                    .map_err(Box::<dyn Error + Send + Sync>::from)
-            })
-            .await??;
-            assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
-            assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
-        }
-        Ok(())
-    }
-
-    #[tokio::test]
-    async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, true, true).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, false, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(true, 2, true, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(false, 1, false, false).await
-    }
-
-    #[tokio::test]
-    async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
-        verify_two_node_directory_recovery(false, 2, false, false).await
-    }
 }

crates/e2e_test/src/internode_rpc_signature_e2e_test.rs

@@ -264,7 +264,7 @@ async fn call_make_volume_response(
     request: MakeVolumeRequest,
     headers: HeaderMap,
 ) -> Result<Response<MakeVolumeResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(url)
+    let mut client = node_service_time_out_client_no_auth(&url.to_string())
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
 }
 
 async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
-    let mut client = node_service_time_out_client_no_auth(url)
+    let mut client = node_service_time_out_client_no_auth(&url.to_string())
         .await
         .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
     let mut rpc_request = Request::new(PingRequest {

crates/e2e_test/src/lib.rs

@@ -179,9 +179,6 @@ mod compression_test;
 #[cfg(test)]
 mod delete_objects_versioning_test;
 
-#[cfg(test)]
-mod conditional_delete_objects_test;
-
 #[cfg(test)]
 mod delete_authorization_test;
 

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use crate::common::RustFSTestClusterEnvironment;
 use aws_sdk_s3::Client;
 use aws_sdk_s3::error::SdkError;
 use bytes::Bytes;
@@ -185,113 +185,21 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
             return Ok(());
         }
         if tokio::time::Instant::now() >= deadline {
-            let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
-            return Err(format!(
-                "read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
-            )
-            .into());
+            return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
         }
         tokio::time::sleep(std::time::Duration::from_millis(100)).await;
     }
 }
 
-#[tokio::test]
-async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult {
-    crate::common::init_logging();
-    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
-    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
-    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-    cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false");
-    cluster.start().await?;
-    let clients = cluster
-        .create_all_clients()?
-        .into_iter()
-        .map(|client| {
-            Client::from_conf(
-                client
-                    .config()
-                    .to_builder()
-                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                    .build(),
-            )
-        })
-        .collect::<Vec<_>>();
-    let bucket = "cold-degraded-mutations";
-    clients[0].create_bucket().bucket(bucket).send().await?;
-    // No object mutation may warm a process-wide configuration probe before
-    // the missing peer is introduced. Three shards still satisfy EC:2 writes.
-    cluster.stop_node(3)?;
-    let admission_deadline = Instant::now() + Duration::from_secs(30);
-    for client in &clients[..3] {
-        loop {
-            match client
-                .get_object()
-                .bucket(bucket)
-                .key("never-created-readiness-probe")
-                .send()
-                .await
-            {
-                Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break,
-                Err(error)
-                    if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
-                        && Instant::now() < admission_deadline =>
-                {
-                    tokio::time::sleep(Duration::from_millis(100)).await;
-                }
-                Err(error) => return Err(error.into()),
-                Ok(_) => panic!("a cold readiness probe must not create an object"),
-            }
-        }
-    }
-    let body = b"first write while one peer is offline";
-    clients[0]
-        .put_object()
-        .bucket(bucket)
-        .key("object")
-        .body(Bytes::from_static(body).into())
-        .send()
-        .await?;
-    for client in &clients[..3] {
-        assert_quorum_object_body(client, bucket, "object", body).await?;
-    }
-    // This coordinator has not previously committed an object mutation either.
-    clients[1].delete_object().bucket(bucket).key("object").send().await?;
-    let error = clients[2]
-        .get_object()
-        .bucket(bucket)
-        .key("object")
-        .send()
-        .await
-        .expect_err("acknowledged deletion must be visible at another entrypoint");
-    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey"));
-
-    cluster.stop_node(2)?;
-    let error = clients[0]
-        .put_object()
-        .bucket(bucket)
-        .key("below-quorum")
-        .body(Bytes::from_static(body).into())
-        .send()
-        .await
-        .expect_err("removing the cache gate must not weaken storage write quorum");
-    assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
-    cluster.stop();
-    Ok(())
-}
-
 #[tokio::test]
 async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
     crate::common::init_logging();
 
-    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
+    for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
         let read_quorum = node_count - parity;
         let write_quorum = read_quorum + usize::from(read_quorum == parity);
         let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
         cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
-        if parity == 0 {
-            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
-        }
         // Wait for every seed fanout before removing any physical shard.
         cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
         cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -430,277 +338,6 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
     Ok(())
 }
 
-#[tokio::test]
-async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
-    crate::common::init_logging();
-    for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
-        let read_quorum = node_count - parity;
-        let write_quorum = read_quorum + usize::from(read_quorum == parity);
-        let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
-        cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
-        if parity == 0 {
-            cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
-        }
-        cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-        cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-        cluster.set_env("RUST_LOG", "warn");
-        cluster.start().await?;
-        let clients = cluster
-            .create_all_clients()?
-            .into_iter()
-            .map(|client| {
-                Client::from_conf(
-                    client
-                        .config()
-                        .to_builder()
-                        .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                        .build(),
-                )
-            })
-            .collect::<Vec<_>>();
-        let bucket = format!("cold-read-quorum-{node_count}-{parity}");
-        clients[0].create_bucket().bucket(&bucket).send().await?;
-        let small = b"cold startup follows the stored EC layout".to_vec();
-        let large = vec![73; 256 * 1024];
-        for (key, body) in [
-            ("warm-small", &small),
-            ("warm-large", &large),
-            ("cold-large", &large),
-            ("below-quorum", &large),
-        ] {
-            clients[0]
-                .put_object()
-                .bucket(&bucket)
-                .key(key)
-                .body(Bytes::copy_from_slice(body).into())
-                .send()
-                .await?;
-            for node in &cluster.nodes {
-                let census =
-                    crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
-                assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
-                assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
-            }
-        }
-        for node in 0..node_count {
-            cluster.stop_node(node)?;
-        }
-        // Rebuild every process's in-memory state below read quorum, then
-        // cross the EC boundary. A format bootstrap must not admit S3 reads
-        // merely because half the peers are online (e.g. six disks, EC:2).
-        for node in 0..read_quorum - 1 {
-            cluster.start_node_process(node).await?;
-        }
-        let error = clients[0]
-            .get_object()
-            .bucket(&bucket)
-            .key("below-quorum")
-            .send()
-            .await
-            .expect_err("cold startup below the configured EC read quorum must stay unavailable");
-        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-        cluster.start_node_process(read_quorum - 1).await?;
-        wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
-        for client in clients.iter().take(read_quorum) {
-            assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
-            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
-            let write = client
-                .put_object()
-                .bucket(&bucket)
-                .key("cold-quorum-write")
-                .body(Bytes::copy_from_slice(&small).into())
-                .send()
-                .await;
-            if read_quorum >= write_quorum {
-                write?;
-                assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
-            } else {
-                let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
-                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            }
-        }
-        for node in read_quorum..node_count {
-            cluster.start_node(node).await?;
-        }
-        for client in &clients {
-            assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
-        }
-    }
-    Ok(())
-}
-
-#[tokio::test]
-async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
-    crate::common::init_logging();
-    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
-    // Explicit URLs are consumed in order. Place one drive from each node in
-    // every set, matching a four-host, six-drive-per-host deployment.
-    let volumes = (0..6)
-        .flat_map(|drive| {
-            cluster
-                .nodes
-                .iter()
-                .map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
-        })
-        .collect::<Vec<_>>()
-        .join(" ");
-    cluster.set_env("RUSTFS_VOLUMES", volumes);
-    cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
-    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
-    cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
-    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
-    cluster.set_env("RUST_LOG", "error");
-    cluster.start().await?;
-    let clients = cluster
-        .create_all_clients()?
-        .into_iter()
-        .map(|client| {
-            Client::from_conf(
-                client
-                    .config()
-                    .to_builder()
-                    .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
-                    .build(),
-            )
-        })
-        .collect::<Vec<_>>();
-    let bucket = "cold-start-read-quorum";
-    clients[0].create_bucket().bucket(bucket).send().await?;
-    let mut fixtures = Vec::new();
-    let mut covered = [[false; 2]; 6];
-    for index in 0..128 {
-        let key = match index {
-            0 => "warm-small".to_string(),
-            1 => "warm-large".to_string(),
-            _ => format!("cold-{index}"),
-        };
-        let size_class = index % 2;
-        let size = if size_class == 0 { 71 } else { 256 * 1024 };
-        let body = (0..size)
-            .map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
-            .collect::<Vec<_>>();
-        clients[0]
-            .put_object()
-            .bucket(bucket)
-            .key(&key)
-            .body(Bytes::copy_from_slice(&body).into())
-            .send()
-            .await?;
-        for node in &cluster.nodes {
-            let mut complete_shards = 0;
-            for (drive, path) in node.data_dirs.iter().enumerate() {
-                let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
-                if census.has_xl_meta {
-                    assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
-                    assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
-                    covered[drive][size_class] = true;
-                    complete_shards += 1;
-                }
-            }
-            assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
-        }
-        fixtures.push((key, body));
-        if covered.iter().flatten().all(|covered| *covered) {
-            break;
-        }
-    }
-    assert!(
-        covered.iter().flatten().all(|covered| *covered),
-        "fixtures must cover all six sets with inline and large objects"
-    );
-
-    for first in 0..4 {
-        for second in first + 1..4 {
-            for node in 0..4 {
-                cluster.stop_node(node)?;
-            }
-            cluster.start_node_process(first).await?;
-            let error = clients[first]
-                .get_object()
-                .bucket(bucket)
-                .key("warm-large")
-                .send()
-                .await
-                .expect_err("one node cannot satisfy read quorum after a cold start");
-            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            cluster.start_node_process(second).await?;
-            wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
-            for node in [first, second] {
-                for (key, body) in &fixtures {
-                    assert_quorum_object_body(&clients[node], bucket, key, body).await?;
-                }
-                let range = clients[node]
-                    .get_object()
-                    .bucket(bucket)
-                    .key("warm-large")
-                    .range("bytes=17-1023")
-                    .send()
-                    .await?;
-                assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
-                let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
-                for (key, _) in &fixtures {
-                    assert!(
-                        listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
-                        "listing omitted {key}"
-                    );
-                }
-                let error = clients[node]
-                    .put_object()
-                    .bucket(bucket)
-                    .key("recovery-write")
-                    .body(Bytes::from_static(b"after recovery").into())
-                    .send()
-                    .await
-                    .expect_err("two nodes must not authorize a write");
-                assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            }
-            let error = clients[first]
-                .delete_object()
-                .bucket(bucket)
-                .key("warm-large")
-                .send()
-                .await
-                .expect_err("two nodes must not authorize a delete");
-            assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
-            assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
-
-            let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
-            cluster.start_node(remaining[0]).await?;
-            let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
-            loop {
-                match clients[first]
-                    .put_object()
-                    .bucket(bucket)
-                    .key("recovery-write")
-                    .body(Bytes::from_static(b"after recovery").into())
-                    .send()
-                    .await
-                {
-                    Ok(_) => break,
-                    Err(error)
-                        if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
-                            && tokio::time::Instant::now() < deadline =>
-                    {
-                        tokio::time::sleep(Duration::from_millis(200)).await
-                    }
-                    Err(error) => return Err(error.into()),
-                }
-            }
-            for node in [first, second, remaining[0]] {
-                assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
-            }
-            cluster.start_node(remaining[1]).await?;
-            for client in &clients {
-                for (key, body) in &fixtures {
-                    assert_quorum_object_body(client, bucket, key, body).await?;
-                }
-                assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
-            }
-        }
-    }
-    Ok(())
-}
-
 async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
     client
         .put_object()

crates/e2e_test/src/quota_test.rs

@@ -851,20 +851,13 @@ mod integration_tests {
             .delete(
                 aws_sdk_s3::types::Delete::builder()
                     .set_objects(Some(objects))
-                    // Quiet mode hides successful entries, so the batch must be
-                    // verbose for the assertion on the deleted list below.
-                    .quiet(false)
+                    .quiet(true)
                     .build()
                     .unwrap(),
             )
             .send()
             .await?;
 
-        assert!(
-            delete_result.errors().is_empty(),
-            "batch delete must not report errors: {:?}",
-            delete_result.errors()
-        );
         assert_eq!(delete_result.deleted().len(), 2);
 
         // Now should be able to upload again (quota freed up)

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -113,13 +113,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs"))
     }
 
-    async fn acquire_quota_mutation_fence(
-        &self,
-        _request: Request<SnapshotLeaseRequest>,
-    ) -> Result<Response<SnapshotLeaseResponse>, Status> {
-        Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs"))
-    }
-
     async fn renew_snapshot_lease(
         &self,
         _request: Request<SnapshotLeaseRenewRequest>,
@@ -134,13 +127,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs"))
     }
 
-    async fn release_quota_mutation_fence(
-        &self,
-        _request: Request<SnapshotLeaseReleaseRequest>,
-    ) -> Result<Response<SnapshotLeaseMutationResponse>, Status> {
-        Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs"))
-    }
-
     async fn acquire_scanner_publication_lease(
         &self,
         _request: Request<ScannerPublicationLeaseRequest>,
@@ -479,13 +465,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
-    async fn recover_delete_transaction(
-        &self,
-        _request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
-    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
-        Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
-    }
-
     async fn settle_part_transaction(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,
@@ -555,13 +534,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
-    async fn rename_data_with_quota_fence(
-        &self,
-        _request: Request<rustfs_protos::proto_gen::node_service::RenameDataRequest>,
-    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RenameDataResponse>, Status> {
-        Err(Status::unimplemented("lock-only test server"))
-    }
-
     async fn make_volumes(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::MakeVolumesRequest>,
@@ -660,13 +632,6 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
-    async fn guarded_delete_versions(
-        &self,
-        _request: Request<rustfs_protos::proto_gen::node_service::GuardedDeleteVersionsRequest>,
-    ) -> Result<Response<rustfs_protos::proto_gen::node_service::DeleteVersionsResponse>, Status> {
-        Err(Status::unimplemented("lock-only test server"))
-    }
-
     async fn delete_versions(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::DeleteVersionsRequest>,

crates/e2e_test/src/replication_extension_test.rs

@@ -3945,15 +3945,7 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         (sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()),
         (large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()),
     ];
-    // Managed-SSE listings may omit ETags. Check the replicated version's body
-    // before deleting it rather than comparing keyless listing fingerprints.
-    for (key, version_id) in &purged {
-        let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?;
-        let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into();
-        wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?;
-        let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?;
-        assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}");
-    }
+    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
 
     for (key, version_id) in &purged {
         source_client
@@ -3964,6 +3956,10 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
             .send()
             .await?;
     }
+    assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
+    let target_state = list_replication_state(&target_client, &target_bucket).await?;
+    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
+
     // The purge state is applied on the source asynchronously after the target
     // acknowledges the delete; only then does the retained version go away and
     // the bucket become deletable. A listing that is empty while DeleteBucket
@@ -3988,9 +3984,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
         }
     }
 
-    let target_state = list_replication_state(&target_client, &target_bucket).await?;
-    assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
-
     Ok(())
 }
 

crates/e2e_test/src/replication_target_matrix_test.rs

@@ -45,10 +45,9 @@ use aws_sdk_s3::error::ProvideErrorMetadata;
 use aws_sdk_s3::primitives::{ByteStream, DateTime};
 use aws_sdk_s3::types::{
     Checksum, ChecksumAlgorithm, CompletedMultipartUpload, CompletedPart, ObjectAttributes, ObjectLockLegalHold,
-    ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ServerSideEncryption, Tag, Tagging,
+    ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, Tag, Tagging,
 };
 use bytes::Bytes;
-use md5::{Digest as _, Md5};
 use std::error::Error;
 use std::time::{SystemTime, UNIX_EPOCH};
 use tokio::time::{Duration, sleep, timeout};
@@ -74,20 +73,14 @@ enum TargetMode {
     /// land, and every version-addressed mutation must resolve the replica
     /// through the target-version ledger.
     MintOwnVersionIds,
-    /// AWS S3 and other generic S3 services: ignores the replication
-    /// source-ETag headers and reports the MD5 of the plaintext it received.
-    /// Every single-part replica ETag is therefore checked against the source
-    /// ETag for real (the other modes echo the source ETag back).
-    ComputesOwnEtags,
 }
 
 impl TargetMode {
-    const ALL: [TargetMode; 5] = [
+    const ALL: [TargetMode; 4] = [
         TargetMode::Baseline,
         TargetMode::RejectAwsChunked,
         TargetMode::RequireChecksumWithObjectLock,
         TargetMode::MintOwnVersionIds,
-        TargetMode::ComputesOwnEtags,
     ];
 
     fn apply(self, target: &FakeS3Target) {
@@ -99,7 +92,6 @@ impl TargetMode {
                 target.assign_own_version_ids(true);
                 target.reject_unknown_version_deletes(true);
             }
-            TargetMode::ComputesOwnEtags => target.ignore_source_etags(true),
         }
     }
 
@@ -109,7 +101,6 @@ impl TargetMode {
             TargetMode::RejectAwsChunked => "reject-aws-chunked",
             TargetMode::RequireChecksumWithObjectLock => "require-checksum-object-lock",
             TargetMode::MintOwnVersionIds => "mint-own-version-ids",
-            TargetMode::ComputesOwnEtags => "computes-own-etags",
         }
     }
 }
@@ -135,14 +126,10 @@ enum ObjectShape {
     /// Single-part object uploaded with `x-amz-checksum-sha256`; the replica
     /// must carry the same header (rustfs/backlog#2340).
     Checksummed,
-    /// Single-part SSE-S3 object. Its ETag is a keyed fingerprint, not the
-    /// plaintext MD5 (docs/operations/encrypted-etags.md), while the replica
-    /// travels as plaintext under `x-amz-server-side-encryption: AES256`.
-    SseS3,
 }
 
 impl ObjectShape {
-    const ALL: [ObjectShape; 9] = [
+    const ALL: [ObjectShape; 8] = [
         ObjectShape::Empty,
         ObjectShape::Plain,
         ObjectShape::Retention,
@@ -151,7 +138,6 @@ impl ObjectShape {
         ObjectShape::LockedMultipart,
         ObjectShape::OdmPreservedMd5Multipart,
         ObjectShape::Checksummed,
-        ObjectShape::SseS3,
     ];
 
     fn key(self) -> &'static str {
@@ -164,7 +150,6 @@ impl ObjectShape {
             ObjectShape::LockedMultipart => "matrix/locked-multipart.bin",
             ObjectShape::OdmPreservedMd5Multipart => "matrix/odm-preserved-md5.bin",
             ObjectShape::Checksummed => "matrix/checksummed.bin",
-            ObjectShape::SseS3 => "matrix/sse-s3.bin",
         }
     }
 
@@ -248,24 +233,6 @@ impl ObjectShape {
                     .await?;
                 Ok(body)
             }
-            ObjectShape::SseS3 => {
-                let body = payload(56 * 1024, 0x77);
-                let put = client
-                    .put_object()
-                    .bucket(bucket)
-                    .key(key)
-                    .body(ByteStream::from(body.clone()))
-                    .server_side_encryption(ServerSideEncryption::Aes256)
-                    .send()
-                    .await?;
-                // The cell only proves something while the source ETag is
-                // not the content MD5 a generic target reports back.
-                let plaintext_md5 = hex_simd::encode_to_string(Md5::digest(&body), hex_simd::AsciiCase::Lower);
-                if put.e_tag().map(|etag| etag.trim_matches('"')) == Some(plaintext_md5.as_str()) {
-                    return Err("SSE-S3 source ETag is the plaintext MD5; the cell no longer exercises encrypted ETags".into());
-                }
-                Ok(body)
-            }
         }
     }
 }
@@ -303,40 +270,14 @@ fn expectation(mode: TargetMode, shape: ObjectShape) -> Expectation {
 /// - the first PUT lands but its response is lost, so the object is FAILED
 ///   and the scanner heal pass re-drives it;
 /// - an existing-object resync re-drives a COMPLETED object unconditionally.
-///
-/// A permanent delete of the healed version then has to remove exactly the
-/// replica the heal pass located.
 #[tokio::test]
 async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult {
-    run_redrive_does_not_duplicate(RedriveObjects::Plaintext).await
-}
-
-/// The same re-drives for SSE-S3 objects against a target that also computes
-/// its own ETags (AWS S3). The source ETag is a keyed fingerprint
-/// (docs/operations/encrypted-etags.md) while the replica reports the MD5 of
-/// the plaintext it received, so no exact-ETag lookup can ever find the
-/// replica: the lost-response heal, the resync and the purge must locate it
-/// without comparing ETags.
-#[tokio::test]
-async fn matrix_mint_own_version_ids_encrypted_redrive_does_not_duplicate() -> TestResult {
-    run_redrive_does_not_duplicate(RedriveObjects::SseS3OnEtagComputingTarget).await
-}
-
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-enum RedriveObjects {
-    Plaintext,
-    SseS3OnEtagComputingTarget,
-}
-
-async fn run_redrive_does_not_duplicate(objects: RedriveObjects) -> TestResult {
     init_logging();
-    let encrypted = objects == RedriveObjects::SseS3OnEtagComputingTarget;
 
     let target = FakeS3Target::start().await?;
     let target_bucket = "matrix-mint-own-redrive-dst".to_string();
     target.create_bucket_with_object_lock(target_bucket.clone());
     target.assign_own_version_ids(true);
-    target.ignore_source_etags(encrypted);
 
     let mut env_vars = replication_fast_env();
     env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
@@ -350,7 +291,6 @@ async fn run_redrive_does_not_duplicate(objects: RedriveObjects) -> TestResult {
     ]);
     let env = OdmTestEnv::start_with(OdmEnvOptions {
         env: env_vars,
-        local_kms: encrypted,
         ..OdmEnvOptions::default()
     })
     .await?;
@@ -379,63 +319,45 @@ async fn run_redrive_does_not_duplicate(objects: RedriveObjects) -> TestResult {
         },
     )
     .await?;
-    put_bucket_replication_with_delete_statuses(source_env, source_bucket, &target_arn, "Enabled", Some("Enabled")).await?;
-
-    let put = |key: &'static str, fill: u8, len: usize| {
-        let request = source_client
-            .put_object()
-            .bucket(source_bucket)
-            .key(key)
-            .body(ByteStream::from(payload(len, fill)));
-        let request = if encrypted {
-            request.server_side_encryption(ServerSideEncryption::Aes256)
-        } else {
-            request
-        };
-        async move { request.send().await }
-    };
+    put_bucket_replication(source_env, source_bucket, &target_arn).await?;
 
     // Teach the worker the target's identity contract with one ordinary
     // write, exactly as production learns it (the PUT response carries the
     // minted id).
     let probe_key = "redrive/identity-probe.bin";
-    let probe = put(probe_key, 0x01, 4 * 1024).await?;
+    source_client
+        .put_object()
+        .bucket(source_bucket)
+        .key(probe_key)
+        .body(ByteStream::from(payload(4 * 1024, 0x01)))
+        .send()
+        .await?;
     assert_eq!(
         wait_for_terminal_replication_status(&source_client, source_bucket, probe_key).await?,
         "COMPLETED"
     );
-    if encrypted {
-        // The variant only proves something while the replica's ETag differs
-        // from the source's encrypted-ETag fingerprint.
-        let replica = single_target_version(&target, &target_bucket, probe_key)?;
-        let replica_etag = fake_source_client(&target)
-            .head_object()
-            .bucket(&target_bucket)
-            .key(probe_key)
-            .version_id(&replica)
-            .send()
-            .await?
-            .e_tag()
-            .map(str::to_string);
-        if replica_etag.is_none() || replica_etag.as_deref() == probe.e_tag() {
-            return Err(format!(
-                "the replica ETag {replica_etag:?} must differ from the source fingerprint {:?}",
-                probe.e_tag()
-            )
-            .into());
-        }
-    }
 
     // Shape 1: the PUT is stored, its response never arrives, heal re-drives.
     let heal_key = "redrive/heal.bin";
     target.inject_for_key(FakeTargetOperation::PutObject, heal_key, FakeTargetFault::DisconnectAfterResponse, 1);
-    let healed = put(heal_key, 0x02, 8 * 1024).await?;
-    let healed_version = healed.version_id().ok_or("source PUT returned no version id")?.to_string();
+    source_client
+        .put_object()
+        .bucket(source_bucket)
+        .key(heal_key)
+        .body(ByteStream::from(payload(8 * 1024, 0x02)))
+        .send()
+        .await?;
     wait_for_replication_status_and_single_version(&source_client, source_bucket, &target, &target_bucket, heal_key).await?;
 
     // Shape 2: an existing-object resync re-drives a COMPLETED object.
     let resync_key = "redrive/resync.bin";
-    put(resync_key, 0x03, 8 * 1024).await?;
+    source_client
+        .put_object()
+        .bucket(source_bucket)
+        .key(resync_key)
+        .body(ByteStream::from(payload(8 * 1024, 0x03)))
+        .send()
+        .await?;
     assert_eq!(
         wait_for_terminal_replication_status(&source_client, source_bucket, resync_key).await?,
         "COMPLETED"
@@ -466,20 +388,6 @@ async fn run_redrive_does_not_duplicate(objects: RedriveObjects) -> TestResult {
         );
     }
 
-    // The heal pass recorded the replica it located; a permanent delete of
-    // that version must remove exactly it.
-    source_client
-        .delete_object()
-        .bucket(source_bucket)
-        .key(heal_key)
-        .version_id(&healed_version)
-        .send()
-        .await?;
-    wait_until("permanent delete of the healed replica", || async {
-        Ok(target.stored_versions(&target_bucket, heal_key).is_empty())
-    })
-    .await?;
-
     target.shutdown().await;
     Ok(())
 }
@@ -963,11 +871,6 @@ async fn matrix_mint_own_version_ids_target() -> TestResult {
     run_row(TargetMode::MintOwnVersionIds).await
 }
 
-#[tokio::test]
-async fn matrix_computes_own_etags_target() -> TestResult {
-    run_row(TargetMode::ComputesOwnEtags).await
-}
-
 /// Every known-red entry must name a real cell and an issue, and the lookup
 /// must round-trip, so a stale or mistyped entry cannot silently pin nothing.
 #[test]
@@ -1004,8 +907,6 @@ async fn run_row(mode: TargetMode) -> TestResult {
     env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
     let env = OdmTestEnv::start_with(OdmEnvOptions {
         env: env_vars,
-        // The SseS3 shape needs a KMS for SSE-S3.
-        local_kms: true,
         ..OdmEnvOptions::default()
     })
     .await?;

crates/e2e_test/src/sts_query_compat_test.rs

@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
         .version_id(version)
         .send()
         .await?;
-    opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
+    opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
     client.list_objects_v2().bucket(source).send().await?;
     opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
     client.list_object_versions().bucket(source).send().await?;

crates/ecstore/Cargo.toml

@@ -31,7 +31,6 @@ workspace = true
 
 [features]
 default = []
-diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
 gcs = ["dep:google-cloud-storage", "dep:google-cloud-auth"]
 # Compiles the controlled list-objects namespace-journal chaos injector into a
 # production binary (it is always available to tests). Off by default so the
@@ -184,7 +183,6 @@ sha2 = { workspace = true }
 hex-simd = { workspace = true }
 tempfile.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
-h2 = { workspace = true }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
 hostname.workspace = true
@@ -202,8 +200,8 @@ md-5.workspace = true
 memmap2 = { workspace = true }
 libc.workspace = true
 # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
-# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
-rustix = { workspace = true, features = ["process", "fs", "thread"] }
+# (backlog#1178); "fs" comes from the workspace default.
+rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
@@ -250,7 +248,6 @@ windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
 
 [dev-dependencies]
 aws-smithy-async.workspace = true
-rustfs-s3-ops.workspace = true
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true, features = ["async_closure"] }

crates/ecstore/src/api/mod.rs

@@ -14,15 +14,6 @@
 
 //! Explicit ECStore public facades for outer crate compatibility boundaries.
 
-#[cfg(feature = "diagnostic-read-version-caller")]
-pub mod diagnostic {
-    pub use crate::cluster::rpc::runtime_sources::{
-        current_read_version_entry_source, observe_read_version_context_boundary, with_read_version_entry,
-        with_read_version_spawn_context,
-    };
-    pub use rustfs_io_metrics::internode_metrics::{ReadVersionContextBoundary, ReadVersionEntrySource};
-}
-
 pub mod admin {
     pub use crate::diagnostics::admin_server_info::{get_local_server_property, get_server_info};
 }
@@ -208,10 +199,10 @@ pub mod bucket {
             get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
             get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
             get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
-            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
-            remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
-            update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
-            update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
+            init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
+            update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
+            update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
+            update_under_transaction_lock_at,
         };
         #[cfg(feature = "test-util")]
         pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -219,8 +210,7 @@ pub mod bucket {
 
     pub mod migration {
         pub use crate::bucket::migration::{
-            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
-            try_migrate_iam_config,
+            LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
         };
     }
 
@@ -381,6 +371,24 @@ pub mod config {
             with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
             with_server_config_write_lock,
         };
+
+        /// Read config bytes and metadata without treating an empty payload as absent.
+        pub async fn read_config_preserve_empty_with_metadata_opts(
+            api: std::sync::Arc<crate::store::ECStore>,
+            file: &str,
+            opts: &crate::object_api::ObjectOptions,
+        ) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
+            crate::config::com::read_config_preserve_empty_with_metadata_opts(api, file, opts).await
+        }
+
+        pub async fn read_config_limited_preserve_empty_with_metadata_opts(
+            api: std::sync::Arc<crate::store::ECStore>,
+            file: &str,
+            opts: &crate::object_api::ObjectOptions,
+            max_bytes: usize,
+        ) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
+            crate::config::com::read_config_limited_preserve_empty_with_metadata_opts(api, file, opts, max_bytes).await
+        }
     }
 
     pub mod storageclass {
@@ -432,14 +440,11 @@ pub mod disk {
     pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, CurrentDeleteTransition,
-        DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest,
-        DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DeleteRecoveryState,
-        Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions, FileReader, FileWriter,
-        GuardedDeleteVersionsPayload, GuardedFileInfoVersions, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
-        PartTransactionAction, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp,
-        ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
-        is_quota_mutation_fence_path, new_disk, validate_batch_read_version_item_count,
+        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
+        DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
+        PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
+        STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
+        validate_batch_read_version_item_count,
     };
     pub use bytes::Bytes;
     pub use endpoint::Endpoint;
@@ -508,19 +513,17 @@ pub mod metrics {
 }
 
 pub mod notification {
+    #[cfg(any(test, feature = "test-util"))]
+    pub use crate::services::notification_sys::rotate_cross_pool_fence_fleet_proof_for_test;
     pub use crate::services::notification_sys::{
         ClusterTierDailyStats, CrossPoolFenceFleetProofToken, IlmRecoveryExportFleetProofToken,
         LegacyTransitionStateReconcileFleetProofToken, NotificationPeerErr, NotificationSys, ScannerPublicationLeaseGrant,
         acquire_cross_pool_fence_fleet_proof, acquire_ilm_recovery_export_fleet_proof,
-        acquire_legacy_transition_state_reconcile_fleet_proof, configured_fleet_topology_fingerprint,
-        cross_pool_fence_fleet_proof_matches, get_global_notification_sys, ilm_recovery_export_fleet_proof_matches,
-        ilm_recovery_export_local_process_epoch, ilm_recovery_export_member_epochs_sha256,
-        ilm_recovery_export_topology_generation, legacy_transition_state_reconcile_fleet_proof_matches,
-        new_global_notification_sys, scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe,
-    };
-    #[cfg(any(test, feature = "test-util"))]
-    pub use crate::services::notification_sys::{
-        rotate_cross_pool_fence_fleet_proof_for_test, without_cross_pool_fence_fleet_proof_for_test,
+        acquire_legacy_transition_state_reconcile_fleet_proof, cross_pool_fence_fleet_proof_matches, get_global_notification_sys,
+        ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_local_process_epoch,
+        ilm_recovery_export_member_epochs_sha256, ilm_recovery_export_topology_generation,
+        legacy_transition_state_reconcile_fleet_proof_matches, new_global_notification_sys,
+        scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe,
     };
 }
 
@@ -534,14 +537,13 @@ pub mod integrity {
 pub mod object {
     pub use crate::object_api::{
         BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
-        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
-        NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
-        PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
+        ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
+        RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
         SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
         ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
         get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
         register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
-        with_get_object_read_condition,
     };
     pub use crate::store::{
         PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -606,7 +608,6 @@ pub mod set_disk {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
-        pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
         pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
 
         /// Keep a namespace commit pending until the returned owner is dropped.

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -142,11 +142,11 @@ fn same_replication_service(edited: &BucketTarget, previous: &BucketTarget) -> b
         && access_key(edited) == access_key(previous)
 }
 
-/// Page size and page budget for [`TargetClient::replica_candidates`].
+/// Page size and page budget for [`TargetClient::locate_replica_by_etag`].
 const FIND_VERSION_BY_ETAG_PAGE_SIZE: i32 = 1000;
 const FIND_VERSION_BY_ETAG_MAX_PAGES: usize = 8;
-/// Candidate cap for [`TargetClient::replica_candidates`]: more than this
-/// many matching versions of one key is ambiguity by any measure.
+/// Candidate cap for [`TargetClient::replica_candidates_by_etag`]: more than
+/// this many same-content versions of one key is ambiguity by any measure.
 const FIND_VERSION_BY_ETAG_MAX_MATCHES: usize = 16;
 pub type GetObjectSdkError = Box<SdkError<GetObjectError>>;
 pub type GetObjectTaggingSdkError = Box<SdkError<GetObjectTaggingError>>;
@@ -2030,28 +2030,27 @@ impl TargetClient {
 
     /// Candidate replicas by content identity on a target that mints its own
     /// version ids: page `ListObjectVersions` under the exact key and report
-    /// the live versions that match `identity`, newest first. Delete markers
-    /// and prefix siblings never match. Bounded to
+    /// the live versions whose ETag matches `source_etag`, newest first.
+    /// Delete markers and prefix siblings never match. Bounded to
     /// [`FIND_VERSION_BY_ETAG_MAX_PAGES`] pages and
-    /// [`FIND_VERSION_BY_ETAG_MAX_MATCHES`] candidates per kind so a key with
-    /// a very deep history cannot turn one convergence check into an
-    /// unbounded scan; a replica beyond that window reads as missing, which
-    /// only costs a re-PUT (today's behaviour), never a lost object.
+    /// [`FIND_VERSION_BY_ETAG_MAX_MATCHES`] candidates so a key with a very
+    /// deep history cannot turn one convergence check into an unbounded scan;
+    /// a replica beyond that window reads as missing, which only costs a
+    /// re-PUT (today's behaviour), never a lost object.
     ///
     /// Content identity is not version identity: two source generations with
-    /// the same bytes have the same ETag, and two with the same size have the
-    /// same shape. Callers drop the candidates other source versions already
-    /// claim through their ledgers and decide through
-    /// [`ReplicaCandidates::locate`].
-    pub async fn replica_candidates(
+    /// the same bytes have the same ETag. Callers drop the candidates other
+    /// source versions already claim through their ledgers and refuse an
+    /// [`ReplicaLocation::Ambiguous`] remainder before mutating or deleting.
+    pub async fn replica_candidates_by_etag(
         &self,
         bucket: &str,
         object: &str,
-        identity: &ReplicaIdentity<'_>,
-    ) -> Result<ReplicaCandidates, Box<SdkError<aws_sdk_s3::operation::list_object_versions::ListObjectVersionsError>>> {
+        source_etag: &str,
+    ) -> Result<Vec<String>, Box<SdkError<aws_sdk_s3::operation::list_object_versions::ListObjectVersionsError>>> {
         let mut key_marker: Option<String> = None;
         let mut version_id_marker: Option<String> = None;
-        let mut candidates = ReplicaCandidates::default();
+        let mut matches: Vec<String> = Vec::new();
         for _ in 0..FIND_VERSION_BY_ETAG_MAX_PAGES {
             let page = self
                 .client
@@ -2064,27 +2063,19 @@ impl TargetClient {
                 .send()
                 .await
                 .map_err(Box::new)?;
-            for version in page.versions() {
-                let Some(version_id) = version.version_id().filter(|id| !id.is_empty()) else {
-                    continue;
-                };
-                if version.key() != Some(object) {
-                    continue;
-                }
-                match identity.evidence(version.e_tag(), version.size(), version.last_modified()) {
-                    Some(ReplicaEvidence::Etag) => candidates.exact.push(version_id.to_string()),
-                    // An unversioned target overwrites the key in place: its
-                    // `null` version holds whichever generation was written
-                    // last, so a shape match there identifies nothing.
-                    Some(ReplicaEvidence::Shape) if version_id != "null" => candidates.shaped.push(version_id.to_string()),
-                    Some(ReplicaEvidence::Shape) => {}
-                    None => {}
-                }
-            }
+            matches.extend(
+                page.versions()
+                    .iter()
+                    .filter(|version| {
+                        version.key() == Some(object)
+                            && version.version_id().is_some_and(|id| !id.is_empty())
+                            && replication_etags_match(Some(source_etag), version.e_tag())
+                    })
+                    .filter_map(|version| version.version_id().map(str::to_string)),
+            );
             // A listing that moved past the exact key (every listed key is >=
-            // the prefix), ended, or already filled a candidate cap decides.
-            if candidates.exact.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES
-                || candidates.shaped.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES
+            // the prefix), ended, or already filled the candidate cap decides.
+            if matches.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES
                 || page
                     .versions()
                     .iter()
@@ -2099,9 +2090,8 @@ impl TargetClient {
                 break;
             }
         }
-        candidates.exact.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES);
-        candidates.shaped.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES);
-        Ok(candidates)
+        matches.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES);
+        Ok(matches)
     }
 
     /// PutObjectRetention against a replica version on a target that does not
@@ -2624,18 +2614,19 @@ impl TargetClient {
 }
 
 /// Where a replica stands on a target that mints its own version ids, by
-/// content identity after the candidates other source versions claim were
-/// removed. See [`TargetClient::replica_candidates`].
+/// content identity (exact key + ETag) after the candidates other source
+/// versions claim were removed. See
+/// [`TargetClient::replica_candidates_by_etag`].
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub enum ReplicaLocation {
-    /// No live version under the key matches the source.
+    /// No live version under the key carries the source ETag.
     Missing,
-    /// Exactly one live version matches: safe to address.
+    /// Exactly one live version carries it: safe to address.
     Unique(String),
-    /// More than one live version carries the source ETag (same bytes
-    /// replicated for several source generations). `newest` is the most
-    /// recently listed one — good enough to prove the replica exists, never
-    /// good enough to pick which one to mutate or delete.
+    /// More than one live version carries it (same bytes replicated for
+    /// several source generations). `newest` is the most recently listed
+    /// one — good enough to prove the replica exists, never good enough to
+    /// pick which one to mutate or delete.
     Ambiguous { newest: String },
 }
 
@@ -2661,107 +2652,6 @@ impl ReplicaLocation {
     }
 }
 
-/// What a replica of an encrypted source version still has in common with
-/// it on a target that computes its own ETags. The source ETag is a keyed
-/// fingerprint (docs/operations/encrypted-etags.md) or a legacy plaintext
-/// MD5 the target's re-encrypted replica never reports, so the replica is
-/// recognised by the byte count the replication PUT sent and by a write time
-/// no earlier than the source version's own.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub struct EncryptedReplicaShape {
-    /// Bytes on the replication wire: the plaintext size for managed SSE,
-    /// the stored ciphertext size for an SSE-C passthrough.
-    pub size: i64,
-    /// The source version's mtime in whole seconds; listings and HEAD report
-    /// second-precision times.
-    pub not_before_secs: i64,
-}
-
-impl EncryptedReplicaShape {
-    pub fn matches(&self, size: Option<i64>, last_modified: Option<&aws_sdk_s3::primitives::DateTime>) -> bool {
-        size == Some(self.size) && last_modified.is_some_and(|written| written.secs() >= self.not_before_secs)
-    }
-}
-
-/// How a source version recognises its replica among a key's target
-/// versions.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum ReplicaIdentity<'a> {
-    /// Unencrypted source: its ETag is the MD5 of the replicated bytes, which
-    /// every target reports back — exact identity.
-    Etag(&'a str),
-    /// Encrypted source: a target that adopts the source ETag (RustFS,
-    /// MinIO) still reports `etag`; any other target only keeps the shape.
-    Encrypted {
-        etag: Option<&'a str>,
-        shape: EncryptedReplicaShape,
-    },
-}
-
-/// Which kind of evidence tied a target version to the source.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum ReplicaEvidence {
-    /// The target reports the source ETag.
-    Etag,
-    /// Encrypted source on a target that computes its own ETag: same size,
-    /// written no earlier than the source version.
-    Shape,
-}
-
-impl ReplicaIdentity<'_> {
-    pub fn evidence(
-        &self,
-        etag: Option<&str>,
-        size: Option<i64>,
-        last_modified: Option<&aws_sdk_s3::primitives::DateTime>,
-    ) -> Option<ReplicaEvidence> {
-        match *self {
-            Self::Etag(source_etag) => replication_etags_match(Some(source_etag), etag).then_some(ReplicaEvidence::Etag),
-            Self::Encrypted {
-                etag: source_etag,
-                shape,
-            } => {
-                if replication_etags_match(source_etag, etag) {
-                    Some(ReplicaEvidence::Etag)
-                } else {
-                    shape.matches(size, last_modified).then_some(ReplicaEvidence::Shape)
-                }
-            }
-        }
-    }
-}
-
-/// Content-identity candidates of one key, newest first, by evidence kind.
-#[derive(Debug, Clone, Default, PartialEq, Eq)]
-pub struct ReplicaCandidates {
-    /// Versions that report the source ETag.
-    pub exact: Vec<String>,
-    /// Encrypted sources only: versions without the source ETag whose shape
-    /// matches.
-    pub shaped: Vec<String>,
-}
-
-impl ReplicaCandidates {
-    pub fn is_empty(&self) -> bool {
-        self.exact.is_empty() && self.shaped.is_empty()
-    }
-
-    /// Drop the candidates other source versions claim and decide. An ETag
-    /// match outranks any shape match. A shape is weaker evidence than an
-    /// ETag — two generations of one size share it — so several shape
-    /// matches are never resolved by picking one: they read as
-    /// [`ReplicaLocation::Missing`] for convergence (the caller re-PUTs, a
-    /// duplicate at worst) and as ambiguous for a purge (it refuses).
-    pub fn locate(mut self, claimed: &HashSet<String>) -> (ReplicaLocation, ReplicaEvidence) {
-        self.exact.retain(|candidate| !claimed.contains(candidate));
-        if !self.exact.is_empty() {
-            return (ReplicaLocation::from_matches(self.exact), ReplicaEvidence::Etag);
-        }
-        self.shaped.retain(|candidate| !claimed.contains(candidate));
-        (ReplicaLocation::from_matches(self.shaped), ReplicaEvidence::Shape)
-    }
-}
-
 #[derive(Debug)]
 pub enum BucketTargetError {
     BucketRemoteTargetNotFound {

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -2417,16 +2417,6 @@ fn is_slow_down(err: &Error) -> bool {
     matches!(err, Error::SlowDown)
 }
 
-/// Lifecycle expiry is idempotent when the exact object or version has already
-/// disappeared between the scanner listing and the delete admission.
-///
-/// Keep this predicate narrower than bucket/volume "not found" helpers: an
-/// unavailable bucket or disk is an operational failure and must not be
-/// reported as a completed expiry.
-fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
-    is_err_object_not_found(err) || is_err_version_not_found(err)
-}
-
 /// Resolves the expiry worker count from the single documented knob,
 /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
 /// anything else falls back to `min(cpus, 16)`. The historical
@@ -2461,13 +2451,7 @@ fn spawn_recovery_disposition_maintenance_once(api: Arc<ECStore>) -> Option<Join
         return None;
     }
     Some(tokio::spawn(async move {
-        let maintenance = run_recovery_disposition_maintenance_loop(api, cancel_token);
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        let maintenance = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::RecoveryDispositionMaintenance,
-            maintenance,
-        );
-        maintenance.await;
+        run_recovery_disposition_maintenance_loop(api, cancel_token).await;
     }))
 }
 
@@ -3350,13 +3334,7 @@ fn spawn_tier_delete_journal_recovery_once(api: Arc<ECStore>) {
         return;
     }
     tokio::spawn(async move {
-        let recovery = run_tier_delete_journal_recovery_loop(api, cancel_token);
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery,
-            recovery,
-        );
-        recovery.await;
+        run_tier_delete_journal_recovery_loop(api, cancel_token).await;
     });
 }
 
@@ -3375,13 +3353,7 @@ fn spawn_transition_transaction_recovery_once(api: Arc<ECStore>) {
         return;
     }
     tokio::spawn(async move {
-        let recovery = run_transition_transaction_recovery_loop(api, cancel_token);
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-            rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery,
-            recovery,
-        );
-        recovery.await;
+        run_transition_transaction_recovery_loop(api, cancel_token).await;
     });
 }
 
@@ -3966,28 +3938,44 @@ pub(crate) async fn enqueue_immediate_expiry(api: Arc<ECStore>, oi: &ObjectInfo,
         return;
     };
 
-    let object_infos = match api
-        .get_object_versions_for_lifecycle(&oi.bucket, &oi.name, opts, configs.bucket_incarnation_id)
-        .await
-    {
-        Ok(objects) => objects,
-        Err(err) => {
-            warn!(
-                event = EVENT_LIFECYCLE_EVALUATION_FAILED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                result = "failed",
-                bucket = %oi.bucket,
-                object = %oi.name,
-                error = %err,
-                reason = "read_versions_failed",
-                "Failed to load lifecycle version group"
-            );
-            return;
+    let mut marker = None;
+    let mut version_marker = None;
+    let mut object_infos = Vec::new();
+
+    loop {
+        let page = match api
+            .clone()
+            .list_object_versions_for_lifecycle(&oi.bucket, &oi.name, marker.clone(), version_marker.clone(), None, 1000)
+            .await
+        {
+            Ok(page) => page,
+            Err(err) => {
+                warn!(
+                    event = EVENT_LIFECYCLE_EVALUATION_FAILED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    error = %err,
+                    reason = "list_versions_failed",
+                    "Failed to load lifecycle version group"
+                );
+                return;
+            }
+        };
+
+        object_infos.extend(page.objects.into_iter().filter(|object| object.name == oi.name));
+
+        if !page.is_truncated {
+            break;
         }
-    };
+
+        marker = page.next_marker;
+        version_marker = page.next_version_idmarker;
+    }
+
     if object_infos.is_empty() {
-        return;
+        object_infos.push(oi.clone());
     }
 
     let object_opts = object_infos
@@ -4932,20 +4920,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
-            Err(err) if is_lifecycle_expiry_already_gone(&err) => {
-                debug!(
-                    event = EVENT_LIFECYCLE_DELETE_COMPLETED,
-                    component = LOG_COMPONENT_ECSTORE,
-                    subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                    bucket = %oi.bucket,
-                    object = %oi.name,
-                    operation = "delete_restored_object",
-                    reason = "already_gone",
-                    "Lifecycle delete target was already absent"
-                );
-                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-                Ok(oi.clone())
-            }
             Err(err) => Err(std::io::Error::other(err)),
         };
     }
@@ -4956,8 +4930,6 @@ async fn expire_transitioned_object_with_lock_lost_signal(
     // pointing to a remote tier version that has already been removed. If this
     // only creates a delete marker, remote cleanup must be driven by persisted
     // free-version recovery rather than the visible delete result.
-    // Keep remote transitioned not-found errors visible: local metadata
-    // absence alone does not prove that the tier object was cleaned up.
     let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
         Ok(obj) => obj,
         Err(e) => {
@@ -5830,20 +5802,6 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
         .await
     {
         Ok(dobj) => dobj,
-        Err(err) if is_lifecycle_expiry_already_gone(&err) => {
-            debug!(
-                event = EVENT_LIFECYCLE_DELETE_COMPLETED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
-                bucket = %oi.bucket,
-                object = %oi.name,
-                operation = "delete_object",
-                reason = "already_gone",
-                "Lifecycle delete target was already absent"
-            );
-            crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-            return true;
-        }
         Err(e) => {
             error!(
                 event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -6083,19 +6041,18 @@ mod tests {
         cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
         enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
         enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
-        get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
-        lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
-        lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
-        manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
-        manual_transition_version_marker, manual_transition_worker_failure_reason,
-        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
-        persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
-        recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
-        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
-        resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
-        select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
-        should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
-        wait_for_tier_free_version_recovery,
+        get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
+        lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
+        lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
+        manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
+        manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
+        merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
+        persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
+        resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
+        resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator,
+        run_tier_free_version_recovery_loop, select_restore_s3_location, set_lifecycle_observability_observer,
+        set_recovered_free_version_enqueue_observer, should_defer_date_expiry_for_recent_config_update,
+        transitioned_cleanup_tuple, transitioned_object_delete_opts, wait_for_tier_free_version_recovery,
     };
     #[cfg(feature = "test-util")]
     use super::{delete_free_version_remote_object_then, encode_dir_object, get_transitioned_object_reader_with_tier_manager};
@@ -10554,26 +10511,6 @@ mod tests {
         );
     }
 
-    #[test]
-    fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
-        assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
-        assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
-            "bucket".to_string(),
-            "object".to_string(),
-        )));
-        assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
-        assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
-            "bucket".to_string(),
-            "object".to_string(),
-            "version".to_string(),
-        )));
-
-        assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
-        assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
-    }
-
     #[tokio::test]
     async fn manual_transition_counts_already_transitioned_object() {
         let lc = latest_transition_lifecycle();
@@ -13205,81 +13142,6 @@ mod tests {
         assert_eq!(event.action, IlmAction::NoneAction);
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn lifecycle_exact_immediate_expiry_preserves_guards_and_skips_stale_writes() {
-        let (paths, ecstore) = setup_test_env().await;
-        let bucket = format!("lifecycle-exact-expiry-{}", Uuid::new_v4().simple());
-        let object = "object";
-        create_test_bucket(&ecstore, &bucket).await;
-        let oi = ecstore
-            .put_object(
-                &bucket,
-                object,
-                &mut PutObjReader::from_vec(b"payload".to_vec()),
-                &ObjectOptions::default(),
-            )
-            .await
-            .expect("create object before configuring expiry");
-
-        // Exercise persisted legacy immediate rules without relaxing the S3
-        // validator, which correctly rejects newly submitted Expiration.Days=0.
-        let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system");
-        let sys = sys.read().await.clone();
-        let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata")).clone();
-        let mut lifecycle = latest_expiration_lifecycle();
-        lifecycle.rules[0].expiration.as_mut().expect("expiration rule").days = Some(0);
-        metadata.lifecycle_config_xml = crate::bucket::utils::serialize(&lifecycle).expect("encode legacy policy");
-        metadata.lifecycle_config = Some(lifecycle);
-        sys.persist_and_set(metadata).await.expect("persist legacy policy");
-
-        let state = install_unconsumed_runtime_expiry_worker(&ecstore, 4).await;
-        super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await;
-        assert_eq!(state.read().await.stats.pending_tasks(), 1, "exact reads must preserve immediate expiry");
-
-        let state = install_unconsumed_runtime_expiry_worker(&ecstore, 4).await;
-        for path in &paths {
-            let metadata_path = path.join(&bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE);
-            let mut meta = rustfs_filemeta::FileMeta::load(&fs::read(&metadata_path).await.expect("read object metadata"))
-                .expect("decode object metadata");
-            let mut version = meta
-                .get_all_file_info_versions(&bucket, object, true)
-                .expect("version group")
-                .versions
-                .remove(0);
-            version.metadata.insert("x-amz-object-lock-legal-hold".into(), "ON".into());
-            meta.add_version(version).expect("persist legal hold");
-            fs::write(metadata_path, meta.marshal_msg().expect("encode held metadata"))
-                .await
-                .expect("write legal hold");
-        }
-        super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await;
-        assert_eq!(
-            state.read().await.stats.pending_tasks(),
-            0,
-            "use current lock metadata, not the stale write result"
-        );
-
-        for path in &paths {
-            fs::write(path.join(&bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE), b"corrupt")
-                .await
-                .expect("inject corrupt metadata");
-        }
-        super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await;
-        assert_eq!(state.read().await.stats.pending_tasks(), 0, "read failure must never enqueue expiry");
-        for path in &paths {
-            fs::remove_dir_all(path.join(&bucket).join(object))
-                .await
-                .expect("simulate concurrent object removal");
-        }
-        super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await;
-        assert_eq!(
-            state.read().await.stats.pending_tasks(),
-            0,
-            "confirmed absence must not fall back to the old ObjectInfo"
-        );
-    }
-
     #[tokio::test]
     #[serial]
     async fn lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata() {
@@ -13523,9 +13385,7 @@ mod tests {
                 .bucket_incarnation_id_from_disk(&bucket)
                 .await
                 .expect("bucket incarnation");
-            // Replaying the same scanner snapshot after the delete is an
-            // already-gone success and must not be reported as a failure.
-            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
                 assert_eq!(
                     super::apply_expiry_on_non_transitioned_objects(
                         ecstore.clone(),

crates/ecstore/src/bucket/metadata_sys.rs

@@ -178,13 +178,6 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
     Ok(ObjectLockConfigState::ConfirmedAbsent)
 }
 
-#[derive(Debug, Clone, PartialEq)]
-pub(crate) struct CachedQuotaMetadataSnapshot {
-    pub(crate) quota: Option<BucketQuota>,
-    pub(crate) bucket_incarnation: Uuid,
-    pub(crate) quota_revision: OffsetDateTime,
-}
-
 /// Convert the persisted serving-layer configuration into the storage-level
 /// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention)
 /// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a
@@ -321,33 +314,6 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
     Ok(lock.remove(bucket).await)
 }
 
-/// Remove a peer's cached metadata only when the bucket is absent under the
-/// namespace read fence. A delayed delete notification must not withdraw the
-/// metadata of a same-name bucket that has already been recreated.
-pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
-    if is_meta_bucketname(bucket) {
-        return Ok(false);
-    }
-    // Lock order: bucket namespace read -> metadata system read (released)
-    // -> metadata publish, matching peer reloads.
-    let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
-    let namespace_guard = namespace_lock
-        .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
-        .await?;
-    let sys = bucket_metadata_sys_of(&api.ctx)?;
-    let sys = sys.read().await.clone();
-    if sys
-        .bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
-        .await?
-    {
-        return Ok(false);
-    }
-    await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
-        Ok(sys.remove(bucket).await)
-    })
-    .await
-}
-
 fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
     let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
         warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -525,36 +491,6 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance:
     sys.get_bucket_incarnation_id_from_disk(bucket).await
 }
 
-/// Directory loss is recoverable only for a committed, non-retired generation.
-/// The caller retains the bucket lifecycle fence through the ensuing reconstruction.
-pub(crate) async fn validate_bucket_reconstruction_in(
-    ctx: &crate::runtime::instance::InstanceContext,
-    bucket: &str,
-    expected: Uuid,
-) -> Result<()> {
-    let api = bucket_metadata_sys_of(ctx)?.read().await.object_store();
-    let lock = api
-        .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
-        .await?;
-    let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-    let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?;
-    if !persisted
-        || !metadata.bucket_incarnation_sidecar
-        || expected.is_nil()
-        || metadata.bucket_incarnation_id != expected
-        || !metadata.bucket_creation_committed
-    {
-        return Err(Error::PreconditionFailed);
-    }
-    if crate::bucket::retirement::is_retired(api, bucket, expected).await? {
-        return Err(Error::BucketNotFound(bucket.to_owned()));
-    }
-    if guard.is_lock_lost() {
-        return Err(Error::PreconditionFailed);
-    }
-    Ok(())
-}
-
 /// Validate against disk while retaining an already-held Object Lock fence.
 pub(crate) async fn get_bucket_incarnation_id_for_options_in(
     ctx: &crate::runtime::instance::InstanceContext,
@@ -1487,9 +1423,6 @@ fn quota_config_and_incarnation_from_authority(
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
-            if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
-                return Err(Error::other("persisted bucket quota configuration is invalid"));
-            }
             Ok((
                 metadata.quota_config.clone(),
                 metadata.bucket_incarnation_id,
@@ -1504,15 +1437,6 @@ fn quota_config_and_incarnation_from_authority(
     }
 }
 
-pub(crate) async fn get_cached_quota_config_and_incarnation_in(
-    ctx: &crate::runtime::instance::InstanceContext,
-    bucket: &str,
-) -> Result<Option<CachedQuotaMetadataSnapshot>> {
-    let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
-    let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
-    bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
-}
-
 pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -3028,27 +2952,6 @@ impl BucketMetadataSys {
         }
     }
 
-    async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result<Option<CachedQuotaMetadataSnapshot>> {
-        let metadata = {
-            let metadata_map = self.metadata_map.read().await;
-            metadata_map.get(bucket).cloned()
-        };
-        let Some(metadata) = metadata else {
-            return Ok(None);
-        };
-        if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() {
-            return Ok(None);
-        }
-        if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
-            return Err(Error::other("persisted bucket quota configuration is invalid"));
-        }
-        Ok(Some(CachedQuotaMetadataSnapshot {
-            quota: metadata.quota_config.clone(),
-            bucket_incarnation: metadata.bucket_incarnation_id,
-            quota_revision: metadata.quota_config_updated_at,
-        }))
-    }
-
     pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
@@ -4953,154 +4856,6 @@ mod tests {
         );
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn quota_admission_rejects_stale_cached_no_quota() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system");
-        let bucket = "stale-no-quota-admission";
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("create bucket");
-        let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata");
-        store
-            .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec())
-            .await
-            .expect("publish quota to authoritative storage");
-        // Model a peer that has not received the newer configuration snapshot.
-        sys.read().await.set(bucket.to_owned(), stale).await;
-        assert!(
-            get_cached_quota_config_and_incarnation_in(&store.ctx, bucket)
-                .await
-                .expect("read stale cache")
-                .expect("cached bucket identity")
-                .quota
-                .is_none()
-        );
-
-        temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async {
-            let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0)
-                .await
-                .err()
-                .expect("persisted quota must reject admission without its snapshot");
-            assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}");
-        })
-        .await;
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
-        let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
-        let bucket = "cross-node-negative-cache-refresh";
-        let node_b_sys = node_b.read().await.clone();
-
-        assert_eq!(
-            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
-            Error::BucketNotFound(bucket.to_string()),
-            "the probe must populate node B's missing-bucket cache"
-        );
-
-        for dir in &dirs {
-            std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
-        }
-        let created = BucketMetadata::new(bucket);
-        let created_incarnation = created.bucket_incarnation_id;
-        node_a
-            .read()
-            .await
-            .persist_new_and_set(created)
-            .await
-            .expect("node A should publish the created bucket metadata");
-
-        node_b_sys
-            .reload_from_store(bucket)
-            .await
-            .expect("peer reload should clear the stale negative cache");
-        assert_eq!(
-            node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
-            created_incarnation,
-            "node B should resolve the newly created bucket immediately"
-        );
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    #[serial]
-    async fn peer_delete_clears_metadata_for_absent_bucket() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
-        let bucket = "peer-delete-absent-bucket";
-
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("initial bucket creation");
-        let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
-        store
-            .delete_bucket(bucket, &DeleteBucketOptions::default())
-            .await
-            .expect("bucket deletion");
-
-        // Model a peer that still caches the metadata after the physical delete.
-        sys.read().await.set(bucket.to_string(), cached).await;
-        assert!(
-            remove_bucket_metadata_if_absent(store.clone(), bucket)
-                .await
-                .expect("conditional peer delete"),
-            "the peer must evict the deleted bucket's cached metadata"
-        );
-        let sys = sys.read().await;
-        assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
-        assert_eq!(
-            sys.get_bucket_incarnation_id(bucket)
-                .await
-                .expect_err("deleted bucket lookup"),
-            Error::BucketNotFound(bucket.to_string()),
-        );
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    #[serial]
-    async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
-        let bucket = "delayed-peer-delete-recreated-bucket";
-
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("initial bucket creation");
-        let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
-        store
-            .delete_bucket(bucket, &DeleteBucketOptions::default())
-            .await
-            .expect("bucket deletion");
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("bucket recreation");
-        let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
-        assert_ne!(old_incarnation, new_incarnation);
-
-        let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
-            .await
-            .expect("conditional peer delete");
-        assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
-        assert_eq!(
-            sys.read()
-                .await
-                .get(bucket)
-                .await
-                .expect("recreated metadata should remain cached")
-                .bucket_incarnation_id,
-            new_incarnation,
-        );
-    }
-
     #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
     #[serial]
     async fn stale_config_request_cannot_mutate_a_recreated_bucket() {

crates/ecstore/src/bucket/migration.rs

@@ -273,30 +273,6 @@ where
 
     let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
 
-    try_migrate_bucket_metadata_for_buckets(store, &buckets).await
-}
-
-/// Migrate the already discovered startup candidates without repeating a
-/// majority-only namespace listing. Every migration write retains its quorum.
-pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
-where
-    S: ObjectIO<
-            Error = Error,
-            RangeSpec = HTTPRangeSpec,
-            HeaderMap = HeaderMap,
-            ObjectOptions = ObjectOptions,
-            ObjectInfo = ObjectInfo,
-            GetObjectReader = GetObjectReader,
-            PutObjectReader = PutObjReader,
-        > + ObjectOperations<
-            Error = Error,
-            ObjectInfo = ObjectInfo,
-            ObjectOptions = ObjectOptions,
-            FileInfo = FileInfo,
-            ObjectToDelete = ObjectToDelete,
-            DeletedObject = DeletedObject,
-        >,
-{
     if buckets.is_empty() {
         debug!("No migrating bucket metadata found");
         return Ok(());

crates/ecstore/src/bucket/quota/reservation.rs

@@ -419,6 +419,7 @@ pub(crate) struct QuotaContext {
     quota_revision: Option<OffsetDateTime>,
     quota_limit: Option<u64>,
     reservation_protocol: Option<u32>,
+    capability_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
     snapshot_admission: Option<QuotaAdmission>,
     legacy_data_movement: bool,
     metadata_guard: Option<Arc<NamespaceLockGuard>>,
@@ -478,6 +479,7 @@ impl QuotaContext {
             reservation: reservation.clone(),
         };
         let metadata_guard = self.metadata_guard;
+        let capability_proof = self.capability_proof;
 
         tokio::spawn(async move {
             reap_stale_reservations(Arc::clone(&store), &ledger_data.bucket, &ledger_data.ledger_object).await?;
@@ -524,6 +526,7 @@ impl QuotaContext {
                 ledger: Some(ReservationLedgerData::Single(ledger_data)),
                 operation_guard: Some(operation_guard),
                 metadata_guard,
+                capability_proof,
                 state: ReservationState::Pending,
             })
         })
@@ -552,6 +555,7 @@ async fn reserve_sharded(context: QuotaContext, old_size: u64, new_size: u64) ->
         commit_started: false,
     };
     let metadata_guard = context.metadata_guard;
+    let capability_proof = context.capability_proof;
     let data = ShardedReservationData {
         store: Arc::clone(&store),
         bucket: context.bucket,
@@ -636,6 +640,7 @@ async fn reserve_sharded(context: QuotaContext, old_size: u64, new_size: u64) ->
                 ledger: Some(ReservationLedgerData::Sharded(data)),
                 operation_guard: Some(operation_guard),
                 metadata_guard,
+                capability_proof,
                 state: ReservationState::Pending,
             });
         }
@@ -720,12 +725,20 @@ impl ReservationLedgerData {
             Self::Sharded(data) => &data.bucket,
         }
     }
+
+    fn error_object(&self) -> &str {
+        match self {
+            Self::Single(data) => &data.ledger_object,
+            Self::Sharded(data) => &data.shard_object,
+        }
+    }
 }
 
 pub(crate) struct QuotaReservation {
     ledger: Option<ReservationLedgerData>,
     operation_guard: Option<NamespaceLockGuard>,
     metadata_guard: Option<Arc<NamespaceLockGuard>>,
+    capability_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
     state: ReservationState,
 }
 
@@ -743,6 +756,7 @@ impl QuotaReservation {
             ledger: None,
             operation_guard: None,
             metadata_guard,
+            capability_proof: None,
             state: ReservationState::Pending,
         }
     }
@@ -752,7 +766,17 @@ impl QuotaReservation {
             || self.metadata_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
     }
 
+    pub(crate) fn capability_proof_matches(&self) -> bool {
+        self.capability_proof
+            .as_ref()
+            .is_none_or(crate::services::notification_sys::cross_pool_fence_fleet_proof_matches)
+    }
+
     pub(crate) async fn mark_commit_started(&mut self) -> Result<()> {
+        if !self.capability_proof_matches() {
+            let ledger = self.ledger.as_ref().ok_or(StorageError::PartMissingOrCorrupt)?;
+            return Err(quota_capability_error(ledger.bucket(), ledger.error_object()));
+        }
         if let Some(ledger) = self.ledger.as_ref() {
             mark_commit_started(ledger).await?;
         }
@@ -826,11 +850,6 @@ pub(crate) async fn begin(
     let snapshot_admission = opts.quota_admission;
     let data_movement = opts.data_movement;
     if crate::bucket::utils::is_meta_bucketname(bucket) {
-        let meta_bucket_fast_path_started = rustfs_io_metrics::put_stage_timer();
-        rustfs_io_metrics::record_put_object_stage_duration_from(
-            rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH,
-            meta_bucket_fast_path_started,
-        );
         return Ok(QuotaContext {
             store: None,
             bucket: bucket.to_string(),
@@ -840,6 +859,7 @@ pub(crate) async fn begin(
             quota_revision: None,
             quota_limit: None,
             reservation_protocol: None,
+            capability_proof: None,
             snapshot_admission: None,
             legacy_data_movement: false,
             metadata_guard: None,
@@ -858,6 +878,7 @@ pub(crate) async fn begin(
             quota_revision: None,
             quota_limit: Some(snapshot_admission.quota_limit()),
             reservation_protocol: None,
+            capability_proof: None,
             snapshot_admission: Some(snapshot_admission),
             legacy_data_movement: false,
             metadata_guard: None,
@@ -876,6 +897,7 @@ pub(crate) async fn begin(
             quota_revision: None,
             quota_limit: None,
             reservation_protocol: None,
+            capability_proof: None,
             snapshot_admission: None,
             legacy_data_movement: false,
             metadata_guard: None,
@@ -884,63 +906,9 @@ pub(crate) async fn begin(
         });
     }
 
-    let metadata_lock_started = rustfs_io_metrics::put_stage_timer();
     let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK,
-        metadata_lock_started,
-    );
-    let cache_lookup_started = rustfs_io_metrics::put_stage_timer();
-    let cached_quota_snapshot = if rustfs_io_metrics::put_stage_metrics_enabled() {
-        match metadata_sys::get_cached_quota_config_and_incarnation_in(ctx, bucket).await {
-            Ok(Some(snapshot)) => {
-                rustfs_io_metrics::record_put_object_quota_cache_decision(
-                    "candidate",
-                    quota_cache_reason(snapshot.quota.as_ref()),
-                );
-                Some(snapshot)
-            }
-            Ok(None) => {
-                rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "no_authoritative_cache");
-                None
-            }
-            Err(_) => {
-                rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "invalid_cached_quota");
-                None
-            }
-        }
-    } else {
-        None
-    };
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP,
-        cache_lookup_started,
-    );
-    let config_read_started = rustfs_io_metrics::put_stage_timer();
     let (quota, bucket_incarnation, quota_revision) =
         metadata_sys::get_quota_config_and_incarnation_from_disk_for_options_in(ctx, bucket, opts, &metadata_guard).await?;
-    if let Some(cached) = cached_quota_snapshot.as_ref() {
-        let reason = quota_cache_reason(quota.as_ref());
-        if cached.quota == quota && cached.bucket_incarnation == bucket_incarnation && cached.quota_revision == quota_revision {
-            rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_match", reason);
-        } else {
-            rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch", reason);
-            if cached.quota != quota {
-                rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota");
-            }
-            if cached.bucket_incarnation != bucket_incarnation {
-                rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "bucket_incarnation");
-            }
-            if cached.quota_revision != quota_revision {
-                rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota_revision");
-            }
-        }
-    }
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ,
-        config_read_started,
-    );
-    let lock_lost_check_started = rustfs_io_metrics::put_stage_timer();
     if metadata_guard.is_lock_lost() {
         return Err(StorageError::NamespaceLockQuorumUnavailable {
             mode: "quota_config",
@@ -950,11 +918,6 @@ pub(crate) async fn begin(
             achieved: 0,
         });
     }
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK,
-        lock_lost_check_started,
-    );
-    let policy_evaluate_started = rustfs_io_metrics::put_stage_timer();
     if quota
         .as_ref()
         .is_some_and(|quota| quota.has_unsupported_reservation_protocol())
@@ -963,15 +926,15 @@ pub(crate) async fn begin(
         return Err(StorageError::PartMissingOrCorrupt);
     }
     let durable_quota = quota.as_ref().filter(|quota| quota.uses_durable_reservations());
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE,
-        policy_evaluate_started,
-    );
-    // Durable writes are fenced by a write quorum of participating disks that
-    // enforce the configured topology. Fleet-wide proof remains a configuration gate;
-    // losing an unrelated peer must not revoke an admitted object write.
+    let capability_proof = if durable_quota.is_some() {
+        Some(
+            crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
+                .ok_or_else(|| quota_capability_error(bucket, &ledger_object(bucket)))?,
+        )
+    } else {
+        None
+    };
     let durable_quota_limit = durable_quota.and_then(|quota| quota.quota);
-    let snapshot_admission_started = rustfs_io_metrics::put_stage_timer();
     let reservation_protocol = durable_quota.and_then(|quota| quota.reservation_protocol);
     let snapshot_admission = match quota.as_ref().filter(|quota| !quota.uses_durable_reservations()) {
         Some(quota) => match (quota.quota, snapshot_admission) {
@@ -991,10 +954,6 @@ pub(crate) async fn begin(
         },
         None => None,
     };
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION,
-        snapshot_admission_started,
-    );
     let legacy_data_movement = durable_quota_limit.is_none()
         && quota.as_ref().and_then(|quota| quota.quota).is_some()
         && snapshot_admission.is_none()
@@ -1006,21 +965,11 @@ pub(crate) async fn begin(
                 .then(|| quota.as_ref().and_then(|quota| quota.quota))
                 .flatten()
         });
-    let object_store_lookup_started = rustfs_io_metrics::put_stage_timer();
     let store = if durable_quota_limit.is_some() {
         Some(metadata_sys::object_store_in(ctx).await?)
     } else {
         None
     };
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP,
-        object_store_lookup_started,
-    );
-    let context_build_started = rustfs_io_metrics::put_stage_timer();
-    rustfs_io_metrics::record_put_object_stage_duration_from(
-        rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD,
-        context_build_started,
-    );
     Ok(QuotaContext {
         store,
         bucket: bucket.to_string(),
@@ -1030,6 +979,7 @@ pub(crate) async fn begin(
         quota_revision: Some(quota_revision),
         quota_limit,
         reservation_protocol,
+        capability_proof,
         snapshot_admission,
         legacy_data_movement,
         metadata_guard: Some(metadata_guard),
@@ -1038,12 +988,13 @@ pub(crate) async fn begin(
     })
 }
 
-fn quota_cache_reason(quota: Option<&crate::bucket::quota::BucketQuota>) -> &'static str {
-    match quota {
-        None => "no_quota",
-        Some(quota) if quota.uses_durable_reservations() => "durable_quota",
-        Some(quota) if quota.quota.is_some() => "snapshot_quota",
-        Some(_) => "quota_without_limit",
+fn quota_capability_error(bucket: &str, object: &str) -> StorageError {
+    StorageError::NamespaceLockQuorumUnavailable {
+        mode: "quota_capability",
+        bucket: bucket.to_string(),
+        object: object.to_string(),
+        required: 1,
+        achieved: 0,
     }
 }
 

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -37,9 +37,8 @@ use time::OffsetDateTime;
 use time::format_description::well_known::Rfc3339;
 
 pub(crate) use crate::bucket::bucket_target_sys::{
-    AdvancedPutOptions, EncryptedReplicaShape, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions,
-    RemotePutObjectResponse, RemoveObjectOptions, ReplicaCandidates, ReplicaEvidence, ReplicaIdentity, ReplicaLocation,
-    S3ClientError, TargetClient, resolve_read_api_version_id,
+    AdvancedPutOptions, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, RemotePutObjectResponse, RemoveObjectOptions,
+    ReplicaLocation, S3ClientError, TargetClient, resolve_read_api_version_id,
 };
 #[cfg(test)]
 pub(crate) use crate::bucket::target::BucketTarget;
@@ -158,19 +157,6 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
     rustfs_replication::is_ssec_encrypted(user_defined)
 }
 
-/// Whether the stored ETag of a replication source can be compared with the
-/// ETag a target returns for the bytes it received. Only an unencrypted source
-/// stores the MD5 of the bytes that go on the wire. A managed-SSE (SSE-S3 /
-/// SSE-KMS) or SSE-C source stores a keyed fingerprint (new writes) or a legacy
-/// plaintext MD5, and the target re-encrypts what it receives, so neither
-/// equality nor inequality of the two ETags proves anything about the replica.
-pub(crate) fn replication_source_etag_is_comparable(user_defined: &HashMap<String, String>) -> bool {
-    matches!(
-        classify_replication_source_encryption(user_defined),
-        ReplicationSourceEncryption::Plaintext
-    )
-}
-
 /// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
 /// judgment owned by `rustfs-replication`: extract the echoed
 /// customer-algorithm header and let the crate-owned policy decide.
@@ -592,22 +578,6 @@ mod tests {
     use time::Duration;
     use uuid::Uuid;
 
-    #[test]
-    fn replication_source_etag_is_comparable_only_for_plaintext_sources() {
-        assert!(replication_source_etag_is_comparable(&HashMap::new()));
-        for (key, value) in [
-            (metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"),
-            (metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"),
-            (SSEC_ALGORITHM_HEADER, "AES256"),
-        ] {
-            let metadata = HashMap::from([(key.to_string(), value.to_string())]);
-            assert!(
-                !replication_source_etag_is_comparable(&metadata),
-                "{key}={value} must not expose its stored ETag to the replica audit"
-            );
-        }
-    }
-
     /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the
     /// SSE intent as lowercase `x-amz-server-side-encryption`
     /// (`encryption_material_to_metadata`; header-derived keys come from a

crates/ecstore/src/cluster/rpc/client.rs

@@ -24,11 +24,10 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
 #[cfg(test)]
 use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
 use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
-#[cfg(test)]
 use crate::runtime::sources as runtime_sources;
 use http::{Request as HttpRequest, Response as HttpResponse, Uri};
 use rustfs_protos::{
-    ChannelClass, get_channel_for_class,
+    ChannelClass, create_new_channel, get_channel_for_class,
     proto_gen::node_service::{
         heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
         scanner_control_service_client::ScannerControlServiceClient,
@@ -54,7 +53,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
 /// 3. Subsequent calls will attempt fresh connections
 /// 4. If node is still down, connection will fail fast (3s timeout)
 pub async fn node_service_time_out_client(
-    addr: &str,
+    addr: &String,
     interceptor: TonicInterceptor,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     // Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -67,9 +66,12 @@ pub(crate) async fn scanner_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control)
-        .await
-        .map_err(|err| crate::error::Error::other(err.to_string()))?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr)
+            .await
+            .map_err(|err| crate::error::Error::other(err.to_string()))?,
+    };
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
     Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -82,7 +84,10 @@ pub async fn heal_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
     let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -95,7 +100,10 @@ pub async fn tier_mutation_control_time_out_client(
     interceptor: TonicInterceptor,
 ) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
     let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
     Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -110,12 +118,22 @@ pub async fn tier_mutation_control_time_out_client(
 /// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
 /// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
 pub async fn node_service_time_out_client_for_class(
-    addr: &str,
+    addr: &String,
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     let interceptor = interceptor.with_rpc_audience(addr)?;
-    let channel = get_channel_for_class(addr, class).await?;
+    let channel = match class {
+        ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
+            Some(channel) => {
+                debug!("Using cached gRPC channel for: {}", addr);
+                channel
+            }
+            // No cached connection, create new one.
+            None => create_new_channel(addr).await?,
+        },
+        ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
+    };
 
     let max_message_size = rustfs_protos::internode_rpc_max_message_size();
     let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -124,24 +142,8 @@ pub async fn node_service_time_out_client_for_class(
         .max_encoding_message_size(max_message_size))
 }
 
-pub(crate) async fn node_service_delete_recovery_client(
-    addr: &str,
-    interceptor: TonicInterceptor,
-    timeout: std::time::Duration,
-) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
-    let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
-    let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
-        .await
-        .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
-    let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
-    let max_message_size = rustfs_protos::internode_rpc_max_message_size();
-    Ok(NodeServiceClient::with_interceptor(channel, interceptor)
-        .max_decoding_message_size(max_message_size)
-        .max_encoding_message_size(max_message_size))
-}
-
 pub async fn node_service_time_out_client_no_auth(
-    addr: &str,
+    addr: &String,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
     node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
 }
@@ -353,15 +355,6 @@ fn apply_peer_replay_response(
     }
 }
 
-fn rename_data_grpc_stage(path: &str) -> bool {
-    matches!(
-        path,
-        "/node_service.NodeService/RenameData"
-            | "/node_service.NodeService/RenameDataAtIncarnation"
-            | "/node_service.NodeService/RenameDataWithQuotaFence"
-    )
-}
-
 impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
 where
     S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
@@ -379,12 +372,6 @@ where
     }
 
     fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
-        let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path());
-        let request_scope_started = if observe_rename_data {
-            rustfs_io_metrics::put_stage_timer()
-        } else {
-            None
-        };
         let authenticated = self.audience.as_ref().is_some_and(|_| {
             request
                 .headers()
@@ -419,30 +406,11 @@ where
                 }
             }
         }
-        rustfs_io_metrics::record_put_object_stage_duration_from(
-            rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE,
-            request_scope_started,
-        );
 
         let audience = self.audience.clone();
         let future = self.inner.call(request);
         Box::pin(async move {
-            let response = if observe_rename_data {
-                rustfs_io_metrics::observe_put_stage_future(
-                    future,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY,
-                )
-                .await?
-            } else {
-                future.await?
-            };
-            let replay_response_started = if observe_rename_data {
-                rustfs_io_metrics::put_stage_timer()
-            } else {
-                None
-            };
+            let response = future.await?;
             if let (Some(audience), Some(challenge)) = (audience, challenge) {
                 let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
                 if let Err(error) = &response_state
@@ -460,10 +428,6 @@ where
                 }
                 apply_peer_replay_response(audience, sent_state, response_state);
             }
-            rustfs_io_metrics::record_put_object_stage_duration_from(
-                rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE,
-                replay_response_started,
-            );
             Ok(response)
         })
     }
@@ -582,126 +546,6 @@ mod tests {
     use tracing_opentelemetry::OpenTelemetrySpanExt;
     use tracing_subscriber::{Registry, layer::SubscriberExt};
 
-    #[tokio::test]
-    async fn delete_recovery_channel_uses_its_scoped_deadline() {
-        use bytes::Bytes;
-        use http_body_util::StreamBody;
-        use hyper::{body::Frame, service::service_fn};
-        use hyper_util::rt::{TokioExecutor, TokioIo};
-        use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
-        use tonic_prost::prost::Message;
-        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
-            .await
-            .expect("bind recovery peer");
-        let addr = format!("http://{}", listener.local_addr().expect("peer address"));
-        let server = tokio::spawn(async move {
-            loop {
-                let (stream, _) = listener.accept().await.expect("accept recovery connection");
-                tokio::spawn(async move {
-                    let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
-                        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
-                        let payload = RecoverDeleteTransactionResponse {
-                            success: true,
-                            protocol_version: 1,
-                            result: Bytes::new(),
-                            error: None,
-                        }
-                        .encode_to_vec();
-                        let mut data = vec![0];
-                        data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
-                        data.extend_from_slice(&payload);
-                        let mut trailers = http::HeaderMap::new();
-                        trailers.insert("grpc-status", "0".parse().expect("status"));
-                        let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
-                            vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
-                        let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
-                        response
-                            .headers_mut()
-                            .insert("content-type", "application/grpc".parse().expect("content type"));
-                        Ok::<_, std::convert::Infallible>(response)
-                    });
-                    let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
-                        .serve_connection(TokioIo::new(stream), service)
-                        .await;
-                });
-            }
-        });
-        let request = || RecoverDeleteTransactionRequest {
-            disk: "disk".into(),
-            volume: "bucket".into(),
-            protocol_version: 1,
-            operation: Bytes::new(),
-        };
-        // Seed an existing short-lived control channel to reproduce the inherited deadline.
-        let control = tonic::transport::Endpoint::from_shared(addr.clone())
-            .expect("test endpoint")
-            .timeout(std::time::Duration::from_millis(50))
-            .connect()
-            .await
-            .expect("control connect");
-        rustfs_common::cache_connection(addr.clone(), control).await;
-        let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
-            .await
-            .expect("control client");
-        let error = foreground
-            .recover_delete_transaction(request())
-            .await
-            .expect_err("short control deadline");
-        assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
-        // The 2-second test override bypasses the production range parser, which rejects 2.
-        let mut recovery = node_service_delete_recovery_client(
-            &addr,
-            TonicInterceptor::NoOp(NoOpInterceptor),
-            std::time::Duration::from_secs(2),
-        )
-        .await
-        .expect("scoped recovery client");
-        assert!(
-            recovery
-                .recover_delete_transaction(request())
-                .await
-                .expect("scoped deadline must survive control deadline")
-                .into_inner()
-                .success
-        );
-        server.abort();
-        rustfs_protos::evict_failed_connection(&addr).await;
-    }
-
-    #[tokio::test]
-    async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
-        let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
-        let addr = format!("https://{}", listener.local_addr().expect("local address"));
-        drop(listener);
-        for factory in 0..4 {
-            let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
-            runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
-            let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
-            let _result = match factory {
-                0 => scanner_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                1 => heal_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                2 => tier_mutation_control_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-                _ => node_service_time_out_client(&addr, interceptor)
-                    .await
-                    .map(|_| ())
-                    .map_err(|e| e.to_string()),
-            };
-            assert!(
-                !runtime_sources::test_node_channel_is_cached(&addr).await,
-                "factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
-            );
-        }
-    }
-
     #[derive(Clone)]
     struct EpochProofService {
         audience: String,

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -289,9 +289,13 @@ async fn client(
     >,
     NetworkPeerProbeError,
 > {
-    node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
-        .await
-        .map_err(|_| NetworkPeerProbeError::Unreachable)
+    node_service_time_out_client_for_class(
+        &address.to_owned(),
+        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
+        class,
+    )
+    .await
+    .map_err(|_| NetworkPeerProbeError::Unreachable)
 }
 
 fn ping_request(payload: &[u8]) -> PingRequest {
@@ -325,24 +329,6 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
     match status.code() {
         Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
         Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
-        Code::Cancelled | Code::Internal => {
-            let mut source = std::error::Error::source(&status);
-            while let Some(error) = source {
-                if error
-                    .downcast_ref::<h2::Error>()
-                    .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
-                {
-                    return NetworkPeerProbeError::Unreachable;
-                }
-                source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
-                    // io::Error::source skips its immediate custom error.
-                    error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
-                } else {
-                    error.source()
-                };
-            }
-            NetworkPeerProbeError::ProtocolFailure
-        }
         _ => NetworkPeerProbeError::ProtocolFailure,
     }
 }
@@ -353,35 +339,6 @@ mod tests {
     use crate::layout::endpoint::Endpoint;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
-    #[test]
-    fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
-        for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
-            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
-            assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
-            let code = status.code();
-            assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
-            let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
-            wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
-            assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
-        }
-        for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
-            let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
-            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
-        }
-        for status in [
-            tonic::Status::cancelled("peer cancelled the application request"),
-            tonic::Status::internal("peer rejected the probe"),
-            tonic::Status::unauthenticated("authentication rejected"),
-            tonic::Status::permission_denied("authorization rejected"),
-        ] {
-            assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
-        }
-        assert_eq!(
-            map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
-            NetworkPeerProbeError::TimedOut
-        );
-    }
-
     fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
         NetworkPeerProbeClient::from_endpoint_pools(&topology())
             .with_diagnostic_pacing(Instant::now(), duration)

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -14,8 +14,7 @@
 
 use crate::cluster::rpc::client::{
     AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
-    node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
-    node_service_time_out_client_no_auth,
+    node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
 };
 use crate::cluster::rpc::internode_data_transport::{
     InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -23,12 +22,10 @@ use crate::cluster::rpc::internode_data_transport::{
 };
 use crate::disk::error::{Error, Result};
 use crate::disk::{
-    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
-    DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
-    DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
-    DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, GuardedFileInfoVersions,
-    PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken,
-    UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
+    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
+    DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
+    ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
+    batch_read_version_one_by_one,
     disk_store::{
         DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
         get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -60,12 +57,12 @@ use rustfs_protos::proto_gen::node_service::RenamePartRequest;
 use rustfs_protos::proto_gen::node_service::{
     BatchReadVersionRequest, BatchReadVersionResponse, CheckPartsRequest, CompareAndUpdateFileOutcome,
     CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
-    DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, GuardedDeleteVersionsRequest, ListDirRequest,
-    ListVolumesRequest, MakeVolumeRequest, MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest,
-    ReadMetadataRequest, ReadMultipleRequest, ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest,
-    RecoverDeleteTransactionRequest, RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest,
-    SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest,
-    UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, node_service_client::NodeServiceClient,
+    DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
+    MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
+    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
+    SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
+    SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
+    node_service_client::NodeServiceClient,
 };
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
@@ -145,45 +142,6 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
         .collect()
 }
 
-fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
-    if codes.len() != expected_len {
-        return Err(Error::other("malformed recoverable batch delete response length"));
-    }
-    codes
-        .into_iter()
-        .map(|code| match code {
-            None => Ok(None),
-            Some(code) => DiskError::from_u32(code)
-                .map(Some)
-                .ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
-        })
-        .collect()
-}
-
-fn validate_delete_recovery_inspection(
-    volume: &str,
-    object: &str,
-    transaction: Uuid,
-    inspection: &DeleteRecoveryInspection,
-) -> Result<()> {
-    if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-        || inspection.id.object != object
-        || inspection.id.transaction != transaction
-        || transaction.is_nil()
-    {
-        return Err(Error::other("delete recovery inspection does not match request"));
-    }
-    if let Some(journal) = &inspection.journal
-        && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
-            || journal.volume != volume
-            || journal.object != object
-            || journal.transaction != transaction)
-    {
-        return Err(Error::other("delete recovery journal does not match request"));
-    }
-    Ok(())
-}
-
 fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
     if !response.success {
         return Err(response.error.unwrap_or_default().into());
@@ -218,24 +176,6 @@ fn attach_mutation_body_digest<T>(
     crate::cluster::rpc::set_tonic_rolling_canonical_body_digest(request, &canonical_body).map_err(Error::other)
 }
 
-fn attach_quota_mutation_body_digest<T>(
-    request: &mut Request<T>,
-    canonical_body: std::result::Result<Vec<u8>, std::num::TryFromIntError>,
-) -> Result<()> {
-    let topology = crate::services::notification_sys::configured_fleet_topology_fingerprint()
-        .ok_or_else(|| Error::other("quota mutation topology is not initialized or conflicts with startup"))?;
-    let body = canonical_body.map_err(|_| Error::other("quota request length cannot be represented"))?;
-    let body = rustfs_protos::canonical_quota_mutation_request_body(topology, &body)
-        .map_err(|_| Error::other("quota topology envelope length cannot be represented"))?;
-    request.metadata_mut().insert(
-        rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER,
-        topology
-            .parse()
-            .map_err(|_| Error::other("quota topology cannot be encoded"))?,
-    );
-    crate::cluster::rpc::set_tonic_canonical_body_digest(request, &body).map_err(Error::other)
-}
-
 fn decode_volume_infos(volume_infos: Vec<String>) -> Result<Vec<VolumeInfo>> {
     volume_infos
         .into_iter()
@@ -918,132 +858,6 @@ fn spawn_control_channel_prewarm(addr: String) {
 }
 
 impl RemoteDisk {
-    async fn guarded_delete_versions_rpc(
-        &self,
-        volume: &str,
-        payload: Vec<u8>,
-        probe_only: bool,
-    ) -> Result<DeleteVersionsResponse> {
-        if payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES {
-            return Err(DiskError::FileCorrupt);
-        }
-        self.execute_with_timeout(
-            || async {
-                let mut client = self.get_client().await?;
-                let mut request = Request::new(GuardedDeleteVersionsRequest {
-                    disk: self.endpoint.to_string(),
-                    volume: volume.to_owned(),
-                    payload: payload.into(),
-                    probe_only,
-                });
-                let body = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref())
-                    .map_err(|_| DiskError::FileCorrupt)?;
-                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?;
-                // The additive method is mandatory: an old peer cannot safely delete
-                // a guarded request through any legacy batch method.
-                let response = client
-                    .guarded_delete_versions(request)
-                    .await
-                    .map_err(|status| {
-                        if status.code() == Code::Unimplemented {
-                            DiskError::GuardedDeleteUnsupported
-                        } else {
-                            status.into()
-                        }
-                    })?
-                    .into_inner();
-                if !response.success {
-                    return Err(response.error.ok_or(DiskError::FileCorrupt)?.into());
-                }
-                if response.error.is_some() {
-                    return Err(DiskError::FileCorrupt);
-                }
-                Ok(response)
-            },
-            get_max_timeout_duration(),
-        )
-        .await
-    }
-
-    pub(crate) async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result<bool> {
-        match self
-            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: path.to_owned() })
-            .await?
-        {
-            DeleteRecoveryRpcReply::OrphanDataDir { plain } => Ok(plain),
-            _ => Err(Error::other("unexpected orphan data directory inspection reply")),
-        }
-    }
-
-    async fn recover_delete_transaction(
-        &self,
-        volume: &str,
-        operation: DeleteRecoveryRpcOperation,
-    ) -> Result<DeleteRecoveryRpcReply> {
-        let payload = rmp_serde::to_vec_named(&operation)?;
-        if payload.len() > operation.max_wire_payload_bytes() {
-            return Err(Error::other("delete recovery operation exceeds payload limit"));
-        }
-        let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
-            Some(crate::disk::get_delete_recovery_timeout()?)
-        } else {
-            None
-        };
-        self.execute_with_timeout(
-            || async {
-                let mut client = if let Some(timeout) = settle_timeout {
-                    if let Some(error) = self.offline_bypass_error() {
-                        return Err(error);
-                    }
-                    node_service_delete_recovery_client(
-                        &self.addr,
-                        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
-                        timeout,
-                    )
-                    .await?
-                } else {
-                    self.get_client().await?
-                };
-                let mut request = Request::new(RecoverDeleteTransactionRequest {
-                    disk: self.endpoint.to_string(),
-                    volume: volume.to_owned(),
-                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-                    operation: payload.into(),
-                });
-                let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
-                    .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
-                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
-                // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
-                let response = client
-                    .recover_delete_transaction(request)
-                    .await
-                    .map_err(|status| {
-                        if status.code() == Code::Unimplemented {
-                            DiskError::UnsupportedDisk
-                        } else {
-                            status.into()
-                        }
-                    })?
-                    .into_inner();
-                if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
-                    return Err(DiskError::UnsupportedDisk);
-                }
-                if !response.success {
-                    return Err(response
-                        .error
-                        .ok_or_else(|| Error::other("missing delete recovery error"))?
-                        .into());
-                }
-                if response.error.is_some() {
-                    return Err(Error::other("malformed delete recovery success"));
-                }
-                rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
-            },
-            settle_timeout.unwrap_or_else(get_max_timeout_duration),
-        )
-        .await
-    }
-
     async fn rename_file_with_durability(
         &self,
         src_volume: &str,
@@ -2106,12 +1920,6 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
 }
 
-// Optional fields are omitted by serde; named maps keep rollback flags and
-// transaction UUIDs independent of field order across peer versions.
-fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
-    Ok(rmp_serde::to_vec_named(value)?)
-}
-
 fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
 }
@@ -2355,7 +2163,6 @@ impl RemoteDisk {
         self.execute_with_timeout_for_op(
             "rename_data",
             || async {
-                let prepare_started = rustfs_io_metrics::put_stage_timer();
                 let file_info = compat_json(fi)?;
                 let file_info_bin = encode_file_info_msgpack(fi)?;
                 let mut client = self.get_client().await?;
@@ -2376,83 +2183,31 @@ impl RemoteDisk {
                 });
                 let canonical_body = rustfs_protos::canonical_rename_data_request_body(request.get_ref());
                 let incarnation_bound = !request.get_ref().bucket_incarnation_id.is_empty();
-                let quota_fenced = rustfs_utils::http::metadata_compat::contains_key_str(
-                    &fi.metadata,
-                    crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX,
-                );
-                if quota_fenced {
-                    attach_quota_mutation_body_digest(&mut request, canonical_body)?;
-                } else if scanner_publication_lease_token.is_some() || incarnation_bound {
+                if scanner_publication_lease_token.is_some() || incarnation_bound {
                     let canonical_body =
                         canonical_body.map_err(|_| Error::other("rename_data request length cannot be represented"))?;
                     crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
                 } else {
                     attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
                 }
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
-                    prepare_started,
-                );
 
-                let rpc_started = rustfs_io_metrics::put_stage_timer();
-                let response = rustfs_io_metrics::observe_put_stage_future(
-                    async {
-                        if quota_fenced {
-                            // The method itself proves quota-token enforcement; an old peer must fail closed.
-                            client.rename_data_with_quota_fence(request).await
-                        } else if incarnation_bound {
-                            // Older peers return Unimplemented before mutation; never downgrade.
-                            client.rename_data_at_incarnation(request).await
-                        } else {
-                            client.rename_data(request).await
-                        }
-                    },
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
-                )
-                .await?;
-                let response_process_started = rustfs_io_metrics::put_stage_timer();
-                let response_headers_started = rustfs_io_metrics::put_stage_timer();
-                let _ = response.metadata().len();
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS,
-                    response_headers_started,
-                );
-                let into_inner_started = rustfs_io_metrics::put_stage_timer();
-                let response = response.into_inner();
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER,
-                    into_inner_started,
-                );
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC,
-                    rpc_started,
-                );
+                let response = if incarnation_bound {
+                    // Older peers return Unimplemented before mutation; never downgrade.
+                    client.rename_data_at_incarnation(request).await?
+                } else {
+                    client.rename_data(request).await?
+                }
+                .into_inner();
 
-                let success_check_started = rustfs_io_metrics::put_stage_timer();
                 if !response.success {
                     return Err(response.error.unwrap_or_default().into());
                 }
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK,
-                    success_check_started,
-                );
 
-                let decode_started = rustfs_io_metrics::put_stage_timer();
                 let rename_data_resp = decode_msgpack_or_json::<RenameDataResp>(
                     &response.rename_data_resp_bin,
                     &response.rename_data_resp,
                     "RenameDataResp",
                 )?;
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE,
-                    decode_started,
-                );
-                rustfs_io_metrics::record_put_object_stage_duration_from(
-                    rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS,
-                    response_process_started,
-                );
 
                 Ok(rename_data_resp)
             },
@@ -2533,39 +2288,6 @@ impl RemoteDisk {
 
 #[async_trait::async_trait]
 impl DiskAPI for RemoteDisk {
-    async fn supports_guarded_delete_versions(&self) -> Result<()> {
-        let response = self.guarded_delete_versions_rpc("", Vec::new(), true).await?;
-        if !response.errors.is_empty() || !response.item_errors.is_empty() {
-            return Err(DiskError::FileCorrupt);
-        }
-        Ok(())
-    }
-    async fn delete_versions_guarded_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<GuardedFileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        let count = versions.len();
-        let payload = GuardedDeleteVersionsPayload { versions, opts };
-        if let Err(error) = payload.validate() {
-            return vec![Some(error); count];
-        }
-        let encoded = match rmp_serde::to_vec_named(&payload) {
-            Ok(encoded) => encoded,
-            Err(error) => return vec![Some(error.into()); count],
-        };
-        match self.guarded_delete_versions_rpc(volume, encoded, false).await {
-            Ok(response) => {
-                // New guarded peers must return typed errors, never lossy strings.
-                if response.item_errors.len() != count {
-                    return vec![Some(DiskError::FileCorrupt); count];
-                }
-                decode_delete_versions_errors(response, count)
-            }
-            Err(error) => vec![Some(error); count],
-        }
-    }
     #[tracing::instrument(level = "trace", skip_all)]
     fn to_string(&self) -> String {
         self.endpoint.to_string()
@@ -2844,7 +2566,7 @@ impl DiskAPI for RemoteDisk {
                 // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
                 // JSON + msgpack until its fallback counter has read zero across a release window.
                 let file_info_bin = encode_file_info_msgpack(&fi)?;
-                let opts_bin = encode_delete_options_msgpack(&opts)?;
+                let opts_bin = encode_msgpack(&opts)?;
                 let conditional_marker = opts.expected_delete_marker.is_some();
                 let file_info = serde_json::to_string(&fi)?;
                 let opts = serde_json::to_string(&opts)?;
@@ -2916,7 +2638,7 @@ impl DiskAPI for RemoteDisk {
 
         // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
         // msgpack until its fallback counter has read zero across a release window.
-        let opts_bin = match encode_delete_options_msgpack(&opts) {
+        let opts_bin = match encode_msgpack(&opts) {
             Ok(opts_bin) => opts_bin,
             Err(err) => {
                 let mut errors = Vec::with_capacity(versions.len());
@@ -3073,15 +2795,8 @@ impl DiskAPI for RemoteDisk {
                         .map_err(|_| Error::other("snapshot lease TTL cannot be represented"))?,
                 });
                 let canonical_body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref());
-                let quota_fence = volume == crate::disk::RUSTFS_META_BUCKET && crate::disk::is_quota_mutation_fence_path(path);
-                let response = if quota_fence {
-                    attach_quota_mutation_body_digest(&mut request, canonical_body)?;
-                    client.acquire_quota_mutation_fence(request).await?
-                } else {
-                    attach_mutation_body_digest(&mut request, canonical_body, "acquire_snapshot_lease")?;
-                    client.acquire_snapshot_lease(request).await?
-                }
-                .into_inner();
+                attach_mutation_body_digest(&mut request, canonical_body, "acquire_snapshot_lease")?;
+                let response = client.acquire_snapshot_lease(request).await?.into_inner();
                 snapshot_lease_token_from_response(response)
             },
             get_max_timeout_duration(),
@@ -3122,15 +2837,8 @@ impl DiskAPI for RemoteDisk {
                     token: token.as_bytes().to_vec().into(),
                 });
                 let canonical_body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref());
-                let quota_fence = volume == crate::disk::RUSTFS_META_BUCKET && crate::disk::is_quota_mutation_fence_path(path);
-                let response = if quota_fence {
-                    attach_quota_mutation_body_digest(&mut request, canonical_body)?;
-                    client.release_quota_mutation_fence(request).await?
-                } else {
-                    attach_mutation_body_digest(&mut request, canonical_body, "release_snapshot_lease")?;
-                    client.release_snapshot_lease(request).await?
-                }
-                .into_inner();
+                attach_mutation_body_digest(&mut request, canonical_body, "release_snapshot_lease")?;
+                let response = client.release_snapshot_lease(request).await?.into_inner();
                 if !response.success {
                     return Err(response.error.unwrap_or_default().into());
                 }
@@ -3325,7 +3033,7 @@ impl DiskAPI for RemoteDisk {
                     opts_bin: opts_bin.into(),
                 });
 
-                crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts);
+                crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request();
                 if let Some(request_payload_bytes) = request_payload_bytes {
                     crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes);
                 }
@@ -3403,7 +3111,6 @@ impl DiskAPI for RemoteDisk {
         record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started);
         let request_payload_bytes = batch_read_version_attribution_enabled
             .then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin));
-        let request_opts = req.opts;
         let batch_result = self
             .execute_with_timeout_for_op(
                 "batch_read_version",
@@ -3417,7 +3124,7 @@ impl DiskAPI for RemoteDisk {
                         batch_read_version_req_bin: batch_read_version_req_bin.into(),
                     });
 
-                    crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts);
+                    crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request();
                     if let Some(request_payload_bytes) = request_payload_bytes {
                         crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes(
                             request_payload_bytes.saturating_add(disk_len),
@@ -3910,153 +3617,6 @@ impl DiskAPI for RemoteDisk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        opts: DeleteOptions,
-    ) -> Result<()> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::ApplyVersion {
-                    object: object.to_owned(),
-                    file_info: Box::new(file_info),
-                    force_del_marker,
-                    opts,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
-            _ => Err(Error::other("unexpected recoverable delete reply")),
-        }
-    }
-
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        let expected_len = versions.len();
-        let result = match self
-            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
-            .await
-        {
-            Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
-            Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
-            Err(error) => Err(error),
-        };
-        match result {
-            Ok(errors) => errors,
-            Err(error) => vec![Some(error); expected_len],
-        }
-    }
-
-    async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result<bool> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Exists {
-                    object: object.to_owned(),
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Evidence { present } => Ok(present),
-            _ => Err(Error::other("unexpected delete recovery existence reply")),
-        }
-    }
-
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<DeleteRecoveryPage> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Discover {
-                    object: object.map(str::to_owned),
-                    cursor,
-                    limit,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Discovered(page) => {
-                if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
-                    return Err(Error::other("delete recovery discovery escaped requested object"));
-                }
-                if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
-                    || page
-                        .transactions
-                        .iter()
-                        .any(|id| id.transaction.is_nil() || id.object.is_empty())
-                {
-                    return Err(Error::other("malformed delete discovery page"));
-                }
-                Ok(page)
-            }
-            _ => Err(Error::other("unexpected delete discovery reply")),
-        }
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Inspect {
-                    object: object.to_owned(),
-                    transaction,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Inspected(inspection) => {
-                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
-                Ok(inspection)
-            }
-            _ => Err(Error::other("unexpected delete inspection reply")),
-        }
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: Uuid,
-        expected_fingerprint: DeleteRecoveryDigest,
-        action: DeleteRecoveryAction,
-    ) -> Result<DeleteRecoveryInspection> {
-        match self
-            .recover_delete_transaction(
-                volume,
-                DeleteRecoveryRpcOperation::Settle {
-                    object: object.to_owned(),
-                    transaction,
-                    expected_fingerprint,
-                    action,
-                },
-            )
-            .await?
-        {
-            DeleteRecoveryRpcReply::Settled(inspection) => {
-                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
-                Ok(inspection)
-            }
-            _ => Err(Error::other("unexpected delete settlement reply")),
-        }
-    }
-
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
         self.execute_with_timeout(
             || async {
@@ -4470,140 +4030,6 @@ mod tests {
 
     static INIT: Once = Once::new();
 
-    #[tokio::test]
-    async fn quota_fence_old_peer_rejects_without_legacy_fallback() {
-        runtime_sources::ensure_test_rpc_secret();
-        crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test();
-        let _without_proof = crate::services::notification_sys::without_live_cross_pool_fence_fleet_proof_for_test();
-        assert!(crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_none());
-        assert!(crate::services::notification_sys::configured_fleet_topology_fingerprint().is_some());
-        let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new())
-            .await
-            .expect("old peer must bind");
-        let remote = remote_disk_for_addr(&peer.addr).await;
-        let fence_path = crate::disk::quota_mutation_fence_path("bucket", "object");
-        assert!(
-            remote
-                .acquire_snapshot_lease(crate::disk::RUSTFS_META_BUCKET, &fence_path)
-                .await
-                .is_err()
-        );
-        assert!(
-            remote
-                .release_snapshot_lease(crate::disk::RUSTFS_META_BUCKET, &fence_path, SnapshotLeaseToken::revoke_all())
-                .await
-                .is_err()
-        );
-        let mut file_info = sample_rename_data_file_info();
-        rustfs_utils::http::metadata_compat::insert_str(
-            &mut file_info.metadata,
-            crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX,
-            Uuid::new_v4().to_string(),
-        );
-        assert!(
-            remote
-                .rename_data("tmp", "source", file_info, "bucket", "object")
-                .await
-                .is_err()
-        );
-        let methods = peer
-            .peer
-            .unsupported_methods
-            .lock()
-            .expect("unsupported method list lock")
-            .clone();
-        assert_eq!(
-            methods,
-            vec![
-                "/node_service.NodeService/AcquireQuotaMutationFence",
-                "/node_service.NodeService/ReleaseQuotaMutationFence",
-                "/node_service.NodeService/RenameDataWithQuotaFence",
-            ],
-            "quota operations must never fall back to snapshot or unfenced rename methods"
-        );
-        peer.stop().await;
-    }
-
-    #[tokio::test]
-    async fn guarded_delete_old_peer_rejects_without_legacy_fallback() {
-        runtime_sources::ensure_test_rpc_secret();
-        let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new())
-            .await
-            .expect("old peer must bind");
-        let remote = remote_disk_for_addr(&peer.addr).await;
-        assert_eq!(remote.supports_guarded_delete_versions().await, Err(DiskError::GuardedDeleteUnsupported));
-        let group = GuardedFileInfoVersions {
-            group: FileInfoVersions {
-                name: "object".into(),
-                versions: vec![FileInfo::default()],
-                ..Default::default()
-            },
-            expected_current: crate::disk::CurrentDeleteGeneration::Absent,
-        };
-        let errors = remote
-            .delete_versions_guarded_recoverable(
-                "bucket",
-                vec![group],
-                DeleteOptions {
-                    old_data_dir: Some(Uuid::new_v4()),
-                    ..Default::default()
-                },
-            )
-            .await;
-        assert_eq!(errors, vec![Some(DiskError::GuardedDeleteUnsupported)]);
-        let methods = peer
-            .peer
-            .unsupported_methods
-            .lock()
-            .expect("unsupported method list lock")
-            .clone();
-        assert_eq!(
-            methods,
-            vec!["/node_service.NodeService/GuardedDeleteVersions"; 2],
-            "neither capability failure nor guarded application may call a legacy deletion method"
-        );
-        peer.stop().await;
-    }
-
-    #[test]
-    fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
-        let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
-            .expect("valid per-item errors");
-        assert!(errors[0].is_none());
-        assert_eq!(errors[1], Some(DiskError::FileNotFound));
-        assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
-        assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
-        assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
-    }
-
-    #[test]
-    fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
-        let transaction = Uuid::new_v4();
-        let inspection = DeleteRecoveryInspection {
-            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
-            id: crate::disk::DeleteTransactionId {
-                object: "object".into(),
-                transaction,
-            },
-            state: crate::disk::DeleteRecoveryState::Absent,
-            journal: None,
-            current_digest: None,
-            current_versions: Vec::new(),
-            backup_digest: None,
-            rollback_data_complete: false,
-            commit_data_safe: false,
-            explicit_delete_safe: false,
-            fingerprint: [0; 32],
-        };
-        validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
-        assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
-        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
-        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
-        let mut incompatible = inspection;
-        incompatible.protocol_version += 1;
-        assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
-    }
-
     #[test]
     fn compare_and_update_wire_outcome_fails_closed() {
         assert_eq!(
@@ -4637,19 +4063,32 @@ mod tests {
     }
 
     #[test]
-    fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
+    fn retired_marker_options_preserve_legacy_positional_wire_shape() {
         #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
         struct LegacyDeleteOptions {
             recursive: bool,
             immediate: bool,
             undo_write: bool,
-            #[serde(default)]
             undo_delete: bool,
             old_data_dir: Option<Uuid>,
-            #[serde(default, skip_serializing_if = "Option::is_none")]
-            expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
         }
-        let transaction = Uuid::new_v4();
+        let legacy = LegacyDeleteOptions {
+            recursive: false,
+            immediate: false,
+            undo_write: false,
+            undo_delete: false,
+            old_data_dir: None,
+        };
+        let original = encode_msgpack(&legacy).unwrap();
+        let current = encode_msgpack(&DeleteOptions::default()).unwrap();
+        assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
+        assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
+        assert!(
+            rmp_serde::from_slice::<DeleteOptions>(&original)
+                .unwrap()
+                .expected_delete_marker
+                .is_none()
+        );
         let mut marker = FileInfo {
             deleted: true,
             version_id: Some(Uuid::new_v4()),
@@ -4658,69 +4097,12 @@ mod tests {
         };
         marker.set_delete_marker_incarnation(Uuid::new_v4());
         let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
-        for expected_delete_marker in [None, Some(marker)] {
-            for (undo_write, undo_delete) in [(true, false), (false, true)] {
-                let legacy = LegacyDeleteOptions {
-                    recursive: false,
-                    immediate: false,
-                    undo_write,
-                    undo_delete,
-                    old_data_dir: Some(transaction),
-                    expected_delete_marker: expected_delete_marker.clone(),
-                };
-                // Older senders use positional arrays. Their unchanged field prefix and
-                // missing optional fields must also decode on the named-map sender's peer.
-                for (encoding, payload) in [
-                    ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
-                    ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
-                ] {
-                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
-                    assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
-                    assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
-                    assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
-                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
-                    assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
-                    assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
-                }
-                for incarnation in [None, Some(Uuid::new_v4())] {
-                    let current = DeleteOptions {
-                        undo_write,
-                        undo_delete,
-                        old_data_dir: Some(transaction),
-                        expected_delete_marker: expected_delete_marker.clone(),
-                        delete_recovery_bucket_incarnation: incarnation,
-                        recoverable_delete: true,
-                        ..Default::default()
-                    };
-                    let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
-                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
-                    assert_eq!(decoded.undo_write, undo_write);
-                    assert_eq!(decoded.undo_delete, undo_delete);
-                    assert_eq!(decoded.old_data_dir, Some(transaction));
-                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
-                    assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
-                    assert!(decoded.recoverable_delete);
-                    let decoded: LegacyDeleteOptions =
-                        rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
-                    assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
-                }
-            }
-        }
-    }
-
-    #[test]
-    fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
         let options = DeleteOptions {
-            undo_write: true,
-            old_data_dir: Some(Uuid::new_v4()),
+            expected_delete_marker: Some(marker.clone()),
             ..Default::default()
         };
-        let json = serde_json::to_string(&options).expect("valid compatibility JSON");
-        assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
-        let decoded: DeleteOptions =
-            decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
-        assert!(decoded.undo_write);
-        assert_eq!(decoded.old_data_dir, options.old_data_dir);
+        let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
+        assert_eq!(decoded.expected_delete_marker, Some(marker));
     }
 
     #[test]
@@ -4953,7 +4335,6 @@ mod tests {
         format_data: Bytes,
         read_all_data: Bytes,
         next_format_read_gate: Arc<StdMutex<Option<Arc<TestFormatReadGate>>>>,
-        unsupported_methods: Arc<StdMutex<Vec<String>>>,
     }
 
     #[derive(Debug, Default)]
@@ -4972,7 +4353,6 @@ mod tests {
                 format_data,
                 read_all_data,
                 next_format_read_gate: Arc::default(),
-                unsupported_methods: Arc::default(),
             }
         }
 
@@ -5119,20 +4499,13 @@ mod tests {
                         Ok(grpc.unary(method, request).await)
                     })
                 }
-                _ => {
-                    self.peer
-                        .unsupported_methods
-                        .lock()
-                        .expect("unsupported method list lock")
-                        .push(request.uri().path().to_owned());
-                    Box::pin(async move {
-                        let mut response = http::Response::new(tonic::body::Body::default());
-                        let headers = response.headers_mut();
-                        headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into());
-                        headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE);
-                        Ok(response)
-                    })
-                }
+                _ => Box::pin(async move {
+                    let mut response = http::Response::new(tonic::body::Body::default());
+                    let headers = response.headers_mut();
+                    headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into());
+                    headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE);
+                    Ok(response)
+                }),
             }
         }
     }
@@ -5412,8 +4785,6 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![0x14, 0x35]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)),
-            old_current_source_checked: false,
-            old_current_source: None,
         };
         let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode");
         let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test();
@@ -5816,8 +5187,6 @@ mod tests {
             cleanup_data_dir: Some(Uuid::new_v4()),
             sign: Some(vec![1_u8; 32]),
             old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)),
-            old_current_source_checked: false,
-            old_current_source: None,
         };
         let json = serde_json::to_vec(&response).expect("rename data response json should encode");
         let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode");

crates/ecstore/src/config/com.rs

@@ -495,10 +495,22 @@ pub(crate) async fn read_config_preserve_empty<S>(api: Arc<S>, file: &str) -> Re
 where
     S: EcstoreObjectIO,
 {
-    let (data, _obj) = read_config_with_metadata_inner(api, file, &ObjectOptions::default(), true, None).await?;
+    let (data, _obj) = read_config_preserve_empty_with_metadata_opts(api, file, &ObjectOptions::default()).await?;
     Ok(data)
 }
 
+/// Preserve an empty config payload, its metadata, and caller-supplied read options.
+pub(crate) async fn read_config_preserve_empty_with_metadata_opts<S>(
+    api: Arc<S>,
+    file: &str,
+    opts: &ObjectOptions,
+) -> Result<(Vec<u8>, ObjectInfo)>
+where
+    S: EcstoreObjectIO,
+{
+    read_config_with_metadata_inner(api, file, opts, true, None).await
+}
+
 pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
 where
     S: ObjectIO<
@@ -527,15 +539,13 @@ pub(crate) async fn read_config_no_lock_preserve_empty_with_metadata<S>(api: Arc
 where
     S: EcstoreObjectIO,
 {
-    read_config_with_metadata_inner(
+    read_config_preserve_empty_with_metadata_opts(
         api,
         file,
         &ObjectOptions {
             no_lock: true,
             ..Default::default()
         },
-        true,
-        None,
     )
     .await
 }
@@ -574,20 +584,17 @@ where
         >,
 {
     let h = HeaderMap::new();
-    let read = api.get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts);
-    #[cfg(feature = "diagnostic-read-version-caller")]
-    let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigRead,
-        read,
-    );
-    let mut rd = read.await.map_err(|err| {
-        if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
-            Error::ConfigNotFound
-        } else {
-            warn!("read_config_with_metadata: err: {:?}, file: {}", err, file);
-            err
-        }
-    })?;
+    let mut rd = api
+        .get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts)
+        .await
+        .map_err(|err| {
+            if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) {
+                Error::ConfigNotFound
+            } else {
+                warn!("read_config_with_metadata: err: {:?}, file: {}", err, file);
+                err
+            }
+        })?;
 
     let data = if let Some(max_bytes) = max_bytes {
         let object_size = usize::try_from(rd.object_info.size).map_err(|_| Error::CorruptedFormat)?;
@@ -810,18 +817,7 @@ where
         >,
 {
     let mut put_data = PutObjReader::from_vec(data);
-    let write = api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts);
-    #[cfg(feature = "diagnostic-read-version-caller")]
-    let write = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigWrite,
-        async {
-            crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary(
-                rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ConfigWrite,
-            );
-            write.await
-        },
-    );
-    match write.await {
+    match api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await {
         Ok(object_info) => Ok(object_info),
         Err(err) => {
             if log_error {
@@ -2829,9 +2825,10 @@ mod tests {
         config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
         heal_config_descriptor, is_standard_object_server_config, lookup_configs, map_system_metadata_write_error,
         new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty,
-        read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot, save_config_with_opts_inner,
-        save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
-        server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
+        read_config_preserve_empty_with_metadata_opts, read_config_with_metadata, read_config_without_migrate,
+        read_server_config_snapshot, save_config_with_opts_inner, save_server_config, save_server_config_snapshot,
+        save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
+        storage_class_kvs_mut,
     };
     use crate::config::{audit, heal, notify, oidc, scanner};
     use crate::disk::{RUSTFS_META_BUCKET, endpoint::Endpoint};
@@ -2853,7 +2850,7 @@ mod tests {
     use rustfs_config::{
         DEFAULT_DELIMITER, ENABLE_KEY, EnableState, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, MYSQL_DSN_STRING,
         MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_TABLE, SCANNER_CYCLE, SCANNER_IDLE_MODE, SCANNER_KEYS, SCANNER_SPEED,
-        SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK,
+        SCANNER_SUB_SYS,
     };
 
     #[tokio::test]
@@ -2883,11 +2880,6 @@ mod tests {
     #[derive(Debug, Default)]
     struct MetadataWriteStore {
         error: Option<Error>,
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        observations:
-            std::sync::Mutex<Vec<(Option<rustfs_io_metrics::internode_metrics::ReadVersionEntrySource>, Option<&'static str>)>>,
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        pending: bool,
     }
 
     #[async_trait::async_trait]
@@ -2918,20 +2910,7 @@ mod tests {
             _data: &mut Self::PutObjectReader,
             _opts: &Self::ObjectOptions,
         ) -> core::result::Result<Self::ObjectInfo, Self::Error> {
-            #[cfg(feature = "diagnostic-read-version-caller")]
-            {
-                self.observations.lock().expect("observations lock").push((
-                    crate::cluster::rpc::runtime_sources::current_read_version_entry_source(),
-                    rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
-                ));
-                if self.pending {
-                    std::future::pending::<()>().await;
-                }
-            }
-            match self.error.clone() {
-                Some(error) => Err(error),
-                None => Ok(ObjectInfo::default()),
-            }
+            Err(self.error.clone().expect("test store error should be configured"))
         }
     }
 
@@ -2939,10 +2918,6 @@ mod tests {
     async fn save_config_preserves_retryable_system_volume_errors() {
         let store = Arc::new(MetadataWriteStore {
             error: Some(Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())),
-            #[cfg(feature = "diagnostic-read-version-caller")]
-            observations: Default::default(),
-            #[cfg(feature = "diagnostic-read-version-caller")]
-            pending: false,
         });
         let error =
             save_config_with_opts_inner(store, "buckets/example/.metadata.bin", Vec::new(), &ObjectOptions::default(), false)
@@ -2954,59 +2929,6 @@ mod tests {
             Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), "buckets/example/.metadata.bin".to_string())
         );
     }
-
-    #[cfg(feature = "diagnostic-read-version-caller")]
-    #[tokio::test]
-    async fn diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation() {
-        use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry};
-        use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource;
-        use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context};
-
-        let mut guard = S3HttpRequestGuard::new("PUT");
-        let put = guard.in_scope(|| {
-            rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject);
-            current_s3_http_operation_context().expect("captured PutObject")
-        });
-        for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] {
-            for api in [None, Some(put)] {
-                for case in ["success", "error", "cancelled"] {
-                    let store = Arc::new(MetadataWriteStore {
-                        error: (case == "error").then_some(Error::VolumeNotFound),
-                        pending: case == "cancelled",
-                        ..Default::default()
-                    });
-                    let opts = ObjectOptions::default();
-                    let write = async {
-                        let result = tokio::time::timeout(
-                            std::time::Duration::from_secs(1),
-                            save_config_with_opts_inner(store.clone(), "buckets/diagnostic/state", Vec::new(), &opts, false),
-                        )
-                        .await;
-                        match case {
-                            "success" => assert!(result.expect("write completes").is_ok()),
-                            "error" => {
-                                assert!(matches!(result.expect("error completes"), Err(Error::InsufficientWriteQuorum(_, _))))
-                            }
-                            _ => assert!(result.is_err(), "pending write is cancelled"),
-                        }
-                    };
-                    rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async {
-                        match earlier {
-                            Some(source) => with_read_version_entry(source, write).await,
-                            None => write.await,
-                        }
-                    })
-                    .await;
-                    assert_eq!(
-                        *store.observations.lock().expect("observations lock"),
-                        vec![(Some(earlier.unwrap_or(ReadVersionEntrySource::ConfigWrite)), api.map(|_| "s3:PutObject"))]
-                    );
-                    assert_eq!(current_read_version_entry_source(), None, "write scope restores: {case}");
-                    assert_eq!(current_s3_http_operation_context(), None, "API scope restores: {case}");
-                }
-            }
-        }
-    }
     use rustfs_lock::client::LockClient;
     use rustfs_lock::client::local::LocalClient;
     use rustfs_lock::{LockError, LockInfo, LockResponse, LockStats};
@@ -4201,8 +4123,6 @@ mod tests {
             let value = match *key {
                 SCANNER_SPEED => "fast",
                 SCANNER_IDLE_MODE => "false",
-                // "1" is this key's default, which the diff encoding omits.
-                SCANNER_WALKERS_PER_DISK => "2",
                 _ => "1",
             };
             scanner_kvs.insert((*key).to_string(), value.to_string());
@@ -4224,8 +4144,6 @@ mod tests {
             let expected = match *key {
                 SCANNER_SPEED => "fast",
                 SCANNER_IDLE_MODE => "false",
-                // "1" is this key's default, which the diff encoding omits.
-                SCANNER_WALKERS_PER_DISK => "2",
                 _ => "1",
             };
             assert_eq!(decoded_scanner.get(key), expected, "scanner key {key} should survive roundtrip");
@@ -5158,6 +5076,7 @@ mod tests {
         heal_replacement: Option<Vec<u8>>,
         heal_calls: AtomicUsize,
         write_calls: AtomicUsize,
+        last_get_no_lock: AtomicBool,
         last_put_no_lock: AtomicBool,
         last_put_preconditions: Mutex<Option<HTTPPreconditions>>,
         revision: AtomicUsize,
@@ -5173,6 +5092,7 @@ mod tests {
                 heal_replacement,
                 heal_calls: AtomicUsize::new(0),
                 write_calls: AtomicUsize::new(0),
+                last_get_no_lock: AtomicBool::new(false),
                 last_put_no_lock: AtomicBool::new(false),
                 last_put_preconditions: Mutex::new(None),
                 revision: AtomicUsize::new(1),
@@ -5228,8 +5148,9 @@ mod tests {
             _object: &str,
             _range: Option<HTTPRangeSpec>,
             _h: HeaderMap,
-            _opts: &ObjectOptions,
+            opts: &ObjectOptions,
         ) -> Result<GetObjectReader> {
+            self.last_get_no_lock.store(opts.no_lock, Ordering::SeqCst);
             let data = match &*self.state.lock().expect("state lock poisoned") {
                 RecoveryReadState::Missing => return Err(Error::ConfigNotFound),
                 RecoveryReadState::Blob(data) => data.clone(),
@@ -5524,6 +5445,60 @@ mod tests {
         assert!(data.is_empty());
     }
 
+    #[tokio::test]
+    async fn read_config_preserve_empty_metadata_keeps_empty_and_options() {
+        let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None));
+        for no_lock in [false, true] {
+            let opts = ObjectOptions {
+                no_lock,
+                ..Default::default()
+            };
+            let (data, info) = read_config_preserve_empty_with_metadata_opts(store.clone(), "config/empty.json", &opts)
+                .await
+                .expect("an existing empty config must retain its payload and metadata");
+            assert!(data.is_empty());
+            assert_eq!(info.size, 0);
+            assert_eq!(info.actual_size, 0);
+            assert_eq!(info.etag.as_deref(), Some("config-1"));
+            assert_eq!(info.data_dir, Some(uuid::Uuid::from_u128(1)));
+            assert_eq!(store.last_get_no_lock.load(Ordering::SeqCst), no_lock);
+        }
+        let err = read_config(store, "config/empty.json")
+            .await
+            .expect_err("the existing reader must still treat empty configs as missing");
+        assert!(matches!(err, Error::ConfigNotFound));
+    }
+
+    #[tokio::test]
+    async fn read_config_preserve_empty_metadata_keeps_payload_and_metadata() {
+        let payload = b"original config bytes".to_vec();
+        let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(payload.clone()), None));
+        let (data, info) =
+            read_config_preserve_empty_with_metadata_opts(store, "config/nonempty.json", &ObjectOptions::default())
+                .await
+                .expect("a nonempty config must retain its payload and metadata");
+        assert_eq!(data, payload);
+        let size = i64::try_from(payload.len()).expect("test payload size fits i64");
+        assert_eq!(info.size, size);
+        assert_eq!(info.actual_size, size);
+        assert_eq!(info.etag.as_deref(), Some("config-1"));
+        assert_eq!(info.data_dir, Some(uuid::Uuid::from_u128(1)));
+    }
+
+    #[tokio::test]
+    async fn read_config_preserve_empty_metadata_preserves_read_errors() {
+        let missing = Arc::new(RecoveryMockStore::new(RecoveryReadState::Missing, None));
+        let err = read_config_preserve_empty_with_metadata_opts(missing, "config/missing.json", &ObjectOptions::default())
+            .await
+            .expect_err("physical absence must remain ConfigNotFound");
+        assert!(matches!(err, Error::ConfigNotFound));
+        let quorum = Arc::new(RecoveryMockStore::new(RecoveryReadState::QuorumError, None));
+        let err = read_config_preserve_empty_with_metadata_opts(quorum, "config/quorum.json", &ObjectOptions::default())
+            .await
+            .expect_err("a quorum failure must retain its typed read error");
+        assert!(matches!(err, Error::ErasureReadQuorum));
+    }
+
     #[async_trait::async_trait]
     impl StorageAdminApi for RecoveryMockStore {
         type BackendInfo = rustfs_madmin::BackendInfo;

crates/ecstore/src/config/scanner.rs

@@ -17,11 +17,11 @@ use rustfs_config::{
     DEFAULT_SCANNER_BITROT_CYCLE_SECS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES,
     DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE,
     DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED,
-    DEFAULT_SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS,
-    SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT,
-    SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY,
-    SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED,
-    SCANNER_START_DELAY, SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS,
+    DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE,
+    SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
 };
 use std::sync::LazyLock;
 
@@ -92,11 +92,6 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
             value: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(),
             hidden_if_empty: false,
         },
-        KV {
-            key: SCANNER_WALKERS_PER_DISK.to_owned(),
-            value: DEFAULT_SCANNER_WALKERS_PER_DISK.to_string(),
-            hidden_if_empty: false,
-        },
         KV {
             key: SCANNER_YIELD_EVERY_N_OBJECTS.to_owned(),
             value: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(),

crates/ecstore/src/core/pools.rs

@@ -3691,55 +3691,6 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error {
     }
 }
 
-fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error {
-    match err {
-        rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable {
-            mode: "read",
-            bucket: RUSTFS_META_BUCKET.to_string(),
-            object: object.to_string(),
-            required,
-            achieved,
-        },
-        other => Error::Lock(other),
-    }
-}
-
-/// Read-only admission to an already durable rebalance run. Keeping this
-/// separate from the activation writer prevents admission from authorizing a
-/// pool or rebalance metadata write under shared namespace locks.
-pub(crate) struct PoolRebalanceWorkerFence {
-    pool_meta_guard: rustfs_lock::NamespaceLockGuard,
-    rebalance_meta_guard: rustfs_lock::NamespaceLockGuard,
-    fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
-}
-
-impl PoolRebalanceWorkerFence {
-    pub(crate) fn set_fleet_proof(
-        &mut self,
-        fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
-    ) {
-        self.fleet_proof = fleet_proof;
-    }
-
-    pub(crate) fn ensure_held(&self) -> Result<()> {
-        if self.pool_meta_guard.is_lock_lost()
-            || self.pool_meta_guard.is_released()
-            || self.rebalance_meta_guard.is_lock_lost()
-            || self.rebalance_meta_guard.is_released()
-        {
-            return Err(Error::other("worker admission lock lost before rebalance activation"));
-        }
-        if self
-            .fleet_proof
-            .as_ref()
-            .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
-        {
-            return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
-        }
-        Ok(())
-    }
-}
-
 pub(crate) struct PoolRebalanceActivationFence {
     pool_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
     rebalance_meta_guard: Arc<rustfs_lock::NamespaceLockGuard>,
@@ -3811,35 +3762,6 @@ tokio::task_local! {
     pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc<tokio::sync::Notify>;
 }
 
-pub(crate) async fn acquire_pool_rebalance_worker_locks<S>(pool: Arc<S>) -> Result<PoolRebalanceWorkerFence>
-where
-    S: crate::storage_api_contracts::namespace::NamespaceLocking<
-            Error = Error,
-            NamespaceLock = rustfs_lock::NamespaceLockWrapper,
-        >,
-{
-    // Match activation and entry admission: rebalance.bin -> pool.bin. Joining
-    // the same durable run must coexist with entries retaining their run fence.
-    let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?;
-    #[cfg(test)]
-    let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one());
-    let rebalance_meta_guard = rebalance_meta_lock
-        .get_read_lock(get_lock_acquire_timeout())
-        .await
-        .map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?;
-    let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?;
-    let pool_meta_guard = pool_meta_lock
-        .get_read_lock(get_lock_acquire_timeout())
-        .await
-        .map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?;
-
-    Ok(PoolRebalanceWorkerFence {
-        pool_meta_guard,
-        rebalance_meta_guard,
-        fleet_proof: None,
-    })
-}
-
 pub(crate) async fn acquire_pool_rebalance_activation_locks<S>(
     pool: Arc<S>,
     fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
@@ -6763,12 +6685,12 @@ fn pool_meta_cas_options(object: &str, token: &PoolMetaCasToken, fence: &PoolMet
     fence.ensure_held()?;
     let mut opts = ObjectOptions {
         max_parity: true,
+        write_completion: crate::object_api::WriteCompletion::TailDrained,
         no_lock: true,
         http_preconditions: Some(pool_meta_cas_preconditions(token, object)?),
         ..Default::default()
     };
     fence.add_to_options(&mut opts, object);
-    opts.write_completion = crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair;
     Ok(opts)
 }
 
@@ -6813,11 +6735,7 @@ where
             "payload_sha256": rustfs_utils::crypto::hex(Sha256::digest(&data)),
             "if_match": opts.http_preconditions.as_ref().and_then(|p| p.if_match.as_deref()),
             "if_none_match": opts.http_preconditions.as_ref().and_then(|p| p.if_none_match.as_deref()),
-            "tail_drained": matches!(
-                opts.write_completion,
-                crate::object_api::WriteCompletion::TailDrained
-                    | crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair
-            ),
+            "tail_drained": opts.write_completion == crate::object_api::WriteCompletion::TailDrained,
             "no_lock": opts.no_lock,
         })
     });
@@ -12512,18 +12430,6 @@ impl ECStore {
         Ok(pool_meta)
     }
 
-    pub(crate) async fn load_runtime_pool_meta_under_worker_fence(
-        &self,
-        write_state: &mut PoolMetaWriteState,
-        worker_fence: &PoolRebalanceWorkerFence,
-        operation: &str,
-    ) -> Result<PoolMeta> {
-        worker_fence.ensure_held()?;
-        let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?;
-        worker_fence.ensure_held()?;
-        Ok(pool_meta)
-    }
-
     pub(crate) async fn load_runtime_pool_meta_under_activation_fence(
         &self,
         write_state: &mut PoolMetaWriteState,
@@ -24786,7 +24692,7 @@ mod pools_tests {
                     _ => panic!("phase fixture must receive valid pool metadata"),
                 };
                 assert!(opts.max_parity && opts.no_lock);
-                assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair);
+                assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrained);
                 self.trace.calls.lock().expect("trace lock").push(WriteCall {
                     pool: self.pool,
                     phase,

crates/ecstore/src/data_movement/mod.rs

@@ -893,8 +893,6 @@ fn is_data_movement_rewritten_metadata(object_info: &ObjectInfo, key: &str, norm
         SUFFIX_ACTUAL_SIZE,
         SUFFIX_CRC,
         SUFFIX_PART_CHECKSUMS,
-        // Inline placement is local to each physical copy, like its data directory.
-        rustfs_utils::http::SUFFIX_INLINE_DATA,
         crate::object_api::ENCRYPTED_PART_LAYOUT_QUORUM_SUFFIX,
     ]
     .iter()
@@ -3534,23 +3532,6 @@ mod tests {
         assert!(overwrite_resume_for_target(&source, source.clone()));
     }
 
-    #[test]
-    fn test_data_movement_identity_accepts_inline_layout_change_only() {
-        let source = overwrite_equivalence_source();
-        let mut target = source.clone();
-        rustfs_utils::http::insert_str(
-            Arc::make_mut(&mut target.user_defined),
-            rustfs_utils::http::SUFFIX_INLINE_DATA,
-            "true".to_string(),
-        );
-        assert!(is_equivalent_data_movement_object(&source, &target));
-        assert!(is_equivalent_data_movement_object(&target, &source));
-
-        Arc::make_mut(&mut target.user_defined).insert("x-amz-meta-inline-data".to_string(), "user value".to_string());
-        assert!(!is_equivalent_data_movement_object(&source, &target));
-        assert!(!is_equivalent_data_movement_object(&target, &source));
-    }
-
     #[test]
     fn test_data_movement_overwrite_resume_accepts_destination_marker() {
         let source = overwrite_equivalence_source();

crates/ecstore/src/data_movement/scanner_backlog.rs

@@ -110,17 +110,19 @@ async fn read_replica(set: Arc<SetDisks>) -> Result<ScannerPauseBacklogRetiremen
         set_index: set.set_index,
         data: None,
     };
-    let opts = ObjectOptions {
-        no_lock: true,
-        ..Default::default()
-    };
-    let read = set.get_object_reader(RUSTFS_META_BUCKET, SCANNER_PAUSE_BACKLOG_PATH, None, HeaderMap::new(), &opts);
-    #[cfg(feature = "diagnostic-read-version-caller")]
-    let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ScannerBacklogRead,
-        read,
-    );
-    let reader = match read.await {
+    let reader = match set
+        .get_object_reader(
+            RUSTFS_META_BUCKET,
+            SCANNER_PAUSE_BACKLOG_PATH,
+            None,
+            HeaderMap::new(),
+            &ObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    {
         Ok(reader) => reader,
         Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {
             return Ok(ScannerPauseBacklogRetirementRead { replica, etag: None });

crates/ecstore/src/data_usage/mod.rs

@@ -1325,18 +1325,6 @@ fn select_admin_data_usage_snapshot(
                 && authoritative.is_complete_bucket_usage_snapshot()
                 && observed_data_usage_is_newer(&observed, &authoritative) =>
         {
-            // A bucket can span several sets. Without a current or retained
-            // generation for every live set, its observed sum is only a subset,
-            // not a replacement for the last complete bucket total. Keep the
-            // baseline's timestamp and provenance until coverage is available.
-            if authoritative_format
-                && observed
-                    .usage_snapshot_set_states
-                    .iter()
-                    .any(|state| !state.tombstone && state.scanner_cycle.is_none())
-            {
-                return (authoritative, authoritative_format);
-            }
             (merge_partial_observation_for_admin(authoritative, observed), true)
         }
         Some(observed) if observed_data_usage_is_newer(&observed, &authoritative) => (observed, true),
@@ -2541,17 +2529,19 @@ where
     use crate::object_api::ObjectOptions;
     use http::HeaderMap;
 
-    let opts = ObjectOptions {
-        no_lock: true,
-        ..Default::default()
-    };
-    let read = store.get_object_reader(RUSTFS_META_BUCKET, key, None, HeaderMap::new(), &opts);
-    #[cfg(feature = "diagnostic-read-version-caller")]
-    let read = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-        rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::DataUsageCacheRead,
-        read,
-    );
-    match read.await {
+    match store
+        .get_object_reader(
+            RUSTFS_META_BUCKET,
+            key,
+            None,
+            HeaderMap::new(),
+            &ObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    {
         // A cache object that fails to decode is treated as absent rather than
         // as an error: a corrupt cache should not stall the caller, and the
         // next scanner pass rewrites it.
@@ -3074,8 +3064,6 @@ mod tests {
         transient_failures: Mutex<usize>,
         reads: Mutex<Vec<String>>,
         terminal_error: Mutex<Option<Error>>,
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        record_context: bool,
     }
 
     impl UsageCacheReadStore {
@@ -3084,8 +3072,6 @@ mod tests {
                 transient_failures: Mutex::new(n),
                 reads: Mutex::new(Vec::new()),
                 terminal_error: Mutex::new(None),
-                #[cfg(feature = "diagnostic-read-version-caller")]
-                record_context: false,
             }
         }
 
@@ -3094,8 +3080,6 @@ mod tests {
                 transient_failures: Mutex::new(0),
                 reads: Mutex::new(Vec::new()),
                 terminal_error: Mutex::new(Some(error)),
-                #[cfg(feature = "diagnostic-read-version-caller")]
-                record_context: false,
             }
         }
 
@@ -3122,12 +3106,6 @@ mod tests {
             _h: Self::HeaderMap,
             _opts: &Self::ObjectOptions,
         ) -> Result<Self::GetObjectReader, Self::Error> {
-            #[cfg(feature = "diagnostic-read-version-caller")]
-            if self.record_context {
-                crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(
-                    &crate::disk::ReadOptions::default(),
-                );
-            }
             self.reads.lock().await.push(object.to_string());
             if let Some(error) = self.terminal_error.lock().await.clone() {
                 return Err(error);
@@ -3227,89 +3205,6 @@ mod tests {
         );
     }
 
-    #[cfg(feature = "diagnostic-read-version-caller")]
-    #[test]
-    fn diagnostic_usage_cache_read_preserves_api_and_earlier_entry() {
-        use crate::cluster::rpc::runtime_sources::with_read_version_entry;
-        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
-        use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource;
-        use rustfs_io_metrics::s3_http_metrics::{
-            S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context,
-        };
-
-        let runtime = tokio::runtime::Builder::new_current_thread()
-            .enable_all()
-            .build()
-            .expect("diagnostic usage cache runtime");
-        let recorder = DebuggingRecorder::new();
-        let snapshots = recorder.snapshotter();
-        rustfs_io_metrics::set_get_stage_metrics_enabled(true);
-        let mut request = S3HttpRequestGuard::new("GET");
-        let get = request.in_scope(|| {
-            rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject);
-            current_s3_http_operation_context().expect("dispatched GET operation")
-        });
-        request.response(200);
-        metrics::with_local_recorder(&recorder, || {
-            runtime.block_on(async {
-                let ambient = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture();
-                for (operation, source) in [
-                    (None, None),
-                    (Some(get), None),
-                    (None, Some(ReadVersionEntrySource::ServerInfoPools)),
-                    (Some(get), Some(ReadVersionEntrySource::StoreObjectReader)),
-                ] {
-                    let store = UsageCacheReadStore {
-                        record_context: true,
-                        ..Default::default()
-                    };
-                    with_s3_http_operation_context(operation, async {
-                        let load = load_data_usage_cache(&store, "usage-cache");
-                        let cache = match source {
-                            Some(source) => with_read_version_entry(source, load).await,
-                            None => load.await,
-                        }
-                        .expect("missing diagnostic cache is harmless");
-                        assert!(cache.cache.is_empty());
-                        assert_eq!(store.read_keys().await.len(), 2, "both prefixed and legacy cache reads are observed");
-                    })
-                    .await;
-                    assert_eq!(
-                        crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(),
-                        ambient,
-                        "cache reads must restore the caller's complete diagnostic scope"
-                    );
-                }
-                assert_eq!(current_s3_http_operation_context(), None);
-            });
-        });
-        let rows = snapshots.snapshot().into_vec();
-        for (source, operation, state) in [
-            ("data_usage_cache_read", "unattributed", "absent"),
-            ("data_usage_cache_read", "s3:GetObject", "preserved"),
-            ("server_info_pools", "unattributed", "absent"),
-            ("store_object_reader", "s3:GetObject", "preserved"),
-        ] {
-            assert!(
-                rows.iter().any(|(key, _, _, value)| {
-                    let labels = key
-                        .key()
-                        .labels()
-                        .map(|label| (label.key(), label.value()))
-                        .collect::<std::collections::HashMap<_, _>>();
-                    key.key().name() == "rustfs_storage_diagnostic_context_transitions_total"
-                        && labels.get("entry_source") == Some(&source)
-                        && labels.get("boundary") == Some(&"remote_rpc")
-                        && labels.get("parent_api_operation") == Some(&operation)
-                        && labels.get("child_api_operation") == Some(&operation)
-                        && labels.get("api_context") == Some(&state)
-                        && matches!(value, DebugValue::Counter(2))
-                }),
-                "cache reads must retain API and the earliest observed entry: {source}/{operation}"
-            );
-        }
-    }
-
     #[tokio::test]
     async fn data_usage_removal_surfaces_missing_system_volume_as_read_quorum() {
         for cause in [Error::BucketNotFound(RUSTFS_META_BUCKET.to_string()), Error::VolumeNotFound] {
@@ -3823,169 +3718,6 @@ mod tests {
         assert!(selected.unknown_tier_stats.is_none());
     }
 
-    fn usage_snapshots_with_missing_observed_set() -> (DataUsageInfo, DataUsageInfo) {
-        let baseline_time = SystemTime::UNIX_EPOCH + Duration::from_secs(10);
-        let mut authoritative = data_usage_info_for_test("large-bucket", 510_000, 580_000_000_000, baseline_time);
-        authoritative.scanner_epoch = Some(4);
-        authoritative.scanner_cycle = Some(10);
-        authoritative.usage_snapshot_converged = Some(true);
-
-        let mut observed = data_usage_info_for_test("large-bucket", 3_000, 3_000_000_000, baseline_time + Duration::from_secs(1));
-        observed.scanner_epoch = Some(4);
-        observed.scanner_cycle = Some(11);
-        observed.usage_snapshot_complete = false;
-        observed.usage_snapshot_partial = true;
-        observed.usage_snapshot_converged = Some(false);
-        observed.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
-        observed.usage_snapshot_set_states = vec![
-            rustfs_data_usage::DataUsageSnapshotSetState {
-                pool_index: 0,
-                set_index: 0,
-                scanner_epoch: Some(4),
-                scanner_cycle: Some(11),
-                scan_plan_digest: Some([1; 32]),
-                complete: true,
-                ..Default::default()
-            },
-            rustfs_data_usage::DataUsageSnapshotSetState {
-                pool_index: 1,
-                set_index: 0,
-                scan_plan_digest: Some([1; 32]),
-                ..Default::default()
-            },
-        ];
-        assert!(
-            observed.is_valid_partial_snapshot(),
-            "a missing set is valid diagnostic progress, not a bucket total"
-        );
-        (authoritative, observed)
-    }
-
-    #[test]
-    fn partial_admin_observation_does_not_replace_totals_with_a_missing_set() {
-        let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
-        let identity = authoritative.snapshot_identity();
-
-        let (selected, _) = select_admin_data_usage_snapshot(authoritative, true, Some(observed));
-
-        assert_eq!(selected.buckets_usage["large-bucket"].objects_count, 510_000);
-        assert_eq!(selected.buckets_usage["large-bucket"].size, 580_000_000_000);
-        assert_eq!(selected.objects_total_count, 510_000);
-        assert_eq!(selected.objects_total_size, 580_000_000_000);
-        assert_eq!(
-            selected.snapshot_identity(),
-            identity,
-            "retained counts must not gain a newer timestamp or generation"
-        );
-        assert!(selected.is_complete_bucket_usage_snapshot());
-    }
-
-    #[tokio::test]
-    async fn partial_admin_observation_does_not_promote_a_legacy_baseline() {
-        let (legacy, observed) = usage_snapshots_with_missing_observed_set();
-        let (selected, current_format) = select_admin_data_usage_snapshot(legacy, false, Some(observed));
-        let (normalized, _) = normalize_loaded_data_usage(selected, current_format).await;
-
-        assert!(normalized.is_valid_partial_snapshot());
-        assert!(!normalized.is_complete_bucket_usage_snapshot());
-        assert_eq!(normalized.usage_snapshot_converged, Some(false));
-        assert_eq!(normalized.objects_total_count, 3_000);
-        assert_eq!(normalized.objects_total_size, 3_000_000_000);
-    }
-
-    #[test]
-    fn partial_admin_observation_accepts_decreases_when_every_set_is_covered() {
-        let (authoritative, mut observed) = usage_snapshots_with_missing_observed_set();
-        // A compatible retained generation is useful even when it is not a
-        // completed scan of the current cycle. A tombstone explicitly removes
-        // a set; neither case is the same as an unavailable live set.
-        for tombstone in [false, true] {
-            observed.usage_snapshot_set_states[1].tombstone = tombstone;
-            observed.usage_snapshot_set_states[1].scanner_epoch = (!tombstone).then_some(4);
-            observed.usage_snapshot_set_states[1].scanner_cycle = (!tombstone).then_some(10);
-            let (selected, _) = select_admin_data_usage_snapshot(authoritative.clone(), true, Some(observed.clone()));
-
-            assert_eq!(selected.objects_total_count, 3_000);
-            assert_eq!(selected.objects_total_size, 3_000_000_000);
-            assert!(selected.is_valid_partial_snapshot());
-            assert!(!selected.is_complete_bucket_usage_snapshot());
-            assert_eq!(selected.usage_snapshot_converged, Some(false));
-        }
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn partial_admin_observation_preserves_persisted_totals_across_cache_reload() {
-        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
-        use crate::config::com::save_config;
-
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
-        let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage");
-        let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation");
-        save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
-            .await
-            .expect("persist complete usage");
-        save_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(), observed_bytes.clone())
-            .await
-            .expect("persist observation with a missing set");
-        clear_usage_memory_cache_for_test().await;
-        invalidate_data_usage_snapshot_cache().await;
-
-        for reload in [false, false, true] {
-            if reload {
-                // Recreate process-local usage state, not an OS process restart.
-                clear_usage_memory_cache_for_test().await;
-                invalidate_data_usage_snapshot_cache().await;
-            }
-            let mut displayed = load_admin_data_usage_from_backend_cached(store.clone())
-                .await
-                .expect("load admin usage through the real snapshot cache");
-            apply_bucket_usage_memory_overlay(&mut displayed).await;
-            assert_eq!(displayed.objects_total_count, 510_000);
-            assert_eq!(displayed.objects_total_size, 580_000_000_000);
-            assert_eq!(displayed.bucket_sizes["large-bucket"], 580_000_000_000);
-            assert_eq!(displayed.snapshot_identity(), authoritative.snapshot_identity());
-        }
-        let quota = load_data_usage_from_backend_cached(store.clone())
-            .await
-            .expect("load quota authority");
-        assert_eq!(quota.objects_total_size, 580_000_000_000);
-        assert_eq!(
-            read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
-                .await
-                .expect("read primary"),
-            primary_bytes
-        );
-        assert_eq!(
-            read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str())
-                .await
-                .expect("read diagnostic observation"),
-            observed_bytes,
-            "admin reads must not rewrite either persisted snapshot"
-        );
-
-        let mut reconciled = data_usage_info_for_test("large-bucket", 2, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(12));
-        reconciled.scanner_epoch = Some(4);
-        reconciled.scanner_cycle = Some(12);
-        reconciled.usage_snapshot_converged = Some(true);
-        save_config(
-            store.clone(),
-            DATA_USAGE_OBJ_NAME_PATH.as_str(),
-            serde_json::to_vec(&reconciled).expect("encode confirmed deletion"),
-        )
-        .await
-        .expect("persist a later complete scan after deletion");
-        invalidate_data_usage_snapshot_cache().await;
-        let displayed = load_admin_data_usage_from_backend_cached(store)
-            .await
-            .expect("load confirmed decrease");
-        assert_eq!((displayed.objects_total_count, displayed.objects_total_size), (2, 42));
-        assert_eq!(displayed.snapshot_identity(), reconciled.snapshot_identity());
-        clear_usage_memory_cache_for_test().await;
-        invalidate_data_usage_snapshot_cache().await;
-    }
-
     #[tokio::test]
     async fn authoritative_save_cleanup_removes_observed_snapshot_best_effort() {
         let store = UsageCasStore::default();

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -15,9 +15,7 @@
 use crate::cluster::rpc::{
     ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
 };
-use crate::data_usage::{
-    DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
-};
+use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
 use crate::error::{Error, Result};
 use crate::{
     disk::endpoint::{Endpoint, EndpointType},
@@ -39,16 +37,12 @@ use rustfs_protos::{
 };
 use std::{
     collections::{HashMap, HashSet},
-    time::{Duration, Instant},
+    time::Duration,
 };
+use time::OffsetDateTime;
 use tokio::time::timeout;
 use tonic::Request;
-use tracing::{error, trace, warn};
-
-const EVENT_SERVER_PING: &str = "server_ping";
-const EVENT_SERVER_INFO: &str = "server_info";
-const LOG_COMPONENT_ECSTORE: &str = "ecstore";
-const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics";
+use tracing::warn;
 
 use shadow_rs::shadow;
 
@@ -163,22 +157,11 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
 
         let response: PingResponse = client.ping(request).await?.into_inner();
 
-        if flatbuffers::root::<PingBody>(&response.body).is_err() {
-            error!(
-                event = EVENT_SERVER_PING,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
-                result = "invalid_response",
-                "Server ping response decoding failed"
-            );
+        let ping_response_body = flatbuffers::root::<PingBody>(&response.body);
+        if let Err(e) = ping_response_body {
+            eprintln!("{e}");
         } else {
-            trace!(
-                event = EVENT_SERVER_PING,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
-                result = "success",
-                "Server ping completed"
-            );
+            println!("ping_resp:body(flatbuffer): {ping_response_body:?}");
         }
 
         Ok(())
@@ -276,10 +259,16 @@ fn collect_runtime_server_stats() -> (MemStats, u64, u64) {
 }
 
 pub async fn get_server_info(get_pools: bool) -> InfoMessage {
-    let started = tracing::enabled!(tracing::Level::TRACE).then(Instant::now);
+    let nowt: OffsetDateTime = OffsetDateTime::now_utc();
+
+    warn!("get_server_info start {:?}", nowt);
 
     let local = get_local_server_property().await;
 
+    let after1 = OffsetDateTime::now_utc();
+
+    warn!("get_local_server_property end {:?}", after1 - nowt);
+
     let mut servers = {
         if let Some(sys) = runtime_sources::notification_sys() {
             sys.server_info().await
@@ -288,6 +277,9 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
         }
     };
 
+    let after2 = OffsetDateTime::now_utc();
+
+    warn!("server_info end {:?}", after2 - after1);
     servers.push(local);
 
     let mut buckets = rustfs_madmin::Buckets::default();
@@ -302,7 +294,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
         apply_data_usage_result(
-            load_server_info_usage_snapshot(store.clone()).await,
+            load_data_usage_from_backend_cached(store.clone()).await,
             &mut buckets,
             &mut objects,
             &mut versions,
@@ -322,8 +314,15 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
             );
         }
 
+        let after3 = OffsetDateTime::now_utc();
+
+        warn!("load_data_usage_from_backend end {:?}", after3 - after2);
+
         let backend_info = StorageAdminApi::backend_info(store.as_ref()).await;
 
+        let after4 = OffsetDateTime::now_utc();
+
+        warn!("backend_info end {:?}", after4 - after3);
         if let Some(endpoints) = runtime_sources::endpoint_pools() {
             let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints);
             if added > 0 || !report.is_complete() {
@@ -345,6 +344,9 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
         }
         let (online_disks, offline_disks, unknown_disks) = get_online_offline_disks_stats(&all_disks);
 
+        let after5 = OffsetDateTime::now_utc();
+
+        warn!("get_online_offline_disks_stats end {:?}", after5 - after4);
         backend = rustfs_madmin::ErasureBackend {
             backend_type: rustfs_madmin::BackendType::ErasureType,
             online_disks: online_disks.sum(),
@@ -357,24 +359,14 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
         };
         if get_pools {
             pools = get_pools_info(&all_disks).await.unwrap_or_default();
+            let after6 = OffsetDateTime::now_utc();
+
+            warn!("get_pools_info end {:?}", after6 - after5);
         }
     }
 
     let services = rustfs_madmin::Services::default();
 
-    if let Some(started) = started {
-        trace!(
-            event = EVENT_SERVER_INFO,
-            component = LOG_COMPONENT_ECSTORE,
-            subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
-            state = "completed",
-            mode,
-            include_pools = get_pools,
-            duration_ms = started.elapsed().as_secs_f64() * 1000.0,
-            "Server info collected"
-        );
-    }
-
     InfoMessage {
         mode: Some(mode.to_string()),
         domain: None,
@@ -686,16 +678,12 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32,
 
         if erasure_set.id == 0 {
             erasure_set.id = d.set_index;
-            let load = load_data_usage_cache(
+            match load_data_usage_cache(
                 store.pools[d.pool_index as usize].disk_set[d.set_index as usize].as_ref(),
                 DATA_USAGE_CACHE_NAME,
-            );
-            #[cfg(feature = "diagnostic-read-version-caller")]
-            let load = crate::cluster::rpc::runtime_sources::with_read_version_entry(
-                rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ServerInfoPools,
-                load,
-            );
-            match load.await {
+            )
+            .await
+            {
                 Ok(cache) => apply_erasure_set_usage(&cache, erasure_set),
                 Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()),
             }
@@ -979,93 +967,6 @@ mod tests {
         assert_eq!(usage.size, 6);
     }
 
-    #[tokio::test]
-    #[serial]
-    async fn server_info_reads_complete_observation_without_promoting_authority() {
-        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
-        use crate::config::com::{read_config, save_config};
-        use crate::data_usage::{
-            DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
-            load_data_usage_from_backend_cached,
-        };
-        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
-        use std::time::{Duration, SystemTime};
-
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        let primary = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
-            scanner_epoch: Some(1),
-            usage_snapshot_bootstrap_pending: true,
-            usage_snapshot_converged: Some(false),
-            ..Default::default()
-        };
-        let observed = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
-            scanner_epoch: Some(1),
-            scanner_cycle: Some(1),
-            buckets_count: 1,
-            objects_total_count: 3,
-            versions_total_count: 4,
-            delete_markers_total_count: 1,
-            objects_total_size: 6,
-            buckets_usage: [(
-                "bucket".to_string(),
-                BucketUsageInfo {
-                    objects_count: 3,
-                    versions_count: 4,
-                    delete_markers_count: 1,
-                    size: 6,
-                    ..Default::default()
-                },
-            )]
-            .into_iter()
-            .collect(),
-            usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
-            ..Default::default()
-        };
-        let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
-        save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
-            .await
-            .expect("bootstrap should persist");
-        save_config(
-            store.clone(),
-            DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
-            serde_json::to_vec(&observed).expect("observation should encode"),
-        )
-        .await
-        .expect("observation should persist");
-        invalidate_data_usage_snapshot_cache().await;
-
-        let displayed = super::load_server_info_usage_snapshot(store.clone())
-            .await
-            .expect("server info should load the complete observation");
-        assert_eq!(displayed.usage_snapshot_converged, Some(false));
-        let mut buckets = rustfs_madmin::Buckets::default();
-        let mut objects = rustfs_madmin::Objects::default();
-        let mut versions = rustfs_madmin::Versions::default();
-        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
-        let mut usage = rustfs_madmin::Usage::default();
-        apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
-        assert!(objects.error.is_none());
-        assert_eq!(
-            (buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
-            (1, 3, 4, 1, 6)
-        );
-        let quota = load_data_usage_from_backend_cached(store.clone())
-            .await
-            .expect("quota should retain its bootstrap state");
-        assert!(!quota.is_complete_bucket_usage_snapshot());
-        assert_eq!(
-            read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
-                .await
-                .expect("primary should remain readable"),
-            primary_bytes
-        );
-        invalidate_data_usage_snapshot_cache().await;
-    }
-
     #[test]
     fn incomplete_data_usage_is_unavailable_in_server_info() {
         let mut buckets = rustfs_madmin::Buckets::default();

crates/ecstore/src/diagnostics/get.rs

@@ -182,7 +182,6 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
 pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
 pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
 pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
-pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
 pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
 pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";

crates/ecstore/src/disk/disk_store.rs

@@ -12,15 +12,14 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::disk::cleanup_runtime;
 use crate::disk::io_admission::{
     AdmissionReader, BoxedChunkReader, DiskIoAdmission, DiskIoClass, DiskIoContext, current_disk_io_context,
     with_disk_io_context, with_disk_io_permit,
 };
 use crate::disk::{
     CheckPartsResp, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskError, DiskInfo, DiskInfoOptions, DiskLocation, Endpoint,
-    Error, FileInfoVersions, GuardedFileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions,
-    RenameDataResp, Result, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
+    Error, FileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, Result,
+    SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
     health_state::{
         RuntimeDriveHealthState, classify_drive_recovery, get_drive_returning_probe_interval,
         get_drive_returning_success_threshold, get_drive_suspect_failure_threshold, record_drive_offline_duration,
@@ -467,11 +466,12 @@ impl LocalDiskWrapper {
         self.track_disk_health_mutation(
             "delete_version",
             DiskMetricMutation::Delete,
-            || {
+            || async {
                 Box::pin(
                     self.disk
                         .delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner),
                 )
+                .await
             },
             get_max_timeout_duration(),
         )
@@ -485,10 +485,10 @@ impl LocalDiskWrapper {
         opts: DeleteOptions,
         namespace_owner: Option<Arc<dyn Send + Sync>>,
     ) -> Result<()> {
-        self.track_cleanup_mutation(
+        self.track_disk_health_mutation(
             "delete",
             DiskMetricMutation::Delete,
-            || async { Box::pin(self.cleanup_delete(volume, path, opts, namespace_owner)).await },
+            || async { Box::pin(self.disk.delete_with_namespace_owner(volume, path, opts, namespace_owner)).await },
             get_max_timeout_duration(),
         )
         .await
@@ -1476,28 +1476,6 @@ impl LocalDiskWrapper {
         )
     }
 
-    async fn cleanup_delete(
-        &self,
-        volume: &str,
-        path: &str,
-        options: DeleteOptions,
-        owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<()> {
-        if !cleanup_runtime::enabled()? {
-            return self.disk.delete_with_namespace_owner(volume, path, options, owner).await;
-        }
-        let disk = self.disk.clone();
-        let volume = volume.to_owned();
-        let path = path.to_owned();
-        let guarded = owner.is_some();
-        let operation = async move { disk.delete_with_namespace_owner(&volume, &path, options, owner).await };
-        if guarded {
-            cleanup_runtime::guarded_disk_operation(operation).await
-        } else {
-            cleanup_runtime::disk_operation(operation).await
-        }
-    }
-
     /// Run a delete under an owned coordinator task when a publication guard
     /// is present. This keeps the guard alive if the RPC waiter is cancelled
     /// while the local namespace mutation is still in progress.
@@ -1516,13 +1494,12 @@ impl LocalDiskWrapper {
         } else {
             get_max_timeout_duration()
         };
-        let cleanup_owner = external_guard.clone();
         run_owned_mutation(external_guard, move || async move {
             operation
-                .track_cleanup_mutation(
+                .track_disk_health_mutation(
                     "delete",
                     DiskMetricMutation::Delete,
-                    || async { operation.cleanup_delete(&volume, &path, options, cleanup_owner).await },
+                    || async { operation.disk.delete(&volume, &path, options).await },
                     timeout_duration,
                 )
                 .await
@@ -1829,7 +1806,19 @@ impl LocalDiskWrapper {
             }
 
             // Clean up
-            disk.delete(test_bucket, test_filename, DeleteOptions::default()).await?;
+            disk.delete(
+                test_bucket,
+                test_filename,
+                DeleteOptions {
+                    recursive: false,
+                    immediate: false,
+                    undo_write: false,
+                    undo_delete: false,
+                    old_data_dir: None,
+                    expected_delete_marker: None,
+                },
+            )
+            .await?;
 
             Ok(())
         })
@@ -2012,138 +2001,10 @@ impl LocalDiskWrapper {
     }
 
     /// Get the current disk ID
-    pub(crate) async fn delete_versions_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        let version_count = versions.len();
-        self.metrics.record_operation_call("delete_versions");
-        // Check if disk is faulty before proceeding
-        if self.health.is_faulty() {
-            self.metrics.record_availability_error();
-            return vec![Some(DiskError::FaultyDisk); versions.len()];
-        }
-
-        // Check if disk is stale
-        if let Err(e) = self.check_disk_stale().await {
-            self.metrics.record_availability_error();
-            return vec![Some(e); versions.len()];
-        }
-
-        let _io_permit = match self.acquire_disk_io_permit().await {
-            Ok(permit) => permit,
-            Err(error) => return vec![Some(error); version_count],
-        };
-
-        // Record operation start
-        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
-        let waiting_guard = self.health.waiting_guard();
-        let metric_waiting_guard = self.metrics.waiting_guard();
-        let started = Instant::now();
-
-        // Execute the operation
-        let result = self
-            .disk
-            .delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
-            .await;
-        self.metrics.record_operation_latency("delete_versions", started.elapsed());
-        self.record_batch_delete_error_metrics(&result);
-
-        drop(waiting_guard);
-        drop(metric_waiting_guard);
-        let has_err = result.iter().any(|e| e.is_some());
-        if !has_err {
-            // Log success and decrement waiting counter
-            self.health.record_operation_success(&self.endpoint(), "operation_success");
-            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
-        }
-
-        result
-    }
-
-    pub(crate) async fn delete_versions_guarded_with_namespace_owner(
-        &self,
-        volume: &str,
-        versions: Vec<GuardedFileInfoVersions>,
-        opts: DeleteOptions,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Vec<Option<Error>> {
-        let version_count = versions.len();
-        self.metrics.record_operation_call("delete_versions");
-        // Check if disk is faulty before proceeding
-        if self.health.is_faulty() {
-            self.metrics.record_availability_error();
-            return vec![Some(DiskError::FaultyDisk); versions.len()];
-        }
-
-        // Check if disk is stale
-        if let Err(e) = self.check_disk_stale().await {
-            self.metrics.record_availability_error();
-            return vec![Some(e); versions.len()];
-        }
-
-        let _io_permit = match self.acquire_disk_io_permit().await {
-            Ok(permit) => permit,
-            Err(error) => return vec![Some(error); version_count],
-        };
-
-        // Record operation start
-        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
-        let waiting_guard = self.health.waiting_guard();
-        let metric_waiting_guard = self.metrics.waiting_guard();
-        let started = Instant::now();
-
-        // Execute the operation
-        let result = self
-            .disk
-            .delete_versions_guarded_with_namespace_owner(volume, versions, opts, namespace_owner)
-            .await;
-        self.metrics.record_operation_latency("delete_versions", started.elapsed());
-        self.record_batch_delete_error_metrics(&result);
-
-        drop(waiting_guard);
-        drop(metric_waiting_guard);
-        let has_err = result.iter().any(|e| e.is_some());
-        if !has_err {
-            // Log success and decrement waiting counter
-            self.health.record_operation_success(&self.endpoint(), "operation_success");
-            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
-        }
-
-        result
-    }
-
     pub async fn get_current_disk_id(&self) -> Option<Uuid> {
         *self.disk_id.read().await
     }
 
-    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: uuid::Uuid,
-        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
-        action: crate::disk::DeleteRecoveryAction,
-        namespace_owner: Option<Arc<dyn Send + Sync>>,
-    ) -> Result<crate::disk::DeleteRecoveryInspection> {
-        self.track_disk_health_mutation(
-            "settle_delete_transaction",
-            DiskMetricMutation::Delete,
-            || {
-                Box::pin(async {
-                    self.disk
-                        .delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, namespace_owner)
-                        .await
-                })
-            },
-            crate::disk::get_delete_recovery_timeout()?,
-        )
-        .await
-    }
-
     /// Track disk health for an operation.
     /// This method should wrap disk operations to ensure health checking.
     pub async fn track_disk_health<T, F, Fut>(&self, operation: F, timeout_duration: Duration) -> Result<T>
@@ -2154,31 +2015,6 @@ impl LocalDiskWrapper {
         self.track_disk_health_with_op("unknown", operation, timeout_duration).await
     }
 
-    /// Cleanup latency includes waiting behind a node-wide execution budget,
-    /// so a waiter deadline alone is not evidence that this disk is faulty.
-    /// The active disk health probe retains its original timeout policy.
-    async fn track_cleanup_mutation<T, F, Fut>(
-        &self,
-        op: &'static str,
-        mutation: DiskMetricMutation,
-        operation: F,
-        timeout_duration: Duration,
-    ) -> Result<T>
-    where
-        F: FnOnce() -> Fut,
-        Fut: std::future::Future<Output = Result<T>>,
-    {
-        // Runtime initialization must not prevent control-plane release of
-        // reader tokens. Disk I/O entries still propagate configuration errors.
-        let action = if cleanup_runtime::enabled().unwrap_or(false) {
-            TimeoutHealthAction::IgnoreFailure
-        } else {
-            TimeoutHealthAction::MarkFailure
-        };
-        self.track_disk_health_with_op_timeout_action_and_mutation(op, operation, timeout_duration, action, mutation)
-            .await
-    }
-
     async fn track_disk_health_mutation<T, F, Fut>(
         &self,
         op: &'static str,
@@ -2359,110 +2195,6 @@ impl LocalDiskWrapper {
 
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
-    async fn supports_guarded_delete_versions(&self) -> Result<()> {
-        // The local implementation shares this binary's capability. Availability
-        // and disk identity remain mutation checks and write-quorum inputs.
-        self.disk.supports_guarded_delete_versions().await
-    }
-    async fn delete_versions_guarded_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<GuardedFileInfoVersions>,
-        opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        self.delete_versions_guarded_with_namespace_owner(volume, versions, opts, None)
-            .await
-    }
-
-    async fn delete_version_recoverable(
-        &self,
-        volume: &str,
-        object: &str,
-        file_info: FileInfo,
-        force_del_marker: bool,
-        mut opts: DeleteOptions,
-    ) -> Result<()> {
-        crate::disk::validate_delete_recovery_options(&opts)?;
-        if object.starts_with('/') {
-            return Err(DiskError::InvalidPath);
-        }
-        opts.recoverable_delete = true;
-        self.delete_version(volume, object, file_info, force_del_marker, opts).await
-    }
-    async fn delete_versions_recoverable(
-        &self,
-        volume: &str,
-        versions: Vec<FileInfoVersions>,
-        mut opts: DeleteOptions,
-    ) -> Vec<Option<DiskError>> {
-        if let Err(err) = crate::disk::validate_delete_recovery_options(&opts) {
-            return vec![Some(err); versions.len()];
-        }
-        if versions.iter().any(|version| version.name.starts_with('/')) {
-            return vec![Some(DiskError::InvalidPath); versions.len()];
-        }
-        opts.recoverable_delete = true;
-        self.delete_versions(volume, versions, opts).await
-    }
-    async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result<bool> {
-        self.track_disk_health_with_op(
-            "has_delete_transaction_evidence",
-            || async { self.disk.has_delete_transaction_evidence(volume, object).await },
-            get_max_timeout_duration(),
-        )
-        .await
-    }
-    async fn discover_delete_transactions(
-        &self,
-        volume: &str,
-        object: Option<&str>,
-        cursor: Option<crate::disk::DeleteRecoveryCursor>,
-        limit: u32,
-    ) -> Result<crate::disk::DeleteRecoveryPage> {
-        self.track_disk_health_with_op(
-            "discover_delete_transactions",
-            || async {
-                self.disk
-                    .discover_delete_transactions(volume, object, cursor.clone(), limit)
-                    .await
-            },
-            get_max_timeout_duration(),
-        )
-        .await
-    }
-    async fn inspect_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: uuid::Uuid,
-    ) -> Result<crate::disk::DeleteRecoveryInspection> {
-        self.track_disk_health_with_op(
-            "inspect_delete_transaction",
-            || async { self.disk.inspect_delete_transaction(volume, object, transaction).await },
-            get_max_timeout_duration(),
-        )
-        .await
-    }
-    async fn settle_delete_transaction(
-        &self,
-        volume: &str,
-        object: &str,
-        transaction: uuid::Uuid,
-        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
-        action: crate::disk::DeleteRecoveryAction,
-    ) -> Result<crate::disk::DeleteRecoveryInspection> {
-        self.track_disk_health_mutation(
-            "settle_delete_transaction",
-            DiskMetricMutation::Delete,
-            || async {
-                self.disk
-                    .settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
-                    .await
-            },
-            crate::disk::get_delete_recovery_timeout()?,
-        )
-        .await
-    }
     fn has_replacement_mount_lease(&self) -> bool {
         self.disk.has_replacement_mount_lease()
     }
@@ -2676,7 +2408,46 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn delete_versions(&self, volume: &str, versions: Vec<FileInfoVersions>, opts: DeleteOptions) -> Vec<Option<Error>> {
-        self.delete_versions_with_namespace_owner(volume, versions, opts, None).await
+        let version_count = versions.len();
+        self.metrics.record_operation_call("delete_versions");
+        // Check if disk is faulty before proceeding
+        if self.health.is_faulty() {
+            self.metrics.record_availability_error();
+            return vec![Some(DiskError::FaultyDisk); versions.len()];
+        }
+
+        // Check if disk is stale
+        if let Err(e) = self.check_disk_stale().await {
+            self.metrics.record_availability_error();
+            return vec![Some(e); versions.len()];
+        }
+
+        let _io_permit = match self.acquire_disk_io_permit().await {
+            Ok(permit) => permit,
+            Err(error) => return vec![Some(error); version_count],
+        };
+
+        // Record operation start
+        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
+        let waiting_guard = self.health.waiting_guard();
+        let metric_waiting_guard = self.metrics.waiting_guard();
+        let started = Instant::now();
+
+        // Execute the operation
+        let result = self.disk.delete_versions(volume, versions, opts).await;
+        self.metrics.record_operation_latency("delete_versions", started.elapsed());
+        self.record_batch_delete_error_metrics(&result);
+
+        drop(waiting_guard);
+        drop(metric_waiting_guard);
+        let has_err = result.iter().any(|e| e.is_some());
+        if !has_err {
+            // Log success and decrement waiting counter
+            self.health.record_operation_success(&self.endpoint(), "operation_success");
+            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
+        }
+
+        result
     }
 
     async fn delete_paths(&self, volume: &str, paths: &[String]) -> Result<()> {
@@ -2699,37 +2470,9 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> {
-        // Quota fence release is control-plane work and may wait for in-flight
-        // mutations. Keep its original waiter lifecycle instead of creating
-        // detached cleanup tasks for repeated fence-release RPCs.
-        if volume == super::RUSTFS_META_BUCKET && super::is_quota_mutation_fence_path(path) {
-            return self
-                .track_disk_health_with_op(
-                    "release_snapshot_lease",
-                    || async { self.disk.release_snapshot_lease(volume, path, token).await },
-                    get_max_timeout_duration(),
-                )
-                .await;
-        }
-        self.track_cleanup_mutation(
+        self.track_disk_health_with_op(
             "release_snapshot_lease",
-            DiskMetricMutation::None,
-            || async {
-                if self.disk.release_snapshot_lease_without_cleanup(volume, path, token).await {
-                    return Ok(());
-                }
-                if !cleanup_runtime::enabled().unwrap_or(false) {
-                    // The raw release removes the token before trying the
-                    // reclaim budget. Initialization failure therefore keeps
-                    // the cleanup intent and data, without leaking a reader
-                    // or falling back to unbudgeted filesystem deletion.
-                    return self.disk.release_snapshot_lease(volume, path, token).await;
-                }
-                let disk = self.disk.clone();
-                let volume = volume.to_owned();
-                let path = path.to_owned();
-                cleanup_runtime::release_lease(async move { disk.release_snapshot_lease(&volume, &path, token).await }).await
-            },
+            || async { self.disk.release_snapshot_lease(volume, path, token).await },
             get_max_timeout_duration(),
         )
         .await
@@ -2754,22 +2497,10 @@ impl DiskAPI for LocalDiskWrapper {
         } else {
             get_max_timeout_duration()
         };
-        self.track_cleanup_mutation(
+        self.track_disk_health_mutation(
             "delete_data_dir",
             DiskMetricMutation::Delete,
-            || async {
-                if !cleanup_runtime::enabled()? {
-                    return self.disk.delete_data_dir(volume, path, opts).await;
-                }
-                let disk = self.disk.clone();
-                let volume = volume.to_owned();
-                let path = path.to_owned();
-                cleanup_runtime::disk_operation(async move {
-                    let _scope = scope;
-                    disk.delete_data_dir(&volume, &path, opts).await
-                })
-                .await
-            },
+            || async { self.disk.delete_data_dir(volume, path, opts).await },
             timeout,
         )
         .await
@@ -2804,7 +2535,6 @@ impl DiskAPI for LocalDiskWrapper {
         .await
     }
 
-    #[track_caller]
     async fn read_version(
         &self,
         org_volume: &str,
@@ -2813,7 +2543,7 @@ impl DiskAPI for LocalDiskWrapper {
         version_id: &str,
         opts: &ReadOptions,
     ) -> Result<FileInfo> {
-        let operation = self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit(
+        self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit(
             "read_version",
             |io_permit| async move {
                 with_disk_io_permit(io_permit, self.disk.read_version(org_volume, volume, path, version_id, opts)).await
@@ -2821,20 +2551,8 @@ impl DiskAPI for LocalDiskWrapper {
             get_max_timeout_duration(),
             TimeoutHealthAction::MarkFailure,
             DiskMetricMutation::None,
-        );
-        #[cfg(feature = "diagnostic-read-version-caller")]
-        {
-            if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_some() {
-                operation.await
-            } else {
-                let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location(
-                    std::panic::Location::caller().file(),
-                );
-                crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await
-            }
-        }
-        #[cfg(not(feature = "diagnostic-read-version-caller"))]
-        operation.await
+        )
+        .await
     }
 
     async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result<RawFileInfo> {
@@ -3029,10 +2747,10 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> {
-        self.track_cleanup_mutation(
+        self.track_disk_health_mutation(
             "delete",
             DiskMetricMutation::Delete,
-            || async { self.cleanup_delete(volume, path, opt, None).await },
+            || async { self.disk.delete(volume, path, opt).await },
             get_max_timeout_duration(),
         )
         .await
@@ -3065,25 +2783,11 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> {
-        let cleanup = path.rsplit('/').next() == Some(cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE);
-        self.track_disk_health_with_op_timeout_action_and_mutation(
+        self.track_disk_health_mutation(
             "write_all",
-            || async {
-                if !cleanup || !cleanup_runtime::enabled()? {
-                    return self.disk.write_all(volume, path, data).await;
-                }
-                let disk = self.disk.clone();
-                let volume = volume.to_owned();
-                let path = path.to_owned();
-                cleanup_runtime::disk_operation(async move { disk.write_all(&volume, &path, data).await }).await
-            },
-            get_max_timeout_duration(),
-            if cleanup && cleanup_runtime::enabled()? {
-                TimeoutHealthAction::IgnoreFailure
-            } else {
-                TimeoutHealthAction::MarkFailure
-            },
             DiskMetricMutation::Write,
+            || async { self.disk.write_all(volume, path, data).await },
+            get_max_timeout_duration(),
         )
         .await
     }
@@ -3476,256 +3180,6 @@ mod tests {
         let _ = walk.await;
     }
 
-    #[tokio::test]
-    async fn invalid_cleanup_config_does_not_leak_snapshot_readers() {
-        const CHILD: &str = "RUSTFS_CLEANUP_INVALID_LEASE_TEST_CHILD";
-        if std::env::var_os(CHILD).is_none() {
-            let output = tokio::task::spawn_blocking(|| {
-                std::process::Command::new(std::env::current_exe().expect("test executable"))
-                    .args([
-                        "--exact",
-                        "disk::disk_store::tests::invalid_cleanup_config_does_not_leak_snapshot_readers",
-                        "--nocapture",
-                    ])
-                    .env(CHILD, "1")
-                    .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true")
-                    .env("RUSTFS_CLEANUP_ASYNC_THREADS", "0")
-                    .output()
-                    .expect("run with invalid cleanup configuration")
-            })
-            .await
-            .expect("child process task");
-            assert!(
-                output.status.success(),
-                "{}\n{}",
-                String::from_utf8_lossy(&output.stdout),
-                String::from_utf8_lossy(&output.stderr)
-            );
-            assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed"));
-            return;
-        }
-        let dir = tempfile::tempdir().expect("temporary disk");
-        let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint");
-        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk"));
-        let wrapper = LocalDiskWrapper::new(disk.clone(), false);
-        let volume = "invalid-cleanup-config";
-        let path = "object/data";
-        wrapper.make_volume(volume).await.expect("bucket");
-        wrapper
-            .write_all(volume, "object/data/part.1", Bytes::from_static(b"live shard"))
-            .await
-            .expect("write shard");
-        let token = wrapper.acquire_snapshot_lease(volume, path).await.expect("GET lease");
-        wrapper
-            .release_snapshot_lease(volume, path, token)
-            .await
-            .expect("control-only release must not depend on cleanup initialization");
-        assert!(matches!(
-            wrapper.renew_snapshot_lease(volume, path, token).await,
-            Err(DiskError::FileNotFound)
-        ));
-
-        let token = wrapper
-            .acquire_snapshot_lease(volume, path)
-            .await
-            .expect("reader of old data");
-        // Register reclaim intent directly; this only updates the lease
-        // registry while a reader is present and performs no filesystem I/O.
-        assert_eq!(
-            disk.delete_data_dir(
-                volume,
-                path,
-                DeleteOptions {
-                    recursive: true,
-                    ..Default::default()
-                }
-            )
-            .await
-            .expect("defer reclaim"),
-            DataDirDeleteStatus::Deferred
-        );
-        assert!(
-            wrapper.release_snapshot_lease(volume, path, token).await.is_err(),
-            "invalid configuration must still reject reclamation"
-        );
-        assert!(
-            matches!(wrapper.renew_snapshot_lease(volume, path, token).await, Err(DiskError::FileNotFound)),
-            "failed reclamation must not strand a reader token"
-        );
-        assert_eq!(
-            wrapper.read_all(volume, "object/data/part.1").await.expect("retained data"),
-            Bytes::from_static(b"live shard"),
-            "configuration failure must not fall back to unbudgeted deletion"
-        );
-    }
-
-    #[tokio::test]
-    async fn rpc_cleanup_entries_share_disk_budget() {
-        const CHILD: &str = "RUSTFS_CLEANUP_RPC_TEST_CHILD";
-        if std::env::var_os(CHILD).is_none() {
-            let output = tokio::task::spawn_blocking(|| {
-                std::process::Command::new(std::env::current_exe().unwrap())
-                    .args([
-                        "--exact",
-                        "disk::disk_store::tests::rpc_cleanup_entries_share_disk_budget",
-                        "--nocapture",
-                    ])
-                    .env(CHILD, "1")
-                    .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true")
-                    .env("RUSTFS_CLEANUP_ASYNC_THREADS", "1")
-                    .env("RUSTFS_CLEANUP_DISK_MAX_PENDING", "1")
-                    .env("RUSTFS_CLEANUP_DISK_WORKERS", "1")
-                    .env("RUSTFS_CLEANUP_GC_WORKERS", "1")
-                    .env("RUSTFS_CLEANUP_CPUS", "none")
-                    .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP")
-                    .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER")
-                    .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP")
-                    .output()
-                    .unwrap()
-            })
-            .await
-            .unwrap();
-            assert!(
-                output.status.success(),
-                "{}\n{}",
-                String::from_utf8_lossy(&output.stdout),
-                String::from_utf8_lossy(&output.stderr)
-            );
-            assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed"));
-            return;
-        }
-        let dir = tempfile::tempdir().unwrap();
-        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
-        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.unwrap());
-        let wrapper = LocalDiskWrapper::new(disk, false);
-        let volume = "cleanup-receiver";
-        let path = "object/old-data";
-        let body_path = format!("{path}/part.1");
-        let receipt_path = format!("{path}/{}", cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE);
-        wrapper.make_volume(volume).await.unwrap();
-        wrapper
-            .write_all(volume, &body_path, Bytes::from_static(b"old body"))
-            .await
-            .unwrap();
-        let read_lease = wrapper
-            .acquire_snapshot_lease(volume, path)
-            .await
-            .expect("ordinary GET lease");
-        let (entered, entry) = tokio::sync::oneshot::channel();
-        let (release, released) = std::sync::mpsc::channel();
-        let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move {
-            entered.send(()).unwrap();
-            // Deliberately stall the only cleanup scheduler to prove that
-            // ordinary GET completion does not dispatch to it.
-            released
-                .recv_timeout(Duration::from_secs(30))
-                .expect("release stalled scheduler");
-            Ok(())
-        }));
-        tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap();
-        tokio::time::timeout(Duration::from_secs(5), wrapper.release_snapshot_lease(volume, path, read_lease))
-            .await
-            .expect("ordinary GET release must progress while the cleanup scheduler is stalled")
-            .expect("release without reclamation");
-        let fence_path = super::super::quota_mutation_fence_path(volume, "object");
-        let fence = wrapper
-            .acquire_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path)
-            .await
-            .unwrap();
-        wrapper
-            .release_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path, fence)
-            .await
-            .unwrap();
-        let timeout = wrapper
-            .track_cleanup_mutation(
-                "delete",
-                DiskMetricMutation::Delete,
-                std::future::pending::<Result<()>>,
-                Duration::from_millis(10),
-            )
-            .await;
-        assert!(matches!(timeout, Err(DiskError::Timeout)));
-        assert!(!wrapper.health.is_faulty(), "cleanup queue delay must not mark a healthy disk faulty");
-        let options = DeleteOptions {
-            recursive: true,
-            ..Default::default()
-        };
-        let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0));
-        // This is the same entry point used after RPC authentication and
-        // scanner publication-lease validation by handle_delete.
-        let result = wrapper
-            .delete_with_publication_guard(volume, path, options.clone(), Some(Arc::new(DropProbe(drops.clone()))))
-            .await;
-        assert!(matches!(result, Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
-        assert_eq!(drops.load(Ordering::SeqCst), 1);
-        assert!(matches!(wrapper.delete_data_dir(volume, path, options.clone()).await,
-            Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
-        assert!(matches!(wrapper.write_all(volume, &receipt_path, Bytes::from_static(b"receipt")).await,
-            Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
-        assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"old body"));
-        release.send(()).unwrap();
-        occupied.await.unwrap().unwrap();
-        wrapper
-            .write_all(volume, &receipt_path, Bytes::from_static(b"receipt"))
-            .await
-            .unwrap();
-        wrapper
-            .delete_with_publication_guard(volume, path, options, None)
-            .await
-            .unwrap();
-        assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound)));
-        // Snapshot readers defer old-data cleanup. Saturation must release
-        // their token while retaining reclaim intent, rather than leaking a
-        // live token or performing an unbudgeted delete on the RPC runtime.
-        wrapper
-            .write_all(volume, &body_path, Bytes::from_static(b"leased body"))
-            .await
-            .unwrap();
-        let lease = wrapper.acquire_snapshot_lease(volume, path).await.unwrap();
-        assert_eq!(
-            wrapper
-                .delete_data_dir(
-                    volume,
-                    path,
-                    DeleteOptions {
-                        recursive: true,
-                        ..Default::default()
-                    }
-                )
-                .await
-                .unwrap(),
-            DataDirDeleteStatus::Deferred
-        );
-        let (entered, entry) = tokio::sync::oneshot::channel();
-        let (release, released) = tokio::sync::oneshot::channel();
-        let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move {
-            entered.send(()).unwrap();
-            released.await.unwrap();
-            Ok(())
-        }));
-        tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap();
-        assert!(matches!(wrapper.release_snapshot_lease(volume, path, lease).await,
-            Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock));
-        assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"leased body"));
-        release.send(()).unwrap();
-        occupied.await.unwrap().unwrap();
-        assert_eq!(
-            wrapper
-                .delete_data_dir(
-                    volume,
-                    path,
-                    DeleteOptions {
-                        recursive: true,
-                        ..Default::default()
-                    }
-                )
-                .await
-                .unwrap(),
-            DataDirDeleteStatus::Deleted
-        );
-        assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound)));
-    }
-
     #[tokio::test]
     async fn owned_mutation_keeps_publication_guard_after_waiter_cancellation() {
         let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0));
@@ -3995,97 +3449,6 @@ mod tests {
         assert_eq!(snapshot.total_errors_availability, 1);
     }
 
-    #[tokio::test]
-    async fn local_guarded_delete_capability_is_independent_of_faulty_or_stale_mutation_rejection() {
-        for stale in [false, true] {
-            let dir = tempfile::tempdir().expect("create local disk directory");
-            let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse local endpoint");
-            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("create local disk"));
-            let volume = "guarded-capability";
-            let object = "object";
-            disk.make_volume(volume).await.expect("create object volume");
-            let mut info = FileInfo::new(object, 1, 0);
-            info.name = object.to_owned();
-            info.version_id = Some(Uuid::nil());
-            info.data_dir = Some(Uuid::new_v4());
-            info.mod_time = Some(::time::OffsetDateTime::now_utc());
-            info.erasure.index = 1;
-            info.size = 8;
-            info.parts = vec![ObjectPartInfo {
-                number: 1,
-                size: 8,
-                actual_size: 8,
-                ..Default::default()
-            }];
-            info.data = Some(Bytes::from_static(b"original"));
-            info.metadata.insert("etag".into(), "etag-a".into());
-            info.set_inline_data();
-            disk.write_metadata(volume, volume, object, info.clone())
-                .await
-                .expect("publish guarded fixture metadata");
-            let generation = crate::disk::CurrentDeleteGeneration::from_file_info(&info).expect("capture object generation");
-            let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false);
-            let expected_error = if stale {
-                let mut format_info = disk.format_info.write().await;
-                format_info.id = Some(Uuid::new_v4());
-                format_info.file_info = Some(tokio::fs::metadata(dir.path()).await.expect("local disk directory metadata"));
-                format_info.last_check = Some(::time::OffsetDateTime::now_utc());
-                drop(format_info);
-                wrapper.set_disk_id_state(Some(Uuid::new_v4())).await;
-                DiskError::DiskNotFound
-            } else {
-                wrapper.force_offline_for_test();
-                DiskError::FaultyDisk
-            };
-            let object_dir = dir.path().join(volume).join(object);
-            let metadata_path = object_dir.join(crate::disk::STORAGE_FORMAT_FILE);
-            let before = tokio::fs::read(&metadata_path).await.expect("read original metadata");
-            wrapper
-                .supports_guarded_delete_versions()
-                .await
-                .expect("local binary capability is known even when the drive is unavailable");
-            assert_eq!(
-                wrapper.metrics_snapshot().total_errors_availability,
-                0,
-                "capability does not probe drive availability"
-            );
-            let transaction = Uuid::new_v4();
-            let result = wrapper
-                .delete_versions_guarded_recoverable(
-                    volume,
-                    vec![GuardedFileInfoVersions {
-                        group: FileInfoVersions {
-                            name: object.to_owned(),
-                            versions: vec![FileInfo {
-                                name: object.to_owned(),
-                                version_id: info.version_id,
-                                ..Default::default()
-                            }],
-                            ..Default::default()
-                        },
-                        expected_current: generation,
-                    }],
-                    DeleteOptions {
-                        old_data_dir: Some(transaction),
-                        ..Default::default()
-                    },
-                )
-                .await;
-            assert_eq!(result, vec![Some(expected_error)], "mutation availability checks remain enforced");
-            assert_eq!(
-                tokio::fs::read(&metadata_path)
-                    .await
-                    .expect("metadata survives rejected mutation"),
-                before
-            );
-            assert!(
-                !object_dir.join(transaction.to_string()).exists(),
-                "rejected mutation creates no recovery record"
-            );
-            assert_eq!(wrapper.metrics_snapshot().total_errors_availability, 1);
-        }
-    }
-
     #[tokio::test]
     async fn local_disk_health_wrapper_counts_faulty_precheck_rejections() {
         let dir = tempfile::tempdir().expect("temp dir should be created");

crates/ecstore/src/disk/error.rs

@@ -167,12 +167,6 @@ pub enum DiskError {
     #[error("method not allowed")]
     MethodNotAllowed,
 
-    #[error("conditional delete object generation changed")]
-    ConditionalDeleteConflict,
-
-    #[error("guarded batch deletion is not supported by this disk")]
-    GuardedDeleteUnsupported,
-
     #[error("erasure write quorum")]
     ErasureWriteQuorum,
 
@@ -768,8 +762,6 @@ impl Clone for DiskError {
             DiskError::PartMissingOrCorrupt => DiskError::PartMissingOrCorrupt,
             DiskError::NoHealRequired => DiskError::NoHealRequired,
             DiskError::MethodNotAllowed => DiskError::MethodNotAllowed,
-            DiskError::ConditionalDeleteConflict => DiskError::ConditionalDeleteConflict,
-            DiskError::GuardedDeleteUnsupported => DiskError::GuardedDeleteUnsupported,
             DiskError::ErasureWriteQuorum => DiskError::ErasureWriteQuorum,
             DiskError::ErasureReadQuorum => DiskError::ErasureReadQuorum,
             DiskError::ShortWrite => DiskError::ShortWrite,
@@ -827,8 +819,6 @@ impl DiskError {
             DiskError::Timeout => 0x29,
             DiskError::InvalidPath => 0x2A,
             DiskError::RemoteClientUnavailable(_) => 0x2B,
-            DiskError::ConditionalDeleteConflict => 0x2C,
-            DiskError::GuardedDeleteUnsupported => 0x2D,
         }
     }
 
@@ -877,8 +867,6 @@ impl DiskError {
             0x29 => Some(DiskError::Timeout),
             0x2A => Some(DiskError::InvalidPath),
             0x2B => Some(DiskError::RemoteClientUnavailable(String::new())),
-            0x2C => Some(DiskError::ConditionalDeleteConflict),
-            0x2D => Some(DiskError::GuardedDeleteUnsupported),
             _ => None,
         }
     }
@@ -1657,21 +1645,6 @@ mod tests {
         assert!(rendered.contains("transport error: connection refused"));
     }
 
-    #[test]
-    fn guarded_delete_errors_preserve_wire_and_io_types() {
-        for (error, code) in [
-            (DiskError::ConditionalDeleteConflict, 0x2C),
-            (DiskError::GuardedDeleteUnsupported, 0x2D),
-        ] {
-            assert_eq!(error.to_u32(), code);
-            assert_eq!(DiskError::from_u32(code), Some(error.clone()));
-            let wire: rustfs_protos::proto_gen::node_service::Error = error.clone().into();
-            assert_eq!(DiskError::from(wire), error);
-            let io_error: std::io::Error = error.clone().into();
-            assert_eq!(DiskError::from(io_error), error);
-        }
-    }
-
     #[test]
     fn remote_client_unavailable_wire_roundtrip_keeps_variant_and_detail() {
         let original = DiskError::RemoteClientUnavailable("dial tcp: connection refused".to_string());

crates/ecstore/src/disk/error_reduce.rs

@@ -131,15 +131,6 @@ pub fn build_write_quorum_failure_summary(
 
 fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
     let Some(dominant_error) = dominant_error else {
-        // The reducer also returns None when every failure is ignored for
-        // quorum voting. That case contains no successful nil entry.
-        if !errors.is_empty() && errors.iter().all(Option::is_some) {
-            return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
-                "disk_not_found"
-            } else {
-                "all_ignored"
-            };
-        }
         return "nil_dominated";
     };
 
@@ -396,9 +387,9 @@ mod tests {
     }
 
     #[test]
-    fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
-        // Ignoring unavailable disks for quorum reduction must not imply a
-        // successful nil entry in the diagnostic label.
+    fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
+        // All disks offline (all DiskNotFound). The dominant error must be None and
+        // the label must be "nil_dominated" rather than falling back to "other_error".
         let errors = vec![
             Some(Error::DiskNotFound),
             Some(Error::DiskNotFound),
@@ -406,21 +397,7 @@ mod tests {
         ];
         let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
         assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.achieved, 0);
-        assert_eq!(summary.dominant_error_label, "disk_not_found");
-    }
-
-    #[test]
-    fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
-        let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
-        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
-        assert_eq!(summary.dominant_error, None);
-        assert_eq!(summary.achieved, 0);
-        assert_eq!(summary.dominant_error_label, "all_ignored");
-        assert_eq!(
-            reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
-            Some(Error::ErasureWriteQuorum)
-        );
+        assert_eq!(summary.dominant_error_label, "nil_dominated");
     }
 
     #[test]

crates/ecstore/src/disk/fs.rs

@@ -223,6 +223,12 @@ pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()
     std::fs::rename(from, to)
 }
 
+#[cfg(any(not(windows), test))]
+#[tracing::instrument(level = "debug", skip_all)]
+pub async fn read_file(path: impl AsRef<Path>) -> io::Result<Vec<u8>> {
+    fs::read(path.as_ref()).await
+}
+
 // Bucket existence cache - reduces statx syscalls for repeated bucket checks
 
 /// Cache for bucket directory existence checks.
@@ -608,6 +614,28 @@ mod tests {
         assert_eq!(content, "test content");
     }
 
+    #[tokio::test]
+    async fn test_read_file() {
+        let temp_dir = TempDir::new().unwrap();
+        let file_path = temp_dir.path().join("test_read.txt");
+
+        let test_content = b"This is test content for reading";
+        tokio::fs::write(&file_path, test_content).await.unwrap();
+
+        // Read file
+        let read_content = read_file(&file_path).await.unwrap();
+        assert_eq!(read_content, test_content);
+    }
+
+    #[tokio::test]
+    async fn test_read_file_nonexistent() {
+        let temp_dir = TempDir::new().unwrap();
+        let file_path = temp_dir.path().join("nonexistent.txt");
+
+        // Should fail for non-existent file
+        assert!(read_file(&file_path).await.is_err());
+    }
+
     #[tokio::test]
     async fn test_same_file() {
         let temp_dir = TempDir::new().unwrap();

crates/ecstore/src/disk/lifecycle_test_io.rs

@@ -1,72 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Bucket-scoped observations distinguish exact metadata reads from walks.
-use std::collections::HashMap;
-use std::sync::{Arc, Mutex, OnceLock};
-
-#[derive(Default, Clone)]
-pub(crate) struct Counts {
-    pub walks: usize,
-    pub reads: Vec<String>,
-}
-
-fn observers() -> &'static Mutex<HashMap<String, Arc<Mutex<Counts>>>> {
-    static OBSERVERS: OnceLock<Mutex<HashMap<String, Arc<Mutex<Counts>>>>> = OnceLock::new();
-    OBSERVERS.get_or_init(Mutex::default)
-}
-
-pub(crate) struct Observer {
-    bucket: String,
-    counts: Arc<Mutex<Counts>>,
-}
-
-impl Observer {
-    pub fn new(bucket: &str) -> Self {
-        let counts = Arc::new(Mutex::new(Counts::default()));
-        assert!(
-            observers()
-                .lock()
-                .expect("I/O observers lock")
-                .insert(bucket.to_owned(), counts.clone())
-                .is_none(),
-            "each test must use its own bucket"
-        );
-        Self {
-            bucket: bucket.to_owned(),
-            counts,
-        }
-    }
-
-    pub fn snapshot(&self) -> Counts {
-        self.counts.lock().expect("I/O counts lock").clone()
-    }
-}
-
-impl Drop for Observer {
-    fn drop(&mut self) {
-        observers().lock().expect("I/O observers lock").remove(&self.bucket);
-    }
-}
-
-pub(crate) fn observe(bucket: &str, object: Option<&str>) {
-    let observer = observers().lock().expect("I/O observers lock").get(bucket).cloned();
-    if let Some(observer) = observer {
-        let mut counts = observer.lock().expect("I/O counts lock");
-        match object {
-            Some(object) => counts.reads.push(object.to_owned()),
-            None => counts.walks += 1,
-        }
-    }
-}

crates/ecstore/src/disk/local/commit.rs

@@ -47,33 +47,6 @@ use tokio::fs;
 use tracing::{info, warn};
 use uuid::Uuid;
 
-fn transitioned_old_current_source(
-    dst_meta_existed: bool,
-    dst_meta_unparsable: bool,
-    xlmeta: &FileMeta,
-    volume: &str,
-    path: &str,
-    version_id: Uuid,
-) -> (bool, Option<FileInfo>) {
-    if dst_meta_unparsable {
-        return (false, None);
-    }
-    if !dst_meta_existed {
-        return (true, None);
-    }
-    let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else {
-        return (false, None);
-    };
-    if old_current.version_id.unwrap_or_default() != version_id {
-        return (true, None);
-    }
-    if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE {
-        (true, Some(old_current))
-    } else {
-        (true, None)
-    }
-}
-
 /// Hold later repair publications after admitting one baseline object. The
 /// fixture arms this on one replacement disk before rejoining the cluster.
 #[cfg(feature = "e2e-test-hooks")]
@@ -241,7 +214,7 @@ pub(super) async fn lock_rename_commit_directories(
     publication_root: &os::PublicationRoot,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<os::RenameCommitGuard> {
-    #[cfg(any(windows, target_os = "linux"))]
+    #[cfg(windows)]
     let result = {
         let source_parent = source_parent.to_path_buf();
         let destination_parent = destination_parent.to_path_buf();
@@ -249,7 +222,7 @@ pub(super) async fn lock_rename_commit_directories(
         let publication_root = publication_root.clone();
         os::run_blocking_namespace_operation(mutation_lease, move || {
             let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root);
-            #[cfg(all(test, windows))]
+            #[cfg(test)]
             if result.is_ok() {
                 run_destination_commit_directory_preparation(&destination_parent);
             }
@@ -257,7 +230,7 @@ pub(super) async fn lock_rename_commit_directories(
         })
         .await
     };
-    #[cfg(not(any(windows, target_os = "linux")))]
+    #[cfg(not(windows))]
     let result = {
         let _ = mutation_lease;
         os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root)
@@ -271,12 +244,12 @@ pub(super) async fn lock_rename_commit_directories(
     result.map_err(to_file_error).map_err(DiskError::from)
 }
 
-async fn prepare_rename_destination_metadata(
+async fn read_rename_destination_metadata(
     file_path: &Path,
     rename_commit_guard: &os::RenameCommitGuard,
     mutation_lease: Arc<os::NamespaceMutationLease>,
 ) -> Result<Option<Bytes>> {
-    #[cfg(any(windows, target_os = "linux"))]
+    #[cfg(windows)]
     let result = {
         let file_path = file_path.to_path_buf();
         let rename_commit_guard = rename_commit_guard.clone();
@@ -285,10 +258,10 @@ async fn prepare_rename_destination_metadata(
         })
         .await
     };
-    #[cfg(not(any(windows, target_os = "linux")))]
+    #[cfg(not(windows))]
     let _ = (rename_commit_guard, mutation_lease);
-    #[cfg(not(any(windows, target_os = "linux")))]
-    let result = match fs::read(file_path).await {
+    #[cfg(not(windows))]
+    let result = match super::super::fs::read_file(file_path).await {
         Ok(data) => Ok(Some(data)),
         Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
         Err(err) => Err(err),
@@ -425,7 +398,7 @@ impl LocalDisk {
 
         // xl.meta path
         let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?;
-        let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
+        let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
 
         // data_dir path
         let has_data_dir_path = {
@@ -480,7 +453,8 @@ impl LocalDisk {
             fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?;
         }
         // Acquire the common trees before reading destination metadata. On
-        // Windows this pins the object directory identity across publication.
+        // Windows this pins the object directory identity across metadata
+        // preparation, data publication, rollback backup, and final commit.
         let rename_commit_guard = lock_rename_commit_directories(
             src_file_parent,
             dst_file_parent,
@@ -498,12 +472,9 @@ impl LocalDisk {
             source_volume_dir: &src_volume_dir,
             data_paths: has_data_dir_path.as_ref(),
         };
-        let has_dst_buf = prepare_rename_destination_metadata(
-            commit.destination_metadata,
-            &commit.directory_guard,
-            commit.mutation_lease.clone(),
-        )
-        .await?;
+        let has_dst_buf =
+            read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
+                .await?;
 
         if no_inline {
             // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -537,14 +508,6 @@ impl LocalDisk {
             }
 
             let version_id = fi.version_id.unwrap_or_default();
-            let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
-                has_dst_buf.is_some(),
-                dst_meta_unparsable,
-                &xlmeta,
-                dst_volume,
-                dst_path,
-                version_id,
-            );
             let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
             let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
             let rollback_data_dir = has_old_data_dir.or_else(|| {
@@ -835,9 +798,7 @@ impl LocalDisk {
                 && let Some(parent) = dst_file_path.parent()
             {
                 let fsync_started = rustfs_io_metrics::put_stage_timer();
-                if let Err(err) =
-                    os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await
-                {
+                if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
                     rustfs_io_metrics::record_put_object_stage_duration_from(
                         rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
                         fsync_started,
@@ -936,8 +897,6 @@ impl LocalDisk {
                 cleanup_data_dir: has_old_data_dir,
                 sign: version_signature,
                 old_current_size,
-                old_current_source_checked,
-                old_current_source,
             })
         } else {
             // Inline metadata preparation is blocking. The transaction lease is
@@ -951,8 +910,6 @@ impl LocalDisk {
                 None
             };
             let dst_path_for_failpoint = dst_path.to_string();
-            let dst_volume_for_source = dst_volume.to_string();
-            let dst_path_for_source = dst_path.to_string();
             #[cfg(windows)]
             let source_parent = src_file_parent.to_path_buf();
             let rename_commit_guard_for_preparation = commit.directory_guard.clone();
@@ -997,14 +954,6 @@ impl LocalDisk {
                 };
 
                 let version_id = fi.version_id.unwrap_or_default();
-                let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
-                    has_dst_buf.is_some(),
-                    dst_meta_unparsable,
-                    &xlmeta,
-                    &dst_volume_for_source,
-                    &dst_path_for_source,
-                    version_id,
-                );
                 let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
                 let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
                 let rollback_data_dir = old_data_dir.or_else(|| {
@@ -1054,8 +1003,6 @@ impl LocalDisk {
                     old_data_dir,
                     version_signature,
                     old_current_size,
-                    old_current_source_checked,
-                    old_current_source,
                     staged_rollback_path,
                     has_dst_buf.is_none(),
                     prepared_metadata_source,
@@ -1075,8 +1022,6 @@ impl LocalDisk {
                 cleanup_data_dir,
                 version_signature,
                 old_current_size,
-                old_current_source_checked,
-                old_current_source,
                 mut local_rollback_path,
                 destination_was_absent,
                 prepared_metadata_source,
@@ -1202,11 +1147,10 @@ impl LocalDisk {
                     && let Some(dst_parent) = dst_file_path.parent()
                 {
                     let fsync_started = rustfs_io_metrics::put_stage_timer();
-                    if let Err(err) = os::fsync_commit_directory(
+                    if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
                         dst_parent,
-                        &commit.directory_guard,
                         commit.mutation_lease.clone(),
-                        Some(admission),
+                        admission,
                     )
                     .await
                     {
@@ -1318,8 +1262,6 @@ impl LocalDisk {
                 cleanup_data_dir,
                 sign: version_signature,
                 old_current_size,
-                old_current_source_checked,
-                old_current_source,
             })
         }
     }

crates/ecstore/src/disk/local/replacement_lease.rs

@@ -26,42 +26,6 @@ pub struct ReplacementExecutionLease {
 // file while it is held would allow a second owner to lock a different inode.
 const EXECUTION_LOCK_FILE: &str = ".rustfs-replacement.lock";
 
-/// A compatibility retry may only admit a disk that has not received data.
-/// The caller retains the mount descriptor and the replacement executor lease.
-pub(crate) async fn is_blank_target(root: PathBuf) -> Result<bool> {
-    tokio::task::spawn_blocking(move || {
-        let mut directories = vec![(root, 0usize)];
-        let mut entries_left = 64usize;
-        while let Some((directory, depth)) = directories.pop() {
-            for entry in std::fs::read_dir(directory)? {
-                let entry = entry?;
-                let Some(remaining) = entries_left.checked_sub(1) else {
-                    return Ok(false);
-                };
-                entries_left = remaining;
-                let kind = entry.file_type()?;
-                let name = entry.file_name();
-                if depth == 0 && name == EXECUTION_LOCK_FILE && kind.is_file() {
-                    if entry.metadata()?.len() != 0 {
-                        return Ok(false);
-                    }
-                } else if depth == 1 && name == crate::disk::HEALING_MARKER_PATH && kind.is_file() {
-                    if entry.metadata()?.len() > 256 {
-                        return Ok(false);
-                    }
-                } else if kind.is_dir() && depth < 4 && (depth > 0 || name == crate::disk::RUSTFS_META_BUCKET) {
-                    directories.push((entry.path(), depth + 1));
-                } else {
-                    return Ok(false);
-                }
-            }
-        }
-        Ok(true)
-    })
-    .await
-    .map_err(DiskError::from)?
-}
-
 pub(super) async fn acquire(root: PathBuf) -> Result<Arc<ReplacementExecutionLease>> {
     #[cfg(unix)]
     {
@@ -92,53 +56,6 @@ pub(super) async fn acquire(root: PathBuf) -> Result<Arc<ReplacementExecutionLea
 mod tests {
     use super::*;
 
-    #[tokio::test]
-    async fn blank_target_probe_rejects_data_symlinks_and_excessive_inventory() {
-        let temp = tempfile::TempDir::new().expect("blank target root");
-        let root = temp.path();
-        let _lease = acquire(root.to_path_buf()).await.expect("replacement execution lease");
-        let metadata = root.join(crate::disk::RUSTFS_META_BUCKET);
-        std::fs::create_dir_all(metadata.join("tmp")).expect("empty system directories");
-        std::fs::write(metadata.join(crate::disk::HEALING_MARKER_PATH), b"owner").expect("healing marker");
-        assert!(
-            is_blank_target(root.to_path_buf())
-                .await
-                .expect("blank target with control files")
-        );
-
-        for path in [
-            root.join("user-bucket"),
-            metadata.join("tmp").join("payload"),
-            metadata.join("format.json"),
-        ] {
-            std::fs::write(&path, b"data").expect("residual content");
-            assert!(!is_blank_target(root.to_path_buf()).await.expect("inspect residual content"));
-            std::fs::remove_file(path).expect("remove residual fixture");
-        }
-        std::fs::create_dir(root.join("user-bucket")).expect("even an empty user bucket is not a fresh target");
-        assert!(!is_blank_target(root.to_path_buf()).await.expect("inspect user bucket"));
-        std::fs::remove_dir(root.join("user-bucket")).expect("remove bucket fixture");
-        std::os::unix::fs::symlink(root, metadata.join("cycle")).expect("symlink fixture");
-        assert!(
-            !is_blank_target(root.to_path_buf())
-                .await
-                .expect("reject symlink without following it")
-        );
-        std::fs::remove_file(metadata.join("cycle")).expect("remove symlink fixture");
-        std::fs::create_dir_all(metadata.join("a/b/c/d")).expect("excessive depth");
-        assert!(!is_blank_target(root.to_path_buf()).await.expect("bounded directory depth"));
-        std::fs::remove_dir_all(metadata.join("a")).expect("remove deep fixture");
-        std::fs::remove_file(metadata.join(crate::disk::HEALING_MARKER_PATH)).expect("remove marker fixture");
-        std::fs::remove_dir(metadata.join("tmp")).expect("remove empty fixture directory");
-        for index in 0..62 {
-            std::fs::create_dir(metadata.join(index.to_string())).expect("bounded empty directories");
-        }
-        assert!(is_blank_target(root.to_path_buf()).await.expect("exact inventory limit"));
-        std::fs::create_dir(metadata.join("extra")).expect("one over inventory limit");
-        assert!(!is_blank_target(root.to_path_buf()).await.expect("bounded inventory"));
-        assert!(is_blank_target(root.join("missing")).await.is_err(), "missing root is not a blank disk");
-    }
-
     #[tokio::test]
     async fn replacement_execution_lease_excludes_independent_openers() {
         let temp = tempfile::TempDir::new().expect("lease root");