diff --git a/.agents/skills/rustfs-release-publish/SKILL.md b/.agents/skills/rustfs-release-publish/SKILL.md index f15c72471..063b77a9a 100644 --- a/.agents/skills/rustfs-release-publish/SKILL.md +++ b/.agents/skills/rustfs-release-publish/SKILL.md @@ -25,7 +25,7 @@ Read only the current phase's procedure; retain completed evidence across turns. | 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. | | Confirmation | Complete the gate below and end the turn. | | 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. | -| 7 | [Maintain milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. | +| 7 | [Reconcile Security Advisories, milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. | ## Human Confirmation @@ -39,4 +39,4 @@ Only a new affirmative user reply to that exact report authorizes Phase 6. The o - Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases. - If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting candidate branches or preview tags requires an explicit decision. -Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization. +Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization. diff --git a/.agents/skills/rustfs-release-publish/references/post-release-updates.md b/.agents/skills/rustfs-release-publish/references/post-release-updates.md index 9506f393f..885275af4 100644 --- a/.agents/skills/rustfs-release-publish/references/post-release-updates.md +++ b/.agents/skills/rustfs-release-publish/references/post-release-updates.md @@ -1,7 +1,17 @@ -# Post-release Milestone, Installation, and Website Updates +# Post-release Security Advisory, Milestone, Installation, and Website Updates Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase. +## Security Advisories + +Reconcile advisories after every completed final release, including Console-only releases. Read-only preparation may start earlier; do not claim an unpublished artifact is an available fix. + +1. Fetch every page of the repository's live Security Advisories, including triage, draft, published, and closed states. A permission error or partial response blocks reconciliation; it does not mean there are no advisories. Keep unpublished advisory details in private task artifacts, never in a public PR, release note, or issue. +2. For each potentially relevant advisory, map its fix to the exact published tag and verify the regression evidence. Main-branch reachability, a merged private fix, or a green build alone does not prove the shipped version is fixed. Check cherry-picked equivalents and reverted fixes explicitly. For embedded Console or dependency fixes, verify the shipped component version. +3. Determine the first available fixed release and affected version ranges from evidence. Preserve an earlier correct patched version; do not relabel all fixes as the current release. An unchanged server baseline such as a Console-only release requires checking the earlier server release as well. Leave unresolved reports and fixes absent from the tag pending; do not close them merely because a release shipped. +4. Prepare any necessary corrections to package identity, affected ranges, patched versions, fix/release links, and advisory text. Preserve unrelated fields, credit, CVE identifiers, and disclosure coordination. Apply updates only within existing user authorization; read back the result to verify it. Publishing a draft, requesting a CVE, or closing a report is a separate action: require explicit authorization for the concrete advisories and respect any embargo. Release-tag approval alone does not authorize those actions. Prepare the exact proposed changes before asking for missing authorization. +5. Record each relevant advisory's URL, current state, fix evidence, first fixed version, action taken or proposed, and remaining blocker in private task artifacts. Report advisory follow-up separately from artifact publication. A successful release with unresolved advisory work is not complete follow-up; retain that work for the release monitor or a concrete user decision. + ## Version milestones 1. Query all pages of both open and closed milestones in the RustFS release repository, matching titles exactly to version strings without a `v` prefix. A failed or incomplete query is not evidence that a milestone is absent; resolve the lookup before making changes. diff --git a/.agents/skills/rustfs-release-version-bump/agents/openai.yaml b/.agents/skills/rustfs-release-version-bump/agents/openai.yaml deleted file mode 100644 index 4bd3e0ff6..000000000 --- a/.agents/skills/rustfs-release-version-bump/agents/openai.yaml +++ /dev/null @@ -1,4 +0,0 @@ -interface: - display_name: "RustFS Release Bump" - short_description: "Prepare RustFS release branches like PR #2957." - default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump." diff --git a/.agents/skills/security-advisory-lessons/agents/openai.yaml b/.agents/skills/security-advisory-lessons/agents/openai.yaml deleted file mode 100644 index a90ec68ac..000000000 --- a/.agents/skills/security-advisory-lessons/agents/openai.yaml +++ /dev/null @@ -1,4 +0,0 @@ -interface: - display_name: "Security Advisory Lessons" - short_description: "Apply advisory lessons in reviews." - default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories." diff --git a/.codex/config.toml b/.codex/config.toml deleted file mode 100644 index f58ee1ace..000000000 --- a/.codex/config.toml +++ /dev/null @@ -1,5 +0,0 @@ -# Bound individual tool outputs retained in context; retrieve relevant ranges -# from task-owned log files when more evidence is needed. -# https://learn.chatgpt.com/docs/config-file/config-reference - -tool_output_token_limit = 4000 diff --git a/.config/e2e-full-selection.txt b/.config/e2e-full-selection.txt index 8669a2126..c05b236bd 100644 --- a/.config/e2e-full-selection.txt +++ b/.config/e2e-full-selection.txt @@ -1,2 +1,2 @@ -sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2 -sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22 +sha256-darwin=3a7f52dd884ed269746029e983006ce69b62f260db2ad1acdcffdb2196c65d11 +sha256-linux=4af4bb607919b96e63e2a3fb7a78bd18fcb08e265f4ca4ce6afb09a8be798305 diff --git a/.config/e2e-nightly-selection.txt b/.config/e2e-nightly-selection.txt index bf672a947..6dcd1745d 100644 --- a/.config/e2e-nightly-selection.txt +++ b/.config/e2e-nightly-selection.txt @@ -1,2 +1,2 @@ -sha256-darwin=5dbb9154b8feb94cedc8fb4aa6895d4b3259cc31b7df1f5c3b9928ef07f04535 -sha256-linux=d3a210be9eccf593d3723ae669b66680e9751dd01a6949f03df20614f913b707 +sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d +sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926 diff --git a/.config/e2e-smoke-selection.txt b/.config/e2e-smoke-selection.txt index eb68a79f1..9430a4d40 100644 --- a/.config/e2e-smoke-selection.txt +++ b/.config/e2e-smoke-selection.txt @@ -1 +1 @@ -sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10 +sha256=ce16185f87c6cd30aa0891ebd22be3352aedc10040d48cc5b00652d72f7be3b5 diff --git a/.config/make/tests.mak b/.config/make/tests.mak index 78ffecfb9..00bb7641f 100644 --- a/.config/make/tests.mak +++ b/.config/make/tests.mak @@ -52,6 +52,8 @@ script-tests: ## Run shell script tests ./scripts/check_embedded_secrets.sh --self-test $(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test $(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py $(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py $(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py $(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test diff --git a/.config/nextest.toml b/.config/nextest.toml index 3881b9036..92674305d 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -76,6 +76,13 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))' threads-required = "num-test-threads" +# These real-disk lock probes must finish durable IO within production's 5s +# acquisition budget. Reserve capacity from sibling test processes while +# preserving the probes' internal concurrency, deadlines, and assertions. +[[profile.default.overrides]] +filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))' +threads-required = "num-test-threads" + [[profile.default.overrides]] filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))' threads-required = "num-test-threads" @@ -88,6 +95,14 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)' setup = 'ecstore-large-stack' +# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests. +# The admin target-repair fixtures compose the full bucket delete and recreate +# path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need +# 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI. +[[profile.default.scripts]] +filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))' +setup = 'ecstore-base-stack' + [[profile.default.scripts]] filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))' setup = 'ecstore-base-stack' @@ -327,6 +342,13 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))' threads-required = "num-test-threads" +# These real-disk lock probes must finish durable IO within production's 5s +# acquisition budget. Reserve capacity from sibling test processes while +# preserving the probes' internal concurrency, deadlines, and assertions. +[[profile.ci.overrides]] +filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))' +threads-required = "num-test-threads" + [[profile.ci.overrides]] filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))' threads-required = "num-test-threads" @@ -339,6 +361,11 @@ threads-required = "num-test-threads" filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)' setup = 'ecstore-large-stack' +# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests. +[[profile.ci.scripts]] +filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))' +setup = 'ecstore-base-stack' + [[profile.ci.scripts]] filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))' setup = 'ecstore-base-stack' @@ -595,6 +622,7 @@ default-filter = """ test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/) | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/) | test(/^reliant::lifecycle::/) + | test(/^conditional_delete_objects_test::/) | test(/^reliant::tiering::/) | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/) | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/) diff --git a/.docker/observability/grafana/dashboards/rustfs.json b/.docker/observability/grafana/dashboards/rustfs.json index 15fbdc34b..1ca9a90a7 100644 --- a/.docker/observability/grafana/dashboards/rustfs.json +++ b/.docker/observability/grafana/dashboards/rustfs.json @@ -1748,7 +1748,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})", + "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})", "legendFormat": "{{bucket}}", "range": true, "refId": "A" @@ -1848,7 +1848,7 @@ "uid": "${datasource}" }, "editorMode": "code", - "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})", + "expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})", "legendFormat": "{{bucket}}", "range": true, "refId": "A" diff --git a/.docker/observability/prometheus.yml b/.docker/observability/prometheus.yml index 1039e2972..4ac483e6e 100644 --- a/.docker/observability/prometheus.yml +++ b/.docker/observability/prometheus.yml @@ -61,6 +61,7 @@ scrape_configs: otlp: promote_resource_attributes: - service.instance.id + - process.pid - service.name - service.namespace - cloud.availability_zone diff --git a/.docker/observability/tests/bucket-usage.test.yml b/.docker/observability/tests/bucket-usage.test.yml new file mode 100644 index 000000000..d886a7dc9 --- /dev/null +++ b/.docker/observability/tests/bucket-usage.test.yml @@ -0,0 +1,48 @@ +evaluation_interval: 1m +tests: +- name: bucket usage keeps clusters separate and deduplicates reporting nodes + interval: 1m + input_series: + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"} + values: '4096 1024' + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"} + values: '4096 1024' + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"} + values: '65536 65536' + - series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"} + values: '0 0' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"} + values: '3 1' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"} + values: '3 1' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"} + values: '70 70' + - series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"} + values: '0 0' + promql_expr_test: + - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"}) + eval_time: 1m + exp_samples: + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}' + value: 1024 + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}' + value: 0 + - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"}) + eval_time: 1m + exp_samples: + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}' + value: 1 + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}' + value: 0 + - expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes) + eval_time: 1m + exp_samples: + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}' + value: 1024 + - labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}' + value: 0 + - labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}' + value: 65536 + - expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"} + eval_time: 1m + exp_samples: [] diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 11b1b6c18..4f29ea348 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -23,6 +23,11 @@ duplicate its rules here. pass by fixing the cause, not by weakening the gate). - If `shellcheck` is installed, actionlint also lints `run:` scripts; treat those findings as part of the gate. +- Runner placement and billing rules live in `.github/workflows/README.md`. + A new job defaults to `ubuntu-latest` (free for this public repository); + a self-hosted `runs-on` needs one of the reasons listed there, stated in a + comment next to it, and a `rust-cache` key is never shared between the + hosted and self-hosted images. ## CI Alignment diff --git a/.github/actions/setup-docker-compose/action.yml b/.github/actions/setup-docker-compose/action.yml new file mode 100644 index 000000000..17fd3f7ab --- /dev/null +++ b/.github/actions/setup-docker-compose/action.yml @@ -0,0 +1,39 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Set up Docker Compose +description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance + +runs: + using: composite + steps: + - name: Install verified Docker Compose CLI plugin + shell: bash + run: | + set -euo pipefail + [[ $(uname -s) == Linux && $(uname -m) == x86_64 ]] + plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins" + mkdir -p "$plugin_dir" + compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX") + compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX") + trap 'rm -f "$compose_download" "$compose_staged"' EXIT + curl --fail --location --retry 3 \ + https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \ + --output "$compose_download" + printf '%s %s\n' \ + dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \ + "$compose_download" | sha256sum --check --strict + install -m 0755 "$compose_download" "$compose_staged" + mv -f "$compose_staged" "$plugin_dir/docker-compose" + [[ $(docker compose version --short) == 2.40.3 ]] diff --git a/.github/actions/setup/action.yml b/.github/actions/setup/action.yml index eb8f8b377..f9e178569 100644 --- a/.github/actions/setup/action.yml +++ b/.github/actions/setup/action.yml @@ -36,6 +36,16 @@ inputs: description: "Install cross-compilation tools" required: false default: "false" + zig-version: + description: >- + Zig version for cargo-zigbuild builds. Pinned so the `.` + baseline semantics cannot drift with an untested zig release. + required: false + default: "0.17.0" + cargo-zigbuild-version: + description: "cargo-zigbuild version installed for cross builds." + required: false + default: "0.23.4" target: description: "Target architecture to add" required: false @@ -99,10 +109,14 @@ runs: - name: Install Zig if: inputs.install-cross-tools == 'true' uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2 + with: + version: ${{ inputs.zig-version }} - name: Install cargo-zigbuild if: inputs.install-cross-tools == 'true' uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild + with: + tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }} - name: Install cargo-nextest if: inputs.install-test-tools == 'true' @@ -126,5 +140,5 @@ runs: # normalises, so cargo fingerprints stay valid. cache-all-crates: false cache-on-failure: true - shared-key: ${{ inputs.cache-shared-key }} + shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }} save-if: ${{ inputs.cache-save-if }} diff --git a/.github/scheduled-validations.json b/.github/scheduled-validations.json index e55ef3d56..060200715 100644 --- a/.github/scheduled-validations.json +++ b/.github/scheduled-validations.json @@ -1,6 +1,6 @@ [ { "workflow": ".github/workflows/audit.yml", "max_age_hours": 36 }, - { "workflow": ".github/workflows/build.yml", "max_age_hours": 192 }, + { "workflow": ".github/workflows/build.yml", "max_age_hours": 36 }, { "workflow": ".github/workflows/ci.yml", "max_age_hours": 192 }, { "workflow": ".github/workflows/coverage.yml", "max_age_hours": 192 }, { "workflow": ".github/workflows/e2e-replication-nightly.yml", "max_age_hours": 36 }, diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 000000000..e5492068d --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,74 @@ +# Workflow runners: what is billed and where a job belongs + +**Use this when:** adding a job, changing a `runs-on`, moving a lane between runner pools, or deciding whether a new check is worth its runner cost. +**Source of truth:** `runs-on` in each workflow file for current placement; `.github/actionlint.yaml` for the declared self-hosted labels; the organisation's Settings → Billing → Usage page for what is actually charged; `docs/testing/ci-gates.md` for which checks block a merge (that does not change when a runner changes). + +## Billing facts + +This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories. + +Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well: + +| Label | Paid by | What it is | Use it for | +|---|---|---|---| +| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job | +| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets | +| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) | +| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds | +| `sm-standard-4-arm` | RustFS | ARC pod, aarch64; reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds | +| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push | +| `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites | + +The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; `ci.yml` builds the debug binary on `sm-standard-4` and runs it on both pools. + +## Placement rule + +A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason goes in a comment next to its `runs-on`: + +1. **Disk.** It restores or produces the workspace `target/`: `cargo build` or `cargo test` with `--all-targets`, `cargo nextest run --all`, or any reader of the `ci-dev` / `ci-feat-*` caches. The workspace test build leaves about 64 GB in `target/`; a hosted runner has about 14 GB. +2. **Critical-path CPU.** It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (`build-rustfs-debug-binary` gates the e2e and s3-tests lanes). +3. **Hardware.** Native aarch64, Docker-in-Docker, or the lab machines. +4. **Cache ownership.** It reads a `rust-cache` key that a self-hosted job writes (see the cache rule below). + +Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason. + +A hosted runner is also the only choice for a few things the pods cannot do: `mount -o loop` and `mount -t tmpfs` fail inside the ARC pods, and a container runtime cannot guarantee that `io_uring_setup` is allowed. `uring-integration` and `e2e-distributed` are on `ubuntu-latest` for that reason, not for cost. + +## Cache rule + +`Swatinem/rust-cache` builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an `sm-standard-4` pod and one written on `ubuntu-latest` would overwrite each other under the same `shared-key`. + +- Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not. +- All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`). +- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. +- `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it. + +## Where self-hosted minutes leak + +These are the patterns that have cost real money here without validating anything more: + +- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`. +- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter. +- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images. +- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further. +- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run. + +## Measuring a run + +To see which pool each job of a run used and for how long: + +```bash +gh api "repos/rustfs/rustfs/actions/runs//jobs?per_page=100" \ + --jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"' +``` + +Only rows whose label starts with `sm-`, `dind-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure. + +## Checklist for a new or moved job + +- `runs-on` is a literal label. A new self-hosted label is declared in `.github/actionlint.yaml` in the same change (`.github/AGENTS.md`). +- The job declares `timeout-minutes` (`scripts/security/check_job_timeouts.sh`). +- `actionlint` reports nothing. +- A self-hosted `runs-on` carries a comment naming its reason from the placement rule. +- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other. +- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies. diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml index c1406bec2..209fffead 100644 --- a/.github/workflows/audit.yml +++ b/.github/workflows/audit.yml @@ -77,7 +77,7 @@ jobs: cancel-closed-pr-runs: name: Cancel Closed PR Runs if: github.event_name == 'pull_request' && github.event.action == 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Explain cancellation run @@ -89,7 +89,7 @@ jobs: cargo-deny: name: Cargo Deny if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed') - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 20 steps: - name: Checkout repository @@ -132,7 +132,7 @@ jobs: workflow-pin-report: name: Workflow Pin Report if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed') - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 5 steps: - name: Checkout repository @@ -169,7 +169,7 @@ jobs: dependency-review: name: Dependency Review - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 if: github.event_name == 'pull_request' && github.event.action != 'closed' permissions: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index a647019c5..41941fbcf 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -22,9 +22,12 @@ # 3. Trigger docker.yml to build and push images using the uploaded binaries # # Platform scope: -# - Pushes to main (development builds) build the Linux targets only — they -# feed the dev download channel and Docker dev images. Tags, the weekly -# schedule and manual dispatch build the full platform matrix. +# - Development builds come from the nightly schedule and manual dispatch and +# cover the full platform matrix; they feed the dev download channel +# (artifacts/rustfs/dev on R2, the *-main-latest packages). Tags build the +# release matrix. Main pushes do not build: at the usual merge rate most +# per-merge builds were cancelled by the next merge before publishing +# anything, and the dev channel does not need per-merge granularity. # # Manual Parameters: # - publish: Publish binaries/releases for manual runs (default: false) @@ -36,24 +39,8 @@ name: Build and Release on: push: tags: [ "*.*.*" ] - branches: [ main ] - paths-ignore: - - "**.md" - - "**.txt" - - ".github/**" - - "docs/**" - - "deploy/**" - - "scripts/dev_*.sh" - - "LICENSE*" - - "README*" - - "**/*.png" - - "**/*.jpg" - - "**/*.svg" - - ".gitignore" - - ".dockerignore" - - "flake.lock" schedule: - - cron: "13 1 * * 0" # Weekly on Sunday 01:13 UTC + - cron: "13 1 * * *" # Nightly 01:13 UTC workflow_dispatch: inputs: publish: @@ -93,7 +80,7 @@ jobs: # Build strategy check - determine build type based on trigger build-check: name: Build Strategy Check - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 outputs: should_build: ${{ steps.check.outputs.should_build }} @@ -144,12 +131,6 @@ jobs: build_type="release" echo "📦 Release build detected: $tag_name" fi - elif [[ "${{ github.ref }}" == "refs/heads/main" ]]; then - # Main branch push - development build - should_build=true - build_type="development" - version="dev-${short_sha}" - echo "🛠️ Development build detected" elif [[ "${{ github.event_name }}" == "schedule" ]] || \ [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then # Scheduled or manual build @@ -178,7 +159,7 @@ jobs: prepare-platform-matrix: name: Prepare Platform Matrix needs: build-check - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 outputs: matrix: ${{ steps.select.outputs.matrix }} @@ -200,25 +181,14 @@ jobs: selected="all" fi - # Per-merge development builds only feed the dev download channel - # and the Docker dev images, which consume the Linux binaries; at - # the usual merge cadence most per-merge builds are cancelled by the - # next merge anyway. macOS and Windows stay covered by tag builds, - # the weekly scheduled build and manual dispatch. - if [[ "${selected}" == "all" \ - && "${{ github.event_name }}" == "push" \ - && "${{ needs.build-check.outputs.build_type }}" == "development" ]]; then - selected="linux-x86_64-musl,linux-aarch64-musl,linux-x86_64-gnu,linux-aarch64-gnu" - echo "Development (main push) build: restricting to Linux targets" - fi all='{"include":[ - {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""}, - {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""}, - {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"}, - {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"}, - {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"}, - {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""} + {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, + {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, + {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, + {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, + {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""}, + {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""} ]}' if [[ "${selected}" == "all" ]]; then @@ -246,7 +216,8 @@ jobs: echo "Selected platforms: ${selected}" build-rustfs: - name: Build RustFS + # Acceptance workflows select artifacts by this stable platform name. + name: Build RustFS (${{ matrix.target_id }}) needs: [ build-check, prepare-platform-matrix ] if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success' runs-on: ${{ matrix.os }} @@ -273,7 +244,7 @@ jobs: with: rust-version: stable target: ${{ matrix.target }} - cache-shared-key: build-${{ matrix.target }} + cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }} # main only. A cache saved on refs/tags/X is scoped to that tag: no # other tag, no main run and no PR can restore it, so every release # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag, @@ -410,35 +381,53 @@ jobs: shell: bash env: BUILD_TARGET: ${{ matrix.target }} + BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }} run: | # Force rebuild by touching build.rs touch rustfs/build.rs - BINARY_NAMES=(rustfs) - if [[ "${{ matrix.platform }}" == "linux" ]]; then - BINARY_NAMES+=(rustfs-cli) - fi FEATURE_ARGS=() if [[ -n "${{ matrix.features }}" ]]; then FEATURE_ARGS=(--features "${{ matrix.features }}") fi - # Build one binary at a time so release LTO links cannot exhaust a - # self-hosted runner by running concurrently. - for binary in "${BINARY_NAMES[@]}"; do - PROFILE_ARGS=() - if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then - # The in-process profiler resolves names from the final executable. - # Retain function symbols without enabling DWARF source information. - PROFILE_ARGS=(--config 'profile.release.strip="none"') - fi - if [[ "${{ matrix.cross }}" == "true" ]]; then - # All cross targets in the matrix are Linux; zigbuild handles them. - cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary" - else - cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary" - fi - done + # gnu targets pin an explicit glibc baseline (`.` for + # cargo-zigbuild) so release binaries link no symbols newer than it + # and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062. + TARGET_ARG="$BUILD_TARGET" + if [[ -n "${{ matrix.glibc }}" ]]; then + TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}" + fi + + PROFILE_ARGS=() + if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then + # The in-process profiler resolves names from the final executable. + # Retain function symbols without enabling DWARF source information. + PROFILE_ARGS=(--config 'profile.release.strip="none"') + fi + if [[ "${{ matrix.cross }}" == "true" ]]; then + cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs + else + cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs + fi + + # All release targets use same-architecture runners, including GNU + # builds that use Zig only to select the glibc ABI floor. + bin="target/${{ matrix.target }}/release/rustfs" + if [[ -f "${bin}.exe" ]]; then + bin="${bin}.exe" + fi + version_output=$("$bin" --version) + if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \ + ! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \ + ! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then + echo "release binary reports an unexpected compile target" >&2 + exit 1 + fi + if [[ -n "${{ matrix.glibc }}" ]]; then + python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \ + "target/${{ matrix.target }}/release/rustfs" + fi - name: Verify macOS binary has no dynamic liblzma dependency if: matrix.platform == 'macos' @@ -554,9 +543,6 @@ jobs: fi BINARY_FILES=("$BINARY_NAME") - if [[ "${{ matrix.platform }}" == "linux" ]]; then - BINARY_FILES+=("rustfs-cli") - fi if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256) @@ -629,7 +615,7 @@ jobs: LATEST_FILES="$LATEST_FILE" fi elif [[ "$BUILD_TYPE" == "development" ]]; then - # Development builds (only main branch triggers development builds) + # Development builds (nightly schedule and manual dispatch) # Create main-latest version filename # Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest MAIN_LATEST_FILE="${PACKAGE_BASENAME}-main-latest.zip" @@ -671,7 +657,7 @@ jobs: echo "🔧 Build type: ${BUILD_TYPE}" echo "📊 Version: ${VERSION}" - - name: Verify packaged Linux binaries + - name: Verify packaged Linux binary if: matrix.platform == 'linux' shell: bash run: | @@ -679,7 +665,7 @@ jobs: package_file="${{ steps.package.outputs.package_file }}" archive_members="$(unzip -Z1 "$package_file" | sort)" - expected_members=$'rustfs\nrustfs-cli' + expected_members='rustfs' target="${{ matrix.target }}" features="${{ matrix.features }}" if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then @@ -691,13 +677,14 @@ jobs: exit 1 fi - # Every Linux target builds natively, so executing the binaries here + # musl targets build natively; the gnu legs are same-arch + # cargo-zigbuild builds pinned to the release glibc baseline. Either + # way the binaries run on their runner, so executing them here # proves each artifact matches its runner architecture. - cli_dir="$(mktemp -d)" - trap 'rm -rf "$cli_dir"' EXIT - unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir" - "$cli_dir/rustfs" --version - "$cli_dir/rustfs-cli" --help >/dev/null + binary_dir="$(mktemp -d)" + trap 'rm -rf "$binary_dir"' EXIT + unzip -q "$package_file" rustfs -d "$binary_dir" + "$binary_dir/rustfs" --version - name: Verify packaged console if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu' @@ -843,7 +830,7 @@ jobs: name: Build Summary needs: [ build-check, build-rustfs ] if: always() && needs.build-check.outputs.should_build == 'true' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Build completion summary @@ -1032,7 +1019,7 @@ jobs: name: Update Latest Version needs: [ build-check, publish-release ] if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease') - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 steps: - name: Update latest.json diff --git a/.github/workflows/cache-warm.yml b/.github/workflows/cache-warm.yml index c01dee315..bd8009b45 100644 --- a/.github/workflows/cache-warm.yml +++ b/.github/workflows/cache-warm.yml @@ -42,26 +42,23 @@ name: Cache Warm on: + schedule: + # Every six hours; the 22:41 UTC run lands before ci.yml's nightly 00:11 + # UTC matrix. A reader restores the newest entry under its key and + # rebuilds whatever its own diff changed, so a cache up to six hours old + # costs a PR the handful of crates merged since. Warming after every merge + # (fourteen completed runs a day, about two sm-standard-4 hours each) + # spent far more than that to avoid it. + - cron: "41 4,10,16,22 * * *" push: branches: [ main ] - # Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml - # compiles, it cannot change what ci.yml needs restored either. - paths-ignore: - - "**.md" - - "docs/**" - - "deploy/**" - - "scripts/dev_*.sh" - - "scripts/probe.sh" - - "LICENSE*" - - ".gitignore" - - ".dockerignore" - - "README*" - - "**/*.png" - - "**/*.jpg" - - "**/*.svg" - - ".github/workflows/build.yml" - - ".github/workflows/docker.yml" - - ".github/workflows/audit.yml" + # A toolchain or lockfile change invalidates most of every key; those + # re-warm at once instead of waiting for the next slot. + paths: + - "Cargo.lock" + - "rust-toolchain.toml" + - ".github/actions/setup/**" + - ".github/workflows/cache-warm.yml" workflow_dispatch: inputs: emit_timings: @@ -75,7 +72,7 @@ on: permissions: contents: read -# Scoped by event. A push run and a dispatch run do not compete: GitHub keeps +# Scoped by event. Scheduled, push and dispatch runs do not compete: GitHub keeps # one running plus one pending per group, so with a single shared group a # manually dispatched run was displaced as pending by the next merge and # cancelled — observed three times in a row, which made the --timings gate in @@ -162,6 +159,17 @@ jobs: cargo build --workspace --all-targets cargo build -p rustfs --bins --features e2e-test-hooks + # ci.yml runs Clippy on every pull request. Clippy's check-mode artifacts + # are separate from the build above, so warm them with the exact + # invocation ci.yml uses; a pull request then re-checks only what its diff + # touches. A lint failure on main must not stop the warm-up: the cache is + # still saved (cache-on-failure), and the failure belongs to ci.yml. + - name: Warm Clippy check artifacts + run: | + if ! cargo clippy --all-targets -- -D warnings; then + echo "::warning::cargo clippy reports findings on this commit; check artifacts are still cached" + fi + # Runs before rust-cache's post step, so these are the sizes it is about # to archive. Reported so the cache-all-crates decision stays evidence-led: # registry/src is what that flag prunes, registry/cache is what the pruned @@ -185,9 +193,10 @@ jobs: echo '```' } >> "$GITHUB_STEP_SUMMARY" - # Readers: test-and-lint-rio-v2 (per-PR), build-rustfs-debug-binary-rio-v2 - # (weekly schedule / manual dispatch only — dormant rio-v2 variant, see - # rustfs/backlog#1835 and docs/architecture/minio-file-format-compat.md). + # Readers: test-and-lint-rio-v2 (nightly schedule, merge group, dispatch), + # build-rustfs-debug-binary-rio-v2 (nightly schedule / manual dispatch only — + # dormant rio-v2 variant, see rustfs/backlog#1835 and + # docs/architecture/minio-file-format-compat.md). # The second build below stays despite the reduced cadence: it warms the # rio-v2,e2e-test-hooks feature resolution the scheduled build restores, # which keeps that lane inside its 30-minute timeout. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2e92b226e..8e03643d9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,7 +42,7 @@ on: merge_group: types: [ checks_requested ] schedule: - - cron: "11 0 * * 0" # Weekly on Sunday 00:11 UTC + - cron: "11 0 * * *" # Nightly 00:11 UTC, after the 22:41 UTC cache-warm run workflow_dispatch: permissions: @@ -62,10 +62,15 @@ env: jobs: + # Compile-free lanes, and lanes that only run a prebuilt binary, use + # GitHub-hosted runners: they are free for this public repository, and the + # self-hosted pool is reserved for the lanes that need its CPU count and + # disk (the workspace test build leaves a ~64GB target/, hosted runners + # have 14GB). Placement and billing rules: README.md in this directory. cancel-closed-pr-runs: name: Cancel Closed PR Runs if: github.event_name == 'pull_request' && github.event.action == 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Explain cancellation run @@ -74,7 +79,7 @@ jobs: classify-changes: name: Select CI scope if: github.event_name != 'pull_request' || github.event.action != 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 outputs: mode: ${{ steps.scope.outputs.mode }} @@ -100,16 +105,12 @@ jobs: typos: name: Typos if: github.event_name != 'pull_request' || github.event.action != 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 with: persist-credentials: false - - name: Install typos download dependency - run: | - sudo apt-get update - sudo apt-get install -y --no-install-recommends wget - name: Typos check with custom config file uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2 @@ -117,7 +118,7 @@ jobs: quick-checks: name: Quick Checks if: github.event_name != 'pull_request' || github.event.action != 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Checkout repository @@ -168,10 +169,13 @@ jobs: echo "started_at=$(date --utc --iso-8601=seconds)" } > artifacts/test-and-lint/run-metadata.txt - # Keep full Clippy validation on main, merge groups, and scheduled/manual - # runs. PRs retain the complete default test suite and compile-free checks. + # Clippy gates every event, pull requests included. When it ran only after + # merge, a lint regression merged freely and then failed every main run + # until someone fixed it (10 of 12 main runs on 2026-10-17/18). cache-warm + # writes the same invocation's check artifacts into ci-dev, so a pull + # request re-checks only the crates its diff touches. Main keeps the step + # because pull requests can merge without being up to date with main. - name: Run clippy lints - if: github.event_name != 'pull_request' run: | ./scripts/ci/resource_sampler.sh start clippy trap './scripts/ci/resource_sampler.sh stop' EXIT @@ -179,31 +183,27 @@ jobs: - name: Run nextest tests env: - # #5394 mitigation, now under a measured experiment (backlog#1601). + # #5394 mitigation, settled by the backlog#1601 experiment. # # 2 was chosen when three concurrent workspace test links were believed # to saturate the runner's overlay I/O and wedge Cargo until the 75m # timeout. cgroup v2 readings from the sampler show the pod actually - # has 14 CPUs and 28GB (peak use 2.1GB), so 2 throttles compilation to - # a seventh of what is available and memory was never the constraint — - # the label name "sm-standard-4" had led everyone, including the - # original mitigation, to assume 4 cores. + # has 14 CPUs and 28GB (peak use 2.1GB): the label name "sm-standard-4" + # had led everyone, including the original mitigation, to assume 4 + # cores, and memory was never the constraint. # - # Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the - # merge path is untouched while the experiment runs. - # - # During baseline collection, main pushes cancelled superseded runs: - # only 4 of 20 Test and Lint jobs reached a terminal state. Running - # main validations now finish; dispatch remains an independent group - # for collecting additional samples on demand. - # - # Baseline over 17 samples at 2: - # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2 - # criterion is that ratio dropping at least 10% (below ~1.76) with no - # 75m timeout and no run showing three consecutive samples of - # rustc/collect2/rust-lld in D state. If it does not, the conclusion is - # "this limit is not the bottleneck" — fix it back at 2 and record the - # experiment, which is a result, not a failure. + # Measured on the same ci-dev cache, the test-profile compile took + # 28m21s at 2 on a pull request and 16-23 min at 3 across three + # successful main runs. The first pull-request run at 3 that met a + # busy node (run 39193827281) then hit the 75m timeout before any + # test started: the sampler shows cargo spinning for 70 minutes with + # the pod's io.pressure "full" at 86-88% and CPU pressure at zero, + # while node-level IO pressure was already 36% from co-tenants when + # the step began. The constraint is the pod's shared storage, not + # CPU or memory, and the limit that storage sustains on a contended + # node is 2. Main pushes and dispatches keep 3 (rarer, and three + # successes); pull requests stay at 2 until the pods get local disk + # for target/, which is the lever that would unlock the 14 cores. # # Must stay step-level: rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST # prefixed variables from process.env into the cache key, so promoting @@ -323,10 +323,12 @@ jobs: path: artifacts/ilm-integration retention-days: 3 - # This gate builds into a fresh target directory to isolate the E2E root. - # Give its cold build a separate budget from workspace tests and migration proofs. - offline-enrollment-e2e: - name: Offline Enrollment E2E Root Boundary + # Both Connect boundaries rebuild only the rustfs crate on top of the restored + # ci-dev cache: the two e2e features are empty, so every dependency keeps its + # cached dev-profile artifact and each feature set recompiles rustfs once. + # The budget covers a ci-dev miss, where the first step builds the whole graph. + connect-feature-boundaries: + name: Connect Feature Boundaries if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed') needs: [ quick-checks, classify-changes ] runs-on: sm-standard-4 @@ -351,14 +353,58 @@ jobs: - name: Protect Connect test home run: chmod go-w "$(realpath "$HOME")" + - name: Run short credential behavior tests + env: + CARGO_BUILD_JOBS: "2" + run: | + cargo test -p rustfs --test connect_registration \ + --features connect-e2e-short-credentials \ + registration_enforces_build_profile_credential_lifetime -- --exact + cargo test -p rustfs --test connect_registration \ + --features connect-e2e-short-credentials \ + rotation_waits_for_threshold_and_stops_on_revocation -- --exact + + # The guard in rustfs/src/connect/registration.rs is cfg(not(debug_assertions)), + # and it must be tripped by a profile that turns debug assertions off for + # the whole graph. Handing `-C debug-assertions=off` to the rustfs lib alone + # through `cargo rustc` leaves its dependencies on the dev-profile cfg, so + # the lib also fails on items gated on one side only (E0599 on + # Filenames::Embedded) and "fails solely at the compile_error" can never + # hold. `cargo check --release` is metadata-only and needs no + # link-concurrency limit; it is cold because ci-dev holds no release + # artifacts. + - name: Reject short credentials in release builds + run: | + log="$(mktemp)" + set +e + CARGO_TERM_COLOR=never cargo check -p rustfs --release \ + --features connect-e2e-short-credentials >"$log" 2>&1 + status=$? + set -e + cat "$log" + expected='error: connect-e2e-short-credentials is restricted to debug builds' + summary="error: could not compile \`rustfs\` (lib) due to 1 previous error" + expected_count="$(grep -Fxc "$expected" "$log" || true)" + summary_count="$(grep -Fc "$summary" "$log" || true)" + error_count="$(grep -Ec '^error(:|\[)' "$log" || true)" + if [ "$status" -ne 101 ] || [ "$expected_count" -ne 1 ] \ + || [ "$summary_count" -ne 1 ] || [ "$error_count" -ne 2 ]; then + echo "feature gate did not fail solely at the expected compile_error" >&2 + rm -f "$log" + exit 1 + fi + rm -f "$log" + - name: Check offline enrollment E2E root boundary run: ./scripts/check_offline_enrollment_e2e.sh test-and-lint-rio-v2: name: Test and Lint (rio-v2) - # Alternate feature configurations run on main, merge groups, and scheduled - # or manual validation; the default workspace remains the PR gate. - if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request' + # Alternate feature configurations run on the nightly schedule, merge + # groups and manual dispatch. The default workspace remains the PR gate and + # the per-push main baseline: repeating four feature builds on every merge + # bought nothing the nightly run does not report within a day. + if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["merge_group", "schedule", "workflow_dispatch"]'), github.event_name) needs: [ quick-checks, classify-changes ] runs-on: sm-standard-4 timeout-minutes: 90 @@ -414,67 +460,10 @@ jobs: retention-days: 3 if-no-files-found: error - connect-short-credential-boundary: - name: Connect Short Credential Boundary - if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed') - needs: [ quick-checks, classify-changes ] - runs-on: sm-standard-4 - timeout-minutes: 60 - env: - FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" - steps: - - name: Checkout repository - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 - with: - persist-credentials: false - - - name: Setup Rust environment - uses: ./.github/actions/setup - with: - rust-version: stable - cache-shared-key: ci-dev - cache-save-if: 'false' - install-build-packaging-tools: 'false' - install-test-tools: 'false' - - - name: Run short credential behavior tests - env: - CARGO_BUILD_JOBS: "2" - run: | - cargo test -p rustfs --test connect_registration \ - --features connect-e2e-short-credentials \ - registration_enforces_build_profile_credential_lifetime -- --exact - cargo test -p rustfs --test connect_registration \ - --features connect-e2e-short-credentials \ - rotation_waits_for_threshold_and_stops_on_revocation -- --exact - - - name: Reject short credentials in release builds - env: - CARGO_BUILD_JOBS: "2" - run: | - log="$(mktemp)" - set +e - CARGO_TERM_COLOR=never cargo check -p rustfs --release \ - --features connect-e2e-short-credentials >"$log" 2>&1 - status=$? - set -e - cat "$log" - expected='error: connect-e2e-short-credentials is restricted to debug builds' - summary="error: could not compile \`rustfs\` (lib) due to 1 previous error" - expected_count="$(grep -Fxc "$expected" "$log" || true)" - summary_count="$(grep -Fc "$summary" "$log" || true)" - error_count="$(grep -Ec '^error(:|\[)' "$log" || true)" - if [ "$status" -ne 101 ] || [ "$expected_count" -ne 1 ] \ - || [ "$summary_count" -ne 1 ] || [ "$error_count" -ne 2 ]; then - echo "release feature gate did not fail solely at the expected compile_error" >&2 - rm -f "$log" - exit 1 - fi - rm -f "$log" - test-and-lint-protocols: name: "Test and Lint (${{ matrix.features.name }})" - if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request' + # Same cadence as test-and-lint-rio-v2 above. + if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["merge_group", "schedule", "workflow_dispatch"]'), github.event_name) needs: [ quick-checks, classify-changes ] runs-on: sm-standard-4 timeout-minutes: 90 @@ -630,7 +619,7 @@ jobs: build-rustfs-debug-binary-rio-v2: name: Build RustFS Debug Binary (rio-v2) # Dormant rio-v2 variant (rustfs/backlog#1835): the feature ships in no - # default build, so this full-suite lane runs only on the weekly schedule + # default build, so this full-suite lane runs only on the nightly schedule # and manual dispatch. Per-PR cfg-seam coverage stays with # test-and-lint-rio-v2. Lifecycle and the promote-or-delete condition: # docs/architecture/minio-file-format-compat.md ("rio-v2 variant lifecycle"). @@ -1100,7 +1089,7 @@ jobs: s3-implemented-tests: name: S3 Implemented Tests needs: [ build-rustfs-debug-binary, e2e-tests ] - runs-on: sm-standard-4 + runs-on: ubuntu-latest timeout-minutes: 60 steps: - name: Checkout repository @@ -1169,12 +1158,12 @@ jobs: name: S3 Lifecycle Behavior Tests # Also gated on e2e-tests, matching s3-implemented-tests: when the e2e smoke # suite is already red this lane cannot tell us anything new, and it holds a - # sm-standard-4 for up to 30 minutes doing so. Both lanes only download the + # runner for up to 30 minutes doing so. Both lanes only download the # prebuilt debug binary (no cargo build), and s3-implemented-tests — which # already waits on e2e-tests — finishes later anyway, so a green PR's total # wall clock is unchanged. needs: [ build-rustfs-debug-binary, e2e-tests ] - runs-on: sm-standard-4 + runs-on: ubuntu-latest timeout-minutes: 30 steps: - name: Checkout repository @@ -1231,9 +1220,8 @@ jobs: - typos - quick-checks - test-and-lint - - offline-enrollment-e2e + - connect-feature-boundaries - test-and-lint-rio-v2 - - connect-short-credential-boundary - test-and-lint-protocols - build-rustfs-debug-binary - uring-integration @@ -1243,7 +1231,7 @@ jobs: - build-rustfs-debug-binary-rio-v2 - e2e-tests-rio-v2 - e2e-full - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 @@ -1259,12 +1247,11 @@ jobs: name: Alert on scheduled failure needs: - classify-changes - - connect-short-credential-boundary + - connect-feature-boundaries - required-checks - typos - quick-checks - test-and-lint - - offline-enrollment-e2e - test-and-lint-rio-v2 - test-and-lint-protocols - build-rustfs-debug-binary diff --git a/.github/workflows/cla.yml b/.github/workflows/cla.yml index 13158d288..12ca8a533 100644 --- a/.github/workflows/cla.yml +++ b/.github/workflows/cla.yml @@ -46,7 +46,7 @@ jobs: # Echoes one line; the run exists only so the concurrency group cancels the # in-flight run of a closed PR. permissions: {} - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Explain cancellation run @@ -61,7 +61,7 @@ jobs: checks: write issues: write pull-requests: write - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 steps: - name: Report CLA result for merge queue diff --git a/.github/workflows/connect-diagnostic-schedule-acceptance.yml b/.github/workflows/connect-diagnostic-schedule-acceptance.yml index 4b12335a3..0a69329df 100644 --- a/.github/workflows/connect-diagnostic-schedule-acceptance.yml +++ b/.github/workflows/connect-diagnostic-schedule-acceptance.yml @@ -57,6 +57,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -94,7 +102,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") @@ -137,7 +145,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2 + assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3 assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries) assert inner.testzip() is None pathlib.Path('binary').mkdir() @@ -162,7 +170,7 @@ jobs: connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check (cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh) - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule diff --git a/.github/workflows/connect-health-service-acceptance.yml b/.github/workflows/connect-health-service-acceptance.yml index b0de48ea9..0423f10cc 100644 --- a/.github/workflows/connect-health-service-acceptance.yml +++ b/.github/workflows/connect-health-service-acceptance.yml @@ -57,6 +57,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -127,7 +135,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}") @@ -172,7 +180,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4 + assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3 assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries) assert inner.testzip() is None pathlib.Path('binary').mkdir() @@ -199,7 +207,7 @@ jobs: connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh) - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=health-service diff --git a/.github/workflows/connect-performance-drive-acceptance.yml b/.github/workflows/connect-performance-drive-acceptance.yml index 1513ddfb3..b1624dc3f 100644 --- a/.github/workflows/connect-performance-drive-acceptance.yml +++ b/.github/workflows/connect-performance-drive-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive diff --git a/.github/workflows/connect-performance-network-acceptance.yml b/.github/workflows/connect-performance-network-acceptance.yml index cd4b727af..f686f3288 100644 --- a/.github/workflows/connect-performance-network-acceptance.yml +++ b/.github/workflows/connect-performance-network-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=performance-network diff --git a/.github/workflows/connect-profile-cpu-acceptance.yml b/.github/workflows/connect-profile-cpu-acceptance.yml index c91bd257b..496769dac 100644 --- a/.github/workflows/connect-profile-cpu-acceptance.yml +++ b/.github/workflows/connect-profile-cpu-acceptance.yml @@ -77,6 +77,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -118,7 +126,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.jobs[] | select(.name == $name)] as $matches @@ -173,7 +181,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \ CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \ diff --git a/.github/workflows/connect-profile-memory-acceptance.yml b/.github/workflows/connect-profile-memory-acceptance.yml index 6474f0896..77d226e04 100644 --- a/.github/workflows/connect-profile-memory-acceptance.yml +++ b/.github/workflows/connect-profile-memory-acceptance.yml @@ -57,6 +57,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -127,7 +135,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}") @@ -172,7 +180,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} + expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} assert {entry.filename for entry in entries} == expected and len(entries) == len(expected) assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries) assert inner.testzip() is None @@ -198,7 +206,7 @@ jobs: connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check (cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh) - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory diff --git a/.github/workflows/connect-profile-threads-acceptance.yml b/.github/workflows/connect-profile-threads-acceptance.yml index 25479d60d..5cc0c67f1 100644 --- a/.github/workflows/connect-profile-threads-acceptance.yml +++ b/.github/workflows/connect-profile-threads-acceptance.yml @@ -81,6 +81,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -148,7 +156,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -201,7 +209,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads diff --git a/.github/workflows/connect-top-api-acceptance.yml b/.github/workflows/connect-top-api-acceptance.yml index 2c112dffd..c76184149 100644 --- a/.github/workflows/connect-top-api-acceptance.yml +++ b/.github/workflows/connect-top-api-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-api diff --git a/.github/workflows/connect-top-disk-acceptance.yml b/.github/workflows/connect-top-disk-acceptance.yml index e6b32ce43..58f5ccada 100644 --- a/.github/workflows/connect-top-disk-acceptance.yml +++ b/.github/workflows/connect-top-disk-acceptance.yml @@ -68,6 +68,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -138,7 +146,7 @@ jobs: ' <<<"$run" >/dev/null jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e ' - [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches + [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches | ($matches | length) == 1 and $matches[0].conclusion == "success" ' <<<"$jobs" >/dev/null artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}") @@ -183,7 +191,7 @@ jobs: assert outer.testzip() is None with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner: entries = inner.infolist() - expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} + expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} assert {entry.filename for entry in entries} == expected and len(entries) == len(expected) assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries) assert inner.testzip() is None @@ -213,7 +221,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-disk diff --git a/.github/workflows/connect-top-locks-acceptance.yml b/.github/workflows/connect-top-locks-acceptance.yml index edd8e93b0..6d0051a5a 100644 --- a/.github/workflows/connect-top-locks-acceptance.yml +++ b/.github/workflows/connect-top-locks-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-locks diff --git a/.github/workflows/connect-top-rpc-acceptance.yml b/.github/workflows/connect-top-rpc-acceptance.yml index 5a8cf4da9..7128bc885 100644 --- a/.github/workflows/connect-top-rpc-acceptance.yml +++ b/.github/workflows/connect-top-rpc-acceptance.yml @@ -73,6 +73,14 @@ jobs: ref: ${{ inputs.connect_sha }} token: ${{ secrets.PF_TESTING_GH_TOKEN }} + - name: Set up Docker Compose + uses: ./.github/actions/setup-docker-compose + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Set up Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -107,7 +115,7 @@ jobs: [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] - expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + expected_job='Build RustFS (linux-x86_64-gnu)' jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") jq -e --arg name "$expected_job" ' [.[].jobs[] | select(.name == $name)] as $matches @@ -154,7 +162,7 @@ jobs: npm --prefix connect-harness/web ci connect-harness/web/node_modules/.bin/playwright install --with-deps chromium make -C connect-harness e2e-connected-dispatch-check - RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \ make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index af44d5e35..44cd477a5 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -83,7 +83,7 @@ jobs: (github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' && github.event.workflow_run.head_branch != 'main') - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 outputs: should_build: ${{ steps.check.outputs.should_build }} @@ -501,7 +501,7 @@ jobs: name: Scan Docker Images needs: [ build-check, build-docker ] if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 permissions: contents: read @@ -556,7 +556,7 @@ jobs: name: Docker Build Summary needs: [ build-check, build-docker ] if: always() && needs.build-check.outputs.should_build == 'true' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Docker build completion summary diff --git a/.github/workflows/e2e-s3tests.yml b/.github/workflows/e2e-s3tests.yml index 37a678273..6c0416815 100644 --- a/.github/workflows/e2e-s3tests.yml +++ b/.github/workflows/e2e-s3tests.yml @@ -371,7 +371,7 @@ jobs: name: Upstream HEAD classification canary if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' continue-on-error: true - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 20 steps: - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 diff --git a/.github/workflows/e2e-upgrade.yml b/.github/workflows/e2e-upgrade.yml index b926b613d..10579005f 100644 --- a/.github/workflows/e2e-upgrade.yml +++ b/.github/workflows/e2e-upgrade.yml @@ -46,14 +46,11 @@ concurrency: env: CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 - UPGRADE_SOURCE_VERSION: 1.0.0-rc.5 - UPGRADE_SOURCE_ASSET: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip - UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d jobs: build: name: Build upgrade candidate - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 60 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" @@ -66,19 +63,11 @@ jobs: - name: Setup Rust environment uses: ./.github/actions/setup with: - cache-shared-key: e2e-upgrade-server + cache-shared-key: e2e-upgrade-server-hosted cache-save-if: ${{ github.ref == 'refs/heads/main' }} install-build-packaging-tools: "false" install-test-tools: "false" - - name: Download pinned previous release - run: | - mkdir -p target/debug - archive="target/debug/$UPGRADE_SOURCE_ASSET" - curl --fail --location --retry 3 --output "$archive" \ - "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}" - echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict - - name: Build current RustFS binary run: python3 scripts/e2e_binary.py build @@ -89,19 +78,31 @@ jobs: path: | target/debug/rustfs target/debug/rustfs.e2e.json - target/debug/${{ env.UPGRADE_SOURCE_ASSET }} if-no-files-found: error retention-days: 3 compression-level: 0 overwrite: true upgrade: - name: Upgrade compatibility + name: Upgrade compatibility (${{ matrix.source.version }}) needs: build - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + source: + - version: 1.0.0-rc.5 + asset: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip + sha256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d + - version: 1.0.1 + asset: rustfs-linux-x86_64-gnu-v1.0.1.zip + sha256: 9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + UPGRADE_SOURCE_VERSION: ${{ matrix.source.version }} + UPGRADE_SOURCE_ASSET: ${{ matrix.source.asset }} + UPGRADE_SOURCE_SHA256: ${{ matrix.source.sha256 }} steps: - name: Checkout repository uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 @@ -111,7 +112,7 @@ jobs: - name: Setup Rust environment uses: ./.github/actions/setup with: - cache-shared-key: e2e-upgrade-tests + cache-shared-key: e2e-upgrade-tests-hosted cache-save-if: ${{ github.ref == 'refs/heads/main' }} install-build-packaging-tools: "false" install-test-tools: "false" @@ -132,6 +133,8 @@ jobs: set -euo pipefail mkdir -p "$SOURCE_DIR" archive="target/debug/$UPGRADE_SOURCE_ASSET" + curl --fail --location --retry 3 --output "$archive" \ + "https://github.com/rustfs/rustfs/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}" echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict unzip -q "$archive" -d "$SOURCE_DIR" chmod +x "$SOURCE_DIR/rustfs" @@ -148,14 +151,18 @@ jobs: run: | export no_proxy="$NO_PROXY" cases=( - direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts - mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts + direct-upgrade:direct_upgrade_from_previous_release_preserves_object_contracts + mixed-version-upgrade:rolling_upgrade_from_previous_release_preserves_mixed_version_contracts bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata - odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration - multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts - multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts + multipart-layout-upgrade:direct_upgrade_from_previous_release_preserves_multipart_layouts + multipart-layout-baseline:previous_release_baseline_replicates_multipart_layouts ) + if [[ "$UPGRADE_SOURCE_VERSION" == "1.0.0-rc.5" ]]; then + cases+=(odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration) + else + cases+=(legacy-delete-reservation:stable_upgrade_deletes_foreign_reservations_without_losing_live_objects) + fi status=0 for entry in "${cases[@]}"; do artifact="${entry%%:*}" @@ -180,7 +187,7 @@ jobs: if: always() uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: upgrade-g09-evidence-${{ github.run_number }} + name: upgrade-g09-evidence-${{ matrix.source.version }}-${{ github.run_number }} path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence if-no-files-found: ignore retention-days: 14 @@ -189,7 +196,7 @@ jobs: if: always() uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: upgrade-server-logs-${{ github.run_number }} + name: upgrade-server-logs-${{ matrix.source.version }}-${{ github.run_number }} path: ${{ runner.temp }}/rustfs-upgrade-logs if-no-files-found: warn retention-days: 14 diff --git a/.github/workflows/fuzz.yml b/.github/workflows/fuzz.yml index 55d513445..c9c555471 100644 --- a/.github/workflows/fuzz.yml +++ b/.github/workflows/fuzz.yml @@ -60,7 +60,7 @@ jobs: cancel-closed-pr-runs: name: Cancel Closed PR Runs if: github.event_name == 'pull_request' && github.event.action == 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Explain cancellation run diff --git a/.github/workflows/gateway-integration-sync.yml b/.github/workflows/gateway-integration-sync.yml index f80fbfeb2..2209099e7 100644 --- a/.github/workflows/gateway-integration-sync.yml +++ b/.github/workflows/gateway-integration-sync.yml @@ -6,29 +6,20 @@ name: Gateway Integration Sync # Merges `main` into the long-lived `gateway/integration` branch every weekday # so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736) # never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit -# pushed straight to the branch. A conflicting merge is aborted and surfaced as -# a pull request from `sync/-`, whose head is the un-merged -# source commit, so the conflict is resolved the same day, in review, with no -# other change riding along. The run itself stays green in that case: the pull -# request is the signal. Any other failure (push rejected, pull request not -# created) fails the run and, on the schedule, files the shared failure issue. +# pushed straight to the branch. A conflicting merge is aborted and fails the +# run after its summary records the source and target SHAs, run URL, and exact +# conflicting paths. It creates no sync branch or pull request. Resolve the +# merge in an isolated worktree based on the latest target, preserve the +# two-parent merge commit, and push only after the required checks pass and the +# target still matches the merge's first parent. If the target moves, restart +# from its new head. # -# Resolving a conflict pull request: merge `origin/gateway/integration` into -# the sync branch, resolve, push, get the review. Land it with a fast-forward -# push of the resolved merge commit to `gateway/integration`, not with the -# squash button: a squash drops the merge ancestry and the same hunks conflict -# again on the next run. -# -# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and -# both push targets below are literal refs. The pushes use a one-hour -# installation token of the organisation's own GitHub App (contents: write) -# because that app is the only non-admin bypass actor of the +# Write scope: only `gateway/integration` is ever pushed. The push uses a +# one-hour installation token of the organisation's GitHub App (contents: +# write) because that app is the only non-admin bypass actor of the # `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor, -# so it cannot push through the required reviews and checks. The conflict pull -# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the -# weekly flake.lock pull requests, because this repository keeps "Allow GitHub -# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is -# therefore read-only here. +# so it cannot push through the required reviews and checks. GITHUB_TOKEN is +# read-only here. # # `schedule` fires only from the default branch, so the cron is live once this # file exists on `main`. `workflow_dispatch` runs from any ref that carries the @@ -94,6 +85,7 @@ jobs: id: merge env: SOURCE_REF: ${{ inputs.source_ref || 'main' }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | set -euo pipefail if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \ @@ -119,13 +111,25 @@ jobs: fi exit 0 fi - git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt" - git merge --abort - if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then + conflicts=$(git diff --name-only --diff-filter=U) + if [[ -z "$conflicts" ]]; then + git merge --abort || true echo "::error::git merge failed without reporting conflicting paths; this is not a conflict" exit 1 fi - echo "result=conflict" >> "$GITHUB_OUTPUT" + { + echo "## Gateway integration sync stopped on conflicts" + echo + echo "- Run: ${RUN_URL}" + echo "- Source: \`${SOURCE_REF}\` at ${source_sha}" + echo "- Target: \`${TARGET_BRANCH}\` at ${target_sha}" + echo + echo "Conflicting files:" + printf '%s\n' "$conflicts" | sed 's/^/- /' + } >> "$GITHUB_STEP_SUMMARY" + git merge --abort + echo "::error::${SOURCE_REF} cannot be merged cleanly into ${TARGET_BRANCH}; no branch or pull request was created" + exit 1 - name: Push the merge commit if: steps.merge.outputs.result == 'merged' @@ -137,50 +141,12 @@ jobs: git push origin HEAD:refs/heads/gateway/integration echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY" - - name: Open the conflict pull request - if: steps.merge.outputs.result == 'conflict' - env: - GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }} - SOURCE_REF: ${{ steps.merge.outputs.source_ref }} - SOURCE_SHA: ${{ steps.merge.outputs.source_sha }} - TARGET_SHA: ${{ steps.merge.outputs.target_sha }} - RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: | - set -euo pipefail - date_utc=$(date -u +%Y-%m-%d) - sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}" - git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}" - { - echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged." - echo - echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution." - echo - echo "- Run: ${RUN_URL}" - echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}" - echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}" - echo - echo "## Conflicting files" - echo - echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:" - echo - sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt" - } > "$RUNNER_TEMP/pr-body.md" - title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)" - existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty') - if [[ -n "$existing" ]]; then - gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md" - pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url) - else - pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md") - fi - echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY" - alert-on-failure: name: Alert on scheduled failure needs: [sync] # Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that - # could not push or could not open the conflict pull request must not stay - # visible only in the Actions tab. + # could not push or encountered merge conflicts must not stay visible only + # in the Actions tab. if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')) runs-on: ubuntu-latest timeout-minutes: 10 diff --git a/.github/workflows/helm-package.yml b/.github/workflows/helm-package.yml index 079e37f22..a42925cff 100644 --- a/.github/workflows/helm-package.yml +++ b/.github/workflows/helm-package.yml @@ -31,7 +31,7 @@ permissions: jobs: build-helm-package: - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 if: | (github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) || @@ -166,7 +166,7 @@ jobs: retention-days: 1 publish-helm-package: - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 needs: [ build-helm-package ] if: needs.build-helm-package.result == 'success' diff --git a/.github/workflows/issue-translator.yml b/.github/workflows/issue-translator.yml index 8a48906e2..04a57563d 100644 --- a/.github/workflows/issue-translator.yml +++ b/.github/workflows/issue-translator.yml @@ -32,7 +32,7 @@ permissions: jobs: build: - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7 diff --git a/.github/workflows/nightly-gnu.yml b/.github/workflows/nightly-gnu.yml index afbbb210c..dc2394b1e 100644 --- a/.github/workflows/nightly-gnu.yml +++ b/.github/workflows/nightly-gnu.yml @@ -45,7 +45,7 @@ jobs: resolve-source: if: github.repository == 'rustfs/rustfs' name: Resolve nightly source - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 outputs: source_sha: ${{ steps.source.outputs.sha }} @@ -80,13 +80,24 @@ jobs: - name: Setup Rust environment uses: ./.github/actions/setup with: - cache-shared-key: build-x86_64-unknown-linux-gnu + cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31 cache-save-if: 'false' + install-cross-tools: 'true' + target: x86_64-unknown-linux-gnu install-build-packaging-tools: 'false' install-test-tools: 'false' - name: Build RustFS - run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins + # Pin the glibc baseline (`.` for cargo-zigbuild) so + # nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062. + run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins + + - name: Verify glibc symbol baseline + run: | + python3 scripts/release/check_glibc.py 2.31 \ + target/x86_64-unknown-linux-gnu/release/rustfs \ + target/x86_64-unknown-linux-gnu/release/rustfs-cli + target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu' - name: Build DEB package id: deb diff --git a/.github/workflows/nix-flake-update.yml b/.github/workflows/nix-flake-update.yml index 06e22797b..726b2fcfe 100644 --- a/.github/workflows/nix-flake-update.yml +++ b/.github/workflows/nix-flake-update.yml @@ -34,7 +34,7 @@ jobs: update-flake: if: github.repository == 'rustfs/rustfs' name: Update flake.lock - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 90 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" diff --git a/.github/workflows/nix.yml b/.github/workflows/nix.yml index 42e2644f0..ca532aff3 100644 --- a/.github/workflows/nix.yml +++ b/.github/workflows/nix.yml @@ -54,7 +54,7 @@ jobs: cancel-closed-pr-runs: name: Cancel Closed PR Runs if: github.event_name == 'pull_request' && github.event.action == 'closed' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Explain cancellation run diff --git a/.github/workflows/on-demand-migration-interop.yml b/.github/workflows/on-demand-migration-interop.yml index 172607285..2e1b95e73 100644 --- a/.github/workflows/on-demand-migration-interop.yml +++ b/.github/workflows/on-demand-migration-interop.yml @@ -186,7 +186,7 @@ jobs: cloud-source: name: ${{ matrix.provider }} source (three-case minimum) if: github.repository == 'rustfs/rustfs' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 45 strategy: fail-fast: false diff --git a/.github/workflows/package.yml b/.github/workflows/package.yml index c76a2573e..21a3c4e3f 100644 --- a/.github/workflows/package.yml +++ b/.github/workflows/package.yml @@ -618,8 +618,10 @@ jobs: summary: name: Summary needs: [ resolve, package ] - if: always() - runs-on: sm-standard-2 + # Every completed Build and Release run triggers this workflow; when + # resolve declined it (not a release tag), there is nothing to summarise. + if: always() && needs.resolve.result != 'skipped' + runs-on: ubuntu-latest timeout-minutes: 5 steps: - name: Print summary diff --git a/.github/workflows/rustfs-validation-package.yml b/.github/workflows/rustfs-validation-package.yml index 914602f1d..2e1e3f921 100644 --- a/.github/workflows/rustfs-validation-package.yml +++ b/.github/workflows/rustfs-validation-package.yml @@ -63,7 +63,7 @@ jobs: build: needs: resolve-source name: Build immutable validation DEB - runs-on: sm-standard-4 + runs-on: ubuntu-latest timeout-minutes: 150 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" @@ -83,7 +83,7 @@ jobs: - name: Setup Rust environment uses: ./.github/actions/setup with: - cache-shared-key: validation-package-x86_64-unknown-linux-gnu + cache-shared-key: validation-package-hosted-x86_64-unknown-linux-gnu cache-save-if: "false" install-build-packaging-tools: "false" install-test-tools: "false" diff --git a/.github/workflows/schedule-failure-alert-drill.yml b/.github/workflows/schedule-failure-alert-drill.yml index 735806e3a..17cf13058 100644 --- a/.github/workflows/schedule-failure-alert-drill.yml +++ b/.github/workflows/schedule-failure-alert-drill.yml @@ -44,7 +44,7 @@ permissions: jobs: forced-failure: name: Forced failure - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 5 steps: - name: Fail on purpose diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index f25e696d9..307ec3855 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -27,7 +27,7 @@ on: jobs: stale: if: github.repository == 'rustfs/rustfs' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 30 steps: - uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0 diff --git a/.github/workflows/star-history.yml b/.github/workflows/star-history.yml index e7880bc96..37b342702 100644 --- a/.github/workflows/star-history.yml +++ b/.github/workflows/star-history.yml @@ -15,7 +15,7 @@ concurrency: jobs: update: if: github.repository == 'rustfs/rustfs' - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: overtrue/repo-visuals-action@ee2c632f6ce617e851fb46ea935ee8af762ebb93 # v1.4.0 diff --git a/.github/workflows/targets-integration.yml b/.github/workflows/targets-integration.yml index 6a0d8ac20..e2fb6740e 100644 --- a/.github/workflows/targets-integration.yml +++ b/.github/workflows/targets-integration.yml @@ -42,7 +42,7 @@ jobs: targets-live: if: github.repository == 'rustfs/rustfs' name: PostgreSQL, MySQL, AMQP, and NATS - runs-on: sm-standard-2 + runs-on: ubuntu-latest timeout-minutes: 90 env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" @@ -60,7 +60,7 @@ jobs: - name: Setup Rust environment uses: ./.github/actions/setup with: - cache-shared-key: targets-live-lane + cache-shared-key: targets-live-lane-hosted cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }} install-build-packaging-tools: 'false' install-test-tools: 'false' diff --git a/.github/workflows/windows-filesystem.yml b/.github/workflows/windows-filesystem.yml index 46a395456..fe6ca7000 100644 --- a/.github/workflows/windows-filesystem.yml +++ b/.github/workflows/windows-filesystem.yml @@ -19,8 +19,21 @@ on: branches: [ main ] paths: - "crates/ecstore/src/disk/**" + - "crates/ecstore/src/layout/**" - "crates/ecstore/src/store/init_format.rs" + - "crates/ecstore/src/set_disk/ops/locking.rs" + - "crates/ecstore/src/object_api/write_commit_context.rs" + - "crates/ecstore/src/bucket/utils.rs" - "crates/ecstore/Cargo.toml" + - "crates/utils/src/dunce.rs" + - "crates/utils/src/os/**" + - "crates/utils/Cargo.toml" + - "crates/utils/src/path.rs" + - "rustfs/build.rs" + - "rustfs/Cargo.toml" + - "rustfs/src/main.rs" + - "rustfs/src/server/runtime.rs" + - "rustfs/src/startup_*.rs" - "Cargo.toml" - "Cargo.lock" - ".github/actions/setup/**" @@ -29,8 +42,21 @@ on: branches: [ main ] paths: - "crates/ecstore/src/disk/**" + - "crates/ecstore/src/layout/**" - "crates/ecstore/src/store/init_format.rs" + - "crates/ecstore/src/set_disk/ops/locking.rs" + - "crates/ecstore/src/object_api/write_commit_context.rs" + - "crates/ecstore/src/bucket/utils.rs" - "crates/ecstore/Cargo.toml" + - "crates/utils/src/dunce.rs" + - "crates/utils/src/os/**" + - "crates/utils/Cargo.toml" + - "crates/utils/src/path.rs" + - "rustfs/build.rs" + - "rustfs/Cargo.toml" + - "rustfs/src/main.rs" + - "rustfs/src/server/runtime.rs" + - "rustfs/src/startup_*.rs" - "Cargo.toml" - "Cargo.lock" - ".github/actions/setup/**" @@ -77,14 +103,118 @@ jobs: shell: pwsh run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture + - name: Test Windows path conversion + shell: pwsh + run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture" + + - name: Test Windows volume and disk-space probes + shell: pwsh + run: | + # Map a drive letter to the temp drive's administrative share so the + # remote-volume probe also covers mapped network drives. + $drive = (Split-Path -Qualifier ([System.IO.Path]::GetTempPath())).TrimEnd(':') + net use R: "\\localhost\$drive`$" /persistent:no + if ($LASTEXITCODE -eq 0) { + $env:RUSTFS_TEST_WINDOWS_MAPPED_DRIVE = 'R:' + } else { + Write-Output "::warning::could not map R: to \\localhost\$drive`$; mapped-drive test will skip" + } + cargo test -p rustfs-utils --features os --lib os:: -- --nocapture + $status = $LASTEXITCODE + if ($env:RUSTFS_TEST_WINDOWS_MAPPED_DRIVE) { net use R: /delete /y | Out-Null } + exit $status + - name: Test Windows handle guards shell: pwsh run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture + - name: Test endpoint path parsing + shell: pwsh + run: "cargo test -p rustfs-ecstore --lib layout::endpoint:: -- --nocapture" + - name: Test startup temporary-directory cleanup shell: pwsh run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture + - name: Test object key Windows admission + shell: pwsh + run: "cargo test -p rustfs-ecstore --lib bucket::utils:: -- --nocapture" + + - name: Test Windows file-name rules + shell: pwsh + run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture" + - name: Test fresh format publication shell: pwsh run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture + + # The native Windows main thread only gets the 1 MiB PE stack reserve, while + # Linux and macOS hand out 8 MiB. Startup runs the whole root future on the + # thread that calls `block_on`, so a debug server once overflowed before it + # listened on Windows only. Start a real server to keep that covered. + startup-smoke: + name: Startup Smoke + runs-on: windows-latest + timeout-minutes: 90 + steps: + - name: Checkout repository + uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7 + with: + persist-credentials: false + + - name: Setup Rust environment + uses: ./.github/actions/setup + with: + rust-version: stable + cache-shared-key: build-x86_64-pc-windows-msvc + cache-save-if: 'false' + install-build-packaging-tools: 'false' + install-test-tools: 'false' + + - name: Build debug server binary + shell: pwsh + run: cargo build --locked -p rustfs --bin rustfs + + - name: Print version + shell: pwsh + run: | + & .\target\debug\rustfs.exe --version + if ($LASTEXITCODE -ne 0) { throw "rustfs --version exited with $LASTEXITCODE" } + + - name: Start server and wait for readiness + shell: pwsh + env: + RUSTFS_ACCESS_KEY: rustfsadmin + RUSTFS_SECRET_KEY: rustfsadmin + run: | + $volume = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke' + $stdout = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke.out.log' + $stderr = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke.err.log' + New-Item -ItemType Directory -Force -Path $volume | Out-Null + $server = Start-Process -FilePath .\target\debug\rustfs.exe ` + -ArgumentList @('--address', '127.0.0.1:9000', "`"$volume`"") ` + -RedirectStandardOutput $stdout -RedirectStandardError $stderr ` + -NoNewWindow -PassThru + # Cache the process handle now; otherwise ExitCode reads back empty. + $null = $server.Handle + $ready = $false + try { + for ($i = 0; $i -lt 180; $i++) { + if ($server.HasExited) { break } + try { + $response = Invoke-WebRequest -Uri 'http://127.0.0.1:9000/health/ready' -UseBasicParsing -NoProxy -TimeoutSec 2 + if ($response.StatusCode -eq 200) { $ready = $true; break } + } catch {} + Start-Sleep -Seconds 1 + } + } finally { + if (-not $server.HasExited) { Stop-Process -Id $server.Id -Force } + $server.WaitForExit() + Write-Host '--- server stdout (tail) ---' + Get-Content $stdout -Tail 100 -ErrorAction SilentlyContinue + Write-Host '--- server stderr (tail) ---' + Get-Content $stderr -Tail 100 -ErrorAction SilentlyContinue + } + if (-not $ready) { + throw ("rustfs did not report ready on /health/ready (exit code: {0})" -f $server.ExitCode) + } diff --git a/.gitignore b/.gitignore index 8a26b9e94..1dad3b899 100644 --- a/.gitignore +++ b/.gitignore @@ -90,3 +90,4 @@ worktrees/* # insta scratch files; the accepted .snap files ARE the assertions and are committed *.snap.new *.pending-snap +.mimosa \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index fd799b119..0f299e873 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,6 +3,9 @@ This file contains repository-wide rules. Use the nearest subdirectory `AGENTS.md` for path-specific invariants. +S3-compatible object store in Rust, derived from MinIO. Erasure-coded, +multi-pool, supports ILM tiering/lifecycle. + ## Precedence 1. System/developer instructions. @@ -38,6 +41,27 @@ rules. A skill cannot expand the user's requested scope or grant authorization. pending. Before requesting missing approval, prepare the concrete result that is already authorized; retain explicit merge and release gates. +- Keep individual tool outputs concise. Save large outputs to task-owned logs + and retrieve only the relevant ranges. + +## Common Commands + +```bash +cargo build --release --bin rustfs # production binary +cargo check -p # fast type-check one crate +cargo test -p # test one crate +cargo fmt --all --check # for Rust changes; see AGENTS.md verification tiers +make pre-commit # fast gate: fmt + arch checks + quick-check (NO clippy/tests) +make pre-pr # optional full gate for broad cross-module changes +make build-docker BUILD_OS=ubuntu22.04 +``` + +> **Docker build note**: `buildx build` without `--load` keeps the image in +> the buildx cache only — `docker run` will use a stale local image. The +> Makefile already includes `--load`; if you suspect a stale binary, add +> `--no-cache` to the `buildx build` invocation inside +> `.config/make/build-docker.mak`. + ## Task-Specific Guidance Read only the reference needed for the current task, once per unchanged context: @@ -65,6 +89,19 @@ or agent scratch notes. Durable architecture belongs under `docs/architecture/`, operations under `docs/operations/`, and testing references under `docs/testing/`. `scripts/check_no_planning_docs.sh` enforces this boundary. +## Further References + +- Agent knowledge base index and doc-writing rules: [docs/architecture/README.md](docs/architecture/README.md) +- Crate membership: `Cargo.toml` `[workspace].members` +- Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md) +- Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md) +- CI workflow steps: `.github/workflows/`; event, timeout, and required-status + matrix: [docs/testing/ci-gates.md](docs/testing/ci-gates.md) +- Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake + policy: [docs/testing/README.md](docs/testing/README.md) +- Tier/ILM transition debugging (xl.meta inspection, versionId tracing): + [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md) + ## Verification Select checks from the final task-owned diff. Scoped `AGENTS.md` files may add a diff --git a/CHANGELOG.md b/CHANGELOG.md index 00c2b33b8..02eaf4e62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged. ### Fixed +- **Replication status writebacks no longer invalidate scanner cycles**: every replication status writeback (`persist_replication_state_if_current` in `Update` mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path, advanced the dirty-usage generation and woke the scanner, so a site-replication resync or any steadily replicated bucket ended every cycle `superseded`, the authoritative usage baseline was never certified, and the console reported usage freshness as stale for as long as replication ran (#8375). Those writebacks are now status class: they do not touch the dirty-usage generation, the activity snapshot, the dirty-usage wake or the durable dirty journal, and only the process-local producer coverage is recorded. They need no rescan queue, because a deployment with an active replication rule already walks every bucket on every cycle and that walk refreshes `replication_stats`. Tier-expiration cleanup, which removes data, stays on the namespace path. No persisted format, protocol, admin API or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`. +- **A bucket written during a scanner walk no longer supersedes the whole cycle**: the post-walk cluster activity proof required the entire activity snapshot and the dirty-usage generation to be unchanged, so a single-node deployment with one bucket written every few seconds never reached a `Complete` cycle, `usage_snapshot_converged` never became true and the authoritative usage baseline was never certified (#8494). Dirty usage is now certified per bucket: a structurally complete walk that the guardrail did not veto is `Complete`, buckets whose dirty generation still matches the start snapshot are cleared, and buckets written during the walk stay dirty and are published as the pending set, so `usage_snapshot_converged` means "no bucket pending" and the next cycle is scoped to the pending buckets. Such a cycle writes the authoritative usage object; the observed object is now selected by an explicit observational mark that vetoed cycles carry. The guardrail itself vetoes only on the fields that change which namespace the walk described (peer set, process instance, protocol version, topology, maintenance generation, data movement, movement generation, publication blocked) and reports the field and host in the scanner log and in the new `usage_freshness` fields `guardrail_veto_total`, `last_guardrail_veto_unix_secs` and `last_guardrail_veto_reason`. In a distributed deployment a remote peer whose dirty usage moved during the walk still supersedes the cycle until scoped acknowledgements can name its buckets. `.usage.v2.json` gains the additive, default-when-absent fields `usage_snapshot_pending_buckets`, `usage_snapshot_pending_bucket_count` and `usage_snapshot_observational`; older readers ignore them and treat the snapshot as an authoritative but unconverged publication. Quota is unchanged: an uncertified bucket keeps its last usage value as the floor and live writes do not advance it. No protocol or configuration changes. Design: `docs/architecture/scanner-per-bucket-convergence.md`. - **SSE-C TLS requirement now covers copy-source keys**: with `RUSTFS_SSE_C_REQUIRE_TLS=true`, a `CopyObject` or `UploadPartCopy` that carried only the `x-amz-copy-source-server-side-encryption-customer-*` headers (an SSE-C source copied to a destination that is not SSE-C) was accepted on a plaintext transport and was not counted in `rustfs_ssec_plaintext_requests_total`, although those headers carry the source object's customer key. They now count as SSE-C headers: such requests are counted, and refused with `400 InvalidRequest` when the switch is on. Requests over TLS are unaffected. - **SSE-KMS encryption context could replace the location entry**: data keys are wrapped under an encryption-context entry, keyed by the bucket name, that ties them to the object's location. A client `x-amz-server-side-encryption-context` entry under the same key replaced it, and because the client context is stored with the object, the replacement also applied on every read. SSE-KMS writes now refuse such a context with `400 InvalidArgument`; other keys, including the names of other buckets, are unaffected. Objects stored with such an entry by an earlier release remain readable. See `docs/operations/kms-backend-security.md`. - **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117. @@ -85,6 +87,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary ### Changed +- **Scanner bucket walkers are no longer tied to the disk count**: a new scanner key `walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through `16`, default `1`) spawns that many bucket walkers per local disk of an erasure set, so a single-disk deployment can walk several buckets at once instead of one bucket, one object at a time. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk, and remote disks keep one walker. The walker count is what `max_concurrent_disk_scans` caps and the disk count stays the floor; the `disk_concurrency_budget` log line now reports `walkers_per_disk`, `walkers`, and the `reason` for the effective limit. The default keeps today's I/O profile because a local disk cannot yet report whether it is rotational; the design intends a higher default on flash once it can (`docs/architecture/scanner-per-bucket-convergence.md`). As groundwork for pacing, the raw-enumeration page writer inserts into its building page in place and commits pages without copying them, and a regression test pins the cost of recording 20,000 entries in one prefix so the quadratic bookkeeping removed by [#8114](https://github.com/rustfs/rustfs/pull/8114) (rustfs#8113) cannot return. No persisted format, protocol or admin API changes beyond the additional config key, which older releases preserve and ignore. - **Encryption and KMS work merged since `1.0.0-rc.5`** (entries were missing from this section): - **Persisted KMS configuration secrets** are sealed field-by-field with `RUSTFS_KMS_CONFIG_SECRET`. **When the variable is unset the secrets are persisted in cleartext and the server only warns** (`persisted KMS configuration carries cleartext secrets`); it never refuses the write. Set it, identically, on every node, and re-save the configuration to seal an existing one. - **New v2 ciphertext frame format** with per-frame index binding and final-frame authentication. Its write switch `RUSTFS_ENCRYPTION_FRAME_V2` is **off by default**: v2 frames are unreadable by nodes without v2 read support, and encrypted ciphertext travels verbatim through transition, decommission and SSE-C replication passthrough, so turn it on only after every node — and every RustFS warm/replication target that receives raw ciphertext — runs a release with v2 read support. Reading v2 objects needs no switch. diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 003a1dfcf..000000000 --- a/CLAUDE.md +++ /dev/null @@ -1,46 +0,0 @@ -# RustFS — CLAUDE.md - -S3-compatible object store in Rust, derived from MinIO. Erasure-coded, -multi-pool, supports ILM tiering/lifecycle. - -Agent rules (change style, verification, PR conventions, scoped per-crate -guidance) live in [AGENTS.md](AGENTS.md) — follow it. This file only adds -what Claude Code needs on top: commands and pointers. - -## Commands - -```bash -cargo build --release --bin rustfs # production binary -cargo check -p # fast type-check one crate -cargo test -p # test one crate -cargo fmt --all --check # for Rust changes; see AGENTS.md verification tiers -make pre-commit # fast gate: fmt + arch checks + quick-check (NO clippy/tests) -make pre-pr # optional full gate for broad cross-module changes -make build-docker BUILD_OS=ubuntu22.04 -``` - -> **Docker build note**: `buildx build` without `--load` keeps the image in -> the buildx cache only — `docker run` will use a stale local image. The -> Makefile already includes `--load`; if you suspect a stale binary, add -> `--no-cache` to the `buildx build` invocation inside -> `.config/make/build-docker.mak`. - -## Where to look (do not duplicate here) - -- Agent knowledge base index and doc-writing rules: [docs/architecture/README.md](docs/architecture/README.md) -- Crate membership: `Cargo.toml` `[workspace].members` -- Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md) -- Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md) -- CI workflow steps: `.github/workflows/`; event, timeout, and required-status - matrix: [docs/testing/ci-gates.md](docs/testing/ci-gates.md) -- Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake - policy: [docs/testing/README.md](docs/testing/README.md) -- Tier/ILM transition debugging (xl.meta inspection, versionId tracing): - [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md) - -## Domain conventions worth knowing up front - -Repo-wide domain invariants (dual internal metadata keys, defensive UUID -reads, unversioned tier buckets) live in [AGENTS.md](AGENTS.md) under -"Cross-Cutting Storage Invariants" — read them before touching metadata or -tiering code. diff --git a/Cargo.lock b/Cargo.lock index dc20abde7..c83a71c2e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -30,7 +30,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1973cfbc1a2daf9cf550e74e1f088c28e7f7d8c1e1418fb6c9dc5184b7e84c99" dependencies = [ "crypto-common 0.2.2", - "inout 0.2.2", + "inout 0.2.4", ] [[package]] @@ -328,9 +328,9 @@ dependencies = [ [[package]] name = "archive-trait" -version = "0.0.14" +version = "0.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6080ea14ccf9019d7ce572c319e581c20a805c9bdc6dbfb9b988da003cbd1a3" +checksum = "999bf88cc483664cab77700ca2fdfb0415a73412abe284ef5bc2319d00938e00" dependencies = [ "cap-std", "thiserror 2.0.21", @@ -371,9 +371,9 @@ checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" [[package]] name = "arrow" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c14b3d39f306bc28fd639d59f06e17a0f377d0021e1b7e9054e4d6fedc98774" +checksum = "4c25317ebddb8f2e6f006759336c95e5ded847a446f3524c27853bfe7d77a69b" dependencies = [ "arrow-arith", "arrow-array", @@ -392,9 +392,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce2961626677665b2195eb59242af4c7befe7b8737ca2050295389362380104e" +checksum = "2ab555c0ffe9cb817041c0cee80baa9663a22225d0a035693579771f2693cadc" dependencies = [ "arrow-array", "arrow-buffer", @@ -406,9 +406,9 @@ dependencies = [ [[package]] name = "arrow-array" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e5f6adeffdf587d7a31db5d2266189624b526730cd3627f9ff9fedae97ad584" +checksum = "2dae1d91737421e92c6dd7c7f9d0f2d25198fd6fc539738b628f4dad10adf533" dependencies = [ "ahash", "arrow-buffer", @@ -425,9 +425,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "097d193003ce7995d5d087089069ec2a6e0187faf5a6f8c9f38af2645d987182" +checksum = "1fe9cd0064f2d7cf8ecfaf77cd70e67a980bc71aea92b63102bd37a7d3379a09" dependencies = [ "bytes", "half", @@ -437,9 +437,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "635c9c635668ad26adf76cce8fb276c4be7cf06e63bd516de7da514f9680ee53" +checksum = "ee510bc252781d700400979a96bd1eadebeae23901132bbabd92118da1b376c5" dependencies = [ "arrow-array", "arrow-buffer", @@ -459,9 +459,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c2ebf8d631e79b02c16cf5ae860561272c26024ec88fce389a56aaddd558e86" +checksum = "ace83d5935421b950fea3a0a3f9972c0837c297ef98ade57127a5661b681112c" dependencies = [ "arrow-array", "arrow-cast", @@ -474,9 +474,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ba2f832eaeca24b8f26143dba750e42ee4ab51cf7d65e701ca9607cfda9f358" +checksum = "54f128c73bb7fe9f7137218f910edd5b436f390e0f040b1956f5028390787897" dependencies = [ "arrow-buffer", "arrow-schema", @@ -487,9 +487,9 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dcc41681ea80f521df14c36725b74d4c60702c47f0793af2be469c04527e2599" +checksum = "8e244f39b82fa7a823e8b5984a0bd988234f9c92c1df79378fb85090e6e4c821" dependencies = [ "arrow-array", "arrow-buffer", @@ -503,9 +503,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2f57d7a81969f24ccf80809587b76c09897e6f829d2d65a5976bfb3218851f1" +checksum = "727b21ebf8894dcd9504d6ab66bffdac338391e1f85cde594526a0210299ea40" dependencies = [ "arrow-array", "arrow-buffer", @@ -528,9 +528,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c900759f3bd8354fd4196bc4403eee846894dc2adf66b4225472006a0bf18c5" +checksum = "70d376b9b883239ae0f9963cdeeb59126ccaac08d60adf208a6ad3e2dd4048c3" dependencies = [ "arrow-array", "arrow-buffer", @@ -541,9 +541,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f4c6425032e28266e3fc4ff680805e57e670d6ea92473043f3e65b7ed6ac79f2" +checksum = "48096811f92b5c3a0c59d33cfecd7d7d2ed037d81e54a27a42d232e7700e6de2" dependencies = [ "arrow-array", "arrow-buffer", @@ -554,9 +554,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10fab8d4563491417ba801fab29d205104d20d4bdf37bda6cd1cf425cff598cd" +checksum = "37b6c8a6b12cf3869d299ce36ed8e1cd6fe9836264b68fb6c6538473a36124a1" dependencies = [ "serde_core", "serde_json", @@ -564,9 +564,9 @@ dependencies = [ [[package]] name = "arrow-select" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc58569193c2525915f3cc6310edba3792f1200f65d6e9ed330aa33e691493b8" +checksum = "e523e85fa9fb3181a391bf0edfa6a1ae0dc46df8cbf7f6b96034dcb1e2a7a04a" dependencies = [ "ahash", "arrow-array", @@ -578,9 +578,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e0813f3c35c1cfea65e14c20a953440f7783c088b7ad2d0db162ccdeefcec14" +checksum = "b931a905a0e4b8832d9f54fb369cf6e545f9dc03cb3d9611216a7bd2653df60a" dependencies = [ "arrow-array", "arrow-buffer", @@ -791,7 +791,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -916,9 +916,9 @@ dependencies = [ [[package]] name = "aws-lc-rs" -version = "1.18.1" +version = "1.18.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e" +checksum = "faac5829c2b74c28f830747e7818ccfb684261b5f48a1118b1e2a13d36dfab13" dependencies = [ "aws-lc-sys", "untrusted 0.7.1", @@ -927,9 +927,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.45.0" +version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27" +checksum = "f1622d8446a2d4b2ce0c7eefc73dd43a99779028d5ee5c2dd8073a658ba8a2bc" dependencies = [ "cc", "cmake", @@ -1020,7 +1020,7 @@ dependencies = [ "lru 0.18.5", "percent-encoding", "regex-lite", - "sha2 0.11.0", + "sha2 0.11.1", "tracing", "url", ] @@ -1120,7 +1120,7 @@ dependencies = [ "http 1.5.0", "p256 0.13.2", "percent-encoding", - "sha2 0.11.0", + "sha2 0.11.1", "subtle", "time", "tracing", @@ -1155,7 +1155,7 @@ dependencies = [ "md-5 0.11.0", "pin-project-lite", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "tracing", ] @@ -1518,7 +1518,7 @@ checksum = "144e573728da132683b9488acd528274c790e07fc06ff81ee29f9d8f8b1041e0" dependencies = [ "blowfish", "pbkdf2 0.13.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -1561,19 +1561,13 @@ dependencies = [ [[package]] name = "bit-set" -version = "0.8.0" +version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3" +checksum = "56d87354e4229f54a44f7bf2435906a4656dba36026ab6eaca629a2c436a691c" dependencies = [ - "bit-vec 0.8.0", + "bit-vec 0.10.1", ] -[[package]] -name = "bit-vec" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7" - [[package]] name = "bit-vec" version = "0.9.1" @@ -1583,6 +1577,16 @@ dependencies = [ "serde", ] +[[package]] +name = "bit-vec" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5727b15fa97d4f4fee0a3b7c3d550ed0269f54329207b86388de918604e31269" +dependencies = [ + "borsh", + "serde", +] + [[package]] name = "bitflags" version = "1.3.2" @@ -1703,7 +1707,31 @@ dependencies = [ "prettyplease 0.3.0", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", +] + +[[package]] +name = "borsh" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 3.0.9", ] [[package]] @@ -1950,9 +1978,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.6.0" +version = "1.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f74872d07caf508b30a21f6836e7d7016a2eaf7d9ff4f48deaa58cd8a0407630" +checksum = "6651c9ed80effdc7db0ff72512157f901af5e3549e341e24b1dd4887d836d838" dependencies = [ "find-msvc-tools", "jobserver", @@ -2072,7 +2100,7 @@ checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c" dependencies = [ "block-buffer 0.12.1", "crypto-common 0.2.2", - "inout 0.2.2", + "inout 0.2.4", "zeroize", ] @@ -2107,7 +2135,7 @@ dependencies = [ "heck 0.5.0", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -2151,9 +2179,9 @@ checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" [[package]] name = "combine" -version = "4.6.8" +version = "4.6.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfc320937d09e6de266b31b9afb480f197d7a861be86be7cb2ea7e5d1bfffc5e" +checksum = "4356eba6976435498d05a311ee9fc486da3bc81c200d3f0d7dd9703b41913f92" dependencies = [ "bytes", "futures-core", @@ -2199,8 +2227,8 @@ dependencies = [ "liblzma", "lz4", "memchr", - "zstd 0.14.0", - "zstd-safe 8.0.0", + "zstd 0.14.2", + "zstd-safe 8.1.0", ] [[package]] @@ -2304,9 +2332,12 @@ dependencies = [ [[package]] name = "constant_time_eq" -version = "0.4.2" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" +checksum = "9eb6feb175df65d0a3ccd3e73484e940d0ac664e8034f0519fd49a93db3071d9" +dependencies = [ + "rustversion", +] [[package]] name = "convert_case" @@ -2764,7 +2795,7 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -2808,7 +2839,7 @@ checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785" dependencies = [ "darling_core 0.24.1", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -3310,7 +3341,7 @@ checksum = "14d6df9763c8fb968b2417d797aef4c55ac2e4a895aaef38c72c16a287aa4a40" dependencies = [ "datafusion-doc", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -3577,18 +3608,18 @@ checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" [[package]] name = "defmt" -version = "0.3.100" +version = "0.3.101" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0963443817029b2024136fc4dd07a5107eb8f977eaf18fcd1fdeb11306b64ad" +checksum = "4deef977660e373c458df605a913d4ab121db0ccd18d1ad6976dff7d37cddb53" dependencies = [ - "defmt 1.1.1", + "defmt 1.2.0", ] [[package]] name = "defmt" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +checksum = "e4a0795ac740cb54d7b3775c9c17345f6aaecf884c01733b39d6017738e0e3e1" dependencies = [ "bitflags 1.3.2", "defmt-macros", @@ -3596,21 +3627,21 @@ dependencies = [ [[package]] name = "defmt-macros" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +checksum = "840bb056bd99c495c9188198a258ab787c79f306d864db3e6e8451442d6d8366" dependencies = [ "defmt-parser", "proc-macro2", "quote", - "syn 2.0.119", + "syn 3.0.9", ] [[package]] name = "defmt-parser" -version = "1.0.0" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +checksum = "71958f69fb2de856c5dd51ac9377e986ca1aa5919f931bf5395a807277f889a0" dependencies = [ "thiserror 2.0.21", ] @@ -3677,9 +3708,9 @@ dependencies = [ [[package]] name = "deranged" -version = "0.5.8" +version = "0.5.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" +checksum = "9e9de72ce2ad1f90dc62fa25f0f430ef85eb4b0d8fa0be4f30373bc40a21d28e" dependencies = [ "serde_core", ] @@ -3692,7 +3723,7 @@ checksum = "2e2b94854e8576378ccda7c8de8a66ed8b4e8acbd2c50ec3418ea6c8aaf4b567" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -3769,9 +3800,9 @@ dependencies = [ [[package]] name = "dial9" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b35f95885a2167602b457c77caf898670c02eb0aaba72bef76ac5b0faef86e14" +checksum = "734474ca4e814f893385c8fdea63688a9e6034ea19a5ec9846f8f515a69c9bed" dependencies = [ "bon", "dial9-core", @@ -3786,9 +3817,9 @@ dependencies = [ [[package]] name = "dial9-core" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82f826078c782919cf9a4fb0b2b6755dab24d6c908b5181469b23cf6eb8f6fff" +checksum = "4ff5074fab1f810a73d58336275eb163e795ab0ae923fcc8437ba58bf2994e33" dependencies = [ "ahash", "arc-swap", @@ -3820,9 +3851,9 @@ dependencies = [ [[package]] name = "dial9-metrique" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c59e35e7b38ce255d3dc6310b48c3ea54307c42a0589eda44ea91ba24eadf67" +checksum = "3dd61f8f3d44db27e7a5e970c216bf4c19db66ebb68b7f58db2549f010c3a02e" dependencies = [ "dial9-core", "dial9-trace-format", @@ -3834,9 +3865,9 @@ dependencies = [ [[package]] name = "dial9-perf-self-profile" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c429b209bc07612b2e60267e8bca80cd2b249f498e9a0b83f7caf514d5f5388" +checksum = "d320bef9be4ef9c20ca9800cbfc369c319304b5e11d0a7aa931f5482aa5c1bd8" dependencies = [ "blazesym", "bon", @@ -3853,9 +3884,9 @@ dependencies = [ [[package]] name = "dial9-tokio-telemetry" -version = "0.5.3" +version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aee46c0a97dd88ba151f9688c79de60c41cd7efc6855e6eb369af06326cba8f3" +checksum = "1a98e1e673f2dd9e223811832be7a4344ddfe48282a2e7efbf1289e47e4322ce" dependencies = [ "bon", "bytes", @@ -3949,7 +3980,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3970,7 +4001,7 @@ checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -3999,7 +4030,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" [[package]] name = "e2e_test" -version = "1.0.1" +version = "1.0.2" dependencies = [ "anyhow", "async-compression", @@ -4053,7 +4084,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "suppaftp", "tempfile", "time", @@ -4066,7 +4097,7 @@ dependencies = [ "uuid", "walkdir", "zip", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] @@ -4143,7 +4174,7 @@ dependencies = [ "ed25519 3.0.0", "rand_core 0.10.1", "serde", - "sha2 0.11.0", + "sha2 0.11.1", "signature 3.0.0", "subtle", "zeroize", @@ -4279,9 +4310,9 @@ dependencies = [ [[package]] name = "equivalent" -version = "1.0.2" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +checksum = "00d174d5400e5e8fd687ad1049e2f578285fa914201b1af7e8b112a4546bd826" [[package]] name = "erased-serde" @@ -4299,7 +4330,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4340,7 +4371,7 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6d8d873f78e1d3e1998ed2084da9147e3cd2bb34b67814cd49a3902b970eb124" dependencies = [ - "defmt 0.3.100", + "defmt 0.3.101", "serde", ] @@ -4581,7 +4612,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -4764,7 +4795,7 @@ dependencies = [ "rustls-pki-types", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "time", "tokio", @@ -4926,7 +4957,7 @@ dependencies = [ "serde", "serde_json", "serde_with", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "tokio", "tokio-stream", @@ -5676,9 +5707,9 @@ dependencies = [ [[package]] name = "inout" -version = "0.2.2" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4250ce6452e92010fdf7268ccc5d14faa80bb12fc741938534c58f16804e03c7" +checksum = "3ce983bb6fb399067d101fbfc9e74174e07e9f29a2bc679efefeb8c645819c3f" dependencies = [ "block-padding 0.4.2", "hybrid-array", @@ -5704,7 +5735,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "20fd6de4ccfcc187e38bc21cfa543cb5a302cb86a8b114eb7f0bf0dc9f8ac00f" dependencies = [ "io-lifetimes 3.0.1", - "windows-sys 0.59.0", + "windows-sys 0.60.2", ] [[package]] @@ -5821,7 +5852,7 @@ version = "0.2.38" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b2b005715dcbeb0089a3c0dab99f2ff1cc3b2525323552703d648585d342a383" dependencies = [ - "defmt 1.1.1", + "defmt 1.2.0", "jiff-core", "jiff-static", "jiff-tzdb-platform", @@ -5838,7 +5869,7 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5e52fe76043ccecc9005d2305ebaadf7d7fc0cc89ca6baa10a94d6bc68c7128c" dependencies = [ - "defmt 1.1.1", + "defmt 1.2.0", "log", ] @@ -6021,9 +6052,9 @@ checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" [[package]] name = "lapin" -version = "4.12.0" +version = "4.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b86b2a46f42a1d565c7ba327e03725708c1ff212baa4c271cec291e265ae542" +checksum = "3a585eb3060f00f09a1d231e3030afe59f2c2d63f794787ab1dd87d2f72df6a3" dependencies = [ "amq-protocol", "async-rs", @@ -6057,7 +6088,7 @@ dependencies = [ "proc-macro2", "quote", "regex", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -6349,11 +6380,11 @@ dependencies = [ [[package]] name = "lzma-rust2" -version = "0.16.5" +version = "0.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca93e534d1142d1d0dcca6d25fe302508a5dfb40b302802904577725ea0b695b" +checksum = "fde178a3caf126c440fa15628147772d8ee590a39477d711353fbe2e58a73a5b" dependencies = [ - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -6536,7 +6567,7 @@ dependencies = [ "metrique-core", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -6599,7 +6630,7 @@ dependencies = [ "metrique-core", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "synstructure 0.14.0", ] @@ -6963,7 +6994,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -7226,9 +7257,7 @@ version = "0.39.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2e5a6c098c7a3b6547378093f5cc30bc54fd361ce711e05293a5cc589562739b" dependencies = [ - "flate2", "memchr", - "ruzstd", ] [[package]] @@ -7238,9 +7267,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dd229a0361b9d0d4396176e02d65897f487eebeab7caa6d443855ee152ca0b9c" dependencies = [ "crc32fast", + "flate2", "hashbrown 0.17.1", "indexmap 2.14.2", "memchr", + "ruzstd", ] [[package]] @@ -7339,9 +7370,9 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" [[package]] name = "opentelemetry" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cdb0b1b267eb9db3331b434ed9ddab10d50e280a9adf9d13e5233e2002b61b5" +checksum = "8c1f1a6ac9c875a4941e4c7c085b61e8f4daa326de03b4701768c3491a6ad9ee" dependencies = [ "futures-core", "futures-sink", @@ -7353,9 +7384,9 @@ dependencies = [ [[package]] name = "opentelemetry-appender-tracing" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2b304e8d8f11326ba366a99ca5c7aaf7cd397b67268a332437175aeb42a40db" +checksum = "1f01f1c53c46ce80e5fd8cb7a646f13766fd8f315a8a5b75b9bad59a08d2c5c7" dependencies = [ "opentelemetry", "tracing", @@ -7366,9 +7397,9 @@ dependencies = [ [[package]] name = "opentelemetry-http" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee2c3625b8aa04209f7e01e513bc8044da9687fa38a9eacf59628ca5f3b87300" +checksum = "302249bd8ee804789faaa7b5b2cbf79bc61cb864714079c644e3828af5b7e811" dependencies = [ "async-trait", "bytes", @@ -7379,9 +7410,9 @@ dependencies = [ [[package]] name = "opentelemetry-otlp" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "699a67345e21962a231955b9059a157e428b219fa5df8efe11f08346fb23a344" +checksum = "0756af24841e367128ef31c831999937b8b5c5197d47914590d8f4e2e88c2e1a" dependencies = [ "flate2", "http 1.5.0", @@ -7397,9 +7428,9 @@ dependencies = [ [[package]] name = "opentelemetry-proto" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d" +checksum = "d576f5dddc3299341bcd85a04be17b2564844c0c9563214f02d85f2524d1afae" dependencies = [ "opentelemetry", "opentelemetry_sdk", @@ -7408,15 +7439,15 @@ dependencies = [ [[package]] name = "opentelemetry-semantic-conventions" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d288b1e1bc3590052fbb8f0f635ef248dd9022cf1fbaeb6582b7ad6081a17b87" +checksum = "7a3cf639fefd3e852869114f9925b578da6dd8e99281a2089bb0e773ead46962" [[package]] name = "opentelemetry-stdout" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f71803f481d3f5ccbe477edb42a78f473613712b2327835911a9ea0ab069824" +checksum = "2afbc6661934787ce79bef1e35d0ffe9d62e945f07f664dc1283ce83292b908a" dependencies = [ "chrono", "opentelemetry", @@ -7425,9 +7456,9 @@ dependencies = [ [[package]] name = "opentelemetry_sdk" -version = "0.33.0" +version = "0.33.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cb39533d9d1c912123efd7d41d7e0c29d16917b60ce15b4c8d87cb1af7f67520" +checksum = "372a912db2974d1990207006fd0e3aa0ef3fd4ad3be43c65cfaadf2206866061" dependencies = [ "futures-channel", "futures-executor", @@ -7546,7 +7577,7 @@ dependencies = [ "elliptic-curve 0.14.1", "primefield", "primeorder 0.14.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -7572,7 +7603,7 @@ dependencies = [ "fiat-crypto 0.3.0", "primefield", "primeorder 0.14.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -7586,7 +7617,7 @@ dependencies = [ "elliptic-curve 0.14.1", "primefield", "primeorder 0.14.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -7612,7 +7643,7 @@ dependencies = [ "futures", "log", "rand 0.10.3", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "tokio", "windows", @@ -7650,9 +7681,9 @@ dependencies = [ [[package]] name = "parquet" -version = "59.3.0" +version = "59.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff322f54b1a0f9288e614ed1f2d329b380af5476420db19f46ffb865e1163d73" +checksum = "8b3d2ef0fb136e9341570f219ecb168902e76079c6d81ad555f0a4840b1e08a3" dependencies = [ "ahash", "arrow-array", @@ -8001,7 +8032,7 @@ dependencies = [ "pbkdf2 0.13.0", "rand_core 0.10.1", "scrypt 0.12.0", - "sha2 0.11.0", + "sha2 0.11.1", "spki 0.8.1", ] @@ -8119,7 +8150,7 @@ dependencies = [ "md-5 0.11.0", "memchr", "rand 0.10.3", - "sha2 0.11.0", + "sha2 0.11.1", "stringprep", ] @@ -8204,7 +8235,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2bfe0f4c752e450fc2faf62654f1c134747922825d5b04ca717b8874f41a40c0" dependencies = [ "proc-macro2", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -8301,16 +8332,17 @@ dependencies = [ [[package]] name = "proptest" -version = "1.11.0" +version = "1.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744" +checksum = "b8530004ccb15eae51c7e40009fbe317f341f804db54dc033eec1c50be28cfa0" dependencies = [ "bit-set", - "bit-vec 0.8.0", + "bit-vec 0.10.1", "bitflags 2.13.2", + "chacha20", + "core_detect", "num-traits", - "rand 0.9.5", - "rand_chacha 0.9.0", + "rand 0.10.3", "rand_xorshift", "regex-syntax", "rusty-fork", @@ -8345,7 +8377,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" dependencies = [ "heck 0.5.0", - "itertools 0.10.5", + "itertools 0.14.0", "log", "multimap", "once_cell", @@ -8365,7 +8397,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" dependencies = [ "heck 0.5.0", - "itertools 0.10.5", + "itertools 0.14.0", "log", "multimap", "petgraph 0.8.3", @@ -8386,7 +8418,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" dependencies = [ "anyhow", - "itertools 0.10.5", + "itertools 0.14.0", "proc-macro2", "quote", "syn 2.0.119", @@ -8399,7 +8431,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", - "itertools 0.10.5", + "itertools 0.14.0", "proc-macro2", "quote", "syn 2.0.119", @@ -8497,9 +8529,9 @@ dependencies = [ [[package]] name = "pyroscope" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59b1989a61e856eba4ef903eadad8313f8376e88e7c9cc5d5db8217d832a64d9" +checksum = "357782a4af9b6b43bd3a9caa64dd802967498b1fce1d4e0bda552267d62de7cf" dependencies = [ "aligned-vec", "backtrace", @@ -8510,10 +8542,11 @@ dependencies = [ "log", "memmap2", "nix 0.31.3", - "object 0.39.1", + "object 0.40.0", "once_cell", "prost 0.14.4", "reqwest", + "ruzstd", "serde_json", "smallvec", "spin 0.12.3", @@ -8726,7 +8759,7 @@ dependencies = [ "once_cell", "socket2", "tracing", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -8848,11 +8881,11 @@ dependencies = [ [[package]] name = "rand_xorshift" -version = "0.4.0" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a" +checksum = "60aa6af80be32871323012e02e6e65f8a7cc7890931ae421d217ad8fe0df2ccf" dependencies = [ - "rand_core 0.9.5", + "rand_core 0.10.1", ] [[package]] @@ -9039,7 +9072,7 @@ checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -9257,7 +9290,7 @@ dependencies = [ "pkcs1 0.8.0-rc.4", "pkcs8 0.11.0", "rand_core 0.10.1", - "sha2 0.11.0", + "sha2 0.11.1", "signature 3.0.0", "spki 0.8.1", "zeroize", @@ -9347,7 +9380,7 @@ dependencies = [ "ghash", "hex-literal", "hmac 0.13.0", - "inout 0.2.2", + "inout 0.2.4", "keccak", "log", "md5", @@ -9372,7 +9405,7 @@ dependencies = [ "scrypt 0.12.0", "sec1 0.8.1", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "sha3 0.12.0", "signature 3.0.0", "spki 0.8.1", @@ -9462,7 +9495,7 @@ version = "8.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0d84e8ba78bd384263e5922f084cbe1b081c3b7e69add59c8fb097b879ba968a" dependencies = [ - "sha2 0.11.0", + "sha2 0.11.1", "walkdir", ] @@ -9489,7 +9522,7 @@ dependencies = [ [[package]] name = "rustfs" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "anyhow", @@ -9615,7 +9648,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "shadow-rs", "snap", "socket2", @@ -9643,12 +9676,12 @@ dependencies = [ "xxhash-rust", "zeroize", "zip", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-audit" -version = "1.0.1" +version = "1.0.2" dependencies = [ "const-str", "futures", @@ -9670,7 +9703,7 @@ dependencies = [ [[package]] name = "rustfs-checksums" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "bytes", @@ -9680,13 +9713,13 @@ dependencies = [ "md-5 0.11.0", "pretty_assertions", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "xxhash-rust", ] [[package]] name = "rustfs-common" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "hotpath", @@ -9701,7 +9734,7 @@ dependencies = [ [[package]] name = "rustfs-concurrency" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "insta", @@ -9714,7 +9747,7 @@ dependencies = [ [[package]] name = "rustfs-config" -version = "1.0.1" +version = "1.0.2" dependencies = [ "const-str", "hotpath", @@ -9724,19 +9757,19 @@ dependencies = [ [[package]] name = "rustfs-connect-symbol-catalog" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hex-simd", "object 0.40.0", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "symbolic-demangle", ] [[package]] name = "rustfs-credentials" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "hmac 0.13.0", @@ -9744,13 +9777,13 @@ dependencies = [ "rand 0.10.3", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "time", ] [[package]] name = "rustfs-crypto" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "argon2", @@ -9763,7 +9796,7 @@ dependencies = [ "rsa 0.10.0-rc.18", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "test-case", "thiserror 2.0.21", "time", @@ -9771,7 +9804,7 @@ dependencies = [ [[package]] name = "rustfs-data-usage" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "rmp-serde", @@ -9781,7 +9814,7 @@ dependencies = [ [[package]] name = "rustfs-ecstore" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-channel", @@ -9804,6 +9837,7 @@ dependencies = [ "futures-util", "google-cloud-auth", "google-cloud-storage", + "h2", "hex-simd", "hmac 0.13.0", "hostname", @@ -9859,6 +9893,7 @@ dependencies = [ "rustfs-rio", "rustfs-rio-v2", "rustfs-s3-client", + "rustfs-s3-ops", "rustfs-s3-types", "rustfs-scanner-metrics", "rustfs-storage-api", @@ -9871,7 +9906,7 @@ dependencies = [ "serde", "serde_json", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "shadow-rs", "smallvec", "temp-env", @@ -9913,7 +9948,7 @@ dependencies = [ [[package]] name = "rustfs-extension-schema" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "serde", @@ -9923,7 +9958,7 @@ dependencies = [ [[package]] name = "rustfs-filemeta" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "base64-simd", @@ -9940,7 +9975,7 @@ dependencies = [ "rustfs-utils", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tempfile", "thiserror 2.0.21", "time", @@ -9952,7 +9987,7 @@ dependencies = [ [[package]] name = "rustfs-heal" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "base64-simd", @@ -9966,7 +10001,6 @@ dependencies = [ "rustfs-common", "rustfs-concurrency", "rustfs-config", - "rustfs-data-usage", "rustfs-ecstore", "rustfs-filemeta", "rustfs-heal-contracts", @@ -9980,7 +10014,7 @@ dependencies = [ "serde", "serde_json", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "temp-env", "tempfile", "thiserror 2.0.21", @@ -9994,7 +10028,7 @@ dependencies = [ [[package]] name = "rustfs-heal-contracts" -version = "1.0.1" +version = "1.0.2" dependencies = [ "serde", "serde_json", @@ -10004,7 +10038,7 @@ dependencies = [ [[package]] name = "rustfs-iam" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-trait", @@ -10039,7 +10073,7 @@ dependencies = [ "serde_with", "serial_test", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "subtle", "temp-env", "tempfile", @@ -10054,7 +10088,7 @@ dependencies = [ [[package]] name = "rustfs-io-core" -version = "1.0.1" +version = "1.0.2" dependencies = [ "bytes", "hotpath", @@ -10066,7 +10100,7 @@ dependencies = [ [[package]] name = "rustfs-io-metrics" -version = "1.0.1" +version = "1.0.2" dependencies = [ "criterion", "hotpath", @@ -10096,7 +10130,7 @@ dependencies = [ "pbkdf2 0.13.0", "rand 0.10.3", "rustls", - "sha2 0.11.0", + "sha2 0.11.1", "socket2", "thiserror 2.0.21", "tracing", @@ -10120,7 +10154,7 @@ dependencies = [ "rand 0.10.3", "rustfs-kafka", "rustls", - "sha2 0.11.0", + "sha2 0.11.1", "tokio", "tokio-rustls", "tracing", @@ -10130,7 +10164,7 @@ dependencies = [ [[package]] name = "rustfs-keystone" -version = "1.0.1" +version = "1.0.2" dependencies = [ "bytes", "futures", @@ -10157,7 +10191,7 @@ dependencies = [ [[package]] name = "rustfs-kms" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "anyhow", @@ -10190,7 +10224,7 @@ dependencies = [ "serde", "serde_ignored", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "subtle", "temp-env", "tempfile", @@ -10207,14 +10241,14 @@ dependencies = [ [[package]] name = "rustfs-license" -version = "1.0.1" +version = "1.0.2" dependencies = [ "thiserror 2.0.21", ] [[package]] name = "rustfs-lifecycle" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "hotpath", @@ -10236,7 +10270,7 @@ dependencies = [ [[package]] name = "rustfs-lock" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "compact_str", @@ -10259,7 +10293,7 @@ dependencies = [ [[package]] name = "rustfs-log-analyzer" -version = "1.0.1" +version = "1.0.2" dependencies = [ "chrono", "flate2", @@ -10267,18 +10301,18 @@ dependencies = [ "regex", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tar", "tempfile", "thiserror 2.0.21", "walkdir", "zip", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-madmin" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "http 1.5.0", @@ -10316,7 +10350,7 @@ dependencies = [ [[package]] name = "rustfs-notify" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-trait", @@ -10351,7 +10385,7 @@ dependencies = [ [[package]] name = "rustfs-object-capacity" -version = "1.0.1" +version = "1.0.2" dependencies = [ "criterion", "futures", @@ -10370,7 +10404,7 @@ dependencies = [ [[package]] name = "rustfs-object-data-cache" -version = "1.0.1" +version = "1.0.2" dependencies = [ "bytes", "criterion", @@ -10387,7 +10421,7 @@ dependencies = [ [[package]] name = "rustfs-obs" -version = "1.0.1" +version = "1.0.2" dependencies = [ "chrono", "crossbeam-channel", @@ -10445,12 +10479,12 @@ dependencies = [ "tracing-opentelemetry", "tracing-subscriber", "url", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-policy" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "base64-simd", @@ -10481,7 +10515,7 @@ dependencies = [ [[package]] name = "rustfs-protocols" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-compression", "async-trait", @@ -10524,7 +10558,7 @@ dependencies = [ "serde", "serde_json", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "socket2", "subtle", "tempfile", @@ -10543,7 +10577,7 @@ dependencies = [ [[package]] name = "rustfs-protos" -version = "1.0.1" +version = "1.0.2" dependencies = [ "flatbuffers", "hotpath", @@ -10568,7 +10602,7 @@ dependencies = [ [[package]] name = "rustfs-replication" -version = "1.0.1" +version = "1.0.2" dependencies = [ "byteorder", "bytes", @@ -10586,7 +10620,7 @@ dependencies = [ [[package]] name = "rustfs-rio" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "arc-swap", @@ -10618,7 +10652,7 @@ dependencies = [ "serde", "serde_json", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "temp-env", "thiserror 2.0.21", "tokio", @@ -10630,7 +10664,7 @@ dependencies = [ [[package]] name = "rustfs-rio-v2" -version = "1.0.1" +version = "1.0.2" dependencies = [ "aes-gcm", "bytes", @@ -10646,14 +10680,14 @@ dependencies = [ "rustfs-utils", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tokio", "walkdir", ] [[package]] name = "rustfs-s3-client" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "bytes", @@ -10683,7 +10717,7 @@ dependencies = [ "s3s", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "thiserror 2.0.21", "time", "tokio", @@ -10697,7 +10731,7 @@ dependencies = [ [[package]] name = "rustfs-s3-ops" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "rustfs-s3-types", @@ -10705,7 +10739,7 @@ dependencies = [ [[package]] name = "rustfs-s3-types" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "serde", @@ -10714,7 +10748,7 @@ dependencies = [ [[package]] name = "rustfs-s3select-api" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-compression", @@ -10750,7 +10784,7 @@ dependencies = [ [[package]] name = "rustfs-s3select-query" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "bytes", @@ -10770,7 +10804,7 @@ dependencies = [ [[package]] name = "rustfs-scanner" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "bytes", @@ -10803,7 +10837,7 @@ dependencies = [ "serde", "serde_json", "serial_test", - "sha2 0.11.0", + "sha2 0.11.1", "temp-env", "tempfile", "thiserror 2.0.21", @@ -10817,7 +10851,7 @@ dependencies = [ [[package]] name = "rustfs-scanner-metrics" -version = "1.0.1" +version = "1.0.2" dependencies = [ "chrono", "jiff", @@ -10832,7 +10866,7 @@ dependencies = [ [[package]] name = "rustfs-security-governance" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "thiserror 2.0.21", @@ -10840,7 +10874,7 @@ dependencies = [ [[package]] name = "rustfs-signer" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "bytes", @@ -10858,7 +10892,7 @@ dependencies = [ [[package]] name = "rustfs-storage-api" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "hotpath", @@ -10873,7 +10907,7 @@ dependencies = [ [[package]] name = "rustfs-targets" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "async-nats", @@ -10909,7 +10943,7 @@ dependencies = [ "s3s", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "snap", "sysinfo", "tempfile", @@ -10927,7 +10961,7 @@ dependencies = [ [[package]] name = "rustfs-test-utils" -version = "1.0.1" +version = "1.0.2" dependencies = [ "hotpath", "rustfs-data-usage", @@ -10943,7 +10977,7 @@ dependencies = [ [[package]] name = "rustfs-tls-runtime" -version = "1.0.1" +version = "1.0.2" dependencies = [ "arc-swap", "hotpath", @@ -10955,7 +10989,7 @@ dependencies = [ "rustls-pki-types", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tempfile", "thiserror 2.0.21", "tokio", @@ -10982,7 +11016,7 @@ dependencies = [ [[package]] name = "rustfs-trusted-proxies" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-trait", "axum", @@ -11019,7 +11053,7 @@ dependencies = [ [[package]] name = "rustfs-utils" -version = "1.0.1" +version = "1.0.2" dependencies = [ "base64-simd", "blake2", @@ -11046,7 +11080,7 @@ dependencies = [ "rustix", "serde", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "siphasher", "snap", "temp-env", @@ -11056,12 +11090,12 @@ dependencies = [ "transform-stream", "url", "windows", - "zstd 0.14.0", + "zstd 0.14.2", ] [[package]] name = "rustfs-zip" -version = "1.0.1" +version = "1.0.2" dependencies = [ "async-compression", "futures", @@ -11071,7 +11105,7 @@ dependencies = [ "rustfs-tokio-tar", "serde", "serde_json", - "sha2 0.11.0", + "sha2 0.11.1", "tar-codec", "tar-framing", "thiserror 2.0.21", @@ -11131,7 +11165,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -11146,9 +11180,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.45" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "aws-lc-rs", "log", @@ -11176,9 +11210,9 @@ dependencies = [ [[package]] name = "rustls-native-certs" -version = "0.8.4" +version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" +checksum = "4a073f5dc7469f984c52ad2752b63b0807745133b6de880b7b64c1ac4c48aec4" dependencies = [ "openssl-probe", "rustls-pki-types", @@ -11214,7 +11248,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -11255,9 +11289,9 @@ dependencies = [ [[package]] name = "ruzstd" -version = "0.8.3" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7c1c839d570d835527c9a5e4db7cb2198683a988cb9d7293fc8674e6bd58fc8" +checksum = "69e4ce4a039f42bf25920b95667e4d789e98749ab11bbb8d2c2936f788421bb5" dependencies = [ "twox-hash", ] @@ -11308,7 +11342,7 @@ dependencies = [ "serde_json", "serde_urlencoded", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "smallvec", "std-next", "subtle", @@ -11372,7 +11406,7 @@ dependencies = [ "jiff", "nom 8.0.0", "serde", - "sha2 0.11.0", + "sha2 0.11.1", "smallvec", "std-next", "subtle", @@ -11437,9 +11471,9 @@ dependencies = [ [[package]] name = "schemars" -version = "1.2.2" +version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "687274d293b6cdc6e73e0fee520bf2049650090d7164f87672d212a3c530cf4a" +checksum = "ab508826f74a77ca9d5aba6ff19b522583ee3eaf28a19384ff3d0e5835fadf6e" dependencies = [ "dyn-clone", "ref-cast", @@ -11473,7 +11507,7 @@ dependencies = [ "cfg-if", "pbkdf2 0.13.0", "salsa20 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", ] [[package]] @@ -11596,7 +11630,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11611,10 +11645,11 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.151" +version = "1.0.154" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6" dependencies = [ + "foldhash 0.2.0", "indexmap 2.14.2", "itoa", "memchr", @@ -11660,7 +11695,7 @@ checksum = "8d3b1629de253c70a0508c3899572da79ca359fdab27c7920ff00406df418906" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11689,7 +11724,7 @@ dependencies = [ "indexmap 2.14.2", "jiff", "schemars 0.9.0", - "schemars 1.2.2", + "schemars 1.2.3", "serde_core", "serde_json", "serde_with_macros", @@ -11705,7 +11740,7 @@ dependencies = [ "darling 0.24.1", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11740,7 +11775,7 @@ checksum = "a22144e767da4ddd8416dbf383700542ffd8a5dc493dfecedfe1fe3ad03c98ae" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -11784,9 +11819,9 @@ dependencies = [ [[package]] name = "sha2" -version = "0.11.0" +version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +checksum = "47d7069beb7d6ac7b9acd1039986e73443f24234f41074da099d6f994ac9ad19" dependencies = [ "cfg-if", "cpufeatures 0.3.1", @@ -11987,9 +12022,9 @@ dependencies = [ [[package]] name = "smallvec" -version = "1.16.2" +version = "1.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9395f0f0eee849a9b707b2f06bb92a6a422090e2123bb2ef8e87a0e61892a8e" +checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b" dependencies = [ "serde", ] @@ -12156,7 +12191,7 @@ dependencies = [ "sec1 0.8.1", "serde", "sha1 0.11.0", - "sha2 0.11.0", + "sha2 0.11.1", "signature 3.0.0", "ssh-cipher", "ssh-encoding", @@ -12259,9 +12294,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "suppaftp" -version = "12.1.1" +version = "12.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b161d71217da0cc91bd09c8f98ac5ad2a5e5b2735c6dfe1c6cb08b4471612f35" +checksum = "2b58fcf921745cca6f0127c9f822069947da8c6dbeff52e9004750bcc448171f" dependencies = [ "async-trait", "chrono", @@ -12327,9 +12362,9 @@ dependencies = [ [[package]] name = "syn" -version = "3.0.6" +version = "3.0.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee" +checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b" dependencies = [ "proc-macro2", "quote", @@ -12376,7 +12411,7 @@ checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12435,9 +12470,9 @@ dependencies = [ [[package]] name = "tar-codec" -version = "0.0.14" +version = "0.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ea42eb144d30fcbf32c26dfea8959175bb8335bfb7b18d20c6ed32edecf0551" +checksum = "b9dad615e50d181afea4bafea23dacb8e33ed52dde06503813570634b932d467" dependencies = [ "archive-trait", "tar-framing", @@ -12447,9 +12482,9 @@ dependencies = [ [[package]] name = "tar-framing" -version = "0.0.14" +version = "0.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "783223a7a6590be4227cb821e7ce80372575511f67c824921aba0752d8ad5573" +checksum = "563e8181b47c9cdcce569eae3ad431eec71543d6269dd60c7bd2350f70bcd233" dependencies = [ "thiserror 2.0.21", "tokio", @@ -12485,10 +12520,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.3.4", + "getrandom 0.4.3", "once_cell", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -12518,7 +12553,7 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12529,7 +12564,7 @@ checksum = "f4ce35ece947bccac166e1ded639133827a953f69a48ba134d17f91108333082" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "test-case-core", ] @@ -12570,7 +12605,7 @@ checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12685,9 +12720,9 @@ dependencies = [ [[package]] name = "tokio" -version = "1.53.2" +version = "1.53.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e95f91fcc7a621e8b030f6aa23c71fe9838ae2fb4d8118b75602a328f5144044" +checksum = "ce3335fa71841cda333a58d7615b03901380ecf09d59b3296d21f8bbac0dde4e" dependencies = [ "bytes", "libc", @@ -12707,7 +12742,7 @@ checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] @@ -12743,7 +12778,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c2ad44aa0ae96db89c4742212ed41645b2f597311ff6e1945542a4d9fadc2fb" dependencies = [ "rustls", - "sha2 0.11.0", + "sha2 0.11.1", "tokio", "tokio-postgres", "tokio-rustls", @@ -12785,9 +12820,9 @@ dependencies = [ [[package]] name = "tokio-util" -version = "0.7.19" +version = "0.7.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52" +checksum = "e464cf451ba96ebfc6f9b6542f17ee8b8956e33f1e40d9690624e59d7a7f8a4b" dependencies = [ "bytes", "futures-core", @@ -12823,18 +12858,18 @@ dependencies = [ [[package]] name = "toml_datetime" -version = "1.1.1+spec-1.1.0" +version = "1.1.2+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053" dependencies = [ "serde_core", ] [[package]] name = "toml_edit" -version = "0.25.15+spec-1.1.0" +version = "0.25.17+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1340ea94a5856333492c9064b02c778b191dd2c853778d9609debdcdfea3a614" +checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c" dependencies = [ "indexmap 2.14.2", "toml_datetime", @@ -12844,9 +12879,9 @@ dependencies = [ [[package]] name = "toml_parser" -version = "1.1.3+spec-1.1.0" +version = "1.1.5+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d38ac1cf9b95face32296c0a3ede1fdc270627c9d9c02a7274dd6d960dc4d56" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" dependencies = [ "winnow", ] @@ -13223,9 +13258,9 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" [[package]] name = "unicode-ident" -version = "1.0.26" +version = "1.0.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954" +checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563" [[package]] name = "unicode-normalization" @@ -13315,9 +13350,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.27.0" +version = "1.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97277d36b9c3ace13e58fa6e753f8b0bbbf302a18dd193240d70f9e29681059a" +checksum = "7cc1186384beb7dd8eedea376413fd654937285ea6c9cfbb928dc3043ea4b606" dependencies = [ "getrandom 0.4.3", "js-sys", @@ -13468,7 +13503,7 @@ dependencies = [ "bumpalo", "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "wasm-bindgen-shared", ] @@ -13582,7 +13617,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -14043,24 +14078,24 @@ checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "synstructure 0.14.0", ] [[package]] name = "zerocopy" -version = "0.8.60" +version = "0.8.63" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6400fc4bb7426f6faac3fb07566122bb18622e3022bd2c36426a92ed489b1dbe" +checksum = "e5fe1f8f1b06191a00962174c61aa5005e0bb391a6d80d07e24d115c01a92ed8" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.60" +version = "0.8.63" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "99c27cc7525bad4df59a45e073c45b2c2aa820efb588234cc23a547ec69a34db" +checksum = "863ad3ac83293fb4d740aedbfdc9240dd8d1a50c1099acd76ce80ce7c7230c7f" dependencies = [ "proc-macro2", "quote", @@ -14084,7 +14119,7 @@ checksum = "f75b4683f6c7f45248d4d64056a24298c6281e0993356d7d1b4a1a962ef10d4a" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", "synstructure 0.14.0", ] @@ -14138,14 +14173,14 @@ checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da" dependencies = [ "proc-macro2", "quote", - "syn 3.0.6", + "syn 3.0.9", ] [[package]] name = "zip" -version = "8.6.0" +version = "9.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b" +checksum = "c2d98561b612a84ae525331a7a094429ef8a5772514ed507ff6db5f54badc971" dependencies = [ "aes 0.9.3", "bzip2", @@ -14158,6 +14193,7 @@ dependencies = [ "indexmap 2.14.2", "lzma-rust2", "memchr", + "nix 0.31.3", "pbkdf2 0.13.0", "ppmd-rust", "sha1 0.11.0", @@ -14165,7 +14201,7 @@ dependencies = [ "typed-path", "zeroize", "zopfli", - "zstd 0.13.3", + "zstd 0.14.2", ] [[package]] @@ -14182,9 +14218,9 @@ checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" [[package]] name = "zopfli" -version = "0.8.3" +version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" +checksum = "aaf7fc5d30c28483d93805c4a5e12b05bbb52407fa67c5f8bd552374cd01fb11" dependencies = [ "bumpalo", "crc32fast", @@ -14203,11 +14239,11 @@ dependencies = [ [[package]] name = "zstd" -version = "0.14.0" +version = "0.14.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf06bd8162af0734b344780deb55b42a2429ae430870d13fcc12f238e880fe6e" +checksum = "057cfd910cfac363a0ada849592624b4c9ff2e10bef504c3433810d78ed96f93" dependencies = [ - "zstd-safe 8.0.0", + "zstd-safe 8.1.0", ] [[package]] @@ -14221,18 +14257,18 @@ dependencies = [ [[package]] name = "zstd-safe" -version = "8.0.0" +version = "8.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae42c0555055784c70058d19ba8e275528e8a99a706684868ace5da4e716a4ab" +checksum = "bdd44c6a7284e91f3717755b24315a302edd9153a01f753c3cba3d765e8eafac" dependencies = [ "zstd-sys", ] [[package]] name = "zstd-sys" -version = "2.1.0+zstd.1.5.7" +version = "2.1.1+zstd.1.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ef0a8027ec3ee71300ab3bcbcd0393f434aa72b91ca6d635a39941deae8eea0" +checksum = "aeec9eaf2dffbbd09201e23bd0ffcbaa33bb8e9266a10734fd7ed90a85eca078" dependencies = [ "cc", "pkg-config", diff --git a/Cargo.toml b/Cargo.toml index 6a5791efb..a87735faa 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -74,7 +74,7 @@ edition = "2024" license = "Apache-2.0" repository = "https://github.com/rustfs/rustfs" rust-version = "1.99.0" -version = "1.0.1" +version = "1.0.2" homepage = "https://rustfs.com" description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. " keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"] @@ -91,56 +91,56 @@ redundant_clone = "warn" [workspace.dependencies] # RustFS Internal Crates -rustfs = { path = "./rustfs", version = "1.0.1" } -rustfs-heal = { path = "crates/heal", version = "1.0.1" } -rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" } -rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" } -rustfs-audit = { path = "crates/audit", version = "1.0.1" } -rustfs-checksums = { path = "crates/checksums", version = "1.0.1" } -rustfs-common = { path = "crates/common", version = "1.0.1" } -rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" } -rustfs-config = { path = "./crates/config", version = "1.0.1" } -rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" } -rustfs-credentials = { path = "crates/credentials", version = "1.0.1" } -rustfs-crypto = { path = "crates/crypto", version = "1.0.1" } -rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" } -rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" } -rustfs-iam = { path = "crates/iam", version = "1.0.1" } -rustfs-keystone = { path = "crates/keystone", version = "1.0.1" } -rustfs-license = { path = "crates/license", version = "1.0.1" } -rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" } -rustfs-kms = { path = "crates/kms", version = "1.0.1" } -rustfs-lock = { path = "crates/lock", version = "1.0.1" } -rustfs-madmin = { path = "crates/madmin", version = "1.0.1" } -rustfs-notify = { path = "crates/notify", version = "1.0.1" } -rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" } -rustfs-io-core = { path = "crates/io-core", version = "1.0.1" } -rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" } -rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false } -rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" } -rustfs-obs = { path = "crates/obs", version = "1.0.1" } -rustfs-policy = { path = "crates/policy", version = "1.0.1" } -rustfs-protos = { path = "crates/protos", version = "1.0.1" } -rustfs-protocols = { path = "crates/protocols", version = "1.0.1" } -rustfs-replication = { path = "crates/replication", version = "1.0.1" } -rustfs-rio = { path = "crates/rio", version = "1.0.1" } -rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" } -rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" } -rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" } -rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" } -rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" } -rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" } -rustfs-scanner = { path = "crates/scanner", version = "1.0.1" } -rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" } -rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" } -rustfs-signer = { path = "crates/signer", version = "1.0.1" } -rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" } -rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" } -rustfs-targets = { path = "crates/targets", version = "1.0.1" } -rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" } -rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" } -rustfs-utils = { path = "crates/utils", version = "1.0.1" } -rustfs-zip = { path = "./crates/zip", version = "1.0.1" } +rustfs = { path = "./rustfs", version = "1.0.2" } +rustfs-heal = { path = "crates/heal", version = "1.0.2" } +rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" } +rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" } +rustfs-audit = { path = "crates/audit", version = "1.0.2" } +rustfs-checksums = { path = "crates/checksums", version = "1.0.2" } +rustfs-common = { path = "crates/common", version = "1.0.2" } +rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" } +rustfs-config = { path = "./crates/config", version = "1.0.2" } +rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" } +rustfs-credentials = { path = "crates/credentials", version = "1.0.2" } +rustfs-crypto = { path = "crates/crypto", version = "1.0.2" } +rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" } +rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" } +rustfs-iam = { path = "crates/iam", version = "1.0.2" } +rustfs-keystone = { path = "crates/keystone", version = "1.0.2" } +rustfs-license = { path = "crates/license", version = "1.0.2" } +rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" } +rustfs-kms = { path = "crates/kms", version = "1.0.2" } +rustfs-lock = { path = "crates/lock", version = "1.0.2" } +rustfs-madmin = { path = "crates/madmin", version = "1.0.2" } +rustfs-notify = { path = "crates/notify", version = "1.0.2" } +rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" } +rustfs-io-core = { path = "crates/io-core", version = "1.0.2" } +rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" } +rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false } +rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" } +rustfs-obs = { path = "crates/obs", version = "1.0.2" } +rustfs-policy = { path = "crates/policy", version = "1.0.2" } +rustfs-protos = { path = "crates/protos", version = "1.0.2" } +rustfs-protocols = { path = "crates/protocols", version = "1.0.2" } +rustfs-replication = { path = "crates/replication", version = "1.0.2" } +rustfs-rio = { path = "crates/rio", version = "1.0.2" } +rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" } +rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" } +rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" } +rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" } +rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" } +rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" } +rustfs-scanner = { path = "crates/scanner", version = "1.0.2" } +rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" } +rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" } +rustfs-signer = { path = "crates/signer", version = "1.0.2" } +rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" } +rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" } +rustfs-targets = { path = "crates/targets", version = "1.0.2" } +rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" } +rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" } +rustfs-utils = { path = "crates/utils", version = "1.0.2" } +rustfs-zip = { path = "./crates/zip", version = "1.0.2" } # Async Runtime and Networking async-channel = "2.5.0" @@ -155,9 +155,10 @@ futures-core = "0.3.34" futures-lite = "2.6.1" futures-util = "0.3.34" pollster = "1.0.1" -pulsar = { default-features = false, version = "6.9.0" } -lapin = { default-features = false, version = "4.12.0" } +pulsar = { default-features = false, version = "=6.9.0" } +lapin = { default-features = false, version = "4.12.2" } hyper = { version = "1.12.0" } +h2 = { version = "0.4.20" } hyper-rustls = { default-features = false, version = "0.27.10" } hyper-util = { version = "0.1.21" } http = "1.5.0" @@ -168,11 +169,11 @@ minlz = "1.2.3" reqwest = "0.13.5" rustfs-kafka-async = { version = "1.4.0" } socket2 = { version = "0.6.5" } -tokio = { version = "1.53.2" } +tokio = { version = "1.53.3" } tokio-rustls = { default-features = false, version = "0.26.6" } tokio-stream = { version = "0.1.19" } tokio-test = "0.4.6" -tokio-util = { version = "0.7.19" } +tokio-util = { version = "0.7.20" } tonic = { version = "0.14.6" } tonic-prost = { version = "0.14.6" } tonic-prost-build = { version = "0.14.6" } @@ -180,7 +181,7 @@ tower = { version = "0.5.3" } tower-http = { version = "0.7.1" } h3 = "0.0.8" h3-quinn = "0.0.10" -quinn = { version = "0.11.11", default-features = false, features = [ +quinn = { version = "0.11.12", default-features = false, features = [ "futures-io", "runtime-tokio", "rustls-aws-lc-rs", @@ -200,7 +201,7 @@ rmp-serde = { version = "1.3.1" } serde = { version = "1.0.229" } serde_ignored = { version = "0.1" } serde_with = { version = "3", default-features = false, features = ["macros", "std"] } -serde_json = { version = "1.0.151" } +serde_json = { version = "1.0.154" } serde_urlencoded = "0.7.1" # Cryptography and Security @@ -219,12 +220,12 @@ openidconnect = { default-features = false, version = "4.0" } pbkdf2 = "0.13.0" p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] } rsa = { version = "=0.10.0-rc.18" } -rustls = { default-features = false, version = "0.23.45" } +rustls = { default-features = false, version = "0.23.46" } rustls-native-certs = "0.8" rustls-pki-types = "1.15.1" x509-parser = "0.18.1" sha1 = "0.11.0" -sha2 = "0.11.0" +sha2 = "0.11.1" subtle = "2.6" zeroize = { version = "1.9.1" } proptest = "1" @@ -245,8 +246,8 @@ anyhow = "1.0.104" arc-swap = "1.9.2" rustfs-tokio-tar = { version = "0.7.1" } # Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures. -tar-codec = "0.0.14" -tar-framing = "0.0.14" +tar-codec = "0.0.15" +tar-framing = "0.0.15" atoi = "3.1.0" atomic_enum = "0.3.0" aws-config = { version = "1.12.0" } @@ -268,7 +269,7 @@ crossbeam-queue = "0.3.14" crossbeam-channel = "0.5.17" crossbeam-deque = "0.8.8" crossbeam-utils = "0.8.23" -datafusion = { default-features = false, version = "55.1.0" } +datafusion = { default-features = false, version = "55.2.0" } derive_builder = "0.20.2" ed25519-dalek = "3.0.0" enumset = "1.1.14" @@ -319,14 +320,14 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" } redis = { version = "1.7.1" } rustify = { version = "0.7", default-features = false } rustix = { version = "1.1.5" } -rust-embed = { version = "8.12.0" } +rust-embed = { version = "8.13.0" } rustc-hash = { version = "2.1.3" } s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] } s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" } serial_test = "4.0.1" shadow-rs = { default-features = false, version = "2.0.0" } siphasher = "1.0.4" -smallvec = { version = "1.16.2" } +smallvec = { version = "1.16.3" } compact_str = "0.10.0" snap = "1.1.2" starshard = { version = "2.4.0" } @@ -340,12 +341,12 @@ tracing = { version = "0.1.44" } tracing-appender = "0.2.5" tracing-core = "0.1.36" tracing-error = "0.2.1" -tracing-opentelemetry = { version = "0.34" } +tracing-opentelemetry = { version = "0.34.0" } tracing-subscriber = { version = "0.3.23" } transform-stream = "0.3.1" url = "2.5.8" urlencoding = "2.1.3" -uuid = { version = "1.27.0" } +uuid = { version = "1.28.0" } vaultrs = { version = "0.8.0" } tar = "0.4.46" walkdir = "2.5.0" @@ -353,26 +354,26 @@ winapi-util = "0.1.11" windows = { version = "0.62.2" } windows-sys = "0.61.2" xxhash-rust = { version = "0.8.19" } -zip = "8.6.0" -zstd = "0.14.0" +zip = "9.0.3" +zstd = "0.14.2" # Observability and Metrics metrics = "0.24.6" metrics-util = "0.20" -dial9 = { version = "0.5.3", default-features = false } -opentelemetry = { version = "0.33.0" } -opentelemetry-appender-tracing = { version = "0.33.0" } -opentelemetry-otlp = { version = "0.33.0" } -opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] } -opentelemetry_sdk = { version = "0.33.0" } -opentelemetry-semantic-conventions = { version = "0.33.0" } -opentelemetry-stdout = { version = "0.33.0" } -pyroscope = { version = "2.1.1" } +dial9 = { version = "0.5.4", default-features = false } +opentelemetry = { version = "0.33.1" } +opentelemetry-appender-tracing = { version = "0.33.1" } +opentelemetry-otlp = { version = "0.33.1" } +opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] } +opentelemetry_sdk = { version = "0.33.1" } +opentelemetry-semantic-conventions = { version = "0.33.1" } +opentelemetry-stdout = { version = "0.33.1" } +pyroscope = { version = "2.1.2" } # FTP and SFTP libunftp = { version = "0.23.1" } unftp-core = "0.1.0" -suppaftp = { version = "12.1.1" } +suppaftp = { version = "12.2.1" } rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] } russh = { version = "0.64.1" } russh-sftp = "3.0.1" @@ -383,7 +384,7 @@ dav-server = "0.11.0" # Performance Analysis and Memory Profiling rustfs-mimalloc = { version = "0.5.6" } # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved. -hotpath = { version = "0.28.5", default-features = false } +hotpath = { version = "0.28.6", default-features = false } # Snapshot testing for output format regression detection insta = { version = "1.49" } diff --git a/Dockerfile b/Dockerfile index 87c637928..d505a44ae 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM alpine:3.24.1 AS build +FROM alpine:3.24.2 AS build ARG TARGETARCH ARG RELEASE=latest @@ -70,7 +70,7 @@ RUN set -eux; \ rm -rf rustfs.zip /build/.tmp || true -FROM alpine:3.24.1 +FROM alpine:3.24.2 ARG RELEASE=latest ARG BUILD_DATE diff --git a/README.md b/README.md index bfbeac208..74093efc2 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,7 @@ chown -R 10001:10001 data logs docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest # Using specific version -docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1 +docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.3 ``` If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command @@ -220,7 +220,7 @@ For developers who want to build RustFS Docker images from source with multi-arc ./docker-buildx.sh --push # Build specific version -./docker-buildx.sh --release 1.0.1 --push +./docker-buildx.sh --release 1.0.3 --push # Build for custom registry ./docker-buildx.sh --registry your-registry.com --namespace yourname --push @@ -238,7 +238,7 @@ You can also use Make targets for convenience: ```bash make docker-buildx # Build locally make docker-buildx-push # Build and push -make docker-buildx-version VERSION=1.0.1 # Build specific version +make docker-buildx-version VERSION=1.0.3 # Build specific version make help-docker # Show all Docker-related commands ``` diff --git a/README_ZH.md b/README_ZH.md index 0d2fcee43..9ca9cedec 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -138,7 +138,7 @@ chown -R 10001:10001 data logs docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest # 使用指定版本运行 -docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1 +docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.3 ``` 如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录: @@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d ./docker-buildx.sh --push # 构建指定版本 -./docker-buildx.sh --release 1.0.1 --push +./docker-buildx.sh --release 1.0.3 --push # 构建并推送到自定义仓库 ./docker-buildx.sh --registry your-registry.com --namespace yourname --push @@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d ```bash make docker-buildx # 本地构建 make docker-buildx-push # 构建并推送 -make docker-buildx-version VERSION=1.0.1 # 构建指定版本 +make docker-buildx-version VERSION=1.0.3 # 构建指定版本 make help-docker # 显示所有 Docker 相关命令 ``` diff --git a/_typos.toml b/_typos.toml index 7ff843137..e11c3c036 100644 --- a/_typos.toml +++ b/_typos.toml @@ -51,6 +51,8 @@ unparseable = "unparseable" # Disaster-recovery objectives: recovery time and recovery point. RTO = "RTO" rto = "rto" +# rustix OFlags variant (POSIX O_WRONLY equivalent). +WRONLY = "WRONLY" [files] extend-exclude = [] diff --git a/crates/config/README.md b/crates/config/README.md index 81600cff9..fadcbac0d 100644 --- a/crates/config/README.md +++ b/crates/config/README.md @@ -119,11 +119,19 @@ when restoring `/rustfs/console` for a custom OEM build. - `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical) - `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical) - `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical) +- `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (canonical) +- `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (canonical) +- `RUSTFS_SCANNER_WALKERS_PER_DISK` (canonical) Scanner cycle budget controls: - When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance. - An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`. +Scanner walker controls: + +- `RUSTFS_SCANNER_WALKERS_PER_DISK` sets how many bucket walkers each local disk of an erasure set runs; the default `1` walks one bucket per disk at a time, which is the right value for rotational media. On flash, `2` to `4` lets one disk walk several buckets concurrently, which matters most for single-disk deployments with many buckets, where the walk is otherwise fully serial. Values outside `1` through `16` are rejected by `admin config set`; an out-of-range or non-numeric environment value is logged as a scanner runtime config error (`state = "startup_apply_failed"`) and the scanner keeps the default. The walkers of one disk share that disk's scanner admission, so peers still see one scan session per physical disk, and `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` still caps the total. +- The same value is available as the persistent `scanner.walkers_per_disk` key and is reported under `runtime_config.walkers_per_disk` in the admin scanner status; see [Scanner Runtime Controls](../../docs/operations/scanner-runtime-controls.md). + - A timed-out cycle cancels cooperative scanner work, then fences its leader epoch before releasing the lease. An uncooperative I/O operation is dropped after the bounded shutdown window; its cursor is not claimed to be durable and the scanner reports `recovery-required` when the worker cannot stop cooperatively, the cycle state was not confirmed durable, or epoch fencing cannot be persisted. ## Mmap read environment aliases @@ -266,7 +274,7 @@ Drive timeout profile preset: - `warn` (default): log warning and continue startup. - `fail`: abort startup with an error. -RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. +RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as `\\server\share` and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually `NTFS`) for them. ## 📄 License diff --git a/crates/config/src/constants/console.rs b/crates/config/src/constants/console.rs index 617aa020d..a8dd5f764 100644 --- a/crates/config/src/constants/console.rs +++ b/crates/config/src/constants/console.rs @@ -69,15 +69,17 @@ pub const ENV_CONSOLE_RATE_LIMIT_RPM: &str = "RUSTFS_CONSOLE_RATE_LIMIT_RPM"; pub const DEFAULT_CONSOLE_RATE_LIMIT_RPM: u32 = 100; /// Set the console authentication timeout in seconds -/// Specifies how long a console authentication session remains valid +/// Bounds how long the console waits for a single authenticated HTTP request before it is +/// considered timed out. This is a request timeout, not the session lifetime: the Console +/// credential lifetime issued by the OIDC callback is `RUSTFS_CONSOLE_OIDC_SESSION_DURATION`. /// Default: 3600 seconds (1 hour) /// Minimum: 300 seconds (5 minutes) /// Maximum: 86400 seconds (24 hours) pub const ENV_CONSOLE_AUTH_TIMEOUT: &str = "RUSTFS_CONSOLE_AUTH_TIMEOUT"; /// Default console authentication timeout in seconds -/// This is the default timeout for console authentication sessions. -/// After this timeout, users need to re-authenticate to access the console. +/// This is the default timeout for console authentication requests. +/// After this timeout, a pending console request is abandoned. /// Default value: 3600 seconds (1 hour) /// Environment variable: RUSTFS_CONSOLE_AUTH_TIMEOUT /// Command line argument: --console-auth-timeout @@ -85,6 +87,26 @@ pub const ENV_CONSOLE_AUTH_TIMEOUT: &str = "RUSTFS_CONSOLE_AUTH_TIMEOUT"; /// Example: --console-auth-timeout 3600 pub const DEFAULT_CONSOLE_AUTH_TIMEOUT: u64 = 3600; +/// Console credential lifetime, in seconds, for Console credentials issued by the OIDC callback. +/// +/// Process-level, Console-scoped setting: it applies to every OIDC provider and deliberately +/// lives outside the per-provider `RUSTFS_IDENTITY_OPENID_*` namespace, so it is never +/// suffix-scanned into a provider id and has no `identity_openid` KVS equivalent. The same value +/// also sets how long the Console logout token stays valid. +pub const ENV_CONSOLE_OIDC_SESSION_DURATION: &str = "RUSTFS_CONSOLE_OIDC_SESSION_DURATION"; + +/// Default Console credential lifetime in seconds, used when the environment variable is unset, +/// not a positive integer, or zero. Matches the value this path hard-coded before it was +/// parameterised. +pub const DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS: usize = 3600; + +/// Upper bound on the Console credential lifetime in seconds (7 days). +/// +/// Values above this are clamped with a warning. The bound keeps the moka logout cache's +/// `time_to_live` well under its 1000-year assertion, and stops an operator from minting an +/// effectively permanent Console STS credential on every node. +pub const CONSOLE_OIDC_SESSION_DURATION_MAX_SECS: usize = 604_800; + /// Toggle update check /// It controls whether to check for newer versions of rustfs /// Default value: true diff --git a/crates/config/src/constants/health.rs b/crates/config/src/constants/health.rs index 7e3bfcc73..84b465d4c 100644 --- a/crates/config/src/constants/health.rs +++ b/crates/config/src/constants/health.rs @@ -60,6 +60,20 @@ pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000; pub const ENV_STARTUP_READINESS_MAX_WAIT_SECS: &str = "RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS"; pub const DEFAULT_STARTUP_READINESS_MAX_WAIT_SECS: u64 = 120; +/// Wait for peer re-admission before a distributed node publishes startup +/// readiness. +/// +/// After this node reaches local readiness it asks the peers in each erasure +/// set it serves, over the internode RPC channel, whether they count its +/// drives as online. Startup readiness is published once a strict majority of +/// each set's peers do, or when the startup budget +/// ([`ENV_STARTUP_READINESS_MAX_WAIT_SECS`]) elapses, or when unreachable or +/// older peers make that majority impossible. The gate always fails open; it +/// never affects steady-state `/health/ready` or request handling. Disable it +/// to restore the local-view-only startup readiness. +pub const ENV_STARTUP_PEER_READMISSION_ENABLE: &str = "RUSTFS_STARTUP_PEER_READMISSION_ENABLE"; +pub const DEFAULT_STARTUP_PEER_READMISSION_ENABLE: bool = true; + /// Enable minimal health payload mode for GET `/health*` responses. /// When enabled, only `status` and `ready` fields are returned. pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE"; diff --git a/crates/config/src/constants/internode.rs b/crates/config/src/constants/internode.rs index 9fb78ebe2..da838a888 100644 --- a/crates/config/src/constants/internode.rs +++ b/crates/config/src/constants/internode.rs @@ -242,7 +242,7 @@ const _: () = assert!(!DEFAULT_INTERNODE_OFFLINE_BYPASS); /// read-after-write window can erode the read quorum and surface as a spurious /// `InsufficientReadQuorum` (issue #2761). One bounded retry absorbs that transient failure. /// **Only** idempotent reads use this; write/lock RPCs (`WriteAll`/`RenameData`/`Delete*`/`Lock` -/// /`UnLock`) must never auto-retry, to preserve quorum and idempotency semantics (see `CLAUDE.md`). +/// /`UnLock`) must never auto-retry, to preserve quorum and idempotency semantics (see `AGENTS.md`). pub const ENV_INTERNODE_IDEMPOTENT_READ_RETRIES: &str = "RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES"; pub const DEFAULT_INTERNODE_IDEMPOTENT_READ_RETRIES: usize = 1; diff --git a/crates/config/src/constants/scanner.rs b/crates/config/src/constants/scanner.rs index 8c5b9231b..e17b7d1cd 100644 --- a/crates/config/src/constants/scanner.rs +++ b/crates/config/src/constants/scanner.rs @@ -59,6 +59,15 @@ pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans"; /// Scanner config key capping concurrent scanner disk bucket walks per set. pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans"; +/// Scanner config key setting how many bucket walkers each local disk runs. +pub const SCANNER_WALKERS_PER_DISK: &str = "walkers_per_disk"; + +/// Per-local-disk scanner I/O ceiling in operations per second. +pub const SCANNER_DISK_IO_BUDGET_IOPS: &str = "disk_io_budget_iops"; + +/// Per-local-disk scanner I/O ceiling in bytes per second. +pub const SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC: &str = "disk_io_budget_bytes_per_sec"; + /// Scanner config key controlling how often object loops yield. pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects"; @@ -86,6 +95,9 @@ pub const SCANNER_KEYS: &[&str] = &[ SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_CONCURRENT_DISK_SCANS, + SCANNER_WALKERS_PER_DISK, + SCANNER_DISK_IO_BUDGET_IOPS, + SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_ALERT_EXCESS_VERSION_SIZE, @@ -214,6 +226,18 @@ pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCU /// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1` pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS"; +/// Environment variable that sets how many bucket walkers each local disk runs. +/// The walkers of one disk share its scanner admission, so a peer still sees one +/// scan session per physical disk. Remote disks keep one walker. The product of +/// local disks and this factor is what `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` +/// caps; the disk count stays the floor. Valid range is `1` through `16`. +/// - Example: `export RUSTFS_SCANNER_WALKERS_PER_DISK=4` +pub const ENV_SCANNER_WALKERS_PER_DISK: &str = "RUSTFS_SCANNER_WALKERS_PER_DISK"; + +pub const ENV_SCANNER_DISK_IO_BUDGET_IOPS: &str = "RUSTFS_SCANNER_DISK_IO_BUDGET_IOPS"; + +pub const ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC: &str = "RUSTFS_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC"; + /// Environment variable that controls how often scanner object loops yield to the async runtime. /// A value of `0` disables this extra object-count yield. /// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32` @@ -228,6 +252,36 @@ pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 4; /// Default disk scan concurrency budget. pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4; +/// Default bucket walkers per local disk. +/// +/// One walker per disk keeps today's I/O profile, which is what rotational +/// media needs. Flash is meant to default higher once a local disk can report +/// its media type; `rustfs_utils::os::Info::rotational` is never populated +/// today, so every deployment starts from one walker. +pub const DEFAULT_SCANNER_WALKERS_PER_DISK: usize = 1; + +/// Upper bound for `walkers_per_disk`; a larger factor only multiplies +/// contention on one spindle or NVMe queue without more disks to spread it. +pub const MAX_SCANNER_WALKERS_PER_DISK: usize = 16; + +/// Default spelling of both per-disk scanner I/O budget keys: resolve the +/// rate from the detected media. `off` removes the cap; a number is explicit. +pub const DEFAULT_SCANNER_DISK_IO_BUDGET: &str = "auto"; + +/// Rotational media budget, also used when media cannot be detected: about +/// half of a 7200 rpm spindle's random reads stay available to foreground +/// requests, and the byte ceiling sits well above the steady ~2 MB/s a paced +/// single-disk scan was measured at once admission landed (rustfs/rustfs#8377). +pub const DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_IOPS: u64 = 100; + +pub const DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_BYTES_PER_SEC: u64 = 16 * 1024 * 1024; + +/// Flash media budget: a few percent of a SATA SSD's random-read capacity, +/// enough to keep metadata walks far from saturating the device. +pub const DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_IOPS: u64 = 4000; + +pub const DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_BYTES_PER_SEC: u64 = 256 * 1024 * 1024; + /// Default object interval for cooperative scanner yields. pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128; diff --git a/crates/config/src/observability/mod.rs b/crates/config/src/observability/mod.rs index 9a1acc6c4..ca001fc8b 100644 --- a/crates/config/src/observability/mod.rs +++ b/crates/config/src/observability/mod.rs @@ -51,6 +51,11 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ /// PUT records multiple timers and histograms when attribution is active. pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED"; +/// Enables sampled in-process GetObject stage summaries without OTLP histograms. +pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED"; +pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"; +pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"; + pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL"; pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED"; pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY"; @@ -159,6 +164,15 @@ mod tests { assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED"); assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED"); assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED"); + assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED"); + assert_eq!( + ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE, + "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE" + ); + assert_eq!( + ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS, + "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS" + ); // Test log cleanup related env keys assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES"); assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES"); diff --git a/crates/data-usage/src/data_usage.rs b/crates/data-usage/src/data_usage.rs index 3b0e81699..fd9681064 100644 --- a/crates/data-usage/src/data_usage.rs +++ b/crates/data-usage/src/data_usage.rs @@ -569,6 +569,25 @@ pub struct DataUsageInfo { /// which last-known-good generation contributed to the view. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub usage_snapshot_set_states: Vec, + /// Buckets whose dirty-usage generation moved while this complete + /// snapshot was being walked. Their `buckets_usage` values are the walk + /// result, which the cycle could not certify as current; the next cycle + /// rescans them. Sorted. Empty on a converged snapshot and on an + /// observational one. A bucket created during the walk may be listed + /// here without a `buckets_usage` entry. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub usage_snapshot_pending_buckets: Vec, + /// Length of `usage_snapshot_pending_buckets`, carried for readers that + /// only need the count. The scanner writes both from one set. + #[serde(default, skip_serializing_if = "u64_is_zero")] + pub usage_snapshot_pending_bucket_count: u64, + /// Set by the scanner on a snapshot whose authoritative publication was + /// vetoed or could not be verified. It is routed to the observed object + /// and never replaces the authoritative snapshot. Absent on snapshots + /// written before per-bucket certification, which keep + /// `usage_snapshot_converged == Some(false)` as their observed marker. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub usage_snapshot_observational: bool, /// An observational view may contain only the sets that completed this /// cycle (or retained a compatible last-known-good cache). #[serde(default)] @@ -584,6 +603,10 @@ pub struct DataUsageInfo { pub disk_usage_status: Vec, } +fn u64_is_zero(value: &u64) -> bool { + *value == 0 +} + /// Stable identity fields changed by both coordinated scanner publication and /// backward-compatible bucket namespace cleanup. #[derive(Debug, Default, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] @@ -3011,6 +3034,58 @@ mod tests { assert_eq!(current.usage_snapshot_converged, Some(false)); } + #[test] + fn pending_bucket_certification_fields_round_trip_and_stay_additive() { + let certified = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH), + usage_snapshot_complete: true, + usage_snapshot_converged: Some(false), + usage_snapshot_pending_buckets: vec!["hot-a".to_string(), "hot-b".to_string()], + usage_snapshot_pending_bucket_count: 2, + ..Default::default() + }; + let encoded = rmp_serde::to_vec_named(&certified).expect("encode certified snapshot"); + let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode certified snapshot"); + assert_eq!(decoded, certified); + assert!(!decoded.usage_snapshot_observational); + let legacy: LegacyUsageReader = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore pending fields"); + assert_eq!(legacy.buckets_count, 0); + + let observed = DataUsageInfo { + usage_snapshot_observational: true, + ..certified + }; + let encoded = rmp_serde::to_vec_named(&observed).expect("encode observed snapshot"); + let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode observed snapshot"); + assert!(decoded.usage_snapshot_observational); + + // Older snapshots carry none of the fields and decode to the defaults; + // a converged snapshot writes none of them (named encoding carries + // field names, so a skipped field leaves no trace in the bytes). + let converged = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH), + usage_snapshot_complete: true, + usage_snapshot_converged: Some(true), + ..Default::default() + }; + let encoded = rmp_serde::to_vec_named(&converged).expect("encode converged snapshot"); + for key in [ + "usage_snapshot_pending_buckets", + "usage_snapshot_pending_bucket_count", + "usage_snapshot_observational", + ] { + assert!( + !encoded.windows(key.len()).any(|window| window == key.as_bytes()), + "{key} must be skipped when empty" + ); + } + let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode converged snapshot"); + assert!(decoded.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(decoded.usage_snapshot_pending_bucket_count, 0); + assert!(!decoded.usage_snapshot_observational); + assert!(decoded.is_complete_bucket_usage_snapshot()); + } + #[test] fn convergence_marker_defaults_to_unknown_for_older_snapshots() { let encoded = rmp_serde::to_vec_named(&DataUsageInfo { diff --git a/crates/e2e_test/Cargo.toml b/crates/e2e_test/Cargo.toml index 6b1dd5ccf..63c3d75de 100644 --- a/crates/e2e_test/Cargo.toml +++ b/crates/e2e_test/Cargo.toml @@ -146,4 +146,6 @@ zip.workspace = true clap = { workspace = true, features = ["derive", "env"] } [dev-dependencies] +aws-smithy-http-client = { workspace = true, default-features = false, features = ["test-util"] } tempfile.workspace = true +tokio = { workspace = true, features = ["test-util"] } diff --git a/crates/e2e_test/README.md b/crates/e2e_test/README.md index b5b1e727f..db60e4413 100644 --- a/crates/e2e_test/README.md +++ b/crates/e2e_test/README.md @@ -22,7 +22,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern: | **functional** | top-level `*_test.rs` | S3 data plane: `list_objects_*`, `copy_object_*`, `delete_objects_versioning`, `head_object_*`, `checksum_upload`, `compression`, `content_encoding`, `special_chars`, `leading_slash_key`, `create_bucket_region`, `quota`, `data_usage`, `snowball_auto_extract`, `mc_mirror_small_bucket`, `archive_download_integrity`, `version_id_regression`, `delete_marker_migration_semantics` | | **object_lock** | [`src/object_lock/`](src/object_lock) | Retention / legal-hold / WORM semantics | | **kms** | [`src/kms/`](src/kms) | SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: [`src/kms/README.md`](src/kms/README.md) | -| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) | +| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `conditional_write_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) | | **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) | | **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) | | **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test`, `tier_stats_cluster_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` | diff --git a/crates/e2e_test/src/common.rs b/crates/e2e_test/src/common.rs index e573fb00f..74d7f8729 100644 --- a/crates/e2e_test/src/common.rs +++ b/crates/e2e_test/src/common.rs @@ -1037,6 +1037,22 @@ pub async fn awscurl_post_sts_form_urlencoded( Ok(response) } +fn extract_xml_tag(xml: &str, tag: &str) -> Option { + let open = format!("<{tag}>"); + let close = format!(""); + let start = xml.find(&open)? + open.len(); + let end = xml[start..].find(&close)? + start; + Some(xml[start..end].to_string()) +} + +/// Returns `(access_key, secret_key, session_token)` from an STS `AssumeRole` XML response. +pub fn parse_assume_role_credentials(xml: &str) -> Result<(String, String, String), Box> { + let access_key = extract_xml_tag(xml, "AccessKeyId").ok_or("missing AccessKeyId in AssumeRole response")?; + let secret_key = extract_xml_tag(xml, "SecretAccessKey").ok_or("missing SecretAccessKey in AssumeRole response")?; + let session_token = extract_xml_tag(xml, "SessionToken").ok_or("missing SessionToken in AssumeRole response")?; + Ok((access_key, secret_key, session_token)) +} + /// Helper function for POST requests pub async fn awscurl_post( url: &str, diff --git a/crates/e2e_test/src/conditional_delete_objects_test.rs b/crates/e2e_test/src/conditional_delete_objects_test.rs new file mode 100644 index 000000000..753ed42a7 --- /dev/null +++ b/crates/e2e_test/src/conditional_delete_objects_test.rs @@ -0,0 +1,381 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! S3 wire semantics for per-object conditional bulk deletion. + +use crate::common::RustFSTestEnvironment; +use aws_sdk_s3::Client; +use aws_sdk_s3::error::ProvideErrorMetadata; +use aws_sdk_s3::operation::delete_objects::DeleteObjectsOutput; +use aws_sdk_s3::operation::put_object::PutObjectOutput; +use aws_sdk_s3::primitives::ByteStream; +use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration}; +use std::collections::BTreeSet; +use std::error::Error; + +type TestResult = Result>; +type VersionSnapshot = BTreeSet<(String, bool)>; + +async fn put(client: &Client, bucket: &str, key: &str, body: &'static [u8]) -> TestResult { + let result = client + .put_object() + .bucket(bucket) + .key(key) + .body(ByteStream::from_static(body)) + .send() + .await?; + assert!(result.e_tag().is_some(), "fixture PUT must provide an ETag for the following conditions"); + Ok(result) +} + +fn identifier(key: &str, etag: Option<&str>, version: Option<&str>) -> TestResult { + Ok(ObjectIdentifier::builder() + .key(key) + .set_e_tag(etag.map(str::to_owned)) + .set_version_id(version.map(str::to_owned)) + .build()?) +} + +async fn delete_batch( + client: &Client, + bucket: &str, + objects: Vec, + quiet: bool, +) -> TestResult { + Ok(client + .delete_objects() + .bucket(bucket) + .delete(Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?) + .send() + .await?) +} + +async fn assert_body(client: &Client, bucket: &str, key: &str, expected: &[u8]) -> TestResult { + let result = client.get_object().bucket(bucket).key(key).send().await?; + assert_eq!(result.body.collect().await?.into_bytes().as_ref(), expected); + Ok(()) +} + +async fn assert_absent(client: &Client, bucket: &str, key: &str) { + let error = client + .head_object() + .bucket(bucket) + .key(key) + .send() + .await + .expect_err("deleted key must be absent"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("NotFound")); +} + +async fn versions(client: &Client, bucket: &str, key: &str) -> TestResult { + let result = client.list_object_versions().bucket(bucket).prefix(key).send().await?; + assert!(!result.is_truncated().unwrap_or_default(), "focused version fixture fits a single page"); + let mut snapshot = BTreeSet::new(); + for version in result.versions().iter().filter(|version| version.key() == Some(key)) { + snapshot.insert((version.version_id().expect("version has an identity").to_owned(), false)); + } + for marker in result.delete_markers().iter().filter(|marker| marker.key() == Some(key)) { + snapshot.insert((marker.version_id().expect("delete marker has an identity").to_owned(), true)); + } + Ok(snapshot) +} + +#[tokio::test] +async fn sdk_conditional_delete_objects_mixed_results_overwrites_duplicates_and_quiet() -> TestResult { + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let client = env.create_s3_client(); + let bucket = "conditional-bulk-mixed"; + client.create_bucket().bucket(bucket).send().await?; + + let old = put(&client, bucket, "overwritten", b"generation A").await?; + let current = put(&client, bucket, "overwritten", b"generation B").await?; + let matched = put(&client, bucket, "matched", b"matching object").await?; + put(&client, bucket, "exists", b"existence object").await?; + put(&client, bucket, "unconditional", b"unconditional object").await?; + put(&client, bucket, "empty", b"empty condition must preserve this").await?; + put(&client, bucket, "empty-quoted", b"quoted empty condition must preserve this").await?; + let weak = put(&client, bucket, "weak", b"weak condition must preserve this").await?; + let weak_etag = format!("W/\"{}\"", weak.e_tag().expect("PUT supplies an ETag").trim_matches('"')); + assert_ne!(old.e_tag(), current.e_tag(), "overwrite fixture must have distinct ETags"); + let matched_etag = matched.e_tag().expect("PUT supplies an ETag"); + let result = delete_batch( + &client, + bucket, + vec![ + identifier("matched", Some(matched_etag), None)?, + identifier("overwritten", old.e_tag(), None)?, + identifier("matched", Some("stale-etag"), None)?, + identifier("matched", Some(matched_etag), None)?, + identifier("unconditional", None, None)?, + identifier("exists", Some("*"), None)?, + identifier("missing", Some("*"), None)?, + identifier("empty", Some(""), None)?, + identifier("empty-quoted", Some("\"\""), None)?, + identifier("weak", Some(&weak_etag), None)?, + identifier("unconditional-missing", None, None)?, + identifier("missing-specific", Some("opaque-etag"), None)?, + ], + false, + ) + .await?; + assert_eq!( + result.deleted().iter().map(|object| object.key()).collect::>(), + [ + Some("matched"), + Some("matched"), + Some("unconditional"), + Some("exists"), + Some("unconditional-missing") + ], + "duplicate successful inputs retain their separate response entries" + ); + assert_eq!( + result + .errors() + .iter() + .map(|error| (error.key(), error.code())) + .collect::>(), + [ + (Some("overwritten"), Some("PreconditionFailed")), + (Some("matched"), Some("PreconditionFailed")), + (Some("missing"), Some("NoSuchKey")), + (Some("empty"), Some("InvalidArgument")), + (Some("empty-quoted"), Some("InvalidArgument")), + (Some("weak"), Some("InvalidArgument")), + (Some("missing-specific"), Some("NoSuchKey")), + ] + ); + assert_body(&client, bucket, "overwritten", b"generation B").await?; + assert_body(&client, bucket, "empty", b"empty condition must preserve this").await?; + assert_body(&client, bucket, "empty-quoted", b"quoted empty condition must preserve this").await?; + assert_body(&client, bucket, "weak", b"weak condition must preserve this").await?; + for key in ["matched", "unconditional", "exists"] { + assert_absent(&client, bucket, key).await; + } + + let quiet = put(&client, bucket, "quiet-match", b"quiet matching object").await?; + put(&client, bucket, "quiet-rejected", b"quiet rejected object").await?; + let result = delete_batch( + &client, + bucket, + vec![ + identifier("quiet-match", quiet.e_tag(), None)?, + identifier("quiet-match", quiet.e_tag(), None)?, + identifier("quiet-rejected", Some("stale-etag"), None)?, + identifier("missing", Some("*"), None)?, + ], + true, + ) + .await?; + assert!(result.deleted().is_empty(), "Quiet suppresses successful entries"); + assert_eq!( + result + .errors() + .iter() + .map(|error| (error.key(), error.code())) + .collect::>(), + [ + (Some("quiet-rejected"), Some("PreconditionFailed")), + (Some("missing"), Some("NoSuchKey")) + ] + ); + assert_absent(&client, bucket, "quiet-match").await; + assert_body(&client, bucket, "quiet-rejected", b"quiet rejected object").await?; + Ok(()) +} + +#[tokio::test] +async fn sdk_conditional_delete_objects_versioning_current_markers_and_explicit_versions() -> TestResult { + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let client = env.create_s3_client(); + for suspended in [false, true] { + let bucket = if suspended { + "conditional-bulk-suspended" + } else { + "conditional-bulk-enabled" + }; + client.create_bucket().bucket(bucket).send().await?; + client + .put_bucket_versioning() + .bucket(bucket) + .versioning_configuration( + VersioningConfiguration::builder() + .status(BucketVersioningStatus::Enabled) + .build(), + ) + .send() + .await?; + let historical = put(&client, bucket, "history", b"historical payload").await?; + let invalid_historical = put(&client, bucket, "invalid", b"historical rejected payload").await?; + if suspended { + client + .put_bucket_versioning() + .bucket(bucket) + .versioning_configuration( + VersioningConfiguration::builder() + .status(BucketVersioningStatus::Suspended) + .build(), + ) + .send() + .await?; + } + let current = put(&client, bucket, "history", b"current payload").await?; + let invalid = put(&client, bucket, "invalid", b"current rejected payload").await?; + let invalid_before = versions(&client, bucket, "invalid").await?; + let historical_id = historical.version_id().expect("enabled fixture has a UUID version"); + let invalid_historical_id = invalid_historical + .version_id() + .expect("enabled rejected fixture has a UUID version"); + let result = delete_batch( + &client, + bucket, + vec![ + identifier("history", current.e_tag(), None)?, + identifier("history", None, Some(historical_id))?, + identifier("history", current.e_tag(), None)?, + identifier("invalid", invalid.e_tag(), Some(invalid_historical_id))?, + identifier("invalid", Some("*"), Some("null"))?, + identifier("invalid", Some("stale-etag"), None)?, + ], + false, + ) + .await?; + assert_eq!(result.deleted().len(), 3, "both current input entries and the historical delete succeed"); + assert_eq!( + result + .errors() + .iter() + .map(|error| (error.key(), error.code())) + .collect::>(), + [ + (Some("invalid"), Some("InvalidRequest")), + (Some("invalid"), Some("InvalidRequest")), + (Some("invalid"), Some("PreconditionFailed")), + ] + ); + assert_eq!(result.errors()[0].version_id(), Some(invalid_historical_id)); + assert_eq!(result.errors()[1].version_id(), Some("null")); + assert_eq!(versions(&client, bucket, "invalid").await?, invalid_before); + assert_body(&client, bucket, "invalid", b"current rejected payload").await?; + + let marker = result.deleted()[0] + .delete_marker_version_id() + .expect("current conditional delete returns a marker"); + assert_eq!(result.deleted()[0].delete_marker(), Some(true)); + assert_eq!(result.deleted()[2].delete_marker_version_id(), Some(marker)); + assert_eq!(result.deleted()[1].version_id(), Some(historical_id)); + if suspended { + assert_eq!(marker, "null", "a suspended current delete returns the created null marker identity"); + } + let history_after = versions(&client, bucket, "history").await?; + assert_eq!( + history_after.iter().filter(|(_, marker)| *marker).count(), + 1, + "duplicate actions create one marker" + ); + assert!( + !history_after.contains(&(historical_id.to_string(), false)), + "distinct historical target is removed" + ); + if suspended { + assert_eq!(history_after, BTreeSet::from([("null".to_string(), true)])); + } else { + assert_eq!( + history_after, + BTreeSet::from([ + (current.version_id().expect("current enabled object has a UUID").to_string(), false), + (marker.to_string(), true), + ]), + "enabled conditional current deletion retains the current data version behind the marker" + ); + } + + let result = delete_batch( + &client, + bucket, + vec![ + identifier("history", Some("*"), None)?, + identifier("history", current.e_tag(), None)?, + identifier("missing", Some("*"), None)?, + ], + false, + ) + .await?; + assert!( + result.deleted().is_empty(), + "current markers and absent objects cannot satisfy a live-object condition" + ); + assert_eq!(result.errors().len(), 3); + assert!(result.errors().iter().all(|error| error.code() == Some("NoSuchKey"))); + assert_eq!( + versions(&client, bucket, "history").await?, + history_after, + "failed conditions create no marker" + ); + assert!(versions(&client, bucket, "missing").await?.is_empty()); + + put(&client, bucket, "exists", b"versioned existence payload").await?; + let result = delete_batch(&client, bucket, vec![identifier("exists", Some("*"), None)?], false).await?; + assert!(result.errors().is_empty()); + assert_eq!(result.deleted().len(), 1); + assert_eq!(result.deleted()[0].delete_marker(), Some(true)); + assert_eq!( + versions(&client, bucket, "exists") + .await? + .iter() + .filter(|(_, marker)| *marker) + .count(), + 1 + ); + + if !suspended { + let directory = put(&client, bucket, "dir/", b"directory null-version payload").await?; + assert_eq!( + versions(&client, bucket, "dir/").await?, + BTreeSet::from([("null".to_string(), false)]), + "the directory PUT fixture must identify the synthetic null data version" + ); + let result = delete_batch( + &client, + bucket, + vec![identifier("dir/", directory.e_tag(), None)?, identifier("dir/", None, None)?], + false, + ) + .await?; + assert!(result.errors().is_empty(), "both distinct directory actions must succeed"); + assert_eq!(result.deleted().len(), 2); + assert_eq!(result.deleted()[0].key(), Some("dir/")); + assert_eq!(result.deleted()[1].key(), Some("dir/")); + assert_eq!(result.deleted()[0].delete_marker(), Some(true)); + let marker = result.deleted()[0] + .delete_marker_version_id() + .expect("conditional current-directory deletion creates a versioned marker"); + assert!( + !uuid::Uuid::parse_str(marker)?.is_nil(), + "enabled current-directory deletion creates a UUID marker" + ); + assert_eq!(result.deleted()[1].delete_marker(), None); + assert_eq!(result.deleted()[1].delete_marker_version_id(), None); + assert_eq!(result.deleted()[1].version_id(), None); + assert_eq!( + versions(&client, bucket, "dir/").await?, + BTreeSet::from([(marker.to_string(), true)]), + "current-marker creation and synthetic null-version removal are distinct actions: one marker remains and the null data version is gone" + ); + } + } + Ok(()) +} diff --git a/crates/e2e_test/src/conditional_write_policy_test.rs b/crates/e2e_test/src/conditional_write_policy_test.rs new file mode 100644 index 000000000..5b610b692 --- /dev/null +++ b/crates/e2e_test/src/conditional_write_policy_test.rs @@ -0,0 +1,377 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! E2E: conditional write enforcement with the `s3:if-none-match` and `s3:if-match` +//! condition keys in a **bucket policy**, an **IAM identity policy**, and an **STS +//! AssumeRole session policy**. Covers PutObject and CompleteMultipartUpload, the two +//! upload operations that accept the conditional headers. The plain precondition +//! semantics without a policy live in `reliant/conditional_writes.rs`. + +use crate::common::{ + AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user, + awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging, parse_assume_role_credentials, +}; +use aws_sdk_s3::Client; +use aws_sdk_s3::config::http::HttpResponse; +use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError}; +use aws_sdk_s3::primitives::ByteStream; +use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; +use tracing::info; +use uuid::Uuid; + +type TestResult = Result<(), Box>; + +fn assert_error( + err: &SdkError, + code: &str, + status: u16, + context: &str, +) { + assert_eq!( + err.as_service_error().and_then(ProvideErrorMetadata::code), + Some(code), + "{context}: {err:?}" + ); + assert_eq!( + err.raw_response().map(|response| response.status().as_u16()), + Some(status), + "{context}: {err:?}" + ); +} + +async fn admin_add_and_attach_policy( + env: &RustFSTestEnvironment, + policy_name: &str, + policy_json: &str, + username: &str, +) -> TestResult { + admin_add_canned_policy_via( + AdminTransport::Signed, + &env.url, + &env.access_key, + &env.secret_key, + policy_name, + policy_json, + ) + .await?; + admin_attach_user_policy_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, policy_name, username).await +} + +/// Starts a one-part multipart upload as the admin so that only the complete step +/// runs under the policy being tested. CreateMultipartUpload and UploadPart are both +/// authorized as `s3:PutObject` and cannot carry a conditional header, so a policy +/// that requires `s3:if-none-match` denies them. AWS exempts them with +/// `s3:ObjectCreationOperation`, which RustFS does not support yet. +async fn admin_start_upload( + admin: &Client, + bucket: &str, + key: &str, +) -> Result<(String, CompletedMultipartUpload), Box> { + let upload = admin.create_multipart_upload().bucket(bucket).key(key).send().await?; + let upload_id = upload.upload_id().ok_or("missing upload id")?.to_string(); + let part = admin + .upload_part() + .bucket(bucket) + .key(key) + .upload_id(&upload_id) + .part_number(1) + .body(ByteStream::from_static(b"multipart-conditional-write")) + .send() + .await?; + let completed = CompletedMultipartUpload::builder() + .parts( + CompletedPart::builder() + .part_number(1) + .set_e_tag(part.e_tag().map(str::to_string)) + .build(), + ) + .build(); + Ok((upload_id, completed)) +} + +/// One server for the three policy kinds; each scenario uses its own principal and bucket. +#[tokio::test] +async fn test_e2e_policies_enforce_conditional_write_keys() -> TestResult { + init_logging(); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let admin = env.create_s3_client(); + let suffix = Uuid::new_v4(); + + bucket_policy_requires_if_none_match(&env, &admin, suffix).await?; + iam_policy_requires_if_match(&env, &admin, suffix).await?; + sts_session_policy_requires_if_none_match(&env, &admin, suffix).await?; + + info!("test_e2e_policies_enforce_conditional_write_keys passed"); + Ok(()) +} + +/// Bucket policy in the shape of the AWS enforcement example: PutObject is allowed only +/// when the request carries `If-None-Match`, which makes the bucket create-only. +async fn bucket_policy_requires_if_none_match(env: &RustFSTestEnvironment, admin: &Client, suffix: Uuid) -> TestResult { + let user = format!("e2econdwrite-{suffix}"); + let user_secret = "longSecretKeyForCondWrite1!"; + let bucket = format!("e2e-cond-write-bkt-{suffix}"); + + admin_create_user(env, &user, user_secret).await?; + admin.create_bucket().bucket(&bucket).send().await?; + let bucket_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": { "AWS": [user.clone()] }, + "Action": ["s3:PutObject"], + "Resource": [format!("arn:aws:s3:::{bucket}/*")], + "Condition": { "Null": { "s3:if-none-match": "false" } } + }] + }) + .to_string(); + admin + .put_bucket_policy() + .bucket(&bucket) + .policy(&bucket_policy) + .send() + .await?; + + let client = env.create_s3_client_with_credentials(&user, user_secret); + + let denied = client + .put_object() + .bucket(&bucket) + .key("plain.txt") + .body(ByteStream::from_static(b"plain")) + .send() + .await + .expect_err("PutObject without If-None-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional PutObject"); + + let created = client + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("*") + .body(ByteStream::from_static(b"created")) + .send() + .await?; + let etag = created.e_tag().ok_or("missing ETag on conditional PutObject")?.to_string(); + + let overwrite = client + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("*") + .body(ByteStream::from_static(b"overwrite")) + .send() + .await + .expect_err("If-None-Match: * must not overwrite an existing key"); + assert_error( + &overwrite, + "PreconditionFailed", + 412, + "allowed PutObject still evaluates the precondition", + ); + + // A non-`*` value satisfies `Null: false` but must not act as an ETag comparison + // that lets the write overwrite the key. + let overwrite = client + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("\"0123456789abcdef0123456789abcdef\"") + .body(ByteStream::from_static(b"overwrite")) + .send() + .await + .expect_err("If-None-Match with an ETag value must be rejected"); + assert_error(&overwrite, "NotImplemented", 501, "PutObject with an ETag If-None-Match"); + let current = admin.head_object().bucket(&bucket).key("created.txt").send().await?; + assert_eq!(current.e_tag(), Some(etag.as_str()), "created.txt must not be overwritten"); + + let (upload_id, completed) = admin_start_upload(admin, &bucket, "multipart.bin").await?; + let denied = client + .complete_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .multipart_upload(completed.clone()) + .send() + .await + .expect_err("CompleteMultipartUpload without If-None-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional CompleteMultipartUpload"); + + client + .complete_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .if_none_match("*") + .multipart_upload(completed) + .send() + .await?; + + let (upload_id, completed) = admin_start_upload(admin, &bucket, "multipart.bin").await?; + let overwrite = client + .complete_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .if_none_match("*") + .multipart_upload(completed) + .send() + .await + .expect_err("CompleteMultipartUpload with If-None-Match: * must not overwrite an existing key"); + assert_error( + &overwrite, + "PreconditionFailed", + 412, + "allowed CompleteMultipartUpload still evaluates the precondition", + ); + + let _ = admin + .abort_multipart_upload() + .bucket(&bucket) + .key("multipart.bin") + .upload_id(&upload_id) + .send() + .await; + for key in ["created.txt", "multipart.bin"] { + let _ = admin.delete_object().bucket(&bucket).key(key).send().await; + } + let _ = admin.delete_bucket().bucket(&bucket).send().await; + Ok(()) +} + +/// IAM identity policy: PutObject is allowed only when the request carries `If-Match`. +async fn iam_policy_requires_if_match(env: &RustFSTestEnvironment, admin: &Client, suffix: Uuid) -> TestResult { + let user = format!("e2econdmatch-{suffix}"); + let user_secret = "longSecretKeyForCondMatch2!"; + let policy_name = format!("e2e-cond-match-pol-{suffix}"); + let bucket = format!("e2e-cond-match-bkt-{suffix}"); + let key = "guarded.txt"; + + admin_create_user(env, &user, user_secret).await?; + let policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": ["s3:PutObject"], + "Resource": [format!("arn:aws:s3:::{bucket}/*")], + "Condition": { "Null": { "s3:if-match": "false" } } + }] + }) + .to_string(); + admin_add_and_attach_policy(env, &policy_name, &policy, &user).await?; + + admin.create_bucket().bucket(&bucket).send().await?; + let original = admin + .put_object() + .bucket(&bucket) + .key(key) + .body(ByteStream::from_static(b"v1")) + .send() + .await?; + let etag = original.e_tag().ok_or("missing ETag on initial PutObject")?.to_string(); + + let client = env.create_s3_client_with_credentials(&user, user_secret); + let denied = client + .put_object() + .bucket(&bucket) + .key(key) + .body(ByteStream::from_static(b"blind-overwrite")) + .send() + .await + .expect_err("PutObject without If-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional PutObject"); + + client + .put_object() + .bucket(&bucket) + .key(key) + .if_match(&etag) + .body(ByteStream::from_static(b"v2")) + .send() + .await?; + + let _ = admin.delete_object().bucket(&bucket).key(key).send().await; + let _ = admin.delete_bucket().bucket(&bucket).send().await; + Ok(()) +} + +/// STS session policy: AssumeRole must accept `s3:if-none-match`, and the session may +/// only create objects with `If-None-Match: *`. +async fn sts_session_policy_requires_if_none_match(env: &RustFSTestEnvironment, admin: &Client, suffix: Uuid) -> TestResult { + let parent = format!("e2e-sts-cond-par-{suffix}"); + let parent_secret = "longSecretKeyForCondParent3!"; + let policy_name = format!("e2e-sts-cond-rw-{suffix}"); + let bucket = format!("e2e-sts-cond-bkt-{suffix}"); + + admin_create_user(env, &parent, parent_secret).await?; + let parent_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:*"], "Resource": ["arn:aws:s3:::*"] }, + { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["arn:aws:s3:::*"] } + ] + }) + .to_string(); + admin_add_and_attach_policy(env, &policy_name, &parent_policy, &parent).await?; + admin.create_bucket().bucket(&bucket).send().await?; + + let session_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": ["s3:PutObject"], + "Resource": [format!("arn:aws:s3:::{bucket}/*")], + "Condition": { "StringEquals": { "s3:if-none-match": "*" } } + }] + }) + .to_string(); + let body = format!( + "Action=AssumeRole&Version=2011-06-15&DurationSeconds=3600&Policy={}", + urlencoding::encode(&session_policy) + ); + let url = format!("{}/", env.url.trim_end_matches('/')); + let xml = awscurl_post_sts_form_urlencoded(&url, &body, &parent, parent_secret).await?; + let (access_key, secret_key, session_token) = parse_assume_role_credentials(&xml)?; + let session = Client::from_conf(build_test_s3_config( + &env.url, + &access_key, + &secret_key, + Some(&session_token), + "e2e-sts-session", + )); + + let denied = session + .put_object() + .bucket(&bucket) + .key("plain.txt") + .body(ByteStream::from_static(b"plain")) + .send() + .await + .expect_err("session PutObject without If-None-Match must be denied"); + assert_error(&denied, "AccessDenied", 403, "unconditional session PutObject"); + + session + .put_object() + .bucket(&bucket) + .key("created.txt") + .if_none_match("*") + .body(ByteStream::from_static(b"created")) + .send() + .await?; + + let _ = admin.delete_object().bucket(&bucket).key("created.txt").send().await; + let _ = admin.delete_bucket().bucket(&bucket).send().await; + Ok(()) +} diff --git a/crates/e2e_test/src/distributed/harness.rs b/crates/e2e_test/src/distributed/harness.rs index 364034477..3ce00fa01 100644 --- a/crates/e2e_test/src/distributed/harness.rs +++ b/crates/e2e_test/src/distributed/harness.rs @@ -860,10 +860,12 @@ pub(crate) async fn start_rebalance(cluster: &RustFSTestClusterEnvironment) -> T if status.is_success() { break response; } - if status == StatusCode::INTERNAL_SERVER_ERROR - && response.contains("requires a live fleet capability proof") - && Instant::now() < deadline - { + // The init stage answers 503 + Retry-After while the cluster capability + // proof is still missing; older binaries surfaced the same condition as + // a 500 carrying the proof marker. + let waiting_for_readiness = status == StatusCode::SERVICE_UNAVAILABLE + || (status == StatusCode::INTERNAL_SERVER_ERROR && response.contains("requires a live fleet capability proof")); + if waiting_for_readiness && Instant::now() < deadline { sleep(Duration::from_secs(1)).await; continue; } diff --git a/crates/e2e_test/src/existing_object_tag_policy_test.rs b/crates/e2e_test/src/existing_object_tag_policy_test.rs index 57a21f48e..ff32787d9 100644 --- a/crates/e2e_test/src/existing_object_tag_policy_test.rs +++ b/crates/e2e_test/src/existing_object_tag_policy_test.rs @@ -18,7 +18,7 @@ use crate::common::{ AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, - awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging, + awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging, parse_assume_role_credentials, }; use aws_sdk_s3::Client; use aws_sdk_s3::error::ProvideErrorMetadata; @@ -41,21 +41,6 @@ fn sts_session_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: )) } -fn extract_xml_tag(xml: &str, tag: &str) -> Option { - let open = format!("<{tag}>"); - let close = format!(""); - let start = xml.find(&open)? + open.len(); - let end = xml[start..].find(&close)? + start; - Some(xml[start..end].to_string()) -} - -fn parse_assume_role_credentials(xml: &str) -> Result<(String, String, String), Box> { - let ak = extract_xml_tag(xml, "AccessKeyId").ok_or("missing AccessKeyId in AssumeRole response")?; - let sk = extract_xml_tag(xml, "SecretAccessKey").ok_or("missing SecretAccessKey in AssumeRole response")?; - let token = extract_xml_tag(xml, "SessionToken").ok_or("missing SessionToken in AssumeRole response")?; - Ok((ak, sk, token)) -} - async fn assume_role_with_session_policy( env: &RustFSTestEnvironment, parent_ak: &str, diff --git a/crates/e2e_test/src/fake_s3_target/README.md b/crates/e2e_test/src/fake_s3_target/README.md index bcd2e8ca6..dda6ee019 100644 --- a/crates/e2e_test/src/fake_s3_target/README.md +++ b/crates/e2e_test/src/fake_s3_target/README.md @@ -6,7 +6,7 @@ This module is the shared failure-injection boundary for replication end-to-end Supported data operations are HeadBucket, GetBucketVersioning, ListObjectsV2, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets created with `create_bucket` are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB. -`create_bucket_with_object_lock(name)` creates a versioned bucket whose GetObjectLockConfiguration reports `Enabled`; every other bucket answers `ObjectLockConfigurationNotFoundError`, the code RustFS's replication-check classifies as "not enabled". Three switches model remote-target behaviors the fleet has shown, so the outbound target matrix (`crates/e2e_test/src/replication_target_matrix_test.rs`) can replicate every object shape against each: `assign_own_version_ids(true)` ignores the source version id and mints its own (AWS S3 / Wasabi); `reject_aws_chunked_uploads(true)` refuses any PutObject or UploadPart announcing `aws-chunked` framing (`Content-Encoding: aws-chunked`, an `x-amz-trailer`, or a `STREAMING-*` payload hash) with `InvalidRequest` before the body is read (SeaweedFS 3.97, rustfs#6853); `require_checksum_for_object_lock(true)` rejects a PutObject carrying any `x-amz-object-lock-*` header unless it also carries `Content-MD5`, an `x-amz-checksum-*` header, or `x-amz-sdk-checksum-algorithm` (AWS S3 / MinIO, rustfs#7082). Independently of that switch, a `Content-MD5` header is always verified against the body and a mismatch answers `BadDigest`. +`create_bucket_with_object_lock(name)` creates a versioned bucket whose GetObjectLockConfiguration reports `Enabled`; every other bucket answers `ObjectLockConfigurationNotFoundError`, the code RustFS's replication-check classifies as "not enabled". Four switches model remote-target behaviors the fleet has shown, so the outbound target matrix (`crates/e2e_test/src/replication_target_matrix_test.rs`) can replicate every object shape against each: `assign_own_version_ids(true)` ignores the source version id and mints its own (AWS S3 / Wasabi); `reject_aws_chunked_uploads(true)` refuses any PutObject or UploadPart announcing `aws-chunked` framing (`Content-Encoding: aws-chunked`, an `x-amz-trailer`, or a `STREAMING-*` payload hash) with `InvalidRequest` before the body is read (SeaweedFS 3.97, rustfs#6853); `require_checksum_for_object_lock(true)` rejects a PutObject carrying any `x-amz-object-lock-*` header unless it also carries `Content-MD5`, an `x-amz-checksum-*` header, or `x-amz-sdk-checksum-algorithm` (AWS S3 / MinIO, rustfs#7082). `ignore_source_etags(true)` ignores the `x-rustfs-source-etag` / `x-minio-source-etag` replication headers and reports the ETag of the received bytes (AWS S3 and other generic S3 services; the default adopts the source ETag like RustFS/MinIO). Independently of that switch, a `Content-MD5` header is always verified against the body and a mismatch answers `BadDigest`. `create_bucket_with_mode(name, BucketMode::Unversioned)` models a plain migration source: PUT overwrites in place, DELETE removes the key without a delete marker, GetBucketVersioning reports no status, and no `x-amz-version-id` is returned by PUT, GET, HEAD, tagging, or multipart completion. The only `versionId` such a bucket accepts is `null`; any other value is rejected with `InvalidArgument`. The mode is fixed at creation. diff --git a/crates/e2e_test/src/fake_s3_target/mod.rs b/crates/e2e_test/src/fake_s3_target/mod.rs index 3f8bf88d8..411c4133d 100644 --- a/crates/e2e_test/src/fake_s3_target/mod.rs +++ b/crates/e2e_test/src/fake_s3_target/mod.rs @@ -537,6 +537,10 @@ struct StoreState { /// PutObject carrying any `x-amz-object-lock-*` header must also carry /// `Content-MD5` or an `x-amz-checksum-*` header. require_checksum_for_object_lock: bool, + /// Models AWS S3 and other generic S3 services: the `x-rustfs-source-etag` + /// / `x-minio-source-etag` replication headers are ignored and every + /// stored object reports the ETag computed from the bytes it received. + ignore_source_etags: bool, /// Models Wasabi (rustfs/backlog#2340): a version-addressed DELETE of a /// version id the target never had answers 404 `NoSuchVersion` instead of /// the idempotent 204 RustFS/MinIO give. @@ -1052,6 +1056,13 @@ impl FakeS3Target { lock(&self.backend.store).require_checksum_for_object_lock = enabled; } + /// AWS-like mode: ignore the replication source-ETag headers and report + /// the ETag of the received bytes, as AWS S3 does for SSE-S3 and + /// unencrypted objects. The default adopts the source ETag (RustFS/MinIO). + pub fn ignore_source_etags(&self, enabled: bool) { + lock(&self.backend.store).ignore_source_etags = enabled; + } + pub fn assign_own_multipart_version_ids(&self, enabled: bool) { lock(&self.backend.store).assign_own_multipart_version_ids = enabled; } @@ -2444,12 +2455,13 @@ impl S3 for FakeBackend { .map_err(|_| s3s::s3_error!(ServiceUnavailable, "fake target body limiter closed"))?; let headers = req.headers; let input = req.input; - let (assign_own, drop_unlisted, require_checksum, limits, versioned) = { + let (assign_own, drop_unlisted, require_checksum, ignore_source_etags, limits, versioned) = { let state = lock(&self.store); ( state.assign_own_version_ids, state.drop_unlisted_replication_headers, state.require_checksum_for_object_lock, + state.ignore_source_etags, state.limits, bucket_versioned(&state, &input.bucket), ) @@ -2489,7 +2501,7 @@ impl S3 for FakeBackend { } else { NULL_VERSION_ID.to_string() }; - let e_tag = match source_etag(&headers)? { + let e_tag = match source_etag(&headers)?.filter(|_| !ignore_source_etags) { Some(value) => value, None => { let (digest, permit) = md5_digest(body.clone(), body_permit).await?; @@ -3112,7 +3124,10 @@ impl S3 for FakeBackend { .collect(); let (body, digests, _body_permits) = assemble_multipart(assembly_parts, total_len, _body_permits).await?; let part_count = requested.len(); - let e_tag = source_etag(&headers)?.unwrap_or_else(|| format!("{}-{part_count}", md5_hex(digests))); + let ignore_source_etags = lock(&self.store).ignore_source_etags; + let e_tag = source_etag(&headers)? + .filter(|_| !ignore_source_etags) + .unwrap_or_else(|| format!("{}-{part_count}", md5_hex(digests))); let version = ObjectVersion { version_id: upload.version_id.clone(), body, diff --git a/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs b/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs index aaf45f5de..f5187aa77 100644 --- a/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs +++ b/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs @@ -1032,7 +1032,7 @@ mod tests { let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#; let heal_url = format!("{}/rustfs/admin/v3/heal/{}?forceStart=true", env.url, bucket); - signed_admin_post(&heal_url, Some(heal_body), &env.access_key, &env.secret_key) + start_root_heal_when_control_ready(&heal_url, heal_body, &env.access_key, &env.secret_key) .await .expect("admin deep heal should be accepted"); @@ -1191,11 +1191,59 @@ mod tests { .into()) } + fn scanner_cycle_completion_budget( + cycle_end: u64, + interval: Option, + now: u64, + active: bool, + ) -> Result { + if cycle_end > now { + return Err("scanner cycle timestamp is in the future"); + } + if active { + return Ok(Duration::from_secs(60)); + } + let interval = interval + .filter(|interval| *interval != 0) + .ok_or("scanner schedule has no positive interval")?; + let next_cycle = cycle_end + .checked_add(interval) + .ok_or("scanner schedule timestamp overflows")?; + let budget = next_cycle + .saturating_sub(now) + .checked_add(60) + .ok_or("scanner completion budget overflows")?; + Ok(Duration::from_secs(budget)) + } + + #[test] + fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() { + assert_eq!( + scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false), + Ok(Duration::from_secs(259)), + ); + assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60))); + } + + #[test] + fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() { + assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60))); + assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60))); + } + + #[test] + fn scanner_cycle_completion_budget_rejects_invalid_schedule() { + for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] { + assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err()); + } + } + async fn wait_for_scanner_cycle_after( cluster: &RustFSTestClusterEnvironment, previous_cycle_end: u64, ) -> Result> { - let deadline = Instant::now() + Duration::from_secs(60); + let mut deadline = Instant::now() + Duration::from_secs(60); + let mut schedule_accounted = false; loop { let mut latest_cycle_end = 0; let mut versions_observed = false; @@ -1237,14 +1285,36 @@ mod tests { let versions_scanned = metrics["versions_scanned"] .as_u64() .ok_or("scanner status is missing its version-coverage counter")?; - latest_cycle_end = latest_cycle_end.max(cycle_end); + let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs(); + if cycle_end > now { + return Err("scanner cycle timestamp is in the future".into()); + } + let schedule = &status["cycle_schedule"]; + if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") { + let active = metrics["current_cycle_active"] + .as_bool() + .ok_or("scanner status is missing its active-cycle flag")?; + if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) { + let interval = schedule["effective_interval_seconds"].as_u64(); + let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?; + deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?; + schedule_accounted = true; + } + } + // Superseded walks complete all sets while withholding quota authority. + // Deferred and partial attempts do not certify completed scanning. + if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) { + latest_cycle_end = latest_cycle_end.max(cycle_end); + } versions_observed |= versions_scanned > 0; observations.push(format!( - "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}", + "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}", metrics["current_cycle"], metrics["current_cycle_active"], metrics["leader_lock_state"], metrics["last_cycle_result"], + metrics["last_cycle_partial_reason"], + schedule, )); } // The coordinator records cycle completion, but remote workers @@ -1555,7 +1625,8 @@ mod tests { } if background_enabled { - wait_for_scanner_cycle_after(&cluster, 0).await?; + let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs(); + wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?; } cluster.stop_node(1)?; @@ -1898,11 +1969,16 @@ mod tests { let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?; let status: serde_json::Value = serde_json::from_str(&status_body) .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?; - if background_enabled || only_admin_heal_is_active(&status) { + let admin_active = background_rejoin_heal_evidence + || (status["state"].as_str() == Some("active") + && status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1)); + if admin_active && (background_enabled || only_admin_heal_is_active(&status)) { break status; } if Instant::now() >= partial_deadline { - return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into()); + return Err( + format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(), + ); } sleep(Duration::from_millis(50)).await; }; @@ -2562,4 +2638,157 @@ mod tests { }; assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot)); } + async fn two_node_heal_status( + cluster: &RustFSTestClusterEnvironment, + bucket: Option<&str>, + recreate: bool, + scan_mode: u8, + ) -> Result> { + let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default()); + let options = serde_json::json!({ + "recursive": true, "dryRun": false, "remove": false, + "recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false, + }); + let started = + start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?; + let started: serde_json::Value = serde_json::from_str(&started)?; + let token = started["clientToken"] + .as_str() + .ok_or_else(|| format!("missing heal token: {started}"))?; + let status_url = format!("{url}?clientToken={token}"); + let deadline = Instant::now() + Duration::from_secs(240); + loop { + let body = timeout( + Duration::from_secs(20), + signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key), + ) + .await??; + let status: serde_json::Value = serde_json::from_str(&body)?; + match status["summary"].as_str() { + Some("finished" | "stopped") => return Ok(status), + Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await, + _ => return Err(format!("heal did not reach a terminal state: {status}").into()), + } + } + } + + async fn verify_two_node_directory_recovery( + remove_bucket: bool, + scan_mode: u8, + bucket_scoped: bool, + recreate: bool, + ) -> Result<(), Box> { + init_logging(); + let mut cluster = RustFSTestClusterEnvironment::new(2).await?; + // Isolate explicit administrator recovery from scanner, MRF, and disk healing. + for key in [ + "RUSTFS_HEAL_AUTO_HEAL_ENABLE", + "RUSTFS_HEAL_MRF_ENABLE", + "RUSTFS_SCANNER_ENABLED", + "RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", + ] { + cluster.set_env(key, "false"); + } + cluster.start().await?; + let source = cluster.create_s3_client(0)?; + let target = cluster.create_s3_client(1)?; + let bucket = "two-node-directory-recovery"; + source.create_bucket().bucket(bucket).send().await?; + let objects = [ + ("empty", Vec::new()), + ("nested/inline", vec![0x3c; 1024]), + ("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::>()), + ]; + for (key, bytes) in &objects { + source + .put_object() + .bucket(bucket) + .key(*key) + .body(ByteStream::from(bytes.clone())) + .send() + .await?; + } + let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket); + let original = objects + .iter() + .map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)) + .collect::, _>>()?; + for census in &original { + assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}"); + } + assert!( + !original[2].present_part_fingerprints.is_empty(), + "large fixture must exercise external parts" + ); + let removed = Path::new(&cluster.temp_dir).join("removed"); + tokio::fs::create_dir(&removed).await?; + if remove_bucket { + tokio::fs::rename(&target_bucket, removed.join(bucket)).await?; + } else { + for (index, (key, _)) in objects.iter().enumerate() { + tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?; + } + } + let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?; + assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}"); + assert_eq!(status["outcome"]["execution"]["state"], "completed"); + assert_eq!(status["outcome"]["coverage"], "complete"); + for ((key, _), before) in objects.iter().zip(&original) { + let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?; + assert!(after.has_xl_meta, "metadata must be restored before GET: {key}"); + assert_eq!( + after.present_part_fingerprints, before.present_part_fingerprints, + "external shard bytes must match: {key}" + ); + assert_eq!( + after.inline_data_fingerprint, before.inline_data_fingerprint, + "inline shard bytes must match: {key}" + ); + } + cluster.stop_node(0)?; + for (key, expected) in &objects { + let actual = timeout(Duration::from_secs(30), async { + target + .get_object() + .bucket(bucket) + .key(*key) + .send() + .await? + .body + .collect() + .await + .map(|body| body.into_bytes()) + .map_err(Box::::from) + }) + .await??; + assert_eq!(actual.len(), expected.len(), "target-only read length: {key}"); + assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}"); + } + Ok(()) + } + + #[tokio::test] + async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box> { + verify_two_node_directory_recovery(true, 2, true, true).await + } + + #[tokio::test] + async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box> { + verify_two_node_directory_recovery(true, 2, false, false).await + } + + #[tokio::test] + async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box> { + verify_two_node_directory_recovery(true, 2, true, false).await + } + + #[tokio::test] + async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box> { + verify_two_node_directory_recovery(false, 1, false, false).await + } + + #[tokio::test] + async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box> { + verify_two_node_directory_recovery(false, 2, false, false).await + } } diff --git a/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs b/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs index a311aa76e..dcc911e77 100644 --- a/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs +++ b/crates/e2e_test/src/internode_rpc_signature_e2e_test.rs @@ -264,7 +264,7 @@ async fn call_make_volume_response( request: MakeVolumeRequest, headers: HeaderMap, ) -> Result, Status> { - let mut client = node_service_time_out_client_no_auth(&url.to_string()) + let mut client = node_service_time_out_client_no_auth(url) .await .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?; let mut rpc_request = Request::new(request); @@ -273,7 +273,7 @@ async fn call_make_volume_response( } async fn call_ping_response(url: &str, headers: HeaderMap) -> Result, Status> { - let mut client = node_service_time_out_client_no_auth(&url.to_string()) + let mut client = node_service_time_out_client_no_auth(url) .await .map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?; let mut rpc_request = Request::new(PingRequest { diff --git a/crates/e2e_test/src/lib.rs b/crates/e2e_test/src/lib.rs index b4e0f4ce9..939544ed7 100644 --- a/crates/e2e_test/src/lib.rs +++ b/crates/e2e_test/src/lib.rs @@ -138,6 +138,10 @@ mod admin_auth_test; #[cfg(test)] mod existing_object_tag_policy_test; +/// IAM / bucket / STS session policy with `s3:if-none-match` / `s3:if-match` conditions (E2E). +#[cfg(test)] +mod conditional_write_policy_test; + #[cfg(test)] mod sts_query_compat_test; @@ -179,6 +183,9 @@ mod compression_test; #[cfg(test)] mod delete_objects_versioning_test; +#[cfg(test)] +mod conditional_delete_objects_test; + #[cfg(test)] mod delete_authorization_test; diff --git a/crates/e2e_test/src/multipart_auth_test.rs b/crates/e2e_test/src/multipart_auth_test.rs index 613bf0839..4ce076808 100644 --- a/crates/e2e_test/src/multipart_auth_test.rs +++ b/crates/e2e_test/src/multipart_auth_test.rs @@ -1001,6 +1001,382 @@ async fn test_anonymous_post_object_rejects_sse_kms_params_outside_policy_condit Ok(()) } +#[tokio::test] +async fn test_object_creation_operation_separates_staging_from_publication() -> Result<(), Box> { + use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; + init_logging(); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let client = env.create_s3_client(); + let bucket = "object-creation-operation"; + client.create_bucket().bucket(bucket).send().await?; + client + .put_object() + .bucket(bucket) + .key("source") + .body(ByteStream::from_static(b"copied-part")) + .send() + .await?; + let policy = |deny_creation: bool| { + serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Principal": "*", "Action": "s3:PutObject", "Resource": format!("arn:aws:s3:::{bucket}/*") }, + { "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": format!("arn:aws:s3:::{bucket}/target*"), + "Condition": { "Bool": { "s3:ObjectCreationOperation": deny_creation.to_string() } } }, + { "Effect": "Deny", "Principal": "*", "Action": "s3:GetObject", "Resource": format!("arn:aws:s3:::{bucket}/*"), + "Condition": { "Null": { "s3:ObjectCreationOperation": "false" } } } + ] + }).to_string() + }; + client.put_bucket_policy().bucket(bucket).policy(policy(true)).send().await?; + let upload = client + .create_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .send() + .await?; + let upload_id = upload.upload_id().expect("multipart initiation should return upload ID"); + let first_body = vec![b'a'; 5 * 1024 * 1024]; + let part = client + .upload_part() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(1) + .body(ByteStream::from(first_body.clone())) + .send() + .await?; + let copy = client + .upload_part_copy() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(2) + .copy_source(format!("{bucket}/source")) + .send() + .await?; + let completed = CompletedMultipartUpload::builder() + .parts( + CompletedPart::builder() + .part_number(1) + .e_tag(part.e_tag().expect("part ETag")) + .build(), + ) + .parts( + CompletedPart::builder() + .part_number(2) + .e_tag(copy.copy_part_result().and_then(|part| part.e_tag()).expect("copy part ETag")) + .build(), + ) + .build(); + let error = client + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed.clone()) + .send() + .await + .expect_err("publication must be denied"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let error = client + .put_object() + .bucket(bucket) + .key("target-put") + .body(ByteStream::from_static(b"blocked")) + .customize() + .mutate_request(|req| { + req.headers_mut().insert("objectcreationoperation", "false"); + }) + .send() + .await + .expect_err("a spoofed header must not exempt PutObject"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let error = client + .copy_object() + .bucket(bucket) + .key("target-copy") + .copy_source(format!("{bucket}/source")) + .send() + .await + .expect_err("CopyObject publishes an object"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let archive = make_tar(&[("target-member", b"blocked member")], &[]).await; + let error = client + .put_object() + .bucket(bucket) + .key("archive.tar") + .body(ByteStream::from(archive)) + .customize() + .mutate_request(|req| { + req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true"); + }) + .send() + .await + .expect_err("each extracted member must authorize object creation"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + let missing = client + .head_object() + .bucket(bucket) + .key("target-member") + .send() + .await + .expect_err("denied archive member must not exist"); + assert!(missing.as_service_error().is_some_and(|error| error.is_not_found())); + for deny_creation in [true, false] { + client + .put_bucket_policy() + .bucket(bucket) + .policy(policy(deny_creation)) + .send() + .await?; + let form = reqwest::multipart::Form::new() + .text("key", "target-post") + .text("success_action_status", "201") + .part( + "file", + reqwest::multipart::Part::bytes(b"post body".to_vec()) + .file_name("upload.txt") + .mime_str("text/plain")?, + ); + let response = local_http_client() + .post(format!("{}/{bucket}", env.url)) + .multipart(form) + .send() + .await?; + assert_eq!( + response.status(), + if deny_creation { + reqwest::StatusCode::FORBIDDEN + } else { + reqwest::StatusCode::CREATED + }, + "POST object creation={deny_creation}" + ); + } + let error = client + .create_multipart_upload() + .bucket(bucket) + .key("target-denied") + .send() + .await + .expect_err("Bool=false must deny multipart staging under the inverted policy"); + assert_eq!(error.as_service_error().and_then(|error| error.code()), Some("AccessDenied")); + client + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed) + .send() + .await?; + let object = client + .get_object() + .bucket(bucket) + .key("target-multipart") + .send() + .await? + .body + .collect() + .await? + .into_bytes(); + let mut expected = first_body; + expected.extend_from_slice(b"copied-part"); + assert_eq!(object.as_ref(), expected, "completion must publish both regular and copied parts"); + Ok(()) +} + +/// A vended STS credential whose inline session policy follows AWS's conditional-write +/// example 1: object publication needs a request header, multipart staging is exempt +/// through `s3:ObjectCreationOperation`. `s3:x-amz-storage-class` stands in for +/// `s3:if-none-match` because only the former is a header SDK requests can carry here. +#[tokio::test] +async fn test_object_creation_operation_in_sts_session_policy() -> Result<(), Box> { + use crate::common::{ + AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, + awscurl_post_sts_form_urlencoded, build_test_s3_config, + }; + use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart, StorageClass}; + init_logging(); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; + let admin = env.create_s3_client(); + let bucket = "object-creation-sts"; + admin.create_bucket().bucket(bucket).send().await?; + admin + .put_object() + .bucket(bucket) + .key("source") + .body(ByteStream::from_static(b"copied-part")) + .send() + .await?; + + let (parent, parent_secret, parent_policy) = ("objcreate-sts-parent", "objcreate-sts-parent-secret", "objcreate-sts-rw"); + let (url, access_key, secret_key) = (env.url.as_str(), env.access_key.as_str(), env.secret_key.as_str()); + admin_create_user_via(AdminTransport::Signed, url, access_key, secret_key, parent, parent_secret).await?; + let read_write = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["s3:*"], "Resource": ["arn:aws:s3:::*"] }, + { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": ["arn:aws:s3:::*"] } + ] + }) + .to_string(); + admin_add_canned_policy_via(AdminTransport::Signed, url, access_key, secret_key, parent_policy, &read_write).await?; + admin_attach_user_policy_via(AdminTransport::Signed, url, access_key, secret_key, parent_policy, parent).await?; + + let objects = format!("arn:aws:s3:::{bucket}/*"); + let session_policy = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": "s3:PutObject", "Resource": objects, + "Condition": { "Null": { "s3:x-amz-storage-class": "false" } } }, + { "Effect": "Allow", "Action": "s3:PutObject", "Resource": objects, + "Condition": { "Bool": { "s3:ObjectCreationOperation": "false" } } }, + { "Effect": "Allow", "Action": "s3:GetObject", "Resource": objects } + ] + }) + .to_string(); + let body = format!( + "Action=AssumeRole&Version=2011-06-15&DurationSeconds=3600&Policy={}", + urlencoding::encode(&session_policy) + ); + let xml = + awscurl_post_sts_form_urlencoded(&format!("{}/", env.url.trim_end_matches('/')), &body, parent, parent_secret).await?; + let field = |tag: &str| -> Result> { + let open = format!("<{tag}>"); + let start = xml.find(&open).ok_or(format!("missing {tag} in AssumeRole response"))? + open.len(); + let end = xml[start..].find(&format!("")).ok_or(format!("unterminated {tag}"))? + start; + Ok(xml[start..end].to_string()) + }; + let session = aws_sdk_s3::Client::from_conf(build_test_s3_config( + &env.url, + &field("AccessKeyId")?, + &field("SecretAccessKey")?, + Some(&field("SessionToken")?), + "object-creation-sts-session", + )); + let denied = |error: Option<&str>, what: &str| assert_eq!(error, Some("AccessDenied"), "{what}"); + + let error = session + .put_object() + .bucket(bucket) + .key("target-put") + .body(ByteStream::from_static(b"blocked")) + .send() + .await + .expect_err("PutObject without the header must be denied"); + denied(error.as_service_error().and_then(|error| error.code()), "PutObject without the header"); + session + .put_object() + .bucket(bucket) + .key("target-put") + .storage_class(StorageClass::Standard) + .body(ByteStream::from_static(b"allowed")) + .send() + .await?; + let error = session + .copy_object() + .bucket(bucket) + .key("target-copy") + .copy_source(format!("{bucket}/source")) + .send() + .await + .expect_err("CopyObject without the header must be denied"); + denied(error.as_service_error().and_then(|error| error.code()), "CopyObject without the header"); + session + .copy_object() + .bucket(bucket) + .key("target-copy") + .copy_source(format!("{bucket}/source")) + .storage_class(StorageClass::Standard) + .send() + .await?; + + let upload = session + .create_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .send() + .await?; + let upload_id = upload.upload_id().expect("multipart initiation should return upload ID"); + let first_body = vec![b'a'; 5 * 1024 * 1024]; + let part = session + .upload_part() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(1) + .body(ByteStream::from(first_body.clone())) + .send() + .await?; + let copy = session + .upload_part_copy() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .part_number(2) + .copy_source(format!("{bucket}/source")) + .send() + .await?; + let completed = CompletedMultipartUpload::builder() + .parts( + CompletedPart::builder() + .part_number(1) + .e_tag(part.e_tag().expect("part ETag")) + .build(), + ) + .parts( + CompletedPart::builder() + .part_number(2) + .e_tag(copy.copy_part_result().and_then(|part| part.e_tag()).expect("copy part ETag")) + .build(), + ) + .build(); + let error = session + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed.clone()) + .send() + .await + .expect_err("CompleteMultipartUpload without the header must be denied"); + denied( + error.as_service_error().and_then(|error| error.code()), + "CompleteMultipartUpload without the header", + ); + session + .complete_multipart_upload() + .bucket(bucket) + .key("target-multipart") + .upload_id(upload_id) + .multipart_upload(completed) + .customize() + .mutate_request(|req| { + req.headers_mut() + .insert("x-amz-storage-class", HeaderValue::from_static("STANDARD")); + }) + .send() + .await?; + + let object = session + .get_object() + .bucket(bucket) + .key("target-multipart") + .send() + .await? + .body + .collect() + .await? + .into_bytes(); + let mut expected = first_body; + expected.extend_from_slice(b"copied-part"); + assert_eq!(object.as_ref(), expected, "completion must publish both regular and copied parts"); + Ok(()) +} + #[tokio::test] async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box> { init_logging(); diff --git a/crates/e2e_test/src/namespace_lock_quorum_test.rs b/crates/e2e_test/src/namespace_lock_quorum_test.rs index 1edf47c2b..7f9e8c760 100644 --- a/crates/e2e_test/src/namespace_lock_quorum_test.rs +++ b/crates/e2e_test/src/namespace_lock_quorum_test.rs @@ -195,6 +195,91 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes } } +#[tokio::test] +async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult { + crate::common::init_logging(); + let mut cluster = RustFSTestClusterEnvironment::new(4).await?; + cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2"); + cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false"); + cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false"); + cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false"); + cluster.start().await?; + let clients = cluster + .create_all_clients()? + .into_iter() + .map(|client| { + Client::from_conf( + client + .config() + .to_builder() + .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1)) + .build(), + ) + }) + .collect::>(); + let bucket = "cold-degraded-mutations"; + clients[0].create_bucket().bucket(bucket).send().await?; + // No object mutation may warm a process-wide configuration probe before + // the missing peer is introduced. Three shards still satisfy EC:2 writes. + cluster.stop_node(3)?; + let admission_deadline = Instant::now() + Duration::from_secs(30); + for client in &clients[..3] { + loop { + match client + .get_object() + .bucket(bucket) + .key("never-created-readiness-probe") + .send() + .await + { + Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break, + Err(error) + if error.raw_response().is_some_and(|response| response.status().as_u16() == 503) + && Instant::now() < admission_deadline => + { + tokio::time::sleep(Duration::from_millis(100)).await; + } + Err(error) => return Err(error.into()), + Ok(_) => panic!("a cold readiness probe must not create an object"), + } + } + } + let body = b"first write while one peer is offline"; + clients[0] + .put_object() + .bucket(bucket) + .key("object") + .body(Bytes::from_static(body).into()) + .send() + .await?; + for client in &clients[..3] { + assert_quorum_object_body(client, bucket, "object", body).await?; + } + // This coordinator has not previously committed an object mutation either. + clients[1].delete_object().bucket(bucket).key("object").send().await?; + let error = clients[2] + .get_object() + .bucket(bucket) + .key("object") + .send() + .await + .expect_err("acknowledged deletion must be visible at another entrypoint"); + assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey")); + + cluster.stop_node(2)?; + let error = clients[0] + .put_object() + .bucket(bucket) + .key("below-quorum") + .body(Bytes::from_static(body).into()) + .send() + .await + .expect_err("removing the cache gate must not weaken storage write quorum"); + assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable")); + cluster.stop(); + Ok(()) +} + #[tokio::test] async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult { crate::common::init_logging(); diff --git a/crates/e2e_test/src/quota_test.rs b/crates/e2e_test/src/quota_test.rs index 7acb51e02..eb5593607 100644 --- a/crates/e2e_test/src/quota_test.rs +++ b/crates/e2e_test/src/quota_test.rs @@ -851,13 +851,20 @@ mod integration_tests { .delete( aws_sdk_s3::types::Delete::builder() .set_objects(Some(objects)) - .quiet(true) + // Quiet mode hides successful entries, so the batch must be + // verbose for the assertion on the deleted list below. + .quiet(false) .build() .unwrap(), ) .send() .await?; + assert!( + delete_result.errors().is_empty(), + "batch delete must not report errors: {:?}", + delete_result.errors() + ); assert_eq!(delete_result.deleted().len(), 2); // Now should be able to upload again (quota freed up) diff --git a/crates/e2e_test/src/reliant/conditional_writes.rs b/crates/e2e_test/src/reliant/conditional_writes.rs index 6bc37980c..60505eb35 100644 --- a/crates/e2e_test/src/reliant/conditional_writes.rs +++ b/crates/e2e_test/src/reliant/conditional_writes.rs @@ -70,7 +70,7 @@ async fn test_conditional_put_okay() -> TestResult { let test_key = "conditional-put-ok"; let initial_data = generate_test_data(1024); // 1KB test data let matching_data = generate_test_data(2048); // 2KB updated data - let non_matching_data = generate_test_data(3072); // 3KB updated data + let created_data = generate_test_data(3072); // 3KB new object // Upload initial object and get its ETag let initial_etag = upload_object_with_metadata(&client, TEST_BUCKET, test_key, &initial_data).await?; @@ -86,17 +86,17 @@ async fn test_conditional_put_okay() -> TestResult { .await?; assert_eq!(object_body(&client, test_key).await?.as_ref(), matching_data); - // Test 2: PUT with non-matching If-None-Match condition (should succeed) - let fake_etag = "\"fake-etag-12345\""; + // Test 2: PUT with If-None-Match: * on a missing key (should succeed) + let created_key = "conditional-put-ok-created"; client .put_object() .bucket(TEST_BUCKET) - .key(test_key) - .body(Bytes::from(non_matching_data.clone()).into()) - .if_none_match(fake_etag) + .key(created_key) + .body(Bytes::from(created_data.clone()).into()) + .if_none_match("*") .send() .await?; - assert_eq!(object_body(&client, test_key).await?.as_ref(), non_matching_data); + assert_eq!(object_body(&client, created_key).await?.as_ref(), created_data); Ok(()) } @@ -130,17 +130,30 @@ async fn test_conditional_put_failed() -> TestResult { assert_s3_error_code(response1, "PreconditionFailed"); assert_eq!(object_body(&client, test_key).await?.as_ref(), initial_data); - // Test 2: PUT with matching If-None-Match condition (should fail with 412) - let response2 = client - .put_object() + // Test 2: writes accept If-None-Match only as `*`; RustFS rejects an ETag + // value with 501 NotImplemented whether or not it matches. + for if_none_match in [initial_etag.as_str(), "\"fake-etag-12345\""] { + let response = client + .put_object() + .bucket(TEST_BUCKET) + .key(test_key) + .body(Bytes::from(updated_data.clone()).into()) + .if_none_match(if_none_match) + .send() + .await; + assert_s3_error_code(response, "NotImplemented"); + } + + let response = client + .copy_object() .bucket(TEST_BUCKET) .key(test_key) - .body(Bytes::from(updated_data.clone()).into()) - .if_none_match(&initial_etag) + .copy_source(format!("{TEST_BUCKET}/{test_key}")) + .metadata_directive(aws_sdk_s3::types::MetadataDirective::Replace) + .if_none_match("\"fake-etag-12345\"") .send() .await; - - assert_s3_error_code(response2, "PreconditionFailed"); + assert_s3_error_code(response, "NotImplemented"); assert_eq!(object_body(&client, test_key).await?.as_ref(), initial_data); Ok(()) @@ -252,7 +265,7 @@ async fn test_conditional_multi_part_upload() -> TestResult { assert_s3_error_code(complete_response, "PreconditionFailed"); - // Test 2: Multipart upload with matching If-None-Match, should fail + // Test 2: Multipart upload with an unsupported ETag If-None-Match let complete_response = client .complete_multipart_upload() .bucket(TEST_BUCKET) @@ -263,7 +276,7 @@ async fn test_conditional_multi_part_upload() -> TestResult { .send() .await; - assert_s3_error_code(complete_response, "PreconditionFailed"); + assert_s3_error_code(complete_response, "NotImplemented"); // Test 3: Multipart upload with unmatching If-Match, should fail let complete_response = client diff --git a/crates/e2e_test/src/reliant/grpc_lock_server.rs b/crates/e2e_test/src/reliant/grpc_lock_server.rs index b97e96555..0abd3cc30 100644 --- a/crates/e2e_test/src/reliant/grpc_lock_server.rs +++ b/crates/e2e_test/src/reliant/grpc_lock_server.rs @@ -113,6 +113,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) } + async fn acquire_quota_mutation_fence( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) + } + async fn renew_snapshot_lease( &self, _request: Request, @@ -127,6 +134,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) } + async fn release_quota_mutation_fence( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("MinimalLockNodeService only supports lock RPCs")) + } + async fn acquire_scanner_publication_lease( &self, _request: Request, @@ -541,6 +555,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("lock-only test server")) } + async fn rename_data_with_quota_fence( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("lock-only test server")) + } + async fn make_volumes( &self, _request: Request, @@ -639,6 +660,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("lock-only test server")) } + async fn guarded_delete_versions( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("lock-only test server")) + } + async fn delete_versions( &self, _request: Request, @@ -933,6 +961,13 @@ impl NodeService for MinimalLockNodeService { Err(Status::unimplemented("lock-only test server")) } + async fn peer_drive_admission( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented("lock-only test server")) + } + async fn get_metacache_listing( &self, _request: Request, diff --git a/crates/e2e_test/src/replication_extension_test.rs b/crates/e2e_test/src/replication_extension_test.rs index ab98d0cb0..e4a4c72a1 100644 --- a/crates/e2e_test/src/replication_extension_test.rs +++ b/crates/e2e_test/src/replication_extension_test.rs @@ -14,8 +14,8 @@ use crate::common::{ AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user, - awscurl_post_sts_form_urlencoded, init_logging, local_http_client, replication_fast_env, rustfs_binary_path, signed_request, - signed_request_with_client, signed_request_with_session_token, + awscurl_post_sts_form_urlencoded, init_logging, local_http_client, parse_assume_role_credentials, replication_fast_env, + rustfs_binary_path, signed_request, signed_request_with_client, signed_request_with_session_token, }; use crate::fake_s3_target::{ FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation, @@ -387,21 +387,6 @@ pub(crate) struct ReplicationResetStatusTarget { pub(crate) object: String, } -fn extract_xml_tag(xml: &str, tag: &str) -> Option { - let open = format!("<{tag}>"); - let close = format!(""); - let start = xml.find(&open)? + open.len(); - let end = xml[start..].find(&close)? + start; - Some(xml[start..end].to_string()) -} - -fn parse_assume_role_credentials(xml: &str) -> Result<(String, String, String), Box> { - let access_key = extract_xml_tag(xml, "AccessKeyId").ok_or("missing AccessKeyId in AssumeRole response")?; - let secret_key = extract_xml_tag(xml, "SecretAccessKey").ok_or("missing SecretAccessKey in AssumeRole response")?; - let session_token = extract_xml_tag(xml, "SessionToken").ok_or("missing SessionToken in AssumeRole response")?; - Ok((access_key, secret_key, session_token)) -} - pub(crate) struct ReplicationTargetOptions<'a> { pub(crate) endpoint: &'a str, pub(crate) access_key: &'a str, @@ -3945,7 +3930,15 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou (sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()), (large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()), ]; - assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?; + // Managed-SSE listings may omit ETags. Check the replicated version's body + // before deleting it rather than comparing keyless listing fingerprints. + for (key, version_id) in &purged { + let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?; + let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into(); + wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?; + let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?; + assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}"); + } for (key, version_id) in &purged { source_client @@ -3956,10 +3949,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou .send() .await?; } - assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?; - let target_state = list_replication_state(&target_client, &target_bucket).await?; - assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}"); - // The purge state is applied on the source asynchronously after the target // acknowledges the delete; only then does the retained version go away and // the bucket become deletable. A listing that is empty while DeleteBucket @@ -3984,6 +3973,9 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou } } + let target_state = list_replication_state(&target_client, &target_bucket).await?; + assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}"); + Ok(()) } diff --git a/crates/e2e_test/src/replication_target_matrix_test.rs b/crates/e2e_test/src/replication_target_matrix_test.rs index 2a3cb973e..743650e72 100644 --- a/crates/e2e_test/src/replication_target_matrix_test.rs +++ b/crates/e2e_test/src/replication_target_matrix_test.rs @@ -45,9 +45,10 @@ use aws_sdk_s3::error::ProvideErrorMetadata; use aws_sdk_s3::primitives::{ByteStream, DateTime}; use aws_sdk_s3::types::{ Checksum, ChecksumAlgorithm, CompletedMultipartUpload, CompletedPart, ObjectAttributes, ObjectLockLegalHold, - ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, Tag, Tagging, + ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ServerSideEncryption, Tag, Tagging, }; use bytes::Bytes; +use md5::{Digest as _, Md5}; use std::error::Error; use std::time::{SystemTime, UNIX_EPOCH}; use tokio::time::{Duration, sleep, timeout}; @@ -73,14 +74,20 @@ enum TargetMode { /// land, and every version-addressed mutation must resolve the replica /// through the target-version ledger. MintOwnVersionIds, + /// AWS S3 and other generic S3 services: ignores the replication + /// source-ETag headers and reports the MD5 of the plaintext it received. + /// Every single-part replica ETag is therefore checked against the source + /// ETag for real (the other modes echo the source ETag back). + ComputesOwnEtags, } impl TargetMode { - const ALL: [TargetMode; 4] = [ + const ALL: [TargetMode; 5] = [ TargetMode::Baseline, TargetMode::RejectAwsChunked, TargetMode::RequireChecksumWithObjectLock, TargetMode::MintOwnVersionIds, + TargetMode::ComputesOwnEtags, ]; fn apply(self, target: &FakeS3Target) { @@ -92,6 +99,7 @@ impl TargetMode { target.assign_own_version_ids(true); target.reject_unknown_version_deletes(true); } + TargetMode::ComputesOwnEtags => target.ignore_source_etags(true), } } @@ -101,6 +109,7 @@ impl TargetMode { TargetMode::RejectAwsChunked => "reject-aws-chunked", TargetMode::RequireChecksumWithObjectLock => "require-checksum-object-lock", TargetMode::MintOwnVersionIds => "mint-own-version-ids", + TargetMode::ComputesOwnEtags => "computes-own-etags", } } } @@ -126,10 +135,14 @@ enum ObjectShape { /// Single-part object uploaded with `x-amz-checksum-sha256`; the replica /// must carry the same header (rustfs/backlog#2340). Checksummed, + /// Single-part SSE-S3 object. Its ETag is a keyed fingerprint, not the + /// plaintext MD5 (docs/operations/encrypted-etags.md), while the replica + /// travels as plaintext under `x-amz-server-side-encryption: AES256`. + SseS3, } impl ObjectShape { - const ALL: [ObjectShape; 8] = [ + const ALL: [ObjectShape; 9] = [ ObjectShape::Empty, ObjectShape::Plain, ObjectShape::Retention, @@ -138,6 +151,7 @@ impl ObjectShape { ObjectShape::LockedMultipart, ObjectShape::OdmPreservedMd5Multipart, ObjectShape::Checksummed, + ObjectShape::SseS3, ]; fn key(self) -> &'static str { @@ -150,6 +164,7 @@ impl ObjectShape { ObjectShape::LockedMultipart => "matrix/locked-multipart.bin", ObjectShape::OdmPreservedMd5Multipart => "matrix/odm-preserved-md5.bin", ObjectShape::Checksummed => "matrix/checksummed.bin", + ObjectShape::SseS3 => "matrix/sse-s3.bin", } } @@ -233,6 +248,24 @@ impl ObjectShape { .await?; Ok(body) } + ObjectShape::SseS3 => { + let body = payload(56 * 1024, 0x77); + let put = client + .put_object() + .bucket(bucket) + .key(key) + .body(ByteStream::from(body.clone())) + .server_side_encryption(ServerSideEncryption::Aes256) + .send() + .await?; + // The cell only proves something while the source ETag is + // not the content MD5 a generic target reports back. + let plaintext_md5 = hex_simd::encode_to_string(Md5::digest(&body), hex_simd::AsciiCase::Lower); + if put.e_tag().map(|etag| etag.trim_matches('"')) == Some(plaintext_md5.as_str()) { + return Err("SSE-S3 source ETag is the plaintext MD5; the cell no longer exercises encrypted ETags".into()); + } + Ok(body) + } } } } @@ -270,14 +303,40 @@ fn expectation(mode: TargetMode, shape: ObjectShape) -> Expectation { /// - the first PUT lands but its response is lost, so the object is FAILED /// and the scanner heal pass re-drives it; /// - an existing-object resync re-drives a COMPLETED object unconditionally. +/// +/// A permanent delete of the healed version then has to remove exactly the +/// replica the heal pass located. #[tokio::test] async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult { + run_redrive_does_not_duplicate(RedriveObjects::Plaintext).await +} + +/// The same re-drives for SSE-S3 objects against a target that also computes +/// its own ETags (AWS S3). The source ETag is a keyed fingerprint +/// (docs/operations/encrypted-etags.md) while the replica reports the MD5 of +/// the plaintext it received, so no exact-ETag lookup can ever find the +/// replica: the lost-response heal, the resync and the purge must locate it +/// without comparing ETags. +#[tokio::test] +async fn matrix_mint_own_version_ids_encrypted_redrive_does_not_duplicate() -> TestResult { + run_redrive_does_not_duplicate(RedriveObjects::SseS3OnEtagComputingTarget).await +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum RedriveObjects { + Plaintext, + SseS3OnEtagComputingTarget, +} + +async fn run_redrive_does_not_duplicate(objects: RedriveObjects) -> TestResult { init_logging(); + let encrypted = objects == RedriveObjects::SseS3OnEtagComputingTarget; let target = FakeS3Target::start().await?; let target_bucket = "matrix-mint-own-redrive-dst".to_string(); target.create_bucket_with_object_lock(target_bucket.clone()); target.assign_own_version_ids(true); + target.ignore_source_etags(encrypted); let mut env_vars = replication_fast_env(); env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV); @@ -291,6 +350,7 @@ async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult ]); let env = OdmTestEnv::start_with(OdmEnvOptions { env: env_vars, + local_kms: encrypted, ..OdmEnvOptions::default() }) .await?; @@ -319,45 +379,63 @@ async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult }, ) .await?; - put_bucket_replication(source_env, source_bucket, &target_arn).await?; + put_bucket_replication_with_delete_statuses(source_env, source_bucket, &target_arn, "Enabled", Some("Enabled")).await?; + + let put = |key: &'static str, fill: u8, len: usize| { + let request = source_client + .put_object() + .bucket(source_bucket) + .key(key) + .body(ByteStream::from(payload(len, fill))); + let request = if encrypted { + request.server_side_encryption(ServerSideEncryption::Aes256) + } else { + request + }; + async move { request.send().await } + }; // Teach the worker the target's identity contract with one ordinary // write, exactly as production learns it (the PUT response carries the // minted id). let probe_key = "redrive/identity-probe.bin"; - source_client - .put_object() - .bucket(source_bucket) - .key(probe_key) - .body(ByteStream::from(payload(4 * 1024, 0x01))) - .send() - .await?; + let probe = put(probe_key, 0x01, 4 * 1024).await?; assert_eq!( wait_for_terminal_replication_status(&source_client, source_bucket, probe_key).await?, "COMPLETED" ); + if encrypted { + // The variant only proves something while the replica's ETag differs + // from the source's encrypted-ETag fingerprint. + let replica = single_target_version(&target, &target_bucket, probe_key)?; + let replica_etag = fake_source_client(&target) + .head_object() + .bucket(&target_bucket) + .key(probe_key) + .version_id(&replica) + .send() + .await? + .e_tag() + .map(str::to_string); + if replica_etag.is_none() || replica_etag.as_deref() == probe.e_tag() { + return Err(format!( + "the replica ETag {replica_etag:?} must differ from the source fingerprint {:?}", + probe.e_tag() + ) + .into()); + } + } // Shape 1: the PUT is stored, its response never arrives, heal re-drives. let heal_key = "redrive/heal.bin"; target.inject_for_key(FakeTargetOperation::PutObject, heal_key, FakeTargetFault::DisconnectAfterResponse, 1); - source_client - .put_object() - .bucket(source_bucket) - .key(heal_key) - .body(ByteStream::from(payload(8 * 1024, 0x02))) - .send() - .await?; + let healed = put(heal_key, 0x02, 8 * 1024).await?; + let healed_version = healed.version_id().ok_or("source PUT returned no version id")?.to_string(); wait_for_replication_status_and_single_version(&source_client, source_bucket, &target, &target_bucket, heal_key).await?; // Shape 2: an existing-object resync re-drives a COMPLETED object. let resync_key = "redrive/resync.bin"; - source_client - .put_object() - .bucket(source_bucket) - .key(resync_key) - .body(ByteStream::from(payload(8 * 1024, 0x03))) - .send() - .await?; + put(resync_key, 0x03, 8 * 1024).await?; assert_eq!( wait_for_terminal_replication_status(&source_client, source_bucket, resync_key).await?, "COMPLETED" @@ -388,6 +466,20 @@ async fn matrix_mint_own_version_ids_redrive_does_not_duplicate() -> TestResult ); } + // The heal pass recorded the replica it located; a permanent delete of + // that version must remove exactly it. + source_client + .delete_object() + .bucket(source_bucket) + .key(heal_key) + .version_id(&healed_version) + .send() + .await?; + wait_until("permanent delete of the healed replica", || async { + Ok(target.stored_versions(&target_bucket, heal_key).is_empty()) + }) + .await?; + target.shutdown().await; Ok(()) } @@ -871,6 +963,11 @@ async fn matrix_mint_own_version_ids_target() -> TestResult { run_row(TargetMode::MintOwnVersionIds).await } +#[tokio::test] +async fn matrix_computes_own_etags_target() -> TestResult { + run_row(TargetMode::ComputesOwnEtags).await +} + /// Every known-red entry must name a real cell and an issue, and the lookup /// must round-trip, so a stale or mistyped entry cannot silently pin nothing. #[test] @@ -907,6 +1004,8 @@ async fn run_row(mode: TargetMode) -> TestResult { env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]); let env = OdmTestEnv::start_with(OdmEnvOptions { env: env_vars, + // The SseS3 shape needs a KMS for SSE-S3. + local_kms: true, ..OdmEnvOptions::default() }) .await?; diff --git a/crates/e2e_test/src/upgrade_compatibility_test.rs b/crates/e2e_test/src/upgrade_compatibility_test.rs index 35e0c8eaf..01db59002 100644 --- a/crates/e2e_test/src/upgrade_compatibility_test.rs +++ b/crates/e2e_test/src/upgrade_compatibility_test.rs @@ -25,11 +25,11 @@ use aws_sdk_s3::Client; use aws_sdk_s3::error::ProvideErrorMetadata; use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::types::{ - BucketLifecycleConfiguration, BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DefaultRetention, - ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter, ObjectAttributes, ObjectLockConfiguration, - ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, PublicAccessBlockConfiguration, ServerSideEncryption, - ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, Tag, Tagging, - VersioningConfiguration, + BucketLifecycleConfiguration, BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DefaultRetention, Delete, + ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter, ObjectAttributes, ObjectIdentifier, + ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, PublicAccessBlockConfiguration, + ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, + StorageClass, Tag, Tagging, VersioningConfiguration, }; use http::{Method, StatusCode}; use serde_json::Value; @@ -46,6 +46,7 @@ type BoxError = Box; const SOURCE_BINARY_ENV: &str = "RUSTFS_UPGRADE_SOURCE_BINARY"; const G09_EVIDENCE_DIR_ENV: &str = "RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR"; const RC5_COMMIT: &str = "40a2470feb567201165a5b809b7598bb4b1f68f5"; +const STABLE_1_0_1_COMMIT: &str = "6de965ae3c965a78ff819fbcd7acd4aa44177d92"; const SSE_MASTER_KEY_ENV: &str = "RUSTFS_SSE_S3_MASTER_KEY"; const SSE_MASTER_KEY: &str = "QkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkI="; const PLAIN_BUCKET: &str = "upgrade-plain-data"; @@ -66,6 +67,16 @@ const MIXED_MULTIPART_UPLOAD_TIMEOUT: Duration = Duration::from_secs(30); // probe_interval (2s) x success_threshold (3) after it comes back; 30s // comfortably covers that window plus CI scheduling jitter. const LISTING_CONVERGENCE_TIMEOUT: Duration = Duration::from_secs(30); +// Parity 0 for the reduced-redundancy class only. A parity-0 write on the +// four-drive set needs all four drives (`write_quorum = data_drives`), so a +// PUT with that class succeeds only once the serving node writes through every +// one of its own drive handles. Standard-class objects keep the default layout. +const ALL_DRIVES_STORAGE_CLASS_ENV: (&str, &str) = ("RUSTFS_STORAGE_CLASS_RRS", "EC:0"); +// A peer drops a restarted node's drive to Offline after two failed +// operations and only re-admits it through probe_interval (2s) x +// success_threshold (3) once the node answers again. The budget covers that +// for every node after the slowest restart in the sequence, plus CI jitter. +const DRIVE_READMISSION_TIMEOUT: Duration = Duration::from_secs(60); // Bucket-configuration upgrade/rollback scenarios (rustfs#7172, #7183, #7089). const CONFIG_PLAIN_BUCKET: &str = "upgrade-config-plain"; @@ -509,6 +520,9 @@ async fn observe_mixed_multipart_failure( let network_deadline = started + Duration::from_secs(4); let path = format!("/{MIXED_BUCKET}/{}", failure.key); let _ = tokio::time::timeout_at(deadline, async { + // Every node keeps its own lock table, and the writer that stalls is + // usually a previous-release node; the current node's snapshot alone + // cannot show which holder blocks the upload. let (heads, locks) = tokio::join!( futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure( cluster, @@ -517,7 +531,13 @@ async fn observe_mixed_multipart_failure( true, network_deadline ))), - probe_mixed_multipart_failure(cluster, 0, "/rustfs/admin/v3/top/locks?count=1000", false, network_deadline), + futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure( + cluster, + node, + "/rustfs/admin/v3/top/locks?count=1000", + false, + network_deadline + ))), ); let snapshot = serde_json::json!({ "phase": phase, "bucket": MIXED_BUCKET, "key": failure.key, @@ -579,6 +599,7 @@ async fn exercise_mixed_cluster( ) -> TestResult { let clients = cluster.create_all_clients()?; wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, phase, DRIVE_READMISSION_TIMEOUT).await?; let current_client = &clients[current_node]; let previous_client = &clients[previous_node]; @@ -633,6 +654,26 @@ async fn exercise_mixed_cluster( vec![u8::try_from(MULTIPART_WORKERS - 1)?; 64 * 1024] ); + // Every online participant must understand the delete decision protocol. + // An upgrade phase can serve reads/writes while intentionally refusing a + // modern DELETE; rejecting it must leave the old writer's seed intact. + let error = upgrade_probe_client(current_client) + .delete_object() + .bucket(MIXED_BUCKET) + .key(PREVIOUS_RELEASE_SEED_KEY) + .send() + .await + .expect_err("a modern DELETE must fail closed while an old participant is online"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503), "{error:?}"); + assert_eq!(error.as_service_error().and_then(ProvideErrorMetadata::code), Some("ServiceUnavailable")); + for (node, client) in clients.iter().enumerate() { + assert_eq!( + read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1, + UPGRADE_READINESS_BODY, + "{phase}: rejected DELETE must preserve the seed through node {node}" + ); + } + Ok(()) } @@ -656,16 +697,51 @@ pub(crate) async fn wait_for_upgrade_write_readiness( // health also accepts Returning disks whose write health is still FAULTY. // Probe every writer outside the asserted phase prefix; compatibility // writes still execute once and retain their original assertions. + wait_for_upgrade_writes(clients, bucket, phase, "write readiness", None, budget).await +} + +/// Block until every node commits a parity-0 write, which needs all of the +/// node's own drive handles. +/// +/// Write readiness proves that a node accepts a standard write at quorum; it +/// does not prove that the node has re-admitted the drives it marked Offline +/// while a peer restarted. A writer that still excludes one drive commits at +/// exactly three of four drives and cannot ride out any other slow drive. The +/// cluster must run with [`ALL_DRIVES_STORAGE_CLASS_ENV`]; a write with the +/// reduced-redundancy class then fails with 503 `ServiceUnavailable` until the +/// serving node writes through every drive, on the previous release and on +/// the current build alike. +async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult { + wait_for_upgrade_writes( + clients, + bucket, + context, + "drive re-admission", + Some(StorageClass::ReducedRedundancy), + budget, + ) + .await +} + +async fn wait_for_upgrade_writes( + clients: &[Client], + bucket: &str, + phase: &str, + probe: &str, + storage_class: Option, + budget: Duration, +) -> TestResult { let deadline = Instant::now() + budget; for (node, client) in clients.iter().enumerate() { let client = upgrade_probe_client(client); - let key = format!(".upgrade-readiness/{phase}/node-{node}"); + let key = match storage_class { + Some(_) => format!(".upgrade-readiness/{phase}/all-drives/node-{node}"), + None => format!(".upgrade-readiness/{phase}/node-{node}"), + }; let mut last_response = "no response".to_string(); loop { if Instant::now() >= deadline { - return Err( - format!("{phase}: node {node} write readiness deadline exceeded; last response: {last_response}").into(), - ); + return Err(format!("{phase}: node {node} {probe} deadline exceeded; last response: {last_response}").into()); } let response = tokio::time::timeout_at( deadline, @@ -673,6 +749,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness( .put_object() .bucket(bucket) .key(&key) + .set_storage_class(storage_class.clone()) .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) .send(), ) @@ -683,13 +760,13 @@ pub(crate) async fn wait_for_upgrade_write_readiness( if error.raw_response().map(|response| response.status().as_u16()) != Some(503) || error.as_service_error().and_then(ProvideErrorMetadata::code) != Some("ServiceUnavailable") { - return Err(format!("{phase}: node {node} write readiness failed: {error:?}").into()); + return Err(format!("{phase}: node {node} {probe} failed: {error:?}").into()); } last_response = format!("{error:?}"); } Err(_) => { return Err(format!( - "{phase}: node {node} write readiness deadline exceeded during PutObject; last response: {last_response}" + "{phase}: node {node} {probe} deadline exceeded during PutObject; last response: {last_response}" ) .into()); } @@ -700,6 +777,18 @@ pub(crate) async fn wait_for_upgrade_write_readiness( Ok(()) } +fn previous_baseline_writers(clients: &[Client], restarted_node: usize) -> &[Client] { + // Node 0 has persisted the seed. With restart order [1, 2, 3, 0], only + // this prefix has also cleared rc.5's startup write fence. Every RRS + // probe still needs all drives, including those of not-yet-restarted peers. + let ready_end = if restarted_node == 0 { + clients.len() + } else { + restarted_node + 1 + }; + &clients[..ready_end] +} + async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnvironment, previous_binary: &Path) -> TestResult { let clients: Vec<_> = cluster.create_all_clients()?.iter().map(upgrade_probe_client).collect(); // rc.5 can latch a non-elected node's write fence while its first startup @@ -707,16 +796,30 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro // persist data, then restart each old process once with three peers still // readable. This preparation ends before any current binary is started. wait_for_upgrade_write_readiness(&clients[..1], MIXED_BUCKET, "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?; + // A node that just came back is still replaying its startup traffic against + // every peer. Killing the next node during that burst gives the fresh node + // two failures in a row and it drops that peer's drive to Offline, where + // only the slow recovery probe can bring it back. Let every writable node + // re-admit every drive before the next kill. Nodes still awaiting their + // restart may retain the startup write fence and cannot coordinate a PUT. for node in [1, 2, 3, 0] { cluster.stop_node(node)?; cluster.start_node_from_binary(node, previous_binary).await?; + let context = format!("previous-restart-{node}"); wait_for_upgrade_write_readiness( std::slice::from_ref(&clients[node]), MIXED_BUCKET, - &format!("previous-restart-{node}"), + &context, LISTING_CONVERGENCE_TIMEOUT, ) .await?; + wait_for_drive_readmission( + previous_baseline_writers(&clients, node), + MIXED_BUCKET, + &context, + DRIVE_READMISSION_TIMEOUT, + ) + .await?; } wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?; @@ -738,61 +841,243 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro Ok(()) } +/// The drive re-admission gate assumes that a parity-0 write needs every +/// drive. Prove it on the workspace build: with one node down a standard +/// write still commits, the reduced-redundancy write fails with 503, and once +/// the node is back the gate converges within its budget. +#[tokio::test] +async fn parity_zero_write_needs_every_drive() -> TestResult { + init_logging(); + let mut cluster = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?; + cluster.set_env("RUST_LOG", "rustfs=warn,rustfs_notify=warn"); + cluster.set_env(ALL_DRIVES_STORAGE_CLASS_ENV.0, ALL_DRIVES_STORAGE_CLASS_ENV.1); + cluster.start().await?; + cluster.create_test_bucket(MIXED_BUCKET).await?; + let clients = cluster.create_all_clients()?; + wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "parity-zero-seed", LISTING_CONVERGENCE_TIMEOUT).await?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, "parity-zero-healthy", DRIVE_READMISSION_TIMEOUT).await?; + + cluster.stop_node(3)?; + let writer = upgrade_probe_client(&clients[0]); + writer + .put_object() + .bucket(MIXED_BUCKET) + .key("parity-zero/standard-with-one-node-down") + .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) + .send() + .await?; + let error = writer + .put_object() + .bucket(MIXED_BUCKET) + .key("parity-zero/all-drives-with-one-node-down") + .storage_class(StorageClass::ReducedRedundancy) + .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) + .send() + .await + .expect_err("a parity-0 write must not commit at three of four drives"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503), "{error:?}"); + assert_eq!( + error.as_service_error().and_then(ProvideErrorMetadata::code), + Some("ServiceUnavailable"), + "{error:?}" + ); + + cluster.start_node(3).await?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, "parity-zero-returned", DRIVE_READMISSION_TIMEOUT).await?; + Ok(()) +} + #[cfg(test)] mod upgrade_write_readiness_tests { use super::*; use crate::fake_s3_target::FaultAction; + use aws_sdk_s3::config::interceptors::BeforeTransmitInterceptorContextRef; + use aws_sdk_s3::config::{ConfigBag, Intercept, RuntimeComponents}; + + #[derive(Debug)] + struct RequireAllDrivesProbe; + + impl Intercept for RequireAllDrivesProbe { + fn name(&self) -> &'static str { + "require-all-drives-probe" + } + + fn read_before_transmit( + &self, + context: &BeforeTransmitInterceptorContextRef<'_>, + _runtime_components: &RuntimeComponents, + _cfg: &mut ConfigBag, + ) -> Result<(), BoxError> { + assert_eq!(context.request().method(), "PUT"); + assert_eq!( + context.request().headers().get("x-amz-storage-class"), + Some("REDUCED_REDUNDANCY"), + "every admitted coordinator must use the all-drive storage class" + ); + Ok(()) + } + } #[tokio::test] - async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult { - let target = FakeS3Target::start().await?; - let bucket = "distributed-upgrade-history"; - target.create_bucket(bucket); - let client = fake_source_client(&target); - client.head_bucket().bucket(bucket).send().await?; + async fn previous_baseline_probes_only_writable_coordinators_until_all_restart() -> TestResult { + let mut targets = Vec::new(); + let mut clients = Vec::new(); + for _ in 0..MIXED_NODE_COUNT { + let target = FakeS3Target::start().await?; + target.create_bucket(MIXED_BUCKET); + clients.push(Client::from_conf( + fake_source_client(&target) + .config() + .to_builder() + .interceptor(RequireAllDrivesProbe) + .build(), + )); + targets.push(target); + } + for target in &targets[2..] { + target.inject(FakeTargetOperation::PutObject, FaultAction::Status(StatusCode::INTERNAL_SERVER_ERROR), 10); + } + // Reproduce the original all-coordinator gate: node 0 and restarted + // node 1 can write, but unrestarted node 2 still returns a latched 500. + let error = wait_for_drive_readmission(&clients, MIXED_BUCKET, "premature", Duration::from_secs(5)) + .await + .expect_err("an unrestarted coordinator must retain its startup write failure"); + assert!(error.to_string().contains("node 2 drive re-admission failed"), "{error}"); + let failed = targets[2].requests(); + assert_eq!(failed.len(), 1, "the startup 500 must fail without retrying"); + assert_eq!(failed[0].fault, Some(FaultAction::Status(StatusCode::INTERNAL_SERVER_ERROR))); + for target in &targets { + target.take_requests(); + } + + for (restarted_node, expected_writers) in [(1, 2), (2, 3), (3, 4), (0, 4)] { + // A restart clears only this process's startup write fence. + targets[restarted_node].clear_faults(); + let phase = format!("previous-restart-{restarted_node}"); + wait_for_drive_readmission( + previous_baseline_writers(&clients, restarted_node), + MIXED_BUCKET, + &phase, + Duration::from_secs(5), + ) + .await?; + for (node, target) in targets.iter().enumerate() { + let key = format!(".upgrade-readiness/{phase}/all-drives/node-{node}"); + let admitted = node < expected_writers; + assert_eq!( + target.count_requests(FakeTargetOperation::PutObject, &key), + usize::from(admitted), + "{phase}: only writable node {node} may coordinate an all-drive probe" + ); + assert_eq!(target.has_object(MIXED_BUCKET, &key), admitted); + assert_eq!(target.take_requests().len(), usize::from(admitted)); + } + } + + // Once every node is admitted, its permanent errors must still fail + // the full gate immediately rather than being skipped or retried. + let phase = "previous-ready-error"; + let key = format!(".upgrade-readiness/{phase}/all-drives/node-3"); + targets[3].inject_for_key( + FakeTargetOperation::PutObject, + &key, + FaultAction::Status(StatusCode::INTERNAL_SERVER_ERROR), + 1, + ); + let error = + wait_for_drive_readmission(previous_baseline_writers(&clients, 0), MIXED_BUCKET, phase, Duration::from_secs(5)) + .await + .expect_err("a recovered coordinator's 500 must still fail the final gate"); + assert!(error.to_string().contains("node 3 drive re-admission failed"), "{error}"); + assert_eq!(targets[3].count_requests(FakeTargetOperation::PutObject, &key), 1); + assert!(!targets[3].has_object(MIXED_BUCKET, &key)); + for target in targets { + target.shutdown().await; + } + Ok(()) + } + + #[tokio::test(start_paused = true)] + async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult { + use aws_sdk_s3::config::{Credentials, Region, RequestChecksumCalculation}; + use aws_smithy_http_client::test_util::{ReplayEvent, StaticReplayClient}; + use aws_smithy_types::body::SdkBody; + + let bucket = "distributed-upgrade-history"; let phase = "one-previous-node"; let first_key = format!(".upgrade-readiness/{phase}/node-0"); let second_key = format!(".upgrade-readiness/{phase}/node-1"); - target.inject_for_key( - FakeTargetOperation::PutObject, - &first_key, - FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE), - 2, - ); - target.inject_for_key( - FakeTargetOperation::PutObject, - &second_key, - FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE), - 1, + let endpoint = "http://upgrade-readiness.example.invalid"; + let event = |method: Method, key: &str, status: StatusCode| { + let (body, query) = if method == Method::PUT { + (SdkBody::from(UPGRADE_READINESS_BODY), "?x-id=PutObject") + } else { + (SdkBody::empty(), "") + }; + ReplayEvent::new( + http::Request::builder() + .method(method) + .uri(format!("{endpoint}/{bucket}/{key}{query}")) + .body(body) + .unwrap(), + http::Response::builder() + .status(status) + .header("content-type", "application/xml") + .body(if status == StatusCode::SERVICE_UNAVAILABLE { + SdkBody::from("ServiceUnavailable") + } else { + SdkBody::empty() + }) + .unwrap(), + ) + }; + // Keep transport scheduling outside the probe's time budget. The + // replay checks every writer's key and body without real network I/O. + let transport = StaticReplayClient::new(vec![ + event(Method::HEAD, "", StatusCode::OK), + event(Method::PUT, &first_key, StatusCode::SERVICE_UNAVAILABLE), + event(Method::PUT, &first_key, StatusCode::SERVICE_UNAVAILABLE), + event(Method::PUT, &first_key, StatusCode::OK), + event(Method::PUT, &second_key, StatusCode::SERVICE_UNAVAILABLE), + event(Method::PUT, &second_key, StatusCode::OK), + ]); + let client = Client::from_conf( + aws_sdk_s3::Config::builder() + .behavior_version_latest() + .credentials_provider(Credentials::new(FAKE_ACCESS_KEY, FAKE_SECRET_KEY, None, None, "readiness-test")) + .region(Region::new("us-east-1")) + .endpoint_url(endpoint) + .force_path_style(true) + .retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled()) + .request_checksum_calculation(RequestChecksumCalculation::WhenRequired) + .http_client(transport.clone()) + .build(), ); + client.head_bucket().bucket(bucket).send().await?; + // Metadata readiness does not prove that a data write can succeed. let premature = client .put_object() .bucket(bucket) .key(&first_key) - .body(ByteStream::from_static(b"upgrade write readiness")) + .body(ByteStream::from_static(UPGRADE_READINESS_BODY)) .send() .await .expect_err("metadata readiness does not prove write readiness"); assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503)); - wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], bucket, phase, Duration::from_secs(5)).await?; - assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3); - assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2); - assert!(target.has_object(bucket, &first_key)); - assert!(target.has_object(bucket, &second_key)); - assert!( - client - .list_objects_v2() - .bucket(bucket) - .prefix(format!("{phase}/")) - .send() - .await? - .contents() - .is_empty() + let started = Instant::now(); + wait_for_upgrade_write_readiness(&[client.clone(), client], bucket, phase, Duration::from_secs(5)).await?; + assert_eq!(started.elapsed(), Duration::from_secs(1), "each unavailable writer must be polled again"); + transport.assert_requests_match(&[]); + assert_eq!( + transport + .actual_requests() + .map(|request| request.method().to_string()) + .collect::>(), + ["HEAD", "PUT", "PUT", "PUT", "PUT", "PUT"], ); - target.shutdown().await; Ok(()) } @@ -963,7 +1248,14 @@ mod upgrade_write_readiness_tests { ); assert_eq!(target.count_requests(FakeTargetOperation::HeadObject, stalled_key), MIXED_NODE_COUNT); assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1); - assert_eq!(snapshot["top_locks"]["observation"], "unknown", "the fake peer has no admin lock API"); + let locks = snapshot["top_locks"] + .as_array() + .expect("failure artifact must retain every node's lock probe"); + assert_eq!(locks.len(), MIXED_NODE_COUNT); + for (node, probe) in locks.iter().enumerate() { + assert_eq!(probe["node"], node); + assert_eq!(probe["observation"], "unknown", "the fake peer has no admin lock API"); + } let locks: MixedLockSnapshot = serde_json::from_value(serde_json::json!({ "total": 1, "truncated": true, "capability_note": "partial snapshot", @@ -1082,7 +1374,7 @@ async fn rc5_rollback_requires_restoring_odm_configuration() -> TestResult { #[tokio::test] #[ignore = "requires a pinned previous RustFS release binary"] -async fn direct_upgrade_from_rc2_preserves_object_contracts() -> TestResult { +async fn direct_upgrade_from_previous_release_preserves_object_contracts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let mut env = RustFSTestEnvironment::new().await?; @@ -1232,13 +1524,14 @@ async fn direct_upgrade_from_rc2_preserves_object_contracts() -> TestResult { #[tokio::test] #[ignore = "requires a pinned previous RustFS release binary"] -async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestResult { +async fn rolling_upgrade_from_previous_release_preserves_mixed_version_contracts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let evidence_context = g09_evidence_context(&previous_binary).await?; let current_binary = rustfs_binary_path(); let mut cluster = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?; cluster.set_env("RUST_LOG", "rustfs=warn,rustfs_notify=warn"); + cluster.set_env(ALL_DRIVES_STORAGE_CLASS_ENV.0, ALL_DRIVES_STORAGE_CLASS_ENV.1); configure_cluster_logs(&mut cluster)?; cluster.start_with_binary(&previous_binary).await?; cluster.create_test_bucket(MIXED_BUCKET).await?; @@ -1248,22 +1541,24 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes cluster.start_node_from_binary(0, ¤t_binary).await?; exercise_mixed_cluster(&cluster, "one-current-node", 0, 1).await?; - for node_idx in [1, 2] { + for (node_idx, phase) in [(1, "two-current-nodes"), (2, "one-previous-node")] { cluster.stop_node(node_idx)?; cluster.start_node_from_binary(node_idx, ¤t_binary).await?; + exercise_mixed_cluster(&cluster, phase, 0, 3).await?; } - exercise_mixed_cluster(&cluster, "one-previous-node", 0, 3).await?; cluster.stop_node(3)?; cluster.start_node_from_binary(3, ¤t_binary).await?; + let clients = cluster.create_all_clients()?; + wait_for_drive_readmission(&clients, MIXED_BUCKET, "homogeneous-current", DRIVE_READMISSION_TIMEOUT).await?; - for (node_idx, client) in cluster.create_all_clients()?.iter().enumerate() { + for (node_idx, client) in clients.iter().enumerate() { assert_eq!( read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1, UPGRADE_READINESS_BODY, "current node {node_idx} must retain the previous-release seed" ); - for phase in ["one-current-node", "one-previous-node"] { + for phase in ["one-current-node", "two-current-nodes", "one-previous-node"] { wait_for_phase_listing( client, phase, @@ -1275,14 +1570,14 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes } if let Some(context) = evidence_context.as_ref() { - let phases = ["one-current-node", "one-previous-node"]; + let phases = ["one-current-node", "two-current-nodes", "one-previous-node"]; let objects_per_phase = MULTIPART_WORKERS * MULTIPART_UPLOADS_PER_WORKER + 2; write_g09_evidence( context, "mixed_version_reader_evidence", "mixed-version-reader", &["old-writer-new-reader", "new-writer-old-reader"], - "upgrade_compatibility_test::rolling_upgrade_from_rc2_preserves_mixed_version_contracts", + "upgrade_compatibility_test::rolling_upgrade_from_previous_release_preserves_mixed_version_contracts", serde_json::json!({ "bucket": MIXED_BUCKET, "phases": phases, @@ -1300,7 +1595,7 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes "mixed_version_writer_evidence", "mixed-version-writer", &["old-reader-new-writer", "new-reader-old-writer"], - "upgrade_compatibility_test::rolling_upgrade_from_rc2_preserves_mixed_version_contracts", + "upgrade_compatibility_test::rolling_upgrade_from_previous_release_preserves_mixed_version_contracts", serde_json::json!({ "bucket": MIXED_BUCKET, "phases": phases, @@ -1308,12 +1603,337 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes "assertions": [ "current writer publishes objects readable by previous-release node", "previous-release writer publishes objects readable by current node", - "multipart writers continue under one-current-node and one-previous-node layouts" + "multipart writers continue at every individual node upgrade", + "modern DELETE fails closed and preserves every byte while old peers remain" ], }), )?; } + let clients = cluster.create_all_clients()?; + wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "all-current", LISTING_CONVERGENCE_TIMEOUT).await?; + clients[0] + .delete_object() + .bucket(MIXED_BUCKET) + .key(PREVIOUS_RELEASE_SEED_KEY) + .send() + .await?; + for (node, client) in clients.iter().enumerate() { + let error = client + .get_object() + .bucket(MIXED_BUCKET) + .key(PREVIOUS_RELEASE_SEED_KEY) + .send() + .await + .expect_err("the homogeneous cluster must apply the previously blocked DELETE"); + assert_eq!( + error.as_service_error().and_then(ProvideErrorMetadata::code), + Some("NoSuchKey"), + "node {node}" + ); + } + + Ok(()) +} + +struct LegacyDeleteObject { + bucket: String, + key: String, + version: Option, + etag: String, + body: Vec, + data_dirs: Vec, +} + +fn legacy_payload_directories(object_dir: &Path) -> Result, BoxError> { + let entries = match std::fs::read_dir(object_dir) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(error) => return Err(error.into()), + }; + let mut directories = Vec::new(); + for entry in entries { + let entry = entry?; + if entry.file_type()?.is_dir() && uuid::Uuid::parse_str(&entry.file_name().to_string_lossy()).is_ok() { + for child in std::fs::read_dir(entry.path())? { + let child = child?; + if child.file_type()?.is_file() && child.file_name().to_string_lossy().starts_with("part.") { + directories.push(entry.path()); + break; + } + } + } + } + Ok(directories) +} + +async fn seed_legacy_delete_object( + env: &RustFSTestEnvironment, + client: &Client, + bucket: &str, + key: &str, + seed: u64, +) -> Result { + let object_dir = Path::new(&env.temp_dir).join(bucket).join(key); + let before = legacy_payload_directories(&object_dir)?; + // Incompressible external shards ensure the regression exercises payload + // reclamation, rather than only the inline xl.meta path. + let body = layout_noise(1024 * 1024 + 17, seed); + let response = client + .put_object() + .bucket(bucket) + .key(key) + .body(ByteStream::from(body.clone())) + .send() + .await?; + let data_dirs: Vec<_> = legacy_payload_directories(&object_dir)? + .into_iter() + .filter(|path| !before.contains(path)) + .collect(); + assert_eq!(data_dirs.len(), 1, "the old binary must create one external payload for {bucket}/{key}"); + // A suspended PUT can omit the version header while reads identify the + // stored generation as "null". Preserve the old reader's exact identity. + let head = client + .head_object() + .bucket(bucket) + .key(key) + .set_version_id(response.version_id().map(str::to_owned)) + .send() + .await?; + assert_eq!(head.e_tag(), response.e_tag(), "old writer and reader agree on the stored bytes"); + if let Some(version) = response.version_id() { + assert_eq!(head.version_id(), Some(version), "old reader resolves the written version"); + } + Ok(LegacyDeleteObject { + bucket: bucket.to_owned(), + key: key.to_owned(), + version: head.version_id().map(str::to_owned), + etag: response.e_tag().ok_or("the old writer omitted its ETag")?.to_owned(), + body, + data_dirs, + }) +} + +async fn assert_legacy_delete_object(client: &Client, object: &LegacyDeleteObject) -> TestResult { + let response = client + .get_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await + .map_err(|error| format!("GET {}/{} version {:?}: {error:?}", object.bucket, object.key, object.version))?; + assert_eq!( + response.e_tag(), + Some(object.etag.as_str()), + "{}/{}: original ETag", + object.bucket, + object.key + ); + assert_eq!( + response.version_id(), + object.version.as_deref(), + "{}/{}: original version", + object.bucket, + object.key + ); + assert_eq!(response.body.collect().await?.into_bytes().as_ref(), object.body); + Ok(()) +} + +fn preserve_legacy_upgrade_volume(env: &RustFSTestEnvironment, label: &str) -> TestResult { + let Some(log_dir) = std::env::var_os("RUSTFS_E2E_LOG_DIR") else { return Ok(()) }; + let root = Path::new(&env.temp_dir); + let destination = Path::new(&log_dir).join(label); + for entry in walkdir::WalkDir::new(root).follow_links(false) { + let entry = entry?; + let target = destination.join(entry.path().strip_prefix(root)?); + if entry.file_type().is_dir() { + std::fs::create_dir_all(target)?; + } else if entry.file_type().is_file() { + std::fs::copy(entry.path(), target)?; + } else { + return Err("the stopped upgrade fixture contains a non-regular entry".into()); + } + } + Ok(()) +} + +#[tokio::test] +#[ignore = "requires the pinned stable 1.0.1 release binary"] +async fn stable_upgrade_deletes_foreign_reservations_without_losing_live_objects() -> TestResult { + use sha2::Digest; + + init_logging(); + let previous_binary = source_binary()?; + assert_eq!(binary_source_revision(&previous_binary).await?, STABLE_1_0_1_COMMIT); + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server_from_binary(&previous_binary, vec![], &[]).await?; + let old_client = env.create_s3_client(); + let plain_bucket = "legacy-delete-plain"; + let versioned_bucket = "legacy-delete-versioned"; + let suspended_bucket = "legacy-delete-suspended"; + for bucket in [plain_bucket, versioned_bucket, suspended_bucket] { + env.create_test_bucket(bucket).await?; + } + enable_versioning(&old_client, versioned_bucket).await?; + enable_versioning(&old_client, suspended_bucket).await?; + old_client + .put_bucket_versioning() + .bucket(suspended_bucket) + .versioning_configuration( + VersioningConfiguration::builder() + .status(BucketVersioningStatus::Suspended) + .build(), + ) + .send() + .await?; + + let mut deleted = Vec::new(); + for (index, (bucket, key)) in [ + (plain_bucket, "single"), + (plain_bucket, "batch-one"), + (plain_bucket, "batch-two"), + (versioned_bucket, "versioned"), + (suspended_bucket, "suspended"), + ] + .into_iter() + .enumerate() + { + deleted.push(seed_legacy_delete_object(&env, &old_client, bucket, key, index as u64 + 1).await?); + } + let successor = seed_legacy_delete_object(&env, &old_client, versioned_bucket, "versioned", 20).await?; + let child_key = format!("single/{}/child", uuid::Uuid::new_v4()); + let child = seed_legacy_delete_object(&env, &old_client, plain_bucket, &child_key, 21).await?; + for object in deleted.iter().chain([&successor, &child]) { + assert_legacy_delete_object(&old_client, object).await?; + } + env.stop_server(); + + // The stable release produced the complete volume and every xl.meta/part + // byte above. Only the empty legacy marker is fault-injected after stop; + // this deterministically models an interrupted reservation, not a claimed + // reproduction of the old binary's SIGKILL timing window. + let mut evidence = Vec::new(); + for object in &deleted { + let metadata_path = Path::new(&env.temp_dir) + .join(&object.bucket) + .join(&object.key) + .join("xl.meta"); + let metadata = std::fs::read(&metadata_path)?; + for directory in &object.data_dirs { + let marker = directory.join(format!("reserve-delete-data.{}", uuid::Uuid::new_v4())); + std::fs::File::create(marker)?.sync_all()?; + } + assert_eq!( + std::fs::read(&metadata_path)?, + metadata, + "fault injection must preserve the old xl.meta bytes" + ); + evidence.push(serde_json::json!({ + "bucket": object.bucket, + "key": object.key, + "version": object.version, + "etag": object.etag, + "xl_meta_sha256": hex_simd::encode_to_string(sha2::Sha256::digest(&metadata), hex_simd::AsciiCase::Lower), + })); + } + if let Some(directory) = std::env::var_os("RUSTFS_E2E_LOG_DIR") { + std::fs::create_dir_all(&directory)?; + std::fs::write( + Path::new(&directory).join("stable-delete-source.json"), + serde_json::to_vec_pretty(&serde_json::json!({ + "writer_revision": STABLE_1_0_1_COMMIT, + "candidate_revision": current_source_revision()?, + "fault_injection": "empty foreign reservation marker after stopping the genuine old writer", + "objects": evidence, + }))?, + )?; + } + + preserve_legacy_upgrade_volume(&env, "stable-source-volume")?; + env.restart_server_preserving_data(vec![], &[]).await?; + let client = env.create_s3_client(); + for object in deleted.iter().chain([&successor, &child]) { + if let Err(error) = assert_legacy_delete_object(&client, object).await { + env.stop_server(); + preserve_legacy_upgrade_volume(&env, "failed-upgrade-volume")?; + return Err(error); + } + } + client.delete_object().bucket(plain_bucket).key("single").send().await?; + let response = client + .delete_objects() + .bucket(plain_bucket) + .delete( + Delete::builder() + .objects(ObjectIdentifier::builder().key("batch-one").build()?) + .objects(ObjectIdentifier::builder().key("batch-two").build()?) + .build()?, + ) + .send() + .await?; + assert!(response.errors().is_empty(), "batch deletion must converge: {:?}", response.errors()); + assert_eq!(response.deleted().len(), 2); + for object in &deleted[3..] { + client + .delete_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await?; + } + + // Repeat recovery across process restarts, then retry the same requests. + // The old committed payload must disappear while the successor and the + // UUID-shaped prefix child remain intact through every pass. + for pass in 0..2 { + env.restart_server_preserving_data(vec![], &[]).await?; + let client = env.create_s3_client(); + for object in &deleted { + let error = client + .get_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await + .expect_err("an acknowledged deletion must not resurrect after restart"); + assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(404)); + for directory in &object.data_dirs { + assert!( + !directory.exists(), + "pass {pass}: deleted payload and foreign reservation must be physically reclaimed" + ); + } + client + .delete_object() + .bucket(&object.bucket) + .key(&object.key) + .set_version_id(object.version.clone()) + .send() + .await?; + } + for object in [&successor, &child] { + assert_legacy_delete_object(&client, object).await?; + } + assert_eq!( + read_object(&client, versioned_bucket, "versioned", None).await?.1, + successor.body, + "deleting the old version must preserve the current successor" + ); + let versions = client.list_object_versions().bucket(versioned_bucket).send().await?; + assert_eq!(versions.versions().len(), 1); + assert_eq!(versions.versions()[0].version_id(), successor.version.as_deref()); + assert!(versions.delete_markers().is_empty()); + let listed = client.list_objects_v2().bucket(plain_bucket).send().await?; + assert_eq!(listed.contents().len(), 1); + assert_eq!(listed.contents()[0].key(), Some(child_key.as_str())); + } + + client.delete_object().bucket(plain_bucket).key(&child_key).send().await?; + client.delete_bucket().bucket(plain_bucket).send().await?; Ok(()) } @@ -2028,14 +2648,14 @@ async fn rollback_to_previous_release_reads_current_bucket_metadata() -> TestRes } // --------------------------------------------------------------------------- -// rc.5 multipart layouts under the current build (backlog#2147 follow-up to +// Previous-release multipart layouts under the current build (backlog#2147 follow-up to // rustfs#7305) // --------------------------------------------------------------------------- // // rustfs#7305 changed `ObjectInfo::is_multipart` to consult the stored part // list before the ETag shape. Every earlier check of that change used // synthetic metadata; this scenario writes the layouts with the published -// rc.5 binary and then reads, describes, and replicates them with the +// previous-release binary and then reads, describes, and replicates them with the // current build on the same data directory. const LAYOUT_PLAIN_BUCKET: &str = "upgrade-layout-plain"; @@ -2056,10 +2676,10 @@ struct LayoutCase { /// `false` for layouts whose replication is a known pre-existing failure; /// their outcome is logged, not asserted. assert_replication: bool, - /// Recorded from the rc.5 writer. - rc5_etag: String, - /// Whether rc.5 reported `ObjectParts` for the object. - rc5_reported_parts: Option, + /// Recorded from the previous-release writer. + source_etag: String, + /// Whether the previous release reported `ObjectParts` for the object. + source_reported_parts: Option, } impl LayoutCase { @@ -2164,7 +2784,7 @@ async fn layout_attributes( Ok(request.send().await?) } -/// Write `case` with the rc.5 client; single PUT when `part_sizes` is empty. +/// Write `case` with the source client; single PUT when `part_sizes` is empty. async fn layout_write(client: &Client, case: &LayoutCase) -> Result<(), BoxError> { let content_type = "text/plain"; if case.part_sizes.is_empty() { @@ -2251,8 +2871,8 @@ fn layout_cases() -> Vec { body, ssec, assert_replication: true, - rc5_etag: String::new(), - rc5_reported_parts: None, + source_etag: String::new(), + source_reported_parts: None, }; vec![ case(LAYOUT_PLAIN_BUCKET, "plain/single.bin", vec![], layout_noise(1024 * 1024 + 17, 1), false), @@ -2295,7 +2915,7 @@ fn layout_cases() -> Vec { // compressible first part is stored well below 5 MiB, so the sender // declares each part's plaintext length and the target validates the // 5 MiB minimum against it (rustfs/backlog#2363). rc.5 as the sender - // still fails this layout (see `rc5_baseline_replicates_multipart_layouts`). + // still fails this layout (see `previous_release_baseline_replicates_multipart_layouts`). case( LAYOUT_PLAIN_BUCKET, "plain/ssec-compressed-multipart-2.txt", @@ -2350,8 +2970,8 @@ async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &s let head = layout_head(client, case).await?; assert_eq!( head.e_tag().map(|etag| etag.trim_matches('"')), - Some(case.rc5_etag.as_str()), - "{context}: {label}: the ETag written by rc.5 must be reported unchanged" + Some(case.source_etag.as_str()), + "{context}: {label}: the ETag written by the previous release must be reported unchanged" ); assert_eq!( head.content_length(), @@ -2361,7 +2981,10 @@ async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &s let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes(); assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length"); - assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload"); + assert!( + full == case.body, + "{context}: {label}: full GET body must equal the previous-release upload" + ); Ok(()) } @@ -2403,7 +3026,7 @@ async fn assert_layout_attributes(client: &Client, case: &LayoutCase, context: & let attributes = layout_attributes(client, case).await?; assert_eq!( attributes.e_tag().map(|etag| etag.trim_matches('"')), - Some(case.rc5_etag.as_str()), + Some(case.source_etag.as_str()), "{context}: {label}: attributes ETag" ); assert_eq!( @@ -2602,13 +3225,13 @@ async fn replicate_layouts( Ok((target, transports)) } -/// rc.5 writes single-PUT, multipart, compressed, SSE-C and SSE-S3 layouts; +/// The previous release writes single-PUT, multipart, compressed, SSE-C and SSE-S3 layouts; /// the current build must read every byte, expose the stored part layout /// through GetObjectAttributes and partNumber reads, and replicate the objects /// with the transport that matches their stored parts. #[tokio::test] -#[ignore = "requires the pinned 1.0.0-rc.5 release binary"] -async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { +#[ignore = "requires a pinned previous RustFS release binary"] +async fn direct_upgrade_from_previous_release_preserves_multipart_layouts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let server_env = layout_server_env(); @@ -2622,18 +3245,18 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { enable_versioning(&old_client, LAYOUT_PLAIN_BUCKET).await?; enable_versioning(&old_client, LAYOUT_ENCRYPTED_BUCKET).await?; put_default_sse_s3_encryption(&old_client, LAYOUT_ENCRYPTED_BUCKET).await?; - assert_default_sse_s3_encryption(&old_client, LAYOUT_ENCRYPTED_BUCKET, "rc.5").await?; + assert_default_sse_s3_encryption(&old_client, LAYOUT_ENCRYPTED_BUCKET, "previous release").await?; let mut cases = layout_cases(); for case in cases.iter_mut() { layout_write(&old_client, case).await?; let head = layout_head(&old_client, case).await?; - case.rc5_etag = head + case.source_etag = head .e_tag() - .ok_or_else(|| format!("{}: rc.5 HEAD omitted the ETag", case.label()))? + .ok_or_else(|| format!("{}: previous-release HEAD omitted the ETag", case.label()))? .trim_matches('"') .to_string(); - case.rc5_reported_parts = layout_attributes(&old_client, case) + case.source_reported_parts = layout_attributes(&old_client, case) .await .ok() .and_then(|a| layout_reported_parts(&a)); @@ -2641,16 +3264,16 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { target: "e2e_test::upgrade_compatibility_test", object = %case.label(), parts = case.part_sizes.len(), - etag = %case.rc5_etag, - rc5_reported_parts = ?case.rc5_reported_parts, - "rc.5 wrote a legacy layout" + etag = %case.source_etag, + source_reported_parts = ?case.source_reported_parts, + "previous release wrote a layout" ); } - // Prove the rc.5 writer persisted every original byte before upgrading. + // Prove the previous-release writer persisted every original byte before upgrading. // Its compressed cross-part range reader has a legacy limitation; the // upgraded reader must still satisfy every range and partNumber assertion. for case in &cases { - assert_layout_full_body(&old_client, case, "rc.5").await?; + assert_layout_full_body(&old_client, case, "previous release").await?; } // Upgrade in place. @@ -2705,19 +3328,19 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult { .await? .into_bytes(); assert_eq!(replica.len(), case.body.len(), "{label}: replica length"); - assert!(replica == case.body, "{label}: replica body must equal the rc.5 upload"); + assert!(replica == case.body, "{label}: replica body must equal the previous-release upload"); } } Ok(()) } -/// The same layouts replicated by rc.5 itself, without an upgrade. This is the +/// The same layouts replicated by the previous release itself, without an upgrade. This is the /// baseline that tells a pre-existing transport failure apart from one the /// current build introduced; it records the outcome per layout and only fails /// when the fixture cannot run. #[tokio::test] -#[ignore = "requires the pinned 1.0.0-rc.5 release binary"] -async fn rc5_baseline_replicates_multipart_layouts() -> TestResult { +#[ignore = "requires a pinned previous RustFS release binary"] +async fn previous_release_baseline_replicates_multipart_layouts() -> TestResult { init_logging(); let previous_binary = source_binary()?; let server_env = layout_server_env(); @@ -2736,9 +3359,9 @@ async fn rc5_baseline_replicates_multipart_layouts() -> TestResult { for case in cases.iter_mut() { layout_write(&client, case).await?; let head = layout_head(&client, case).await?; - case.rc5_etag = head + case.source_etag = head .e_tag() - .ok_or_else(|| format!("{}: rc.5 HEAD omitted the ETag", case.label()))? + .ok_or_else(|| format!("{}: previous-release HEAD omitted the ETag", case.label()))? .trim_matches('"') .to_string(); } @@ -2756,7 +3379,7 @@ async fn rc5_baseline_replicates_multipart_layouts() -> TestResult { ) }) .collect(); - tracing::info!(target: "e2e_test::upgrade_compatibility_test", ?summary, "rc.5 baseline replication outcomes"); + tracing::info!(target: "e2e_test::upgrade_compatibility_test", ?summary, "previous-release baseline replication outcomes"); Ok(()) } @@ -2783,7 +3406,7 @@ async fn existing_object_replication_drives_each_layout_once() -> TestResult { for case in cases.iter_mut() { layout_write(&writer, case).await?; let head = layout_head(&writer, case).await?; - case.rc5_etag = head + case.source_etag = head .e_tag() .ok_or_else(|| format!("{}: HEAD omitted the ETag", case.label()))? .trim_matches('"') diff --git a/crates/ecstore/Cargo.toml b/crates/ecstore/Cargo.toml index 5a41e01f4..d659ee917 100644 --- a/crates/ecstore/Cargo.toml +++ b/crates/ecstore/Cargo.toml @@ -31,6 +31,7 @@ workspace = true [features] default = [] +diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"] gcs = ["dep:google-cloud-storage", "dep:google-cloud-auth"] # Compiles the controlled list-objects namespace-journal chaos injector into a # production binary (it is always available to tests). Off by default so the @@ -183,6 +184,7 @@ sha2 = { workspace = true } hex-simd = { workspace = true } tempfile.workspace = true hyper = { workspace = true, features = ["http2", "http1", "server"] } +h2 = { workspace = true } hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] } hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] } hostname.workspace = true @@ -200,8 +202,8 @@ md-5.workspace = true memmap2 = { workspace = true } libc.workspace = true # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate -# (backlog#1178); "fs" comes from the workspace default. -rustix = { workspace = true, features = ["process", "fs"] } +# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default. +rustix = { workspace = true, features = ["process", "fs", "thread"] } rustfs-madmin.workspace = true reqwest = { workspace = true } aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] } @@ -248,6 +250,7 @@ windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] } [dev-dependencies] aws-smithy-async.workspace = true +rustfs-s3-ops.workspace = true tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] } criterion = { workspace = true, features = ["html_reports"] } temp-env = { workspace = true, features = ["async_closure"] } diff --git a/crates/ecstore/src/api/mod.rs b/crates/ecstore/src/api/mod.rs index ee36d0b76..ef3793625 100644 --- a/crates/ecstore/src/api/mod.rs +++ b/crates/ecstore/src/api/mod.rs @@ -14,6 +14,15 @@ //! Explicit ECStore public facades for outer crate compatibility boundaries. +#[cfg(feature = "diagnostic-read-version-caller")] +pub mod diagnostic { + pub use crate::cluster::rpc::runtime_sources::{ + current_read_version_entry_source, observe_read_version_context_boundary, with_read_version_entry, + with_read_version_spawn_context, + }; + pub use rustfs_io_metrics::internode_metrics::{ReadVersionContextBoundary, ReadVersionEntrySource}; +} + pub mod admin { pub use crate::diagnostics::admin_server_info::{get_local_server_property, get_server_info}; } @@ -243,11 +252,11 @@ pub mod bucket { pub mod quota { pub use crate::bucket::quota::{ BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError, - QuotaOperation, + QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration, }; pub mod checker { - pub use crate::bucket::quota::checker::QuotaChecker; + pub use crate::bucket::quota::checker::{QuotaChecker, record_reservation_protocol_migration}; } } @@ -423,12 +432,14 @@ pub mod disk { pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage}; pub use crate::disk::{ BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp, - CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, - DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, - DeleteRecoveryRpcReply, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions, - FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize, PartTransactionAction, - RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, - SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count, + CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, CurrentDeleteTransition, + DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, + DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DeleteRecoveryState, + Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions, FileReader, FileWriter, + GuardedDeleteVersionsPayload, GuardedFileInfoVersions, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize, + PartTransactionAction, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, + ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, + is_quota_mutation_fence_path, new_disk, validate_batch_read_version_item_count, }; pub use bytes::Bytes; pub use endpoint::Endpoint; @@ -497,17 +508,19 @@ pub mod metrics { } pub mod notification { - #[cfg(any(test, feature = "test-util"))] - pub use crate::services::notification_sys::rotate_cross_pool_fence_fleet_proof_for_test; pub use crate::services::notification_sys::{ ClusterTierDailyStats, CrossPoolFenceFleetProofToken, IlmRecoveryExportFleetProofToken, LegacyTransitionStateReconcileFleetProofToken, NotificationPeerErr, NotificationSys, ScannerPublicationLeaseGrant, acquire_cross_pool_fence_fleet_proof, acquire_ilm_recovery_export_fleet_proof, - acquire_legacy_transition_state_reconcile_fleet_proof, cross_pool_fence_fleet_proof_matches, get_global_notification_sys, - ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_local_process_epoch, - ilm_recovery_export_member_epochs_sha256, ilm_recovery_export_topology_generation, - legacy_transition_state_reconcile_fleet_proof_matches, new_global_notification_sys, - scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe, + acquire_legacy_transition_state_reconcile_fleet_proof, configured_fleet_topology_fingerprint, + cross_pool_fence_fleet_proof_matches, get_global_notification_sys, ilm_recovery_export_fleet_proof_matches, + ilm_recovery_export_local_process_epoch, ilm_recovery_export_member_epochs_sha256, + ilm_recovery_export_topology_generation, legacy_transition_state_reconcile_fleet_proof_matches, + new_global_notification_sys, scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe, + }; + #[cfg(any(test, feature = "test-util"))] + pub use crate::services::notification_sys::{ + rotate_cross_pool_fence_fleet_proof_for_test, without_cross_pool_fence_fleet_proof_for_test, }; } @@ -601,6 +614,14 @@ pub mod set_disk { pub fn hold_namespace_commit(store: &crate::store::ECStore) -> impl Send + Sync { store.ctx.begin_namespace_commit() } + + /// Whether an ordinary namespace commit (a PUT rename fanout, including + /// its post-ACK tail) is still pending on this instance. Test fixtures + /// drain tails through this probe; the scanner's publication barrier no + /// longer reports them. + pub fn namespace_commits_pending(store: &crate::store::ECStore) -> bool { + store.ctx.namespace_commits_pending() + } } } diff --git a/crates/ecstore/src/bucket/bucket_target_sys.rs b/crates/ecstore/src/bucket/bucket_target_sys.rs index a8edc849d..a5561717e 100644 --- a/crates/ecstore/src/bucket/bucket_target_sys.rs +++ b/crates/ecstore/src/bucket/bucket_target_sys.rs @@ -142,11 +142,11 @@ fn same_replication_service(edited: &BucketTarget, previous: &BucketTarget) -> b && access_key(edited) == access_key(previous) } -/// Page size and page budget for [`TargetClient::locate_replica_by_etag`]. +/// Page size and page budget for [`TargetClient::replica_candidates`]. const FIND_VERSION_BY_ETAG_PAGE_SIZE: i32 = 1000; const FIND_VERSION_BY_ETAG_MAX_PAGES: usize = 8; -/// Candidate cap for [`TargetClient::replica_candidates_by_etag`]: more than -/// this many same-content versions of one key is ambiguity by any measure. +/// Candidate cap for [`TargetClient::replica_candidates`]: more than this +/// many matching versions of one key is ambiguity by any measure. const FIND_VERSION_BY_ETAG_MAX_MATCHES: usize = 16; pub type GetObjectSdkError = Box>; pub type GetObjectTaggingSdkError = Box>; @@ -2030,27 +2030,28 @@ impl TargetClient { /// Candidate replicas by content identity on a target that mints its own /// version ids: page `ListObjectVersions` under the exact key and report - /// the live versions whose ETag matches `source_etag`, newest first. - /// Delete markers and prefix siblings never match. Bounded to + /// the live versions that match `identity`, newest first. Delete markers + /// and prefix siblings never match. Bounded to /// [`FIND_VERSION_BY_ETAG_MAX_PAGES`] pages and - /// [`FIND_VERSION_BY_ETAG_MAX_MATCHES`] candidates so a key with a very - /// deep history cannot turn one convergence check into an unbounded scan; - /// a replica beyond that window reads as missing, which only costs a - /// re-PUT (today's behaviour), never a lost object. + /// [`FIND_VERSION_BY_ETAG_MAX_MATCHES`] candidates per kind so a key with + /// a very deep history cannot turn one convergence check into an + /// unbounded scan; a replica beyond that window reads as missing, which + /// only costs a re-PUT (today's behaviour), never a lost object. /// /// Content identity is not version identity: two source generations with - /// the same bytes have the same ETag. Callers drop the candidates other - /// source versions already claim through their ledgers and refuse an - /// [`ReplicaLocation::Ambiguous`] remainder before mutating or deleting. - pub async fn replica_candidates_by_etag( + /// the same bytes have the same ETag, and two with the same size have the + /// same shape. Callers drop the candidates other source versions already + /// claim through their ledgers and decide through + /// [`ReplicaCandidates::locate`]. + pub async fn replica_candidates( &self, bucket: &str, object: &str, - source_etag: &str, - ) -> Result, Box>> { + identity: &ReplicaIdentity<'_>, + ) -> Result>> { let mut key_marker: Option = None; let mut version_id_marker: Option = None; - let mut matches: Vec = Vec::new(); + let mut candidates = ReplicaCandidates::default(); for _ in 0..FIND_VERSION_BY_ETAG_MAX_PAGES { let page = self .client @@ -2063,19 +2064,27 @@ impl TargetClient { .send() .await .map_err(Box::new)?; - matches.extend( - page.versions() - .iter() - .filter(|version| { - version.key() == Some(object) - && version.version_id().is_some_and(|id| !id.is_empty()) - && replication_etags_match(Some(source_etag), version.e_tag()) - }) - .filter_map(|version| version.version_id().map(str::to_string)), - ); + for version in page.versions() { + let Some(version_id) = version.version_id().filter(|id| !id.is_empty()) else { + continue; + }; + if version.key() != Some(object) { + continue; + } + match identity.evidence(version.e_tag(), version.size(), version.last_modified()) { + Some(ReplicaEvidence::Etag) => candidates.exact.push(version_id.to_string()), + // An unversioned target overwrites the key in place: its + // `null` version holds whichever generation was written + // last, so a shape match there identifies nothing. + Some(ReplicaEvidence::Shape) if version_id != "null" => candidates.shaped.push(version_id.to_string()), + Some(ReplicaEvidence::Shape) => {} + None => {} + } + } // A listing that moved past the exact key (every listed key is >= - // the prefix), ended, or already filled the candidate cap decides. - if matches.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES + // the prefix), ended, or already filled a candidate cap decides. + if candidates.exact.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES + || candidates.shaped.len() >= FIND_VERSION_BY_ETAG_MAX_MATCHES || page .versions() .iter() @@ -2090,8 +2099,9 @@ impl TargetClient { break; } } - matches.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES); - Ok(matches) + candidates.exact.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES); + candidates.shaped.truncate(FIND_VERSION_BY_ETAG_MAX_MATCHES); + Ok(candidates) } /// PutObjectRetention against a replica version on a target that does not @@ -2614,19 +2624,18 @@ impl TargetClient { } /// Where a replica stands on a target that mints its own version ids, by -/// content identity (exact key + ETag) after the candidates other source -/// versions claim were removed. See -/// [`TargetClient::replica_candidates_by_etag`]. +/// content identity after the candidates other source versions claim were +/// removed. See [`TargetClient::replica_candidates`]. #[derive(Debug, Clone, PartialEq, Eq)] pub enum ReplicaLocation { - /// No live version under the key carries the source ETag. + /// No live version under the key matches the source. Missing, - /// Exactly one live version carries it: safe to address. + /// Exactly one live version matches: safe to address. Unique(String), - /// More than one live version carries it (same bytes replicated for - /// several source generations). `newest` is the most recently listed - /// one — good enough to prove the replica exists, never good enough to - /// pick which one to mutate or delete. + /// More than one live version carries the source ETag (same bytes + /// replicated for several source generations). `newest` is the most + /// recently listed one — good enough to prove the replica exists, never + /// good enough to pick which one to mutate or delete. Ambiguous { newest: String }, } @@ -2652,6 +2661,107 @@ impl ReplicaLocation { } } +/// What a replica of an encrypted source version still has in common with +/// it on a target that computes its own ETags. The source ETag is a keyed +/// fingerprint (docs/operations/encrypted-etags.md) or a legacy plaintext +/// MD5 the target's re-encrypted replica never reports, so the replica is +/// recognised by the byte count the replication PUT sent and by a write time +/// no earlier than the source version's own. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct EncryptedReplicaShape { + /// Bytes on the replication wire: the plaintext size for managed SSE, + /// the stored ciphertext size for an SSE-C passthrough. + pub size: i64, + /// The source version's mtime in whole seconds; listings and HEAD report + /// second-precision times. + pub not_before_secs: i64, +} + +impl EncryptedReplicaShape { + pub fn matches(&self, size: Option, last_modified: Option<&aws_sdk_s3::primitives::DateTime>) -> bool { + size == Some(self.size) && last_modified.is_some_and(|written| written.secs() >= self.not_before_secs) + } +} + +/// How a source version recognises its replica among a key's target +/// versions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReplicaIdentity<'a> { + /// Unencrypted source: its ETag is the MD5 of the replicated bytes, which + /// every target reports back — exact identity. + Etag(&'a str), + /// Encrypted source: a target that adopts the source ETag (RustFS, + /// MinIO) still reports `etag`; any other target only keeps the shape. + Encrypted { + etag: Option<&'a str>, + shape: EncryptedReplicaShape, + }, +} + +/// Which kind of evidence tied a target version to the source. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReplicaEvidence { + /// The target reports the source ETag. + Etag, + /// Encrypted source on a target that computes its own ETag: same size, + /// written no earlier than the source version. + Shape, +} + +impl ReplicaIdentity<'_> { + pub fn evidence( + &self, + etag: Option<&str>, + size: Option, + last_modified: Option<&aws_sdk_s3::primitives::DateTime>, + ) -> Option { + match *self { + Self::Etag(source_etag) => replication_etags_match(Some(source_etag), etag).then_some(ReplicaEvidence::Etag), + Self::Encrypted { + etag: source_etag, + shape, + } => { + if replication_etags_match(source_etag, etag) { + Some(ReplicaEvidence::Etag) + } else { + shape.matches(size, last_modified).then_some(ReplicaEvidence::Shape) + } + } + } + } +} + +/// Content-identity candidates of one key, newest first, by evidence kind. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct ReplicaCandidates { + /// Versions that report the source ETag. + pub exact: Vec, + /// Encrypted sources only: versions without the source ETag whose shape + /// matches. + pub shaped: Vec, +} + +impl ReplicaCandidates { + pub fn is_empty(&self) -> bool { + self.exact.is_empty() && self.shaped.is_empty() + } + + /// Drop the candidates other source versions claim and decide. An ETag + /// match outranks any shape match. A shape is weaker evidence than an + /// ETag — two generations of one size share it — so several shape + /// matches are never resolved by picking one: they read as + /// [`ReplicaLocation::Missing`] for convergence (the caller re-PUTs, a + /// duplicate at worst) and as ambiguous for a purge (it refuses). + pub fn locate(mut self, claimed: &HashSet) -> (ReplicaLocation, ReplicaEvidence) { + self.exact.retain(|candidate| !claimed.contains(candidate)); + if !self.exact.is_empty() { + return (ReplicaLocation::from_matches(self.exact), ReplicaEvidence::Etag); + } + self.shaped.retain(|candidate| !claimed.contains(candidate)); + (ReplicaLocation::from_matches(self.shaped), ReplicaEvidence::Shape) + } +} + #[derive(Debug)] pub enum BucketTargetError { BucketRemoteTargetNotFound { diff --git a/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs b/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs index 06919238f..61461ea98 100644 --- a/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs +++ b/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs @@ -2417,6 +2417,16 @@ fn is_slow_down(err: &Error) -> bool { matches!(err, Error::SlowDown) } +/// Lifecycle expiry is idempotent when the exact object or version has already +/// disappeared between the scanner listing and the delete admission. +/// +/// Keep this predicate narrower than bucket/volume "not found" helpers: an +/// unavailable bucket or disk is an operational failure and must not be +/// reported as a completed expiry. +fn is_lifecycle_expiry_already_gone(err: &Error) -> bool { + is_err_object_not_found(err) || is_err_version_not_found(err) +} + /// Resolves the expiry worker count from the single documented knob, /// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins; /// anything else falls back to `min(cpus, 16)`. The historical @@ -2451,7 +2461,13 @@ fn spawn_recovery_disposition_maintenance_once(api: Arc) -> Option) { return; } tokio::spawn(async move { - run_tier_delete_journal_recovery_loop(api, cancel_token).await; + let recovery = run_tier_delete_journal_recovery_loop(api, cancel_token); + #[cfg(feature = "diagnostic-read-version-caller")] + let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery, + recovery, + ); + recovery.await; }); } @@ -3353,7 +3375,13 @@ fn spawn_transition_transaction_recovery_once(api: Arc) { return; } tokio::spawn(async move { - run_transition_transaction_recovery_loop(api, cancel_token).await; + let recovery = run_transition_transaction_recovery_loop(api, cancel_token); + #[cfg(feature = "diagnostic-read-version-caller")] + let recovery = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::LifecycleRecovery, + recovery, + ); + recovery.await; }); } @@ -3938,44 +3966,28 @@ pub(crate) async fn enqueue_immediate_expiry(api: Arc, oi: &ObjectInfo, return; }; - let mut marker = None; - let mut version_marker = None; - let mut object_infos = Vec::new(); - - loop { - let page = match api - .clone() - .list_object_versions_for_lifecycle(&oi.bucket, &oi.name, marker.clone(), version_marker.clone(), None, 1000) - .await - { - Ok(page) => page, - Err(err) => { - warn!( - event = EVENT_LIFECYCLE_EVALUATION_FAILED, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_LIFECYCLE, - bucket = %oi.bucket, - object = %oi.name, - error = %err, - reason = "list_versions_failed", - "Failed to load lifecycle version group" - ); - return; - } - }; - - object_infos.extend(page.objects.into_iter().filter(|object| object.name == oi.name)); - - if !page.is_truncated { - break; + let object_infos = match api + .get_object_versions_for_lifecycle(&oi.bucket, &oi.name, opts, configs.bucket_incarnation_id) + .await + { + Ok(objects) => objects, + Err(err) => { + warn!( + event = EVENT_LIFECYCLE_EVALUATION_FAILED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LIFECYCLE, + result = "failed", + bucket = %oi.bucket, + object = %oi.name, + error = %err, + reason = "read_versions_failed", + "Failed to load lifecycle version group" + ); + return; } - - marker = page.next_marker; - version_marker = page.next_version_idmarker; - } - + }; if object_infos.is_empty() { - object_infos.push(oi.clone()); + return; } let object_opts = object_infos @@ -4900,6 +4912,14 @@ async fn expire_transitioned_object_with_lock_lost_signal( let (versioned, version_suspended) = snapshot.versioning_config().delete_state(&oi.name); let mut opts = transitioned_object_delete_opts(oi, lc_event.action, versioned, version_suspended, bucket_incarnation_id) .map_err(std::io::Error::other)?; + if lc_event.requires_expiry_policy() { + opts.lifecycle_expiry_policy = Some( + lc_event + .expiry_policy + .clone() + .ok_or_else(|| std::io::Error::other("lifecycle expiration is missing its policy snapshot"))?, + ); + } opts.add_namespace_lock_guard(&publication_guard); if let Some(signal) = lock_lost_signal { opts.add_namespace_lock_lost_signal(signal); @@ -4920,6 +4940,20 @@ async fn expire_transitioned_object_with_lock_lost_signal( //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn); Ok(dobj) } + Err(err) if is_lifecycle_expiry_already_gone(&err) => { + debug!( + event = EVENT_LIFECYCLE_DELETE_COMPLETED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LIFECYCLE, + bucket = %oi.bucket, + object = %oi.name, + operation = "delete_restored_object", + reason = "already_gone", + "Lifecycle delete target was already absent" + ); + crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await; + Ok(oi.clone()) + } Err(err) => Err(std::io::Error::other(err)), }; } @@ -4930,6 +4964,8 @@ async fn expire_transitioned_object_with_lock_lost_signal( // pointing to a remote tier version that has already been removed. If this // only creates a delete marker, remote cleanup must be driven by persisted // free-version recovery rather than the visible delete result. + // Keep remote transitioned not-found errors visible: local metadata + // absence alone does not prove that the tier object was cleaned up. let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await { Ok(obj) => obj, Err(e) => { @@ -5734,6 +5770,9 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal( bucket_incarnation_id: Uuid, lock_lost_signal: Option>, ) -> bool { + let Some(expiry_policy) = lc_event.expiry_policy.clone() else { + return false; + }; let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id).await else { return false; @@ -5764,6 +5803,7 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal( versioned, version_suspended, expiration: ExpirationOptions { expire: true }, + lifecycle_expiry_policy: Some(expiry_policy), delete_replication_config_snapshot: Some(Arc::new(snapshot)), expected_bucket_incarnation_id: Some(bucket_incarnation_id), ..Default::default() @@ -5802,6 +5842,20 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal( .await { Ok(dobj) => dobj, + Err(err) if is_lifecycle_expiry_already_gone(&err) => { + debug!( + event = EVENT_LIFECYCLE_DELETE_COMPLETED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LIFECYCLE, + bucket = %oi.bucket, + object = %oi.name, + operation = "delete_object", + reason = "already_gone", + "Lifecycle delete target was already absent" + ); + crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await; + return true; + } Err(e) => { error!( event = EVENT_LIFECYCLE_DELETE_FAILED, @@ -6031,6 +6085,7 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc, mod tests { use super::expiry_worker_count; use super::lifecycle_transition_publication_guard; + use super::metadata_boundary::get_expiry_configs; use super::{ DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX, DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED, EVENT_LIFECYCLE_EXPIRED_DETECTED, @@ -6041,18 +6096,19 @@ mod tests { cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at, enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped, enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle, - get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication, - lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action, - lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed, - manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker, - manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success, - merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned, - persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs, - resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, - resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, - run_tier_free_version_recovery_loop, select_restore_s3_location, set_lifecycle_observability_observer, - set_recovered_free_version_enqueue_observer, should_defer_date_expiry_for_recent_config_update, - transitioned_cleanup_tuple, transitioned_object_delete_opts, wait_for_tier_free_version_recovery, + get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay, + lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, + lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, + manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, + manual_transition_version_marker, manual_transition_worker_failure_reason, + mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, + persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint, + recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled, + resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count, + resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop, + select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer, + should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts, + wait_for_tier_free_version_recovery, }; #[cfg(feature = "test-util")] use super::{delete_free_version_remote_object_then, encode_dir_object, get_transitioned_object_reader_with_tier_manager}; @@ -6078,7 +6134,7 @@ mod tests { save_manual_transition_scope_admission_if_current, save_manual_transition_task_if_absent, save_manual_transition_worker_result_if_absent, }; - use crate::bucket::lifecycle::replication_sink::{ReplicationStatusType, VersionPurgeStatusType}; + use crate::bucket::lifecycle::replication_sink::{self, ReplicationStatusType, VersionPurgeStatusType}; use crate::bucket::lifecycle::runtime_boundary as runtime_sources; use crate::bucket::lifecycle::tier_free_version_recovery::{ FreeVersionRecoveryStats, RecoveryWalkTestAction, RecoveryWorkBudget, list_tier_free_versions, @@ -10511,6 +10567,26 @@ mod tests { ); } + #[test] + fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() { + assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound)); + assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound( + "bucket".to_string(), + "object".to_string(), + ))); + assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound)); + assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound( + "bucket".to_string(), + "object".to_string(), + "version".to_string(), + ))); + + assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound)); + assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound)); + assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum)); + assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed)); + } + #[tokio::test] async fn manual_transition_counts_already_transitioned_object() { let lc = latest_transition_lifecycle(); @@ -12832,6 +12908,7 @@ mod tests { .await .expect("bucket incarnation should be available"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; test_recorder::install(); // One version that exists and one that never did: `delete_objects` @@ -12850,15 +12927,24 @@ mod tests { ..Default::default() }, ]; + let _ = replication_sink::take_scheduled_deletes_for_test(); let failed = crate::bucket::lifecycle::object_handlers_common::delete_object_versions( &ecstore, &bucket, &targets, - lifecycle::Event::default(), + lifecycle::Event { + action: IlmAction::DeleteVersionAction, + expiry_policy: Some(expiry_policy.into()), + ..Default::default() + }, incarnation, ) .await; assert_eq!(failed, 0, "a missing version is not a batch failure"); + assert!( + replication_sink::take_scheduled_deletes_for_test().is_empty(), + "a bucket without replication must not schedule deletes for removed or missing versions" + ); let events = test_recorder::recorded_for_bucket(&bucket); let announced = events.iter().map(|event| event.version_id).collect::>(); @@ -12885,6 +12971,97 @@ mod tests { assert_eq!(remaining[0].version_id, Some(current_version_id)); } + #[tokio::test] + #[serial] + async fn lifecycle_noncurrent_batch_expiry_schedules_only_matching_delete_rules() { + use crate::bucket::metadata::BUCKET_REPLICATION_CONFIG; + + let (_disk_paths, ecstore) = setup_test_env().await; + for (prefix, delete_status, should_replicate) in [ + ("other/", "Enabled", false), + ("logs/", "Disabled", false), + ("logs/", "Enabled", true), + ] { + let bucket = format!("lifecycle-repl-admission-{}", Uuid::new_v4().simple()); + let object = "logs/object"; + create_test_bucket(&ecstore, &bucket).await; + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_VERSIONING_CONFIG, + b"Enabled".to_vec(), + ) + .await + .expect("versioning should be enabled"); + + let mut versions = Vec::new(); + for body in [b"noncurrent".as_slice(), b"current".as_slice()] { + let mut reader = PutObjReader::from_vec(body.to_vec()); + let version = ecstore + .put_object( + &bucket, + object, + &mut reader, + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("source version should be created"); + versions.push(version.version_id.expect("versioned PUT must have an identity")); + } + let config = format!( + "delete-ruleEnabled\ + {prefix}arn:test\ + {delete_status}\ + " + ); + metadata_sys::update_in(&ecstore.ctx, &bucket, BUCKET_REPLICATION_CONFIG, config.into_bytes()) + .await + .expect("replication configuration should be persisted"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; + let incarnation = ecstore + .bucket_incarnation_id_from_disk(&bucket) + .await + .expect("bucket incarnation should be available"); + let _ = replication_sink::take_scheduled_deletes_for_test(); + let failed = crate::bucket::lifecycle::object_handlers_common::delete_object_versions( + &ecstore, + &bucket, + &[ObjectToDelete { + object_name: object.to_string(), + version_id: Some(versions[0]), + ..Default::default() + }], + lifecycle::Event { + action: IlmAction::DeleteVersionAction, + expiry_policy: Some(expiry_policy.into()), + ..Default::default() + }, + incarnation, + ) + .await; + assert_eq!(failed, 0, "batch expiry should commit"); + let scheduled = replication_sink::take_scheduled_deletes_for_test(); + assert_eq!(scheduled.len(), usize::from(should_replicate), "prefix={prefix}, delete={delete_status}"); + if should_replicate { + assert_eq!(scheduled[0].version_id, Some(versions[0])); + let state = scheduled[0] + .replication_state + .as_ref() + .expect("admitted purge state must survive"); + assert_eq!(state.version_purge_status_internal.as_deref(), Some("arn:test=PENDING;")); + assert!(state.purge_targets.contains_key("arn:test")); + } + let current = ecstore + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("batch expiry must retain the current version"); + assert_eq!(current.version_id, Some(versions[1])); + } + } + #[tokio::test] #[serial] async fn lifecycle_deletes_only_the_historical_null_version_after_versioning_is_reenabled() { @@ -12972,6 +13149,14 @@ mod tests { assert!(!object_infos[null_index].is_latest); let lifecycle = current_and_noncurrent_expiration_lifecycle(); + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_LIFECYCLE_CONFIG, + crate::bucket::utils::serialize(&lifecycle).unwrap(), + ) + .await + .unwrap(); let object_opts = object_infos .iter() .map(lifecycle::object_opts_from_object_info) @@ -13142,6 +13327,81 @@ mod tests { assert_eq!(event.action, IlmAction::NoneAction); } + #[tokio::test] + #[serial] + async fn lifecycle_exact_immediate_expiry_preserves_guards_and_skips_stale_writes() { + let (paths, ecstore) = setup_test_env().await; + let bucket = format!("lifecycle-exact-expiry-{}", Uuid::new_v4().simple()); + let object = "object"; + create_test_bucket(&ecstore, &bucket).await; + let oi = ecstore + .put_object( + &bucket, + object, + &mut PutObjReader::from_vec(b"payload".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("create object before configuring expiry"); + + // Exercise persisted legacy immediate rules without relaxing the S3 + // validator, which correctly rejects newly submitted Expiration.Days=0. + let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system"); + let sys = sys.read().await.clone(); + let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata")).clone(); + let mut lifecycle = latest_expiration_lifecycle(); + lifecycle.rules[0].expiration.as_mut().expect("expiration rule").days = Some(0); + metadata.lifecycle_config_xml = crate::bucket::utils::serialize(&lifecycle).expect("encode legacy policy"); + metadata.lifecycle_config = Some(lifecycle); + sys.persist_and_set(metadata).await.expect("persist legacy policy"); + + let state = install_unconsumed_runtime_expiry_worker(&ecstore, 4).await; + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!(state.read().await.stats.pending_tasks(), 1, "exact reads must preserve immediate expiry"); + + let state = install_unconsumed_runtime_expiry_worker(&ecstore, 4).await; + for path in &paths { + let metadata_path = path.join(&bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE); + let mut meta = rustfs_filemeta::FileMeta::load(&fs::read(&metadata_path).await.expect("read object metadata")) + .expect("decode object metadata"); + let mut version = meta + .get_all_file_info_versions(&bucket, object, true) + .expect("version group") + .versions + .remove(0); + version.metadata.insert("x-amz-object-lock-legal-hold".into(), "ON".into()); + meta.add_version(version).expect("persist legal hold"); + fs::write(metadata_path, meta.marshal_msg().expect("encode held metadata")) + .await + .expect("write legal hold"); + } + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!( + state.read().await.stats.pending_tasks(), + 0, + "use current lock metadata, not the stale write result" + ); + + for path in &paths { + fs::write(path.join(&bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE), b"corrupt") + .await + .expect("inject corrupt metadata"); + } + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!(state.read().await.stats.pending_tasks(), 0, "read failure must never enqueue expiry"); + for path in &paths { + fs::remove_dir_all(path.join(&bucket).join(object)) + .await + .expect("simulate concurrent object removal"); + } + super::enqueue_immediate_expiry(ecstore.clone(), &oi, LcEventSrc::S3PutObject, &ObjectOptions::default()).await; + assert_eq!( + state.read().await.stats.pending_tasks(), + 0, + "confirmed absence must not fall back to the old ObjectInfo" + ); + } + #[tokio::test] #[serial] async fn lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata() { @@ -13251,6 +13511,7 @@ mod tests { .await .expect("table data object should be created"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; let publication_lock = ecstore .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH) .await @@ -13265,6 +13526,7 @@ mod tests { let mut expiry = tokio::spawn(async move { let event = crate::bucket::lifecycle::lifecycle::Event { action: IlmAction::DeleteAction, + expiry_policy: Some(expiry_policy.into()), ..Default::default() }; let bucket_incarnation_id = expiry_store @@ -13379,13 +13641,16 @@ mod tests { queued.size = 1; let event = lifecycle::Event { action: IlmAction::DeleteAction, + expiry_policy: Some(install_test_expiry_policy(&ecstore, &bucket).await.into()), ..Default::default() }; let incarnation = ecstore .bucket_incarnation_id_from_disk(&bucket) .await .expect("bucket incarnation"); - for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] { + // Replaying the same scanner snapshot after the delete is an + // already-gone success and must not be reported as a failure. + for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] { assert_eq!( super::apply_expiry_on_non_transitioned_objects( ecstore.clone(), @@ -13488,6 +13753,7 @@ mod tests { .bucket_incarnation_id_from_disk(&bucket) .await .expect("bucket incarnation should be available"); + let expiry_policy = install_test_expiry_policy(&ecstore, &bucket).await; let publication_lock = ecstore .new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH) .await @@ -13510,7 +13776,11 @@ mod tests { &delete_store, &delete_bucket, std::slice::from_ref(&target), - lifecycle::Event::default(), + lifecycle::Event { + action: IlmAction::DeleteVersionAction, + expiry_policy: Some(expiry_policy.into()), + ..Default::default() + }, incarnation, ) .await @@ -13666,6 +13936,7 @@ mod tests { let queued_event = crate::bucket::lifecycle::lifecycle::Event { action: IlmAction::DelMarkerDeleteAllVersionsAction, rule_id: "delete-marker-history".to_string(), + expiry_policy: get_expiry_configs(&ecstore, &bucket).await.unwrap().lifecycle.map(Into::into), ..Default::default() }; metadata_sys::update(&bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle_xml(30).into_bytes()) @@ -13783,6 +14054,7 @@ mod tests { &crate::bucket::lifecycle::lifecycle::Event { action: IlmAction::DeleteAllVersionsAction, rule_id: "delete-all-versions".to_string(), + expiry_policy: get_expiry_configs(&ecstore, &bucket).await.unwrap().lifecycle.map(Into::into), ..Default::default() }, &LcEventSrc::Scanner, @@ -13969,6 +14241,221 @@ mod tests { (disk_paths, ecstore) } + #[tokio::test] + #[serial] + async fn queued_expiry_rejects_withdrawn_or_changed_policy() { + let (_disk_paths, ecstore) = setup_test_env().await; + for case in [ + "removed", + "disabled", + "longer-days", + "different-prefix", + "unreadable", + "missing-snapshot", + "unchanged", + ] { + let bucket = format!("expiry-policy-{case}"); + create_test_bucket(&ecstore, &bucket).await; + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_VERSIONING_CONFIG, + b"Enabled".to_vec(), + ) + .await + .unwrap(); + for object in ["single", "batch"] { + for age in [40, 2] { + let mut reader = PutObjReader::from_vec(vec![1; 8]); + ecstore + .put_object( + &bucket, + object, + &mut reader, + &ObjectOptions { + versioned: true, + mod_time: Some(OffsetDateTime::now_utc() - time::Duration::days(age)), + ..Default::default() + }, + ) + .await + .unwrap(); + } + } + let policy = current_and_noncurrent_expiration_lifecycle(); + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_LIFECYCLE_CONFIG, + crate::bucket::utils::serialize(&policy).unwrap(), + ) + .await + .unwrap(); + let incarnation = ecstore.bucket_incarnation_id_from_disk(&bucket).await.unwrap(); + let state = install_unconsumed_runtime_expiry_worker(&ecstore, 3).await; + for object in ["single", "batch"] { + let versions = ecstore + .clone() + .list_object_versions(&bucket, object, None, None, None, 10) + .await + .unwrap() + .objects; + let opts = versions + .iter() + .map(lifecycle::object_opts_from_object_info) + .collect::>(); + let events = Evaluator::new(Arc::new(policy.clone())).eval(&opts).await.unwrap(); + let index = versions.iter().position(|version| !version.is_latest).unwrap(); + let mut event = events[index].clone(); + assert_eq!(event.action, IlmAction::DeleteVersionAction); + if case == "missing-snapshot" { + event.expiry_policy = None; + } + let queued = if object == "single" { + state + .write() + .await + .enqueue_by_days(&versions[index], &event, &LcEventSrc::Scanner, incarnation) + } else { + state.write().await.enqueue_by_newer_noncurrent( + &bucket, + vec![lifecycle_version_delete_target(&versions[index]).unwrap()], + event, + &LcEventSrc::Scanner, + incarnation, + ) + }; + assert!(queued); + } + let mut changed = policy.clone(); + match case { + "removed" => { + metadata_sys::delete(&bucket, BUCKET_LIFECYCLE_CONFIG).await.unwrap(); + } + "disabled" => changed.rules[0].status = ExpirationStatus::from_static(ExpirationStatus::DISABLED), + "longer-days" => { + changed.rules[0] + .noncurrent_version_expiration + .as_mut() + .unwrap() + .noncurrent_days = Some(60) + } + "different-prefix" => changed.rules[0].prefix = Some("keep/".to_string()), + "unreadable" => { + let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx) + .unwrap() + .read() + .await + .clone(); + let mut metadata = (*sys.get(&bucket).await.unwrap()).clone(); + metadata.lifecycle_config_xml = b"".to_vec(); + metadata.lifecycle_config = None; + sys.persist_and_set(metadata).await.unwrap(); + } + _ => {} + } + if !matches!(case, "removed" | "unchanged" | "missing-snapshot" | "unreadable") { + metadata_sys::update_in( + &ecstore.ctx, + &bucket, + BUCKET_LIFECYCLE_CONFIG, + crate::bucket::utils::serialize(&changed).unwrap(), + ) + .await + .unwrap(); + } + let (rx, stats, recovery_notify) = { + let state = state.read().await; + state.tasks_tx[0].try_send(None).unwrap(); + (state.tasks_rx[0].clone(), state.stats.clone(), state.recovery_notify.clone()) + }; + tokio::time::timeout(StdDuration::from_secs(30), async { + let mut receiver = rx.lock().await; + ExpiryState::worker(&mut receiver, Arc::downgrade(&ecstore), stats, recovery_notify).await; + }) + .await + .expect("queued expiry worker must drain"); + let versions = ecstore + .clone() + .list_object_versions(&bucket, "", None, None, None, 10) + .await + .unwrap() + .objects; + assert_eq!(versions.len(), if case == "unchanged" { 2 } else { 4 }, "case: {case}"); + assert_eq!(versions.iter().filter(|version| version.is_latest).count(), 2); + } + } + + #[tokio::test] + #[serial] + async fn lifecycle_policy_update_waits_for_admitted_delete_commit() { + let (_disk_paths, ecstore) = setup_test_env().await; + let bucket = "expiry-policy-commit-fence"; + create_test_bucket(&ecstore, bucket).await; + let mut reader = PutObjReader::from_vec(b"expired".to_vec()); + let object = ecstore + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + mod_time: Some(OffsetDateTime::now_utc() - time::Duration::days(40)), + ..Default::default() + }, + ) + .await + .unwrap(); + let policy = install_test_expiry_policy(&ecstore, bucket).await; + let event = eval_action_from_lifecycle(&policy, None, &object).await; + assert_eq!(event.action, IlmAction::DeleteAction); + let incarnation = ecstore.bucket_incarnation_id_from_disk(bucket).await.unwrap(); + let barrier = crate::store::DeleteAfterObjectLockSnapshotBarrier::install(bucket); + let deleting_store = ecstore.clone(); + let deleting = tokio::spawn(async move { + super::apply_expiry_on_non_transitioned_objects(deleting_store, &object, &event, &LcEventSrc::Scanner, incarnation) + .await + }); + tokio::time::timeout(StdDuration::from_secs(30), barrier.wait_until_paused()) + .await + .unwrap(); + let probe = metadata_sys::ConfigWriteLockProbe::install(bucket); + let mut removing = tokio::spawn(async move { metadata_sys::delete(bucket, BUCKET_LIFECYCLE_CONFIG).await }); + probe.wait_until_attempted().await; + assert!( + tokio::time::timeout(StdDuration::from_millis(100), &mut removing) + .await + .is_err(), + "configuration removal must wait for the admitted delete" + ); + barrier.release(); + assert!( + tokio::time::timeout(StdDuration::from_secs(30), deleting) + .await + .unwrap() + .unwrap() + ); + tokio::time::timeout(StdDuration::from_secs(30), removing) + .await + .unwrap() + .unwrap() + .unwrap(); + assert!(get_expiry_configs(&ecstore, bucket).await.unwrap().lifecycle.is_none()); + assert!( + ecstore + .get_object_info(bucket, "object", &ObjectOptions::default()) + .await + .is_err() + ); + } + + async fn install_test_expiry_policy(ecstore: &Arc, bucket: &str) -> Arc { + let xml = br#"expireEnabled00"#; + metadata_sys::update_in(&ecstore.ctx, bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec()) + .await + .expect("install expiration policy"); + get_expiry_configs(ecstore, bucket).await.unwrap().lifecycle.unwrap() + } + async fn create_test_bucket(ecstore: &Arc, bucket: &str) { ecstore .make_bucket(bucket, &MakeBucketOptions::default()) diff --git a/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs b/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs index 5df0d848e..a28d518e2 100644 --- a/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs +++ b/crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs @@ -33,13 +33,17 @@ pub async fn delete_object_versions( api: &Arc, bucket: &str, to_del: &[ObjectToDelete], - _lc_event: lifecycle::Event, + lc_event: lifecycle::Event, bucket_incarnation_id: Uuid, ) -> usize { if to_del.is_empty() { return 0; } + let Some(expiry_policy) = lc_event.expiry_policy else { + return to_del.len(); + }; + if to_del.iter().any(|target| { target.version_id.is_none() || (target.version_id.is_some_and(|version_id| version_id.is_nil()) && target.expected_identity.is_none()) @@ -104,6 +108,7 @@ pub async fn delete_object_versions( let mut delete_options = ObjectOptions { delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)), expected_bucket_incarnation_id: Some(bucket_incarnation_id), + lifecycle_expiry_policy: Some(expiry_policy.clone()), ..Default::default() }; delete_options.add_namespace_lock_guard(&publication_guard); @@ -125,7 +130,11 @@ pub async fn delete_object_versions( // already happened (backlog#2202). emit_noncurrent_expiration_event(bucket, target, deleted_obj, false); } - if deleted_obj.replication_state.is_none() { + if !deleted_obj + .replication_state + .as_ref() + .is_some_and(|state| !state.targets.is_empty() || !state.purge_targets.is_empty()) + { continue; } replication_sink::schedule_delete(bucket.to_string(), deleted_obj.clone()).await; diff --git a/crates/ecstore/src/bucket/lifecycle/replication_sink.rs b/crates/ecstore/src/bucket/lifecycle/replication_sink.rs index 042759176..2334d57b3 100644 --- a/crates/ecstore/src/bucket/lifecycle/replication_sink.rs +++ b/crates/ecstore/src/bucket/lifecycle/replication_sink.rs @@ -70,6 +70,11 @@ pub(crate) async fn schedule_delete(bucket: String, delete_object: DeletedObject ReplicationLifecycleBridge::schedule_delete(bucket, delete_object).await; } +#[cfg(test)] +pub(crate) fn take_scheduled_deletes_for_test() -> Vec { + ReplicationLifecycleBridge::take_scheduled_deletes_for_test() +} + #[cfg(test)] mod tests { use std::collections::HashMap; diff --git a/crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs b/crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs index b8b529e43..718f2ba91 100644 --- a/crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs +++ b/crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs @@ -959,7 +959,7 @@ impl PersistedTierDeleteJournalEntry { fn into_jentry(self) -> Result { // Empty `version_id` is a legal sentinel for objects transitioned to an - // unversioned remote tier (see CLAUDE.md: a tier version of `None`/`""` + // unversioned remote tier (see AGENTS.md: a tier version of `None`/`""` // means the tier bucket is unversioned, so the remote delete is issued // without a versionId). Only reject entries missing the object or tier // name, which are always populated for a TRANSITION_COMPLETE object. diff --git a/crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs b/crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs index 27f767846..bd2aa90f3 100644 --- a/crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs +++ b/crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs @@ -626,7 +626,7 @@ fn mark_scan_truncated_if_needed( fn is_recoverable_tier_free_version(oi: &ObjectInfo) -> bool { // A free version transitioned to an unversioned remote tier legally carries an - // empty `version_id` (see CLAUDE.md: a tier version of `None`/`""` means the + // empty `version_id` (see AGENTS.md: a tier version of `None`/`""` means the // tier bucket is unversioned). The worker path handles that by issuing a // versionless remote delete, so the recovery gate must not treat an empty // `version_id` as unrecoverable — the object name and tier are sufficient to diff --git a/crates/ecstore/src/bucket/metadata_sys.rs b/crates/ecstore/src/bucket/metadata_sys.rs index 1dc3e43e1..cb2ad0fb3 100644 --- a/crates/ecstore/src/bucket/metadata_sys.rs +++ b/crates/ecstore/src/bucket/metadata_sys.rs @@ -178,6 +178,13 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe Ok(ObjectLockConfigState::ConfirmedAbsent) } +#[derive(Debug, Clone, PartialEq)] +pub(crate) struct CachedQuotaMetadataSnapshot { + pub(crate) quota: Option, + pub(crate) bucket_incarnation: Uuid, + pub(crate) quota_revision: OffsetDateTime, +} + /// Convert the persisted serving-layer configuration into the storage-level /// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention) /// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a @@ -518,6 +525,45 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance: sys.get_bucket_incarnation_id_from_disk(bucket).await } +pub(crate) async fn find_bucket_incarnation_id_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, +) -> Result> { + let sys = bucket_metadata_sys_of(ctx)?; + let sys = sys.read().await.clone(); + sys.find_bucket_incarnation_id_from_disk(bucket).await +} + +/// Directory loss is recoverable only for a committed, non-retired generation. +/// The caller retains the bucket lifecycle fence through the ensuing reconstruction. +pub(crate) async fn validate_bucket_reconstruction_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, + expected: Uuid, +) -> Result<()> { + let api = bucket_metadata_sys_of(ctx)?.read().await.object_store(); + let lock = api + .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket)) + .await?; + let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?; + let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?; + if !persisted + || !metadata.bucket_incarnation_sidecar + || expected.is_nil() + || metadata.bucket_incarnation_id != expected + || !metadata.bucket_creation_committed + { + return Err(Error::PreconditionFailed); + } + if crate::bucket::retirement::is_retired(api, bucket, expected).await? { + return Err(Error::BucketNotFound(bucket.to_owned())); + } + if guard.is_lock_lost() { + return Err(Error::PreconditionFailed); + } + Ok(()) +} + /// Validate against disk while retaining an already-held Object Lock fence. pub(crate) async fn get_bucket_incarnation_id_for_options_in( ctx: &crate::runtime::instance::InstanceContext, @@ -981,6 +1027,52 @@ async fn update_quota_if_incarnation_stamped( update_under_config_write_guard(sys, &guard, rustfs_config::QUOTA_CONFIG_FILE, data, updated_at).await } +/// Rewrite the quota configuration only while the bucket still carries +/// `expected_incarnation_id` and `expected_quota_revision` (its +/// `quota_config_updated_at`), under the same guards and fleet proof as +/// [`update_quota_if_incarnation`]. +/// +/// The revision check turns a stale read into a typed +/// [`Error::PreconditionFailed`] instead of a lost update: the reservation +/// protocol migration (rustfs/backlog#2904) rewrites a configuration it read +/// from the cache, and an admin save landing in between must win. The system +/// is passed explicitly so the write publishes into the cache the caller +/// reads from. +pub async fn update_quota_with_sys_if_incarnation_and_revision( + sys: Arc>, + bucket: &str, + data: Vec, + expected_incarnation_id: Uuid, + expected_quota_revision: OffsetDateTime, + proof: &crate::services::notification_sys::CrossPoolFenceFleetProofToken, +) -> Result { + let guard = Box::pin(acquire_config_write_guard_for_incarnation( + sys.clone(), + bucket, + Some(expected_incarnation_id), + )) + .await?; + if !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof) { + return Err(Error::NamespaceLockQuorumUnavailable { + mode: "quota_capability", + bucket: bucket.to_string(), + object: rustfs_config::QUOTA_CONFIG_FILE.to_string(), + required: 1, + achieved: 0, + }); + } + let metadata_sys = sys.read().await.clone(); + match Box::pin(metadata_sys.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)).await? { + BucketMetadataAuthority::Authoritative(current) if current.quota_config_updated_at == expected_quota_revision => {} + BucketMetadataAuthority::Authoritative(_) => return Err(Error::PreconditionFailed), + BucketMetadataAuthority::MissingBucket => return Err(Error::BucketNotFound(bucket.to_string())), + BucketMetadataAuthority::Fabricated => { + return Err(Error::other("bucket quota metadata is not authoritative")); + } + } + update_under_config_write_guard(sys, &guard, rustfs_config::QUOTA_CONFIG_FILE, data, None).await +} + pub async fn update_bucket_targets_under_transaction_lock( guard: &BucketMetadataMutationGuard, bucket: &str, @@ -1330,6 +1422,18 @@ pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_ ctx: &crate::runtime::instance::InstanceContext, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>, +) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> { + get_delete_config_from_disk_in(ctx, bucket, guard, None).await +} + +/// Validate queued expiration against the same fresh metadata used for Object +/// Lock admission. The caller holds the transaction read lock through commit, +/// so a successful lifecycle update cannot be followed by an old-policy delete. +pub(crate) async fn get_delete_config_from_disk_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, + guard: Option<&rustfs_lock::NamespaceLockGuard>, + expiry_policy: Option<&rustfs_lifecycle::ExpiryPolicySnapshot>, ) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> { let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?; let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone(); @@ -1350,6 +1454,13 @@ pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_ BucketMetadataAuthority::Authoritative(metadata) if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() => { + if let Some(expected) = expiry_policy { + let current = ConfigState::of(&metadata.lifecycle_config_xml, &metadata.lifecycle_config) + .require(bucket, BUCKET_LIFECYCLE_CONFIG)?; + if !expected.matches_configuration(current) { + return Err(Error::PreconditionFailed); + } + } let object_lock_config_updated_at = metadata.object_lock_config_updated_at; Ok(( object_lock_config_state_from_authoritative_metadata(&metadata)?, @@ -1450,6 +1561,9 @@ fn quota_config_and_incarnation_from_authority( BucketMetadataAuthority::Authoritative(metadata) if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() => { + if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() { + return Err(Error::other("persisted bucket quota configuration is invalid")); + } Ok(( metadata.quota_config.clone(), metadata.bucket_incarnation_id, @@ -1464,6 +1578,15 @@ fn quota_config_and_incarnation_from_authority( } } +pub(crate) async fn get_cached_quota_config_and_incarnation_in( + ctx: &crate::runtime::instance::InstanceContext, + bucket: &str, +) -> Result> { + let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?; + let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone(); + bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await +} + pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> { let bucket_meta_sys_lock = get_bucket_metadata_sys()?; let bucket_meta_sys = bucket_meta_sys_lock.read().await; @@ -2555,6 +2678,11 @@ impl BucketMetadataSys { } async fn get_bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result { + let incarnation_id = self.find_bucket_incarnation_id_from_disk(bucket).await?; + Self::require_bucket_incarnation_id(bucket, incarnation_id) + } + + async fn find_bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result> { let transaction_lock = self .object_store() .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket)) @@ -2562,7 +2690,7 @@ impl BucketMetadataSys { let _transaction_guard = transaction_lock .get_read_lock(crate::set_disk::get_lock_acquire_timeout()) .await?; - self.get_bucket_incarnation_id_under_transaction_lock(bucket, &_transaction_guard) + self.find_bucket_incarnation_id_under_transaction_lock(bucket, &_transaction_guard) .await } @@ -2571,14 +2699,26 @@ impl BucketMetadataSys { bucket: &str, transaction_guard: &rustfs_lock::NamespaceLockGuard, ) -> Result { + let incarnation_id = self + .find_bucket_incarnation_id_under_transaction_lock(bucket, transaction_guard) + .await?; + Self::require_bucket_incarnation_id(bucket, incarnation_id) + } + + fn require_bucket_incarnation_id(bucket: &str, incarnation_id: Option) -> Result { + incarnation_id.ok_or_else(|| Error::other(format!("bucket incarnation metadata is not authoritative: {bucket}"))) + } + + async fn find_bucket_incarnation_id_under_transaction_lock( + &self, + bucket: &str, + transaction_guard: &rustfs_lock::NamespaceLockGuard, + ) -> Result> { let incarnation_id = load_bucket_incarnation(self.object_store(), bucket).await?; if transaction_guard.is_lock_lost() { return Err(Error::other(format!("bucket incarnation metadata transaction lock was lost: {bucket}"))); } - match incarnation_id { - Some(incarnation_id) if !incarnation_id.is_nil() => Ok(incarnation_id), - _ => Err(Error::other(format!("bucket incarnation metadata is not authoritative: {bucket}"))), - } + Ok(incarnation_id) } async fn get_metadata_authority(&self, bucket: &str) -> Result { @@ -2979,6 +3119,27 @@ impl BucketMetadataSys { } } + async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result> { + let metadata = { + let metadata_map = self.metadata_map.read().await; + metadata_map.get(bucket).cloned() + }; + let Some(metadata) = metadata else { + return Ok(None); + }; + if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() { + return Ok(None); + } + if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() { + return Err(Error::other("persisted bucket quota configuration is invalid")); + } + Ok(Some(CachedQuotaMetadataSnapshot { + quota: metadata.quota_config.clone(), + bucket_incarnation: metadata.bucket_incarnation_id, + quota_revision: metadata.quota_config_updated_at, + })) + } + pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> { let (bm, _) = self.get_config(bucket).await?; @@ -4883,6 +5044,43 @@ mod tests { ); } + #[tokio::test] + #[serial] + async fn quota_admission_rejects_stale_cached_no_quota() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system"); + let bucket = "stale-no-quota-admission"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata"); + store + .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec()) + .await + .expect("publish quota to authoritative storage"); + // Model a peer that has not received the newer configuration snapshot. + sys.read().await.set(bucket.to_owned(), stale).await; + assert!( + get_cached_quota_config_and_incarnation_in(&store.ctx, bucket) + .await + .expect("read stale cache") + .expect("cached bucket identity") + .quota + .is_none() + ); + + temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async { + let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0) + .await + .err() + .expect("persisted quota must reject admission without its snapshot"); + assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}"); + }) + .await; + } + #[tokio::test] #[serial] async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() { diff --git a/crates/ecstore/src/bucket/quota/checker.rs b/crates/ecstore/src/bucket/quota/checker.rs index 8e313c09a..d6b3192ad 100644 --- a/crates/ecstore/src/bucket/quota/checker.rs +++ b/crates/ecstore/src/bucket/quota/checker.rs @@ -12,16 +12,33 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation}; -use crate::bucket::metadata_sys::{BucketMetadataSys, update, update_if_incarnation}; +use super::{ + BucketQuota, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError, QuotaMigrationSkipReason, QuotaOperation, + QuotaProtocolMigration, +}; +use crate::bucket::metadata_sys::{ + BucketMetadataSys, update, update_if_incarnation, update_quota_with_sys_if_incarnation_and_revision, +}; use crate::data_usage::get_bucket_usage_memory; +use crate::error::StorageError; use rustfs_config::QUOTA_CONFIG_FILE; use rustfs_scanner_metrics::metrics::Metric; -use std::sync::Arc; -use std::time::Instant; +use std::collections::HashMap; +use std::sync::{Arc, Mutex, OnceLock, PoisonError}; +use std::time::{Duration, Instant}; use time::OffsetDateTime; use tokio::sync::RwLock; -use tracing::{debug, warn}; +use tracing::{debug, info, warn}; + +const LOG_COMPONENT_ECSTORE: &str = "ecstore"; +const LOG_SUBSYSTEM_QUOTA: &str = "quota"; +const EVENT_QUOTA_USAGE_REPORT: &str = "quota_usage_report"; +const EVENT_QUOTA_PROTOCOL_MIGRATION: &str = "quota_reservation_protocol_migration"; +const METRIC_QUOTA_USAGE_REPORT_UNAVAILABLE_TOTAL: &str = "rustfs_quota_usage_report_unavailable_total"; +const METRIC_QUOTA_PROTOCOL_MIGRATIONS_TOTAL: &str = "rustfs_quota_reservation_protocol_migrations_total"; +/// A lazy migration attempt that did not settle the bucket is retried no +/// sooner than this; the admin command is not rate limited. +const LAZY_MIGRATION_RETRY_INTERVAL: Duration = Duration::from_secs(300); pub struct QuotaChecker { metadata_sys: Arc>, @@ -74,7 +91,15 @@ impl QuotaChecker { Some(q) => q, }; - let current_usage = self.get_real_time_usage(bucket).await?; + // A durable quota is admitted by the reservation ledger at commit, so + // scanner usage is reporting input here: an unavailable lookup is + // reported as absent instead of failing the write (rustfs/backlog#2904). + // The snapshot protocol still admits from this value and fails closed. + let current_usage = if uses_durable_reservations { + self.report_only_usage(bucket).await + } else { + Some(self.get_real_time_usage(bucket).await?) + }; // The reporting path projects this operation; storage mutations reserve it at commit. let admission_size = if uses_durable_reservations && !force_usage_calculation { @@ -82,27 +107,23 @@ impl QuotaChecker { } else { operation_size }; - let expected_usage = match operation { + let expected_usage = current_usage.map(|current_usage| match operation { QuotaOperation::PutObject | QuotaOperation::PostObject | QuotaOperation::CopyObject => { current_usage.saturating_add(admission_size) } QuotaOperation::DeleteObject => current_usage.saturating_sub(operation_size), - }; + }); let allowed = match operation { QuotaOperation::PutObject | QuotaOperation::PostObject | QuotaOperation::CopyObject => { - quota_config.check_operation_allowed(current_usage, admission_size) + current_usage.is_none_or(|current_usage| quota_config.check_operation_allowed(current_usage, admission_size)) } QuotaOperation::DeleteObject => true, }; - let remaining = if quota_limit >= expected_usage { - Some(quota_limit - expected_usage) - } else { - Some(0) - }; + let remaining = expected_usage.map(|expected_usage| quota_limit.saturating_sub(expected_usage)); - if !allowed { + if let (false, Some(current_usage)) = (allowed, current_usage) { warn!( "Quota exceeded for bucket: {}, current: {}, limit: {}, attempted: {}", bucket, current_usage, quota_limit, operation_size @@ -111,7 +132,7 @@ impl QuotaChecker { let result = QuotaCheckResult { allowed, - current_usage: Some(current_usage), + current_usage, quota_limit: Some(quota_limit), operation_size, remaining, @@ -152,6 +173,10 @@ impl QuotaChecker { reason: format!("Failed to parse quota config: {}", e), })?; + if quota.needs_reservation_protocol_migration() { + self.schedule_reservation_protocol_migration(bucket).await; + } + Ok(quota) } @@ -219,9 +244,146 @@ impl QuotaChecker { } let quota = self.get_quota_config(bucket).await?; - let current_usage = self.get_real_time_usage(bucket).await?; + let current_usage = if quota.uses_durable_reservations() { + self.report_only_usage(bucket).await + } else { + Some(self.get_real_time_usage(bucket).await?) + }; - Ok((quota, Some(current_usage))) + Ok((quota, current_usage)) + } + + /// Scanner usage for a bucket the reservation ledger admits is reporting + /// input only, so a failed lookup degrades to "unknown" and is counted + /// instead of becoming an error the write path would fail on. + async fn report_only_usage(&self, bucket: &str) -> Option { + match self.get_real_time_usage(bucket).await { + Ok(usage) => Some(usage), + Err(err) => { + metrics::counter!(METRIC_QUOTA_USAGE_REPORT_UNAVAILABLE_TOTAL).increment(1); + debug!( + event = EVENT_QUOTA_USAGE_REPORT, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_QUOTA, + state = "usage_unknown", + bucket = %bucket, + error = %err, + "durable quota usage report degraded" + ); + None + } + } + } + + /// Re-save `bucket`'s quota under the sharded durable reservation + /// protocol (rustfs/backlog#2904). The limit and type are unchanged. The + /// write takes the incarnation- and revision-checked quota save path with + /// the fleet capability proof, so a mixed-version fleet fails the attempt + /// and a concurrent quota edit or bucket replacement leaves the newer + /// configuration untouched. + pub async fn migrate_reservation_protocol(&self, bucket: &str) -> Result { + let (meta, _) = self + .metadata_sys + .read() + .await + .get_config(bucket) + .await + .map_err(QuotaError::StorageError)?; + if meta.quota_config_json.is_empty() { + return Ok(QuotaProtocolMigration::NoQuota); + } + let quota: BucketQuota = serde_json::from_slice(&meta.quota_config_json).map_err(|e| QuotaError::InvalidConfig { + reason: format!("Failed to parse quota config: {}", e), + })?; + if let Some(reservation_protocol) = quota.reservation_protocol { + return Ok(if quota.uses_durable_reservations() { + QuotaProtocolMigration::AlreadyDurable { reservation_protocol } + } else { + QuotaProtocolMigration::Skipped { + reason: QuotaMigrationSkipReason::UnsupportedReservationProtocol, + } + }); + } + let Some(migrated) = quota.migrated_to_sharded_reservations() else { + return Ok(QuotaProtocolMigration::NoQuota); + }; + if !meta.bucket_incarnation_sidecar || meta.bucket_incarnation_id.is_nil() { + return Err(QuotaError::StorageError(StorageError::other( + "bucket incarnation metadata is not authoritative", + ))); + } + // Same gate as the admin save: a durable protocol must not reach a + // fleet that still has nodes that would fail closed on it. + let proof = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().ok_or_else(|| { + QuotaError::StorageError(StorageError::NamespaceLockQuorumUnavailable { + mode: "quota_capability", + bucket: bucket.to_string(), + object: QUOTA_CONFIG_FILE.to_string(), + required: 1, + achieved: 0, + }) + })?; + let json_data = serde_json::to_vec(&migrated).map_err(|e| QuotaError::InvalidConfig { + reason: format!("Failed to serialize quota config: {}", e), + })?; + let start_time = Instant::now(); + let saved = update_quota_with_sys_if_incarnation_and_revision( + Arc::clone(&self.metadata_sys), + bucket, + json_data, + meta.bucket_incarnation_id, + meta.quota_config_updated_at, + &proof, + ) + .await; + match saved { + Ok(updated_at) => { + rustfs_scanner_metrics::metrics::Metrics::inc_time(Metric::QuotaSync, start_time.elapsed()); + Ok(QuotaProtocolMigration::Migrated { + reservation_protocol: QUOTA_RESERVATION_PROTOCOL_V2, + updated_at, + }) + } + Err(StorageError::PreconditionFailed) => Ok(QuotaProtocolMigration::Skipped { + reason: QuotaMigrationSkipReason::QuotaRevisionChanged, + }), + Err(StorageError::BucketNotFound(_)) => Ok(QuotaProtocolMigration::Skipped { + reason: QuotaMigrationSkipReason::BucketIncarnationChanged, + }), + Err(err) => Err(QuotaError::StorageError(err)), + } + } + + /// Lazy half of the migration: the first quota read of a bucket still on + /// the snapshot protocol re-saves it under the sharded durable protocol + /// in the background; the read itself is neither blocked nor failed by + /// it. Only the metadata system registered on the store's instance + /// context migrates, so the rewrite publishes into the cache every + /// reader of this store consults; a checker over an ad-hoc system + /// (tooling, tests) only reads. An attempt that does not settle the + /// bucket is retried no sooner than `LAZY_MIGRATION_RETRY_INTERVAL`. + async fn schedule_reservation_protocol_migration(&self, bucket: &str) { + let registered = self.metadata_sys.read().await.object_store().ctx.bucket_metadata_sys(); + if !registered.is_some_and(|registered| Arc::ptr_eq(®istered, &self.metadata_sys)) { + return; + } + if !lazy_migration_attempt_due(bucket) { + return; + } + let checker = Self::new(Arc::clone(&self.metadata_sys)); + let bucket = bucket.to_string(); + tokio::spawn(async move { + let outcome = checker.migrate_reservation_protocol(&bucket).await; + record_reservation_protocol_migration(&bucket, "lazy", &outcome); + if matches!( + outcome, + Ok(QuotaProtocolMigration::Migrated { .. } + | QuotaProtocolMigration::AlreadyDurable { .. } + | QuotaProtocolMigration::NoQuota) + ) { + lazy_migration_attempt_settled(&bucket); + } + }); } pub async fn bucket_exists(&self, bucket: &str) -> bool { @@ -262,10 +424,107 @@ impl QuotaChecker { } } +/// One entry per bucket whose lazy migration is in flight or did not settle; +/// the entry is dropped once the bucket settles, so the map is bounded by the +/// number of buckets still on the snapshot protocol. +fn lazy_migration_attempts() -> &'static Mutex> { + static ATTEMPTS: OnceLock>> = OnceLock::new(); + ATTEMPTS.get_or_init(Mutex::default) +} + +fn lazy_migration_attempt_due(bucket: &str) -> bool { + let mut attempts = lazy_migration_attempts().lock().unwrap_or_else(PoisonError::into_inner); + let now = Instant::now(); + if attempts + .get(bucket) + .is_some_and(|last| now.duration_since(*last) < LAZY_MIGRATION_RETRY_INTERVAL) + { + return false; + } + attempts.insert(bucket.to_string(), now); + true +} + +fn lazy_migration_attempt_settled(bucket: &str) { + lazy_migration_attempts() + .lock() + .unwrap_or_else(PoisonError::into_inner) + .remove(bucket); +} + +#[cfg(test)] +fn lazy_migration_attempt_pending(bucket: &str) -> bool { + lazy_migration_attempts() + .lock() + .unwrap_or_else(PoisonError::into_inner) + .contains_key(bucket) +} + +/// One log line and one counter per migration attempt; `trigger` names the +/// caller (`lazy` from the quota read path, `admin` from the admin command). +pub fn record_reservation_protocol_migration( + bucket: &str, + trigger: &'static str, + outcome: &Result, +) { + let result = match outcome { + Ok(QuotaProtocolMigration::Migrated { .. }) => "migrated", + Ok(QuotaProtocolMigration::AlreadyDurable { .. }) | Ok(QuotaProtocolMigration::NoQuota) => "noop", + Ok(QuotaProtocolMigration::Skipped { .. }) => "skipped", + Err(_) => "failed", + }; + metrics::counter!(METRIC_QUOTA_PROTOCOL_MIGRATIONS_TOTAL, "trigger" => trigger, "result" => result).increment(1); + match outcome { + Ok(QuotaProtocolMigration::Migrated { + reservation_protocol, .. + }) => info!( + event = EVENT_QUOTA_PROTOCOL_MIGRATION, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_QUOTA, + result, + trigger, + bucket = %bucket, + reservation_protocol, + "quota reservation protocol migrated" + ), + Ok(QuotaProtocolMigration::Skipped { reason }) => debug!( + event = EVENT_QUOTA_PROTOCOL_MIGRATION, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_QUOTA, + result, + trigger, + bucket = %bucket, + reason = reason.as_str(), + "quota reservation protocol migration skipped" + ), + Ok(_) => debug!( + event = EVENT_QUOTA_PROTOCOL_MIGRATION, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_QUOTA, + result, + trigger, + bucket = %bucket, + "quota reservation protocol migration not needed" + ), + Err(err) => warn!( + event = EVENT_QUOTA_PROTOCOL_MIGRATION, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_QUOTA, + result, + trigger, + bucket = %bucket, + error = %err, + "quota reservation protocol migration failed" + ), + } +} + #[cfg(test)] mod tests { use super::*; use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks; + use crate::bucket::quota::QuotaType; + use crate::error::StorageError; use serial_test::serial; use uuid::Uuid; @@ -378,6 +637,367 @@ mod tests { ); } + async fn isolated_durable_quota_store() -> (Vec, Arc, Arc>) + { + let (dirs, store) = isolated_store_over_temp_disks().await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let sys = crate::bucket::metadata_sys::bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system"); + (dirs, store, sys) + } + + async fn create_bucket_with_quota_json(store: &Arc, bucket: &str, quota_json: Vec) { + use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions}; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create quota test bucket"); + store + .update_bucket_metadata_config(bucket, rustfs_config::QUOTA_CONFIG_FILE, quota_json) + .await + .expect("persist quota config"); + } + + /// Regression (rustfs/backlog#2904, decision D3 in rustfs/backlog#2872): a + /// bucket whose quota is admitted by the durable reservation ledger must + /// not fail its write because scanner usage is unavailable. The checker + /// reports `current_usage = None` instead of `UsageUnavailable`, while the + /// ledger still rejects an over-limit write and a bucket without a + /// reservation protocol keeps failing closed. + #[tokio::test] + #[serial] + async fn durable_quota_check_reports_unknown_usage_without_failing_the_write() { + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::object::ObjectIO as _; + + let (_dirs, store, sys) = isolated_durable_quota_store().await; + let checker = QuotaChecker::new(Arc::clone(&sys)); + let bucket = format!("quota-durable-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json( + &store, + &bucket, + serde_json::to_vec(&BucketQuota::new_sharded(4096)).expect("sharded quota should encode"), + ) + .await; + crate::data_usage::invalidate_data_usage_snapshot_cache().await; + + let result = checker + .check_quota(&bucket, QuotaOperation::PutObject, 512) + .await + .expect("a ledger-admitted bucket must not fail the check without scanner usage"); + assert!(result.allowed); + assert!(result.uses_durable_reservations); + assert_eq!(result.quota_limit, Some(4096)); + assert_eq!(result.current_usage, None, "unknown usage is reported as absent, never as zero"); + assert_eq!(result.remaining, None); + + let (_, usage) = checker + .get_quota_stats(&bucket) + .await + .expect("stats for a ledger-admitted bucket degrade to an absent usage"); + assert_eq!(usage, None); + + // Negative control: the ledger, not the scanner, still refuses an + // over-limit write with the same error. + let mut too_large = PutObjReader::from_vec(vec![0x5a; 8192]); + let err = store + .put_object(&bucket, "too-large", &mut too_large, &ObjectOptions::default()) + .await + .expect_err("the ledger must reject a write above the quota limit"); + assert!(matches!(err, StorageError::QuotaExceeded { .. }), "{err:?}"); + + let mut fits = PutObjReader::from_vec(vec![0x5a; 512]); + store + .put_object(&bucket, "fits", &mut fits, &ObjectOptions::default()) + .await + .expect("a write within the quota limit commits through the ledger"); + + // A bucket without a reservation protocol keeps the fail-closed path. + let legacy = format!("quota-legacy-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json(&store, &legacy, br#"{"quota":4096,"quota_type":"Hard"}"#.to_vec()).await; + assert!(matches!( + checker.check_quota(&legacy, QuotaOperation::PutObject, 512).await, + Err(QuotaError::UsageUnavailable { .. }) + )); + assert!(matches!(checker.get_quota_stats(&legacy).await, Err(QuotaError::UsageUnavailable { .. }))); + + crate::data_usage::prepare_bucket_usage_for_namespace_change(&bucket, None) + .await + .expect("test usage cache cleanup should succeed"); + crate::data_usage::invalidate_data_usage_snapshot_cache().await; + } + + fn legacy_quota_json(limit: u64) -> Vec { + format!(r#"{{"quota":{limit},"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z"}}"#).into_bytes() + } + + async fn persisted_quota(store: &Arc, bucket: &str) -> BucketQuota { + let (quota, _, _) = crate::bucket::metadata_sys::get_quota_config_and_incarnation_from_disk_in(&store.ctx, bucket) + .await + .expect("quota metadata should load from disk"); + quota.expect("a quota should be persisted") + } + + /// rustfs/backlog#2904: a quota saved before the reservation protocols is + /// re-saved under the sharded durable protocol with the same limit, type + /// and audit timestamp; durable and unknown protocols are left alone. + #[tokio::test] + #[serial] + async fn legacy_quota_migrates_to_sharded_reservation_protocol_once() { + let (_dirs, store, sys) = isolated_durable_quota_store().await; + crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test(); + let checker = QuotaChecker::new(Arc::clone(&sys)); + let bucket = format!("quota-migrate-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json(&store, &bucket, legacy_quota_json(4096)).await; + + let outcome = checker + .migrate_reservation_protocol(&bucket) + .await + .expect("migration should succeed"); + assert!( + matches!( + outcome, + QuotaProtocolMigration::Migrated { + reservation_protocol: QUOTA_RESERVATION_PROTOCOL_V2, + .. + } + ), + "{outcome:?}" + ); + let migrated = persisted_quota(&store, &bucket).await; + assert!(migrated.uses_sharded_reservations()); + assert_eq!(migrated.quota, Some(4096)); + assert_eq!(migrated.quota_type, QuotaType::Hard); + assert_eq!(migrated.created_at.map(|at| at.unix_timestamp()), Some(1_704_067_200)); + assert_eq!( + checker.get_quota_config(&bucket).await.expect("cached quota should load"), + migrated, + "the rewrite publishes into the registered cache" + ); + + assert_eq!( + checker + .migrate_reservation_protocol(&bucket) + .await + .expect("second attempt should succeed"), + QuotaProtocolMigration::AlreadyDurable { + reservation_protocol: QUOTA_RESERVATION_PROTOCOL_V2 + } + ); + + let no_quota = format!("quota-none-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json( + &store, + &no_quota, + serde_json::to_vec(&BucketQuota::new(None)).expect("empty quota should encode"), + ) + .await; + assert_eq!( + checker + .migrate_reservation_protocol(&no_quota) + .await + .expect("no quota is a no-op"), + QuotaProtocolMigration::NoQuota + ); + + let unsupported = format!("quota-unsupported-{}", Uuid::new_v4().simple()); + let unsupported_json = br#"{"quota":0,"quota_type":"Hard","reservation_protocol":3,"reservation_quota":2048}"#.to_vec(); + create_bucket_with_quota_json(&store, &unsupported, unsupported_json.clone()).await; + assert_eq!( + checker + .migrate_reservation_protocol(&unsupported) + .await + .expect("an unknown protocol is skipped, never rewritten"), + QuotaProtocolMigration::Skipped { + reason: QuotaMigrationSkipReason::UnsupportedReservationProtocol + } + ); + let (raw, _) = sys.read().await.get_config(&unsupported).await.expect("cached metadata"); + assert_eq!(raw.quota_config_json, unsupported_json); + } + + /// The guarded save refuses a replaced bucket and a quota edited since the + /// cached read; the migration reports the latter as a skip and leaves the + /// newer configuration for the next attempt. + #[tokio::test] + #[serial] + async fn quota_protocol_migration_requires_matching_incarnation_and_revision() { + let (_dirs, store, sys) = isolated_durable_quota_store().await; + crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test(); + let checker = QuotaChecker::new(Arc::clone(&sys)); + let bucket = format!("quota-cas-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json(&store, &bucket, legacy_quota_json(4096)).await; + let (meta, _) = sys.read().await.get_config(&bucket).await.expect("cached metadata"); + let proof = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().expect("fleet proof is installed"); + let data = serde_json::to_vec(&BucketQuota::new_sharded(4096)).expect("sharded quota should encode"); + + let err = update_quota_with_sys_if_incarnation_and_revision( + Arc::clone(&sys), + &bucket, + data.clone(), + Uuid::new_v4(), + meta.quota_config_updated_at, + &proof, + ) + .await + .expect_err("a different incarnation must not be rewritten"); + assert!(matches!(err, StorageError::BucketNotFound(_)), "{err:?}"); + + let err = update_quota_with_sys_if_incarnation_and_revision( + Arc::clone(&sys), + &bucket, + data, + meta.bucket_incarnation_id, + meta.quota_config_updated_at + time::Duration::seconds(1), + &proof, + ) + .await + .expect_err("a different quota revision must not be rewritten"); + assert!(matches!(err, StorageError::PreconditionFailed), "{err:?}"); + assert_eq!(persisted_quota(&store, &bucket).await.quota, Some(4096)); + assert!(persisted_quota(&store, &bucket).await.needs_reservation_protocol_migration()); + + // Model a peer whose cache missed a newer save: the migration reads + // the stale entry, the guarded write sees the newer revision, and the + // newer configuration stays in place. + let (stale, _) = sys.read().await.get_config(&bucket).await.expect("cached metadata"); + store + .update_bucket_metadata_config(&bucket, QUOTA_CONFIG_FILE, legacy_quota_json(8192)) + .await + .expect("newer quota save"); + sys.read().await.set(bucket.clone(), stale).await; + assert_eq!( + checker + .migrate_reservation_protocol(&bucket) + .await + .expect("a stale read is a skip, not a failure"), + QuotaProtocolMigration::Skipped { + reason: QuotaMigrationSkipReason::QuotaRevisionChanged + } + ); + let on_disk = persisted_quota(&store, &bucket).await; + assert_eq!(on_disk.quota, Some(8192)); + assert!( + on_disk.needs_reservation_protocol_migration(), + "the newer save is left for the next attempt" + ); + } + + /// The first quota read through the instance's registered metadata + /// system migrates the bucket in the background and still answers with + /// the configuration it found; a checker over an ad-hoc system only reads. + #[tokio::test] + #[serial] + async fn lazy_quota_protocol_migration_runs_from_the_quota_read_path() { + let (_dirs, store, sys) = isolated_durable_quota_store().await; + crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test(); + let checker = QuotaChecker::new(Arc::clone(&sys)); + let bucket = format!("quota-lazy-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json(&store, &bucket, legacy_quota_json(4096)).await; + + let first = checker.get_quota_config(&bucket).await.expect("legacy quota should load"); + assert!( + first.needs_reservation_protocol_migration(), + "the triggering read answers with what it found" + ); + let deadline = Instant::now() + Duration::from_secs(30); + loop { + let on_disk = persisted_quota(&store, &bucket).await; + if on_disk.uses_sharded_reservations() { + assert_eq!(on_disk.quota, Some(4096)); + break; + } + assert!(Instant::now() < deadline, "lazy migration did not land"); + tokio::time::sleep(Duration::from_millis(50)).await; + } + let settled = Instant::now() + Duration::from_secs(10); + while lazy_migration_attempt_pending(&bucket) { + assert!(Instant::now() < settled, "a migrated bucket must leave the retry map"); + tokio::time::sleep(Duration::from_millis(20)).await; + } + + let ad_hoc = QuotaChecker::new(Arc::new(RwLock::new(BucketMetadataSys::new(Arc::clone(&store))))); + let other = format!("quota-adhoc-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json(&store, &other, legacy_quota_json(4096)).await; + assert!( + ad_hoc + .get_quota_config(&other) + .await + .expect("legacy quota should load") + .needs_reservation_protocol_migration() + ); + assert!( + !lazy_migration_attempt_pending(&other), + "an ad-hoc metadata system never schedules a migration" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + assert!(persisted_quota(&store, &other).await.needs_reservation_protocol_migration()); + } + + /// Migration while writes are in flight: objects committed under the + /// snapshot protocol before and around the rewrite are accounted exactly + /// once by the sharded ledger's exact bootstrap, so the limit holds to + /// the byte afterwards. (Under `cfg(test)` a handler-supplied snapshot + /// admission short-circuits `begin`, so every concurrent write here takes + /// the snapshot path; the bootstrap accounting is what is verified.) + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn concurrent_puts_across_protocol_migration_are_accounted_exactly_once() { + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::object::ObjectIO as _; + + const OBJECTS: u64 = 8; + const SIZE: u64 = 256; + const SLACK: u64 = 64; + const LIMIT: u64 = OBJECTS * SIZE + SLACK; + + let (_dirs, store, sys) = isolated_durable_quota_store().await; + crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test(); + let bucket = format!("quota-race-{}", Uuid::new_v4().simple()); + create_bucket_with_quota_json(&store, &bucket, legacy_quota_json(LIMIT)).await; + + let mut writers = Vec::with_capacity(OBJECTS as usize); + for index in 0..OBJECTS { + let store = Arc::clone(&store); + let bucket = bucket.clone(); + writers.push(tokio::spawn(async move { + // The admission a request handler supplies on the snapshot + // protocol; ignored once the ledger owns admission. + let mut opts = ObjectOptions::default(); + assert!(opts.set_quota_admission(0, LIMIT)); + let mut body = PutObjReader::from_vec(vec![0x5a; SIZE as usize]); + store.put_object(&bucket, &format!("object-{index}"), &mut body, &opts).await + })); + } + let migration = { + let checker = QuotaChecker::new(Arc::clone(&sys)); + let bucket = bucket.clone(); + tokio::spawn(async move { checker.migrate_reservation_protocol(&bucket).await }) + }; + for writer in writers { + writer + .await + .expect("writer task should not panic") + .expect("every write fits the limit"); + } + let outcome = migration + .await + .expect("migration task should not panic") + .expect("migration should succeed"); + assert!(matches!(outcome, QuotaProtocolMigration::Migrated { .. }), "{outcome:?}"); + + let mut over = PutObjReader::from_vec(vec![0x5a; (SLACK + 1) as usize]); + let err = store + .put_object(&bucket, "over", &mut over, &ObjectOptions::default()) + .await + .expect_err("one byte above the remaining quota must be rejected"); + assert!(matches!(err, StorageError::QuotaExceeded { .. }), "{err:?}"); + let mut exact = PutObjReader::from_vec(vec![0x5a; SLACK as usize]); + store + .put_object(&bucket, "exact", &mut exact, &ObjectOptions::default()) + .await + .expect("exactly the remaining bytes fit"); + } + #[tokio::test] async fn test_quota_check_no_limit() { let result = QuotaCheckResult { diff --git a/crates/ecstore/src/bucket/quota/mod.rs b/crates/ecstore/src/bucket/quota/mod.rs index b57494250..1187327e1 100644 --- a/crates/ecstore/src/bucket/quota/mod.rs +++ b/crates/ecstore/src/bucket/quota/mod.rs @@ -164,6 +164,26 @@ impl BucketQuota { .is_some_and(|version| !matches!(version, QUOTA_RESERVATION_PROTOCOL_V1 | QUOTA_RESERVATION_PROTOCOL_V2)) } + /// A limit saved before the reservation protocols existed: its writes + /// are still admitted from scanner usage snapshots. Durable and unknown + /// future protocols are never rewritten. + pub fn needs_reservation_protocol_migration(&self) -> bool { + self.quota.is_some() && self.reservation_protocol.is_none() + } + + /// The same limit and type re-expressed under the sharded durable + /// reservation protocol; `created_at` is kept for audit continuity. + /// `None` when the quota is not a migration candidate. + pub fn migrated_to_sharded_reservations(&self) -> Option { + self.needs_reservation_protocol_migration().then(|| Self { + quota: self.quota, + quota_type: self.quota_type.clone(), + reservation_protocol: Some(QUOTA_RESERVATION_PROTOCOL_V2), + created_at: self.created_at, + updated_at: self.updated_at, + }) + } + pub fn check_operation_allowed(&self, current_usage: u64, operation_size: u64) -> bool { if operation_size == 0 { return true; @@ -200,6 +220,45 @@ pub enum QuotaOperation { DeleteObject, } +/// Outcome of re-saving one bucket's quota under the sharded durable +/// reservation protocol (rustfs/backlog#2904). +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum QuotaProtocolMigration { + /// No quota limit is configured; nothing to migrate. + NoQuota, + /// The quota already uses a durable reservation protocol. + AlreadyDurable { reservation_protocol: u32 }, + /// The quota was re-saved under the sharded protocol at this revision. + Migrated { + reservation_protocol: u32, + updated_at: OffsetDateTime, + }, + /// The quota was left untouched for the stated reason. + Skipped { reason: QuotaMigrationSkipReason }, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum QuotaMigrationSkipReason { + /// A reservation protocol this node does not understand; rewriting it + /// would downgrade a newer fleet's quota. + UnsupportedReservationProtocol, + /// The quota configuration changed between the cached read and the + /// guarded write; the newer configuration wins. + QuotaRevisionChanged, + /// The bucket was replaced or removed since the cached read. + BucketIncarnationChanged, +} + +impl QuotaMigrationSkipReason { + pub fn as_str(self) -> &'static str { + match self { + Self::UnsupportedReservationProtocol => "unsupported_reservation_protocol", + Self::QuotaRevisionChanged => "quota_revision_changed", + Self::BucketIncarnationChanged => "bucket_incarnation_changed", + } + } +} + #[derive(Debug, Error)] pub enum QuotaError { #[error("Bucket quota exceeded: current={current}, limit={limit}, operation={operation}")] diff --git a/crates/ecstore/src/bucket/quota/reservation.rs b/crates/ecstore/src/bucket/quota/reservation.rs index 59c51c301..f28918085 100644 --- a/crates/ecstore/src/bucket/quota/reservation.rs +++ b/crates/ecstore/src/bucket/quota/reservation.rs @@ -419,7 +419,6 @@ pub(crate) struct QuotaContext { quota_revision: Option, quota_limit: Option, reservation_protocol: Option, - capability_proof: Option, snapshot_admission: Option, legacy_data_movement: bool, metadata_guard: Option>, @@ -479,7 +478,6 @@ impl QuotaContext { reservation: reservation.clone(), }; let metadata_guard = self.metadata_guard; - let capability_proof = self.capability_proof; tokio::spawn(async move { reap_stale_reservations(Arc::clone(&store), &ledger_data.bucket, &ledger_data.ledger_object).await?; @@ -526,7 +524,6 @@ impl QuotaContext { ledger: Some(ReservationLedgerData::Single(ledger_data)), operation_guard: Some(operation_guard), metadata_guard, - capability_proof, state: ReservationState::Pending, }) }) @@ -555,7 +552,6 @@ async fn reserve_sharded(context: QuotaContext, old_size: u64, new_size: u64) -> commit_started: false, }; let metadata_guard = context.metadata_guard; - let capability_proof = context.capability_proof; let data = ShardedReservationData { store: Arc::clone(&store), bucket: context.bucket, @@ -640,7 +636,6 @@ async fn reserve_sharded(context: QuotaContext, old_size: u64, new_size: u64) -> ledger: Some(ReservationLedgerData::Sharded(data)), operation_guard: Some(operation_guard), metadata_guard, - capability_proof, state: ReservationState::Pending, }); } @@ -725,20 +720,12 @@ impl ReservationLedgerData { Self::Sharded(data) => &data.bucket, } } - - fn error_object(&self) -> &str { - match self { - Self::Single(data) => &data.ledger_object, - Self::Sharded(data) => &data.shard_object, - } - } } pub(crate) struct QuotaReservation { ledger: Option, operation_guard: Option, metadata_guard: Option>, - capability_proof: Option, state: ReservationState, } @@ -756,7 +743,6 @@ impl QuotaReservation { ledger: None, operation_guard: None, metadata_guard, - capability_proof: None, state: ReservationState::Pending, } } @@ -766,17 +752,7 @@ impl QuotaReservation { || self.metadata_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) } - pub(crate) fn capability_proof_matches(&self) -> bool { - self.capability_proof - .as_ref() - .is_none_or(crate::services::notification_sys::cross_pool_fence_fleet_proof_matches) - } - pub(crate) async fn mark_commit_started(&mut self) -> Result<()> { - if !self.capability_proof_matches() { - let ledger = self.ledger.as_ref().ok_or(StorageError::PartMissingOrCorrupt)?; - return Err(quota_capability_error(ledger.bucket(), ledger.error_object())); - } if let Some(ledger) = self.ledger.as_ref() { mark_commit_started(ledger).await?; } @@ -850,6 +826,11 @@ pub(crate) async fn begin( let snapshot_admission = opts.quota_admission; let data_movement = opts.data_movement; if crate::bucket::utils::is_meta_bucketname(bucket) { + let meta_bucket_fast_path_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH, + meta_bucket_fast_path_started, + ); return Ok(QuotaContext { store: None, bucket: bucket.to_string(), @@ -859,7 +840,6 @@ pub(crate) async fn begin( quota_revision: None, quota_limit: None, reservation_protocol: None, - capability_proof: None, snapshot_admission: None, legacy_data_movement: false, metadata_guard: None, @@ -878,7 +858,6 @@ pub(crate) async fn begin( quota_revision: None, quota_limit: Some(snapshot_admission.quota_limit()), reservation_protocol: None, - capability_proof: None, snapshot_admission: Some(snapshot_admission), legacy_data_movement: false, metadata_guard: None, @@ -897,7 +876,6 @@ pub(crate) async fn begin( quota_revision: None, quota_limit: None, reservation_protocol: None, - capability_proof: None, snapshot_admission: None, legacy_data_movement: false, metadata_guard: None, @@ -906,9 +884,63 @@ pub(crate) async fn begin( }); } + let metadata_lock_started = rustfs_io_metrics::put_stage_timer(); let metadata_guard = metadata_sys::acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK, + metadata_lock_started, + ); + let cache_lookup_started = rustfs_io_metrics::put_stage_timer(); + let cached_quota_snapshot = if rustfs_io_metrics::put_stage_metrics_enabled() { + match metadata_sys::get_cached_quota_config_and_incarnation_in(ctx, bucket).await { + Ok(Some(snapshot)) => { + rustfs_io_metrics::record_put_object_quota_cache_decision( + "candidate", + quota_cache_reason(snapshot.quota.as_ref()), + ); + Some(snapshot) + } + Ok(None) => { + rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "no_authoritative_cache"); + None + } + Err(_) => { + rustfs_io_metrics::record_put_object_quota_cache_decision("miss", "invalid_cached_quota"); + None + } + } + } else { + None + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP, + cache_lookup_started, + ); + let config_read_started = rustfs_io_metrics::put_stage_timer(); let (quota, bucket_incarnation, quota_revision) = metadata_sys::get_quota_config_and_incarnation_from_disk_for_options_in(ctx, bucket, opts, &metadata_guard).await?; + if let Some(cached) = cached_quota_snapshot.as_ref() { + let reason = quota_cache_reason(quota.as_ref()); + if cached.quota == quota && cached.bucket_incarnation == bucket_incarnation && cached.quota_revision == quota_revision { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_match", reason); + } else { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch", reason); + if cached.quota != quota { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota"); + } + if cached.bucket_incarnation != bucket_incarnation { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "bucket_incarnation"); + } + if cached.quota_revision != quota_revision { + rustfs_io_metrics::record_put_object_quota_cache_decision("authoritative_mismatch_field", "quota_revision"); + } + } + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ, + config_read_started, + ); + let lock_lost_check_started = rustfs_io_metrics::put_stage_timer(); if metadata_guard.is_lock_lost() { return Err(StorageError::NamespaceLockQuorumUnavailable { mode: "quota_config", @@ -918,6 +950,11 @@ pub(crate) async fn begin( achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK, + lock_lost_check_started, + ); + let policy_evaluate_started = rustfs_io_metrics::put_stage_timer(); if quota .as_ref() .is_some_and(|quota| quota.has_unsupported_reservation_protocol()) @@ -926,15 +963,15 @@ pub(crate) async fn begin( return Err(StorageError::PartMissingOrCorrupt); } let durable_quota = quota.as_ref().filter(|quota| quota.uses_durable_reservations()); - let capability_proof = if durable_quota.is_some() { - Some( - crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof() - .ok_or_else(|| quota_capability_error(bucket, &ledger_object(bucket)))?, - ) - } else { - None - }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE, + policy_evaluate_started, + ); + // Durable writes are fenced by a write quorum of participating disks that + // enforce the configured topology. Fleet-wide proof remains a configuration gate; + // losing an unrelated peer must not revoke an admitted object write. let durable_quota_limit = durable_quota.and_then(|quota| quota.quota); + let snapshot_admission_started = rustfs_io_metrics::put_stage_timer(); let reservation_protocol = durable_quota.and_then(|quota| quota.reservation_protocol); let snapshot_admission = match quota.as_ref().filter(|quota| !quota.uses_durable_reservations()) { Some(quota) => match (quota.quota, snapshot_admission) { @@ -954,6 +991,10 @@ pub(crate) async fn begin( }, None => None, }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION, + snapshot_admission_started, + ); let legacy_data_movement = durable_quota_limit.is_none() && quota.as_ref().and_then(|quota| quota.quota).is_some() && snapshot_admission.is_none() @@ -965,11 +1006,21 @@ pub(crate) async fn begin( .then(|| quota.as_ref().and_then(|quota| quota.quota)) .flatten() }); + let object_store_lookup_started = rustfs_io_metrics::put_stage_timer(); let store = if durable_quota_limit.is_some() { Some(metadata_sys::object_store_in(ctx).await?) } else { None }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP, + object_store_lookup_started, + ); + let context_build_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD, + context_build_started, + ); Ok(QuotaContext { store, bucket: bucket.to_string(), @@ -979,7 +1030,6 @@ pub(crate) async fn begin( quota_revision: Some(quota_revision), quota_limit, reservation_protocol, - capability_proof, snapshot_admission, legacy_data_movement, metadata_guard: Some(metadata_guard), @@ -988,13 +1038,12 @@ pub(crate) async fn begin( }) } -fn quota_capability_error(bucket: &str, object: &str) -> StorageError { - StorageError::NamespaceLockQuorumUnavailable { - mode: "quota_capability", - bucket: bucket.to_string(), - object: object.to_string(), - required: 1, - achieved: 0, +fn quota_cache_reason(quota: Option<&crate::bucket::quota::BucketQuota>) -> &'static str { + match quota { + None => "no_quota", + Some(quota) if quota.uses_durable_reservations() => "durable_quota", + Some(quota) if quota.quota.is_some() => "snapshot_quota", + Some(_) => "quota_without_limit", } } diff --git a/crates/ecstore/src/bucket/replication/replication_pool.rs b/crates/ecstore/src/bucket/replication/replication_pool.rs index f0eb97e2d..1c0d8cd52 100644 --- a/crates/ecstore/src/bucket/replication/replication_pool.rs +++ b/crates/ecstore/src/bucket/replication/replication_pool.rs @@ -3383,7 +3383,7 @@ async fn queue_replicate_deletes(batch: ReplicationHealResyncDeletes) -> Replica #[cfg(test)] mod tests { - use super::super::replication_filemeta_boundary::ReplicateTargetDecision; + use super::super::replication_filemeta_boundary::{ReplicateTargetDecision, ReplicationState, VersionPurgeStatusType}; use super::super::replication_resync_boundary::{decode_mrf_file, encode_mrf_file, encode_resync_file}; use super::super::replication_storage_boundary::{ DeletedObject, FileInfo, GetObjectReader, HTTPRangeSpec, ListOperations, ObjectIO, ObjectOperations, PutObjReader, @@ -3420,6 +3420,7 @@ mod tests { hold_first_read: AtomicBool, allow_first_read: Notify, read_count: AtomicUsize, + delete_storage_calls: AtomicUsize, reported_size: StdMutex>, stream_read_bytes: Arc, write_count: AtomicUsize, @@ -3628,6 +3629,7 @@ mod tests { _object: &str, _opts: &Self::ObjectOptions, ) -> Result { + self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst); Err(EcstoreError::NotImplemented) } @@ -3669,6 +3671,7 @@ mod tests { _object: &str, _opts: Self::ObjectOptions, ) -> Result { + self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst); Err(EcstoreError::NotImplemented) } @@ -3790,6 +3793,7 @@ mod tests { type NamespaceLock = rustfs_lock::NamespaceLockWrapper; async fn new_ns_lock(&self, bucket: &str, object: &str) -> Result { + self.shared.delete_storage_calls.fetch_add(1, Ordering::SeqCst); let lock = rustfs_lock::NamespaceLock::with_local_manager("load-resync-test".to_string(), self.shared.lock_manager.clone()); Ok(rustfs_lock::NamespaceLockWrapper::new( @@ -4166,6 +4170,7 @@ mod tests { hold_first_read: AtomicBool::new(false), allow_first_read: Notify::new(), read_count: AtomicUsize::new(0), + delete_storage_calls: AtomicUsize::new(0), reported_size: StdMutex::new(None), stream_read_bytes: Arc::new(AtomicUsize::new(0)), write_count: AtomicUsize::new(0), @@ -4177,6 +4182,106 @@ mod tests { }) } + #[tokio::test] + async fn delete_worker_empty_admission_skips_storage() { + let mut declined = ReplicateDecision::default(); + declined.set(ReplicateTargetDecision::new("arn:test".to_string(), false, false)); + for state in [ + None, + Some(ReplicationState::default()), + Some(ReplicationState { + replicate_decision_str: declined.to_string(), + ..Default::default() + }), + ] { + for delete_marker in [false, true] { + let shared = empty_resync_shared_state(); + let storage = Arc::new(LoadResyncNodeStore::new("empty-delete", shared.clone())); + let version_id = Uuid::new_v4(); + let delete = DeletedObjectReplicationInfo { + bucket: "no-replication".to_string(), + delete_object: ReplicationDeletedObject { + object_name: "expired-version".to_string(), + version_id: (!delete_marker).then_some(version_id), + delete_marker_version_id: delete_marker.then_some(version_id), + delete_marker, + replication_state: state.clone(), + ..Default::default() + }, + ..Default::default() + }; + let completed = replicate_delete_with_outcome(delete, storage).await; + assert_eq!( + shared.delete_storage_calls.load(Ordering::SeqCst), + 0, + "empty admission must not lock, read, or write storage (marker={delete_marker}, state={state:?})" + ); + assert!(completed, "an empty delete is a completed no-op"); + } + } + } + + #[tokio::test] + async fn delete_worker_preserves_persisted_admission_without_live_config() { + let arn = "arn:test"; + for state in [ + ReplicationState { + replicate_decision_str: test_replicate_decision(&[arn]).to_string(), + ..Default::default() + }, + ReplicationState { + targets: HashMap::from([(arn.to_string(), ReplicationStatusType::Pending)]), + ..Default::default() + }, + ReplicationState { + purge_targets: HashMap::from([(arn.to_string(), VersionPurgeStatusType::Pending)]), + ..Default::default() + }, + ] { + let shared = empty_resync_shared_state(); + let storage = Arc::new(LoadResyncNodeStore::new("pending-delete", shared.clone())); + let delete = DeletedObjectReplicationInfo { + bucket: "no-live-config".to_string(), + delete_object: ReplicationDeletedObject { + object_name: "pending-version".to_string(), + version_id: Some(Uuid::new_v4()), + replication_state: Some(state), + ..Default::default() + }, + ..Default::default() + }; + assert!( + !replicate_delete_with_outcome(delete, storage).await, + "the injected storage failure must retain admitted work for retry" + ); + assert!(shared.delete_storage_calls.load(Ordering::SeqCst) > 0); + } + } + + #[tokio::test] + async fn delete_worker_rejects_malformed_empty_admission() { + let shared = empty_resync_shared_state(); + let storage = Arc::new(LoadResyncNodeStore::new("invalid-delete", shared.clone())); + let delete = DeletedObjectReplicationInfo { + bucket: "no-replication".to_string(), + delete_object: ReplicationDeletedObject { + object_name: "invalid-decision".to_string(), + version_id: Some(Uuid::new_v4()), + replication_state: Some(ReplicationState { + replicate_decision_str: "invalid".to_string(), + ..Default::default() + }), + ..Default::default() + }, + ..Default::default() + }; + assert!( + !replicate_delete_with_outcome(delete, storage).await, + "malformed decisions must not be acknowledged as empty work" + ); + assert_eq!(shared.delete_storage_calls.load(Ordering::SeqCst), 0); + } + #[tokio::test] async fn durable_resync_read_observes_cancellation_after_cached_pending_intent() { let bucket = "startup-cancel-race"; @@ -5198,6 +5303,7 @@ mod tests { hold_first_read: AtomicBool::new(false), allow_first_read: Notify::new(), read_count: AtomicUsize::new(0), + delete_storage_calls: AtomicUsize::new(0), reported_size: StdMutex::new(None), stream_read_bytes: Arc::new(AtomicUsize::new(0)), write_count: AtomicUsize::new(0), diff --git a/crates/ecstore/src/bucket/replication/replication_resyncer.rs b/crates/ecstore/src/bucket/replication/replication_resyncer.rs index f69e662b4..e2b642681 100644 --- a/crates/ecstore/src/bucket/replication/replication_resyncer.rs +++ b/crates/ecstore/src/bucket/replication/replication_resyncer.rs @@ -36,9 +36,9 @@ use super::replication_object_decision_boundary::{ MustReplicateOptions, ReplicationMultipartPartInput, delete_marker_purge_mrf_entry, delete_marker_purge_version_id, delete_replication_creates_marker, delete_replication_target_version_id, heal_uses_delete_replication_path, is_object_lock_denied_delete, is_retryable_delete_replication_head_error, is_version_delete_replication, - replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size, - replication_multipart_part_plan, replication_single_put_size_error, resync_existing_delete_replication_info, - should_retry_delete_marker_purge, single_part_replica_etag_mismatch, + replicate_delete_outcome, replication_multipart_complete_actual_size, replication_multipart_part_plan, + replication_single_put_size_error, resync_existing_delete_replication_info, should_retry_delete_marker_purge, + single_part_replica_etag_mismatch, }; use super::replication_queue_boundary::{DeletedObjectReplicationInfo, ReplicationQueueAdmission}; use super::replication_resync_boundary::ResyncStatusType; @@ -60,12 +60,13 @@ use super::replication_storage_boundary::{ use super::replication_storage_boundary::{NamespaceLockFence, NamespaceLockSignalTestFence, ReplicationDeletedObject}; use super::replication_tagging_boundary::ReplicationTagFilter; use super::replication_target_boundary::{ - ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, - RemotePutObjectResponse, ReplicaLocation, ReplicationTargetStore, S3ClientError, SsecPassthroughCapability, - SsecPassthroughGate, TargetClient, VersionIdentityCapability, is_replication_target_offline_error, - replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options, - replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted, - replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version, + ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED, EncryptedReplicaShape, HeadObjectSdkError, PutObjectOptions, + PutObjectPartOptions, RemotePutObjectResponse, ReplicaCandidates, ReplicaEvidence, ReplicaIdentity, ReplicaLocation, + ReplicationTargetStore, S3ClientError, SsecPassthroughCapability, SsecPassthroughGate, TargetClient, + VersionIdentityCapability, is_replication_target_offline_error, replication_action_for_target_head, + replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options, + replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size, + replication_put_object_options, replication_source_etag_is_comparable, replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put, version_identity_drifted, }; @@ -349,15 +350,23 @@ fn replica_etag_verification_enabled() -> bool { /// target that stores a transformed payload (e.g. undecoded `aws-chunked` /// frames, #6853) returns the ETag of what it actually wrote. Reporting /// COMPLETED over such a replica is silent corruption, so a decidable -/// mismatch fails the replication instead. An SSE-C ciphertext passthrough -/// transfer is exempt: the wire bytes are ciphertext while the source ETag is -/// the plaintext MD5, and that path has its own HEAD-back audit. +/// mismatch fails the replication instead. Encrypted sources are exempt: an +/// SSE-C ciphertext passthrough sends ciphertext while the source ETag is the +/// plaintext MD5 (that path has its own HEAD-back audit), and a managed-SSE +/// (SSE-S3 / SSE-KMS) source stores a keyed fingerprint, or a legacy plaintext +/// MD5, that the ETag of the target's re-encrypted replica can neither confirm +/// nor refute. A managed-SSE source forwards no checksum either +/// (`replication_put_object_options`), so its replica has no post-PUT +/// content audit. fn verify_single_part_replica( object_info: &ObjectInfo, response: &RemotePutObjectResponse, ciphertext_passthrough: bool, ) -> std::result::Result<(), std::io::Error> { - if ciphertext_passthrough || !replica_etag_verification_enabled() { + if ciphertext_passthrough + || !replication_source_etag_is_comparable(&object_info.user_defined) + || !replica_etag_verification_enabled() + { return Ok(()); } if single_part_replica_etag_mismatch(object_info.etag.as_deref(), response.etag.as_deref()) { @@ -448,6 +457,26 @@ fn is_version_id_format_mismatch(err: &SdkError) -> bool { is_version_id_mismatch(code, raw_status) } +/// A version-addressed HEAD rejected because the target cannot address the +/// source version id (AWS S3: bodiless 400). Narrower than +/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine +/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a +/// completion verdict follows instead of another non-destructive HEAD. +fn rejects_source_version_id(err: &SdkError) -> bool { + is_version_id_format_mismatch(err) && !has_raw_status(err, 403) +} + +/// Whether the target's latest version of `object` is a delete marker. For +/// targets that mint their own version ids, which cannot be asked about the +/// source marker id: S3 answers a versionless HEAD on such a key with 404 +/// (405 on some targets) plus `x-amz-delete-marker: true`. +async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool { + match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await { + Ok(head) => head.delete_marker == Some(true), + Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"), + } +} + async fn mark_replication_target_offline_if_needed(target_client: &Arc, err: &(impl Display + ?Sized)) { if is_replication_target_offline_error(err) { ReplicationTargetStore::mark_target_offline(target_client).await; @@ -462,6 +491,7 @@ async fn head_object_fallback( Ok(head) => Ok(Some(LocatedReplica { version_id: head.version_id.clone(), head, + by: LocatedBy::CurrentVersion, })), Err(e) if head_object_not_found(&e) => Ok(None), Err(e) => Err(e), @@ -475,6 +505,63 @@ async fn head_object_fallback( struct LocatedReplica { head: HeadObjectOutput, version_id: Option, + by: LocatedBy, +} + +/// How [`replica_head_fallback`] reached a replica, which decides what may +/// confirm it (see [`located_replica_matches`]). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum LocatedBy { + /// The target version the source ledger recorded for this version. + Ledger, + /// The one candidate content identity left (`ListObjectVersions`). + ContentIdentity, + /// Whatever version is current under the key (AWS-style fallback): + /// nothing ties it to this source version but an ETag match. + CurrentVersion, +} + +/// How this source version recognises its replica on a target. `None` when +/// the source carries nothing to compare (no ETag on a plaintext source). +fn replica_identity(object_info: &ObjectInfo) -> Option> { + let etag = object_info.etag.as_deref().filter(|etag| !etag.trim().is_empty()); + if replication_source_etag_is_comparable(&object_info.user_defined) { + return etag.map(ReplicaIdentity::Etag); + } + // Same byte count the replication PUT sends: SSE-C passes the stored + // ciphertext through, managed SSE sends the decrypted plaintext. + let size = if replication_object_is_ssec_encrypted(&object_info.user_defined) { + object_info.size + } else { + object_info.get_actual_size().ok()? + }; + match object_info.mod_time { + Some(mod_time) if size >= 0 => Some(ReplicaIdentity::Encrypted { + etag, + shape: EncryptedReplicaShape { + size, + not_before_secs: mod_time.unix_timestamp(), + }, + }), + _ => etag.map(ReplicaIdentity::Etag), + } +} + +/// Whether a replica reached through [`replica_head_fallback`] holds this +/// source version. An ETag match always does. For an encrypted source on a +/// target that computes its own ETag, the shape does too — but only for a +/// replica the ledger or content identity singled out, never for whichever +/// version happens to be current. A plaintext source still needs its ETag. +fn located_replica_matches(identity: Option<&ReplicaIdentity<'_>>, located: &LocatedReplica) -> bool { + let Some(identity) = identity else { + return false; + }; + let head = &located.head; + match identity.evidence(head.e_tag(), head.content_length(), head.last_modified()) { + Some(ReplicaEvidence::Etag) => true, + Some(ReplicaEvidence::Shape) => located.by != LocatedBy::CurrentVersion, + None => false, + } } fn head_object_not_found(err: &HeadObjectSdkError) -> bool { @@ -556,15 +643,20 @@ fn ledger_target_version_id(user_defined: &HashMap, arn: &str) - /// Second look at a replica whose version-addressed HEAD failed, for the two /// target shapes where that failure is not a verdict on the replica: /// -/// - AWS-style 400/403 (the RustFS uuid is rejected as malformed): HEAD the -/// current version without a version id; callers compare ETags. +/// - AWS-style 400/403 (the RustFS uuid is rejected as malformed) and /// - 404 on a target that mints its own version ids (the Wasabi shape, -/// rustfs/backlog#2340): the source id never existed there. HEAD the id -/// the source ledger recorded for this target when it has one; otherwise, -/// once the target is known to mint ids, locate the replica by exact key -/// and ETag through ListObjectVersions and HEAD the id the target assigned. -/// Without this, every heal, MRF retry and existing-object resync re-drive -/// PUTs the object again and mints one more target version. +/// rustfs/backlog#2340): the source id never existed there. +/// +/// Both HEAD the id the source ledger recorded for this target first. Without +/// one, the AWS shape HEADs the current version without a version id +/// (callers compare ETags); the Wasabi shape, once the target is known to +/// mint ids, locates the replica by exact key and content identity through +/// ListObjectVersions and HEADs the id the target assigned. An encrypted +/// source takes that listing on the AWS shape too: its ETag never matches a +/// target-computed one (docs/operations/encrypted-etags.md), so only a unique +/// shape match can stand in for it. Without this, every heal, MRF retry and +/// existing-object resync re-drive PUTs the object again and mints one more +/// target version. /// /// `content_identity` is the sibling-ledger view used to disown candidates /// other source versions already claim; `None` disables content identity @@ -581,7 +673,7 @@ struct ReplicaSource<'a> { bucket: &'a str, object: &'a str, version_id: Option, - etag: Option<&'a str>, + identity: Option>, ledger_version_id: Option<&'a str>, } @@ -595,13 +687,11 @@ async fn replica_head_fallback( bucket: source_bucket, object, version_id: source_version_id, - etag: source_etag, + identity, ledger_version_id, } = source; - if is_version_id_format_mismatch(err) { - return Some(head_object_fallback(tgt_client, object).await); - } - if !head_object_not_found(err) { + let format_mismatch = is_version_id_format_mismatch(err); + if !format_mismatch && !head_object_not_found(err) { return None; } if let Some(ledger_version_id) = ledger_version_id { @@ -613,24 +703,45 @@ async fn replica_head_fallback( return Some(Ok(Some(LocatedReplica { head, version_id: Some(ledger_version_id.to_string()), + by: LocatedBy::Ledger, }))); } // The recorded version is gone: fall through to content identity // before concluding that the replica is missing. - Err(e) if head_object_not_found(&e) => {} + Err(e) if head_object_not_found(&e) || is_version_id_format_mismatch(&e) => {} Err(e) => return Some(Err(e)), } } - if !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable() { + let encrypted = matches!(identity, Some(ReplicaIdentity::Encrypted { .. })); + if format_mismatch { + if !(encrypted && content_identity.is_some()) { + return Some(head_object_fallback(tgt_client, object).await); + } + } else if !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable() { return None; } let siblings = content_identity?; - let etag = source_etag.filter(|etag| !etag.trim().is_empty())?; - Some(match tgt_client.replica_candidates_by_etag(&tgt_client.bucket, object, etag).await { + let identity = identity?; + let candidates = match tgt_client.replica_candidates(&tgt_client.bucket, object, &identity).await { + Ok(candidates) => Ok(candidates), + // A target that refused the version-addressed HEAD (a bodiless 403 + // can be a permission denial) may refuse the listing too: keep the + // current-version fallback, which re-PUTs, instead of stalling. + Err(_) if format_mismatch => return Some(head_object_fallback(tgt_client, object).await), + Err(list_err) => Err(list_err), + }; + Some(match candidates { Ok(candidates) => { - let location = + let (location, evidence) = disowned_replica_location(siblings, &tgt_client.arn, source_bucket, object, source_version_id, candidates).await; - match location.any_version_id() { + // Several same-ETag versions all hold the source bytes, so any of + // them proves the replica exists. Several same-shape versions do + // not: re-PUT rather than adopt the wrong generation. + let assigned_version_id = match (&location, evidence) { + (ReplicaLocation::Ambiguous { .. }, ReplicaEvidence::Shape) => None, + _ => location.any_version_id(), + }; + match assigned_version_id { Some(assigned_version_id) => { debug!( event = EVENT_REPLICATION_DRIFTED_REPLICA_LOCATED, @@ -641,6 +752,7 @@ async fn replica_head_fallback( arn = %tgt_client.arn, assigned_version_id = %assigned_version_id, ambiguous = matches!(location, ReplicaLocation::Ambiguous { .. }), + evidence = ?evidence, "Located replica by content identity on a target that mints its own version ids" ); match head_object_for_worker(tgt_client, &tgt_client.bucket, object, Some(assigned_version_id.to_string())) @@ -649,6 +761,7 @@ async fn replica_head_fallback( Ok(head) => Ok(Some(LocatedReplica { head, version_id: Some(assigned_version_id.to_string()), + by: LocatedBy::ContentIdentity, })), // The located version disappeared between LIST and HEAD. Err(e) if head_object_not_found(&e) => Ok(None), @@ -671,16 +784,15 @@ async fn disowned_replica_location( source_bucket: &str, object: &str, source_version_id: Option, - mut candidates: Vec, -) -> ReplicaLocation { + candidates: ReplicaCandidates, +) -> (ReplicaLocation, ReplicaEvidence) { if candidates.is_empty() { - return ReplicaLocation::Missing; + return (ReplicaLocation::Missing, ReplicaEvidence::Etag); } let claimed = siblings .claimed_target_versions(source_bucket, object, source_version_id, arn) .await; - candidates.retain(|candidate| !claimed.contains(candidate)); - ReplicaLocation::from_matches(candidates) + candidates.locate(&claimed) } /// The id the target assigned to a located replica when it is not the source @@ -1827,10 +1939,14 @@ fn spawn_resync_walk_task( /// Classify the target HEAD verification result for one resynced object, /// updating the per-object status counters and returning the accounted size -/// together with any verification error. +/// together with any verification error. `source` is the source version as +/// stored after the resync replicated it (its ledger and encryption decide +/// how the replica is recognised); without it the queued ETag is all there +/// is to go on. async fn verify_resync_head_result( head_result: std::result::Result, roi: &ReplicateObjectInfo, + source: Option<&ObjectInfo>, st: &mut TargetReplicationResyncStatus, target_client: &Arc, ) -> (i64, Option) { @@ -1846,7 +1962,17 @@ async fn verify_resync_head_result( // only absence proves completion. All other outcomes remain // failed (backlog#862 / #799 B13). let retryable = if roi.version_purge_status.is_empty() { - matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous) + match classify_delete_marker_head_error(&err) { + // A target that mints its own version ids never holds the + // source marker id; verify the marker on the latest version. + DeleteMarkerHeadOutcome::Ambiguous => { + let mints_own = rejects_source_version_id(&err) + || ReplicationTargetStore::version_identity_capability(&target_client.arn) + .version_addressing_unreliable(); + !(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await) + } + DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false, + } } else { !matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing) }; @@ -1862,14 +1988,23 @@ async fn verify_resync_head_result( // A version-addressed HEAD is not the last word on every target: // re-verify through the fallback before counting a well-replicated // object as failed (see `replica_head_fallback`). + let ledger_version_id = source.and_then(|info| ledger_target_version_id(&info.user_defined, &target_client.arn)); + let identity = match source { + Some(info) => replica_identity(info), + None => roi + .etag + .as_deref() + .filter(|etag| !etag.trim().is_empty()) + .map(ReplicaIdentity::Etag), + }; match replica_head_fallback( target_client.as_ref(), ReplicaSource { bucket: &roi.bucket, object: &roi.name, version_id: roi.version_id, - etag: roi.etag.as_deref(), - ledger_version_id: None, + identity, + ledger_version_id: ledger_version_id.as_deref(), }, Some(&NoSiblingLedger), &err, @@ -1894,6 +2029,24 @@ async fn verify_resync_head_result( } } +/// The resynced source version as stored now, ledger included, for the +/// verification fallback. `None` when it cannot be read. +async fn resync_source_object_info(storage: &Arc, roi: &ReplicateObjectInfo) -> Option { + storage + .get_object_info( + &roi.bucket, + &roi.name, + &ObjectOptions { + version_id: roi.version_id.map(|version_id| version_id.to_string()), + versioned: ReplicationVersioningStore::prefix_enabled(&roi.bucket, &roi.name).await, + version_suspended: ReplicationVersioningStore::prefix_suspended(&roi.bucket, &roi.name).await, + ..Default::default() + }, + ) + .await + .ok() +} + /// Replicate one existing object (or delete marker / version purge) to the /// resync target, verify the outcome via a target HEAD, and produce the /// per-object resync status update. @@ -1928,7 +2081,12 @@ async fn resync_worker_process_object( roi.version_id.map(|v| v.to_string()), ) .await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, target_client).await; + let source = if head_result.is_err() && !roi.delete_marker { + resync_source_object_info(storage, &roi).await + } else { + None + }; + let (size, err) = verify_resync_head_result(head_result, &roi, source.as_ref(), &mut st, target_client).await; if err.is_some() { debug!( @@ -2269,6 +2427,51 @@ pub(crate) async fn replicate_delete_with_outcome( dobj.delete_object.version_id }; + let dsc = match parse_replicate_decision( + &bucket, + &dobj + .delete_object + .replication_state + .as_ref() + .map(|v| v.replicate_decision_str.clone()) + .unwrap_or_default(), + ) { + Ok(dsc) => dsc, + Err(err) => { + debug!( + event = EVENT_REPLICATION_DELETE_SKIPPED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC, + bucket = %bucket, + arn = %dobj.target_arn, + error = %err, + reason = "replicate_decision_parse_failed", + "Failed to parse replicate decision" + ); + send_local_event(EventArgs { + event_name: EventName::ObjectReplicationNotTracked.to_string(), + bucket_name: bucket.clone(), + object: ObjectInfo { + bucket: bucket.clone(), + name: dobj.delete_object.object_name.clone(), + version_id, + delete_marker: dobj.delete_object.delete_marker, + ..Default::default() + }, + user_agent: "Internal: [Replication]".to_string(), + ..Default::default() + }); + return false; + } + }; + let target_arns = dobj.admitted_target_arns(); + if !dsc.replicate_any() && target_arns.is_empty() { + // Batch expiry can return an empty state after removing the source. + // It owes no remote work or local writeback. Persisted target admission + // must still be honored even when the live configuration is gone. + return true; + } + if dobj.delete_object.delete_marker && let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id { @@ -2355,43 +2558,6 @@ pub(crate) async fn replicate_delete_with_outcome( } } - let dsc = match parse_replicate_decision( - &bucket, - &dobj - .delete_object - .replication_state - .as_ref() - .map(|v| v.replicate_decision_str.clone()) - .unwrap_or_default(), - ) { - Ok(dsc) => dsc, - Err(err) => { - debug!( - event = EVENT_REPLICATION_DELETE_SKIPPED, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC, - bucket = %bucket, - arn = %dobj.target_arn, - error = %err, - reason = "replicate_decision_parse_failed", - "Failed to parse replicate decision" - ); - send_local_event(EventArgs { - event_name: EventName::ObjectReplicationNotTracked.to_string(), - bucket_name: bucket.clone(), - object: ObjectInfo { - bucket: bucket.clone(), - name: dobj.delete_object.object_name.clone(), - version_id, - delete_marker: dobj.delete_object.delete_marker, - ..Default::default() - }, - user_agent: "Internal: [Replication]".to_string(), - ..Default::default() - }); - return false; - } - }; let ns_lock = match storage .new_ns_lock(&bucket, format!("/[replicate]/{}", dobj.delete_object.object_name).as_str()) .await @@ -2471,7 +2637,6 @@ pub(crate) async fn replicate_delete_with_outcome( let mut join_set = JoinSet::new(); // Process each target - let target_arns = dobj.admitted_target_arns(); let expected_targets = dsc .targets_map .values() @@ -3325,6 +3490,9 @@ fn abandoned_purge_target_info(bucket: &str, dobj: &DeletedObjectReplicationInfo /// to resolve the replica on a target that mints its own version ids. struct VersionPurgeSource { etag: Option, + /// Encrypted sources: what identifies the replica when the target + /// computes its own ETag (see [`replica_identity`]). + encrypted_shape: Option, ledger: HashMap, ledger_corrupt: bool, /// Target versions other generations of the key claim, per target ARN: @@ -3370,8 +3538,13 @@ async fn version_purge_source( claimed_by_siblings.insert(target.arn.clone(), claimed); } } + let encrypted_shape = match replica_identity(&info) { + Some(ReplicaIdentity::Encrypted { shape, .. }) => Some(shape), + _ => None, + }; Some(VersionPurgeSource { etag: info.etag, + encrypted_shape, ledger, ledger_corrupt, claimed_by_siblings, @@ -3389,7 +3562,8 @@ enum PurgeReplicaResolution { /// goal is already met there. Absent, /// Refuse: no ledger entry and more than one target version carries the - /// source ETag, so any pick could destroy a live generation. + /// source ETag (or, for an encrypted source on a target that computes its + /// own ETag, its shape), so any pick could destroy a live generation. Ambiguous, /// Refuse: the persisted ledger is inconsistent. LedgerCorrupt, @@ -3417,15 +3591,19 @@ async fn resolve_purge_replica( if !force_lookup && !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable() { return PurgeReplicaResolution::SourceVersion; } - let Some(etag) = source.etag.as_deref().filter(|etag| !etag.trim().is_empty()) else { - return PurgeReplicaResolution::SourceVersion; + let etag = source.etag.as_deref().filter(|etag| !etag.trim().is_empty()); + let identity = match (source.encrypted_shape, etag) { + (Some(shape), etag) => ReplicaIdentity::Encrypted { etag, shape }, + (None, Some(etag)) => ReplicaIdentity::Etag(etag), + (None, None) => return PurgeReplicaResolution::SourceVersion, }; - match tgt_client.replica_candidates_by_etag(&tgt_client.bucket, object, etag).await { - Ok(mut candidates) => { - if let Some(claimed) = source.claimed_by_siblings.get(&tgt_client.arn) { - candidates.retain(|candidate| !claimed.contains(candidate)); - } - match ReplicaLocation::from_matches(candidates) { + match tgt_client.replica_candidates(&tgt_client.bucket, object, &identity).await { + Ok(candidates) => { + let no_claims = HashSet::new(); + let claimed = source.claimed_by_siblings.get(&tgt_client.arn).unwrap_or(&no_claims); + // Several shape matches stay ambiguous here: a purge never picks + // one to delete. + match candidates.locate(claimed).0 { ReplicaLocation::Unique(assigned) => PurgeReplicaResolution::Resolved(assigned), ReplicaLocation::Missing => PurgeReplicaResolution::Absent, ReplicaLocation::Ambiguous { .. } => PurgeReplicaResolution::Ambiguous, @@ -3567,7 +3745,16 @@ async fn replicate_delete_to_target( } } - if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() { + // A target that mints its own version ids never holds the source marker + // id, so a version-addressed pre-check cannot succeed there; the marker + // is created by the versionless DELETE below. + let version_addressable = + !ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable(); + if version_addressable + && !is_version_purge + && dobj.delete_object.delete_marker + && dobj.delete_object.delete_marker_version_id.is_some() + { match head_object_for_worker( tgt_client.as_ref(), &tgt_client.bucket, @@ -3576,6 +3763,11 @@ async fn replicate_delete_to_target( ) .await { + // The verdict is not known yet (e.g. no PUT since restart) and the + // target rejected the source id: learn it as the PUT path does. + Err(e) if rejects_source_version_id(e.as_ref()) => { + ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn); + } Ok(head) if head.delete_marker == Some(true) => { // Some targets expose an existing marker as a successful HEAD // with the marker bit instead of the usual 405. Treat that as @@ -4465,13 +4657,14 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo { // generation with the same bytes would be mistaken for one. // Only the ledger (a lost response of this very version) and // the AWS-style format fallback apply here. + let identity = replica_identity(&object_info); if let Some(fallback) = replica_head_fallback( &tgt_client, ReplicaSource { bucket: &bucket, object: &object, version_id: self.version_id, - etag: object_info.etag.as_deref(), + identity, ledger_version_id: ledger_version_id.as_deref(), }, None, @@ -4480,9 +4673,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo { .await { match fallback { - Ok(Some(located)) - if replication_etags_match(object_info.etag.as_deref(), located.head.e_tag.as_deref()) => - { + Ok(Some(located)) if located_replica_matches(identity.as_ref(), &located) => { if ssec_audit_required && !settle_ssec_passthrough_evidence(&located.head, &tgt_client, &bucket, &object, &mut rinfo) .await @@ -5130,13 +5321,14 @@ async fn resolve_replicate_all_action( } Err(e) => { let ledger_version_id = ledger_target_version_id(&object_info.user_defined, &tgt_client.arn); + let identity = replica_identity(&object_info); if let Some(fallback) = replica_head_fallback( tgt_client, ReplicaSource { bucket, object, version_id: roi.version_id, - etag: object_info.etag.as_deref(), + identity, ledger_version_id: ledger_version_id.as_deref(), }, Some(siblings), @@ -5147,20 +5339,20 @@ async fn resolve_replicate_all_action( match fallback { Ok(Some(located)) => { let oi = &located.head; - let etags_match = replication_etags_match(object_info.etag.as_deref(), oi.e_tag.as_deref()); - if require_existing_target && !etags_match { + let replica_matches = located_replica_matches(identity.as_ref(), &located); + if require_existing_target && !replica_matches { rinfo.error = Some("replica metadata target does not contain matching object data".to_string()); rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs(); return None; } - if etags_match + if replica_matches && ssec_audit_required && !settle_ssec_passthrough_evidence(oi, tgt_client, bucket, object, rinfo).await { rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs(); return None; } - if etags_match && let Some(assigned) = drifted_replica_version_id(&located, roi.version_id) { + if replica_matches && let Some(assigned) = drifted_replica_version_id(&located, roi.version_id) { // The replica lives under an id the target minted: // record it (the ledger every later mutation // resolves through) and apply metadata in place — @@ -5193,7 +5385,7 @@ async fn resolve_replicate_all_action( // records the sync without running the PUT. replication_action = ReplicationAction::None; } else { - replication_action = if etags_match { + replication_action = if replica_matches { ReplicationAction::None } else { ReplicationAction::All @@ -5826,6 +6018,57 @@ mod tests { use super::super::replication_filemeta_boundary::ReplicateTargetDecision; use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError; + #[test] + fn verify_single_part_replica_exempts_managed_sse_sources() { + // A new encrypted write stores a keyed fingerprint; a target that + // computes its own ETag answers with the MD5 of the plaintext it + // received. Both are 32 hex characters, so without the exemption the + // audit would fail every SSE-S3 / SSE-KMS replica (rustfs/backlog). + let fingerprint = "00112233445566778899aabbccddeeff"; + let replica_md5 = "ffeeddccbbaa99887766554433221100"; + let response = RemotePutObjectResponse { + version_id: None, + etag: Some(replica_md5.to_string()), + }; + for sse in ["AES256", "aws:kms"] { + let object_info = ObjectInfo { + bucket: "source".to_string(), + name: "encrypted".to_string(), + etag: Some(fingerprint.to_string()), + user_defined: HashMap::from([("x-amz-server-side-encryption".to_string(), sse.to_string())]).into(), + ..Default::default() + }; + verify_single_part_replica(&object_info, &response, false) + .unwrap_or_else(|err| panic!("{sse}: a managed-SSE source must not be audited against the replica MD5: {err}")); + } + } + + #[test] + fn verify_single_part_replica_still_fails_plaintext_mismatch() { + if !replica_etag_verification_enabled() { + return; + } + let object_info = ObjectInfo { + bucket: "source".to_string(), + name: "plain".to_string(), + etag: Some("00112233445566778899aabbccddeeff".to_string()), + ..Default::default() + }; + let mismatch = RemotePutObjectResponse { + version_id: None, + etag: Some("ffeeddccbbaa99887766554433221100".to_string()), + }; + let err = verify_single_part_replica(&object_info, &mismatch, false) + .expect_err("a plaintext source whose replica MD5 differs must fail the replication"); + assert_eq!(err.to_string(), REPLICA_ETAG_MISMATCH_ERROR); + + let matching = RemotePutObjectResponse { + version_id: None, + etag: object_info.etag.clone(), + }; + verify_single_part_replica(&object_info, &matching, false).expect("matching plaintext MD5s pass the audit"); + } + #[test] fn multipart_read_plan_preserves_legacy_plain_part_ranges() { const MIB: usize = 1024 * 1024; @@ -6552,13 +6795,361 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_none(), "{err:?}"); assert_eq!((size, st.replicated_count, st.failed_count), (4, 1, 0)); server.join().expect("test HTTP server should finish"); } + /// What a target that computes its own ETag (AWS S3) reports for the + /// replica of an encrypted source: the MD5 of the plaintext it received, + /// never the source's encrypted-ETag fingerprint ([`DRIFTED_ETAG`] here). + const COMPUTED_REPLICA_ETAG: &str = "0f0e0d0c0b0a09080706050403020100"; + /// The replica's write time on the scripted targets below. + const REPLICA_WRITTEN: &str = "2026-09-06T10:00:00.000Z"; + const REPLICA_WRITTEN_HTTP: &str = "Sun, 06 Sep 2026 10:00:00 GMT"; + + /// An SSE-S3 source version: 4 plaintext bytes, stored as ciphertext, its + /// ETag a keyed fingerprint, written at `written` (unix seconds). + fn encrypted_roi_and_object(written: i64) -> (ReplicateObjectInfo, ObjectInfo) { + let (roi, mut object_info) = drifted_roi_and_object(); + object_info.size = 36; + object_info.actual_size = 4; + object_info.mod_time = Some(OffsetDateTime::from_unix_timestamp(written).expect("valid timestamp")); + object_info.user_defined = HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())]).into(); + (roi, object_info) + } + + fn replica_written_secs() -> i64 { + OffsetDateTime::parse(REPLICA_WRITTEN, &Rfc3339) + .expect("valid replica time") + .unix_timestamp() + } + + fn computed_etag_versions_response(versions: &[(&str, i64)]) -> String { + let versions = versions + .iter() + .enumerate() + .map(|(index, (version_id, size))| { + format!( + "object{version_id}{}{REPLICA_WRITTEN}"{COMPUTED_REPLICA_ETAG}"{size}STANDARD", + index == 0 + ) + }) + .collect::(); + xml_response( + "200 OK", + format!( + "target-bucketobject1000false{versions}" + ), + ) + } + + fn computed_etag_head_response(version_id: &str, size: i64) -> String { + format!( + "HTTP/1.1 200 OK\r\nETag: \"{COMPUTED_REPLICA_ETAG}\"\r\nContent-Length: {size}\r\nx-amz-version-id: {version_id}\r\nLast-Modified: {REPLICA_WRITTEN_HTTP}\r\nConnection: close\r\n\r\n" + ) + } + + /// A Wasabi/AWS-shaped target that computes its own ETags: 404 on the + /// source uuid, the listed versions, and a HEAD for any listed id. + fn spawn_computed_etag_target( + requests: usize, + versions: &'static [(&'static str, i64)], + ) -> (String, std::thread::JoinHandle>) { + spawn_scripted_target_server(requests, move |line| { + if line.starts_with("GET ") && line.contains("versions") { + return computed_etag_versions_response(versions); + } + if line.starts_with("HEAD ") + && let Some((version_id, size)) = versions + .iter() + .find(|(version_id, _)| line.contains(&format!("versionId={version_id}"))) + { + return computed_etag_head_response(version_id, *size); + } + if line.starts_with("HEAD ") { + return empty_response("404 Not Found"); + } + empty_response("500 Unexpected") + }) + } + + async fn resolve_heal_action( + target: &Arc, + roi: &ReplicateObjectInfo, + object_info: ObjectInfo, + rinfo: &mut ReplicatedTargetInfo, + ) -> Option<(ReplicationAction, ObjectInfo)> { + resolve_replicate_all_action( + ReplicateAllActionContext { + siblings: &NoSiblingLedger, + roi, + tgt_client: target, + bucket: &roi.bucket, + object: &roi.name, + start_time: OffsetDateTime::now_utc(), + ssec_audit_required: false, + }, + object_info, + rinfo, + ) + .await + } + + #[tokio::test] + async fn heal_redrive_locates_encrypted_replica_by_shape_on_target_that_computes_own_etags() { + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(3, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!( + matches!(action, Some((ReplicationAction::None, _))), + "an encrypted replica located by its shape must not be re-driven: {action:?}" + ); + assert!(rinfo.error.is_none(), "{:?}", rinfo.error); + assert_eq!(rinfo.target_version_id.as_deref(), Some(DRIFTED_ASSIGNED_VERSION_ID)); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 3, "HEAD by source id, ListObjectVersions, HEAD by assigned id: {seen:?}"); + } + + #[tokio::test] + async fn heal_redrive_does_not_adopt_an_encrypted_replica_of_another_size_or_an_older_write() { + // Size differs from the 4 plaintext bytes. + static OTHER_SIZE: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 36)]; + let (endpoint, server) = spawn_computed_etag_target(2, OTHER_SIZE); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + server.join().expect("test HTTP server should finish"); + + // Same size, but written before the source version existed. + static SAME_SIZE: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, SAME_SIZE); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() + 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + server.join().expect("test HTTP server should finish"); + } + + #[tokio::test] + async fn heal_redrive_does_not_pick_among_several_encrypted_shape_matches() { + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4), ("another-minted-id", 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!( + matches!(action, Some((ReplicationAction::All, _))), + "two same-shape versions are not evidence of either: {action:?}" + ); + assert!(rinfo.target_version_id.is_none(), "{:?}", rinfo.target_version_id); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "HEAD by source id, ListObjectVersions, no HEAD of a guess: {seen:?}"); + } + + #[tokio::test] + async fn heal_redrive_keeps_exact_etag_identity_for_plaintext_sources() { + // A same-size replica whose ETag differs is not this plaintext + // version's replica: the shape criterion is for encrypted sources only. + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, mut object_info) = drifted_roi_and_object(); + object_info.mod_time = Some(OffsetDateTime::from_unix_timestamp(replica_written_secs() - 1).expect("valid timestamp")); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + server.join().expect("test HTTP server should finish"); + } + + #[tokio::test] + async fn heal_redrive_never_adopts_an_unversioned_null_version_by_shape() { + // An unversioned target's `null` version holds whatever generation + // was written last; a same-size newer generation must not be claimed. + static VERSIONS: &[(&str, i64)] = &[("null", 4)]; + let (endpoint, server) = spawn_computed_etag_target(2, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!(matches!(action, Some((ReplicationAction::All, _))), "{action:?}"); + assert!(rinfo.target_version_id.is_none(), "{:?}", rinfo.target_version_id); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "HEAD by source id, ListObjectVersions, nothing adopted: {seen:?}"); + } + + #[tokio::test] + async fn encrypted_lookup_keeps_the_current_version_fallback_when_listing_is_denied() { + // A bodiless 403 on the version-addressed HEAD may be a permission + // denial; the same credentials then refuse ListObjectVersions. The + // object must still be re-driven, as before content identity existed. + let (endpoint, server) = spawn_scripted_target_server(3, |line| { + let listing = line.starts_with("GET ") && line.contains("versions"); + let version_addressed_head = line.starts_with("HEAD ") && line.contains("versionId="); + if listing || version_addressed_head { + empty_response("403 Forbidden") + } else if line.starts_with("HEAD ") { + computed_etag_head_response(DRIFTED_ASSIGNED_VERSION_ID, 4) + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!( + matches!(action, Some((ReplicationAction::All, _))), + "a denied listing must fall back to the re-PUT, not stall: {action:?}" + ); + assert!(rinfo.error.is_none(), "{:?}", rinfo.error); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!( + seen.len(), + 3, + "HEAD by source id, ListObjectVersions, HEAD of the current version: {seen:?}" + ); + assert!(seen[2].starts_with("HEAD ") && !seen[2].contains("versionId="), "{}", seen[2]); + } + + #[tokio::test] + async fn encrypted_ledger_replica_converges_on_an_aws_style_target() { + // AWS rejects the RustFS uuid as malformed (400). The ledger names the + // replica; its computed ETag cannot confirm it, its shape does. + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.starts_with("HEAD ") && line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}")) { + computed_etag_head_response(DRIFTED_ASSIGNED_VERSION_ID, 4) + } else if line.starts_with("HEAD ") { + empty_response("400 Bad Request") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + let (roi, mut object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut user_defined = (*object_info.user_defined).clone(); + insert_str( + &mut user_defined, + &format!("{SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX}{}", target.arn), + DRIFTED_ASSIGNED_VERSION_ID.to_string(), + ); + object_info.user_defined = Arc::new(user_defined); + let mut rinfo = replicate_all_target_info(&roi, &target); + + let action = resolve_heal_action(&target, &roi, object_info, &mut rinfo).await; + + assert!(matches!(action, Some((ReplicationAction::None, _))), "{action:?}"); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "HEAD by source id, HEAD by the ledger id: {seen:?}"); + assert!(seen[1].contains(DRIFTED_ASSIGNED_VERSION_ID), "{}", seen[1]); + } + + #[tokio::test] + async fn resync_verification_counts_encrypted_replica_by_shape() { + static VERSIONS: &[(&str, i64)] = &[(DRIFTED_ASSIGNED_VERSION_ID, 4)]; + let (endpoint, server) = spawn_computed_etag_target(3, VERSIONS); + let target = test_target_client(endpoint); + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let (roi, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (_, err) = verify_resync_head_result(head_result, &roi, Some(&object_info), &mut st, &target).await; + + assert!(err.is_none(), "{err:?}"); + assert_eq!((st.replicated_count, st.failed_count), (1, 0)); + server.join().expect("test HTTP server should finish"); + } + + fn encrypted_purge_source() -> Arc { + let (_, object_info) = encrypted_roi_and_object(replica_written_secs() - 1); + let Some(ReplicaIdentity::Encrypted { shape, .. }) = replica_identity(&object_info) else { + panic!("an SSE-S3 source must be identified by its shape"); + }; + Arc::new(VersionPurgeSource { + etag: object_info.etag, + encrypted_shape: Some(shape), + ledger: HashMap::new(), + ledger_corrupt: false, + claimed_by_siblings: HashMap::new(), + }) + } + + #[tokio::test] + async fn encrypted_version_purge_without_ledger_deletes_the_unique_shape_match() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.starts_with("GET ") && line.contains("versions") { + computed_etag_versions_response(&[(DRIFTED_ASSIGNED_VERSION_ID, 4)]) + } else if line.starts_with("DELETE ") && line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}")) { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let dobj = version_purge_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), Some(encrypted_purge_source())).await; + + assert_eq!(rinfo.version_purge_status, VersionPurgeStatusType::Complete, "{:?}", rinfo.error); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "ListObjectVersions, then DELETE by the located id: {seen:?}"); + assert!( + seen[1].starts_with("DELETE ") && seen[1].contains(DRIFTED_ASSIGNED_VERSION_ID), + "{}", + seen[1] + ); + } + + #[tokio::test] + async fn encrypted_version_purge_refuses_several_shape_matches() { + let (endpoint, server) = spawn_scripted_target_server(1, |line| { + if line.starts_with("GET ") && line.contains("versions") { + computed_etag_versions_response(&[(DRIFTED_ASSIGNED_VERSION_ID, 4), ("another-minted-id", 4)]) + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let dobj = version_purge_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), Some(encrypted_purge_source())).await; + + assert_eq!(rinfo.version_purge_status, VersionPurgeStatusType::Failed); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "no DELETE of a guessed replica: {seen:?}"); + } + fn delete_marker_roi() -> ReplicateObjectInfo { ReplicateObjectInfo { bucket: "source".to_string(), @@ -6675,6 +7266,93 @@ mod tests { assert!(seen[0].starts_with("HEAD "), "{seen:?}"); } + /// AWS S3 answers a HEAD for a version id it did not mint with a bodiless + /// 400. On a target with no verdict yet, that teaches `MintsOwn` and the + /// marker is still created with one versionless DELETE. + #[tokio::test] + async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.starts_with("HEAD ") { + empty_response("400 Bad Request") + } else if line.starts_with("DELETE ") { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + let dobj = marker_creation_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await; + + assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}"); + assert_eq!( + ReplicationTargetStore::version_identity_capability(&target.arn), + VersionIdentityCapability::MintsOwn + ); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 2, "{seen:?}"); + assert!(seen[0].starts_with("HEAD "), "{seen:?}"); + assert!(seen[1].starts_with("DELETE "), "{seen:?}"); + assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}"); + } + + /// A bodiless 403 on the version-addressed HEAD may be a genuine + /// GetObjectVersion denial on an id-adopting target. It must fail closed: + /// no DELETE, and no `MintsOwn` verdict cached for the ARN. + #[tokio::test] + async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() { + let (endpoint, server) = spawn_scripted_target_server(1, |line| { + if line.starts_with("HEAD ") { + empty_response("403 Forbidden") + } else if line.starts_with("DELETE ") { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + let dobj = marker_creation_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await; + + assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}"); + assert_ne!( + ReplicationTargetStore::version_identity_capability(&target.arn), + VersionIdentityCapability::MintsOwn + ); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}"); + assert!(seen[0].starts_with("HEAD "), "{seen:?}"); + } + + /// A target already known to mint its own version ids is never asked for + /// the source marker id: the marker is created by the versionless DELETE. + #[tokio::test] + async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() { + let (endpoint, server) = spawn_scripted_target_server(1, |line| { + if line.starts_with("DELETE ") { + empty_response("204 No Content") + } else { + empty_response("500 Unexpected") + } + }); + let target = test_target_client(endpoint); + register_test_target(&target).await; + ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn); + let dobj = marker_creation_dobj(&target.arn); + + let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await; + + assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}"); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}"); + assert!(seen[0].starts_with("DELETE "), "{seen:?}"); + assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}"); + } + /// A target answers `HEAD ?versionId=` with a bodiless /// 405, so the SDK reports no error code. That is the marker having /// propagated, not a failure (rustfs/backlog#2479). @@ -6696,7 +7374,7 @@ mod tests { None, "the fixture must reproduce the codeless 405 the SDK yields for a bodiless HEAD" ); - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_none(), "{err:?}"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0)); @@ -6723,7 +7401,7 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_some(), "a 405 must not count as a completed purge ({status:?})"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1), "{status:?}"); @@ -6741,7 +7419,7 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_none(), "{err:?}"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0)); @@ -6758,13 +7436,83 @@ mod tests { let head_result = head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; - let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; assert!(err.is_some(), "a 503 must not count as a propagated delete marker"); assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1)); server.join().expect("test HTTP server should finish"); } + /// AWS answers `HEAD ?versionId=` with a bodiless 400. + /// The marker is verified through the latest version instead: a + /// versionless 404 with `x-amz-delete-marker: true` proves it propagated. + #[tokio::test] + async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.contains("versionId=") { + empty_response("400 Bad Request") + } else { + "HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + .to_string() + } + }); + let target = test_target_client(endpoint); + let roi = delete_marker_roi(); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; + + assert!(err.is_none(), "{err:?}"); + assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0)); + let seen = server.join().expect("test HTTP server should finish"); + assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}"); + } + + /// A live latest version on a target that rejects the source id means the + /// marker has not propagated: the verification stays failed. + #[tokio::test] + async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() { + let (endpoint, server) = spawn_scripted_target_server(2, |line| { + if line.contains("versionId=") { + empty_response("400 Bad Request") + } else { + empty_response("200 OK") + } + }); + let target = test_target_client(endpoint); + let roi = delete_marker_roi(); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; + + assert!(err.is_some(), "a live latest version must not verify the marker"); + assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1)); + server.join().expect("test HTTP server should finish"); + } + + /// A bodiless 403 may be a genuine permission denial: it is not evidence + /// that the target mints its own ids, so no latest-version re-check runs. + #[tokio::test] + async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() { + let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden")); + let target = test_target_client(endpoint); + let roi = delete_marker_roi(); + let mut st = TargetReplicationResyncStatus::default(); + + let head_result = + head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await; + let (size, err) = verify_resync_head_result(head_result, &roi, None, &mut st, &target).await; + + assert!(err.is_some(), "a 403 must not count as a propagated delete marker"); + assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1)); + let seen = server.join().expect("test HTTP server should finish"); + assert_eq!(seen.len(), 1, "{seen:?}"); + } + /// Serves exactly `requests` connections, answering each from /// `respond(request_line)`, and returns the request lines it saw. Reads /// the whole request (headers plus `Content-Length` body, honoring @@ -6874,6 +7622,7 @@ mod tests { fn purge_source(ledger: &[(&str, &str)]) -> Arc { Arc::new(VersionPurgeSource { etag: Some(DRIFTED_ETAG.to_string()), + encrypted_shape: None, ledger: ledger .iter() .map(|(arn, version_id)| (arn.to_string(), version_id.to_string())) @@ -7078,6 +7827,7 @@ mod tests { let dobj = version_purge_dobj(&target.arn); let source = Arc::new(VersionPurgeSource { etag: Some(DRIFTED_ETAG.to_string()), + encrypted_shape: None, ledger: HashMap::new(), ledger_corrupt: false, claimed_by_siblings: HashMap::from([(target.arn.clone(), HashSet::from([DRIFTED_ASSIGNED_VERSION_ID.to_string()]))]), @@ -7165,6 +7915,7 @@ mod tests { let dobj = version_purge_dobj(&target.arn); let source = Arc::new(VersionPurgeSource { etag: Some(DRIFTED_ETAG.to_string()), + encrypted_shape: None, ledger: HashMap::new(), ledger_corrupt: true, claimed_by_siblings: HashMap::new(), diff --git a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs index e71ae9f19..2d7fba2dd 100644 --- a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs +++ b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs @@ -37,8 +37,9 @@ use time::OffsetDateTime; use time::format_description::well_known::Rfc3339; pub(crate) use crate::bucket::bucket_target_sys::{ - AdvancedPutOptions, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, RemotePutObjectResponse, RemoveObjectOptions, - ReplicaLocation, S3ClientError, TargetClient, resolve_read_api_version_id, + AdvancedPutOptions, EncryptedReplicaShape, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, + RemotePutObjectResponse, RemoveObjectOptions, ReplicaCandidates, ReplicaEvidence, ReplicaIdentity, ReplicaLocation, + S3ClientError, TargetClient, resolve_read_api_version_id, }; #[cfg(test)] pub(crate) use crate::bucket::target::BucketTarget; @@ -157,6 +158,19 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap) -> bool { + matches!( + classify_replication_source_encryption(user_defined), + ReplicationSourceEncryption::Plaintext + ) +} + /// HeadObjectOutput adapter over the pure SSE-C passthrough evidence /// judgment owned by `rustfs-replication`: extract the echoed /// customer-algorithm header and let the crate-owned policy decide. @@ -578,6 +592,22 @@ mod tests { use time::Duration; use uuid::Uuid; + #[test] + fn replication_source_etag_is_comparable_only_for_plaintext_sources() { + assert!(replication_source_etag_is_comparable(&HashMap::new())); + for (key, value) in [ + (metadata_keys::SERVER_SIDE_ENCRYPTION, "AES256"), + (metadata_keys::SERVER_SIDE_ENCRYPTION, "aws:kms"), + (SSEC_ALGORITHM_HEADER, "AES256"), + ] { + let metadata = HashMap::from([(key.to_string(), value.to_string())]); + assert!( + !replication_source_etag_is_comparable(&metadata), + "{key}={value} must not expose its stored ETag to the replica audit" + ); + } + } + /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the /// SSE intent as lowercase `x-amz-server-side-encryption` /// (`encryption_material_to_metadata`; header-derived keys come from a diff --git a/crates/ecstore/src/bucket/utils.rs b/crates/ecstore/src/bucket/utils.rs index f659468bb..850e60840 100644 --- a/crates/ecstore/src/bucket/utils.rs +++ b/crates/ecstore/src/bucket/utils.rs @@ -15,6 +15,7 @@ use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET}; use crate::error::{Error, Result, StorageError}; use regex::Regex; +use rustfs_utils::WindowsFileNameError; use rustfs_utils::path::SLASH_SEPARATOR; use s3s::xml; use tracing::instrument; @@ -178,8 +179,7 @@ pub fn is_valid_object_name(object: &str) -> bool { dead_code, reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" )] -pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str = - "object key contains characters unsupported on Windows hosts (one of ':', '*', '?', '\"', '|', '<', '>')"; +pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str = "object key contains characters unsupported on Windows hosts (one of ':', '*', '?', '\"', '|', '<', '>', or a control character)"; /// Client-facing reason for path segments Windows can store but not address /// afterwards (issue #3449): trailing dot/space or reserved DOS device names. @@ -187,37 +187,46 @@ pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str = dead_code, reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" )] -pub const WINDOWS_RESERVED_SEGMENT_REASON: &str = "object key contains a path segment unsupported on Windows hosts (trailing dot or space, or a reserved device name such as NUL/CON/COM1)"; +pub const WINDOWS_RESERVED_SEGMENT_REASON: &str = "object key contains a path segment unsupported on Windows hosts (trailing dot or space, or a reserved device name such as NUL/CON/COM1/CONIN$)"; -/// Reserved DOS device names that shadow regular files on Windows, even when -/// an extension is appended (e.g. `NUL.txt` resolves to the `NUL` device). +/// Client-facing reason for keys containing `\`: Windows hosts treat it as a +/// path separator, so `a\b` would land on the same file as `a/b`. +pub const WINDOWS_BACKSLASH_REASON: &str = + "object key contains '\\', which Windows hosts treat as a path separator (it would alias the same key spelled with '/')"; + +/// Returns the client-facing reason when `object` cannot round-trip through +/// the Win32 path layer, or `None` when it can. +/// +/// Each segment is checked with [`rustfs_utils::check_windows_file_name`]. +/// `\` splits segments here, like on a Windows disk, so this check never +/// rejects it: an existing `a\b` key must stay readable and deletable. New +/// writes reject `\` separately in [`check_new_object_args`]. #[allow( dead_code, reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" )] -const WINDOWS_RESERVED_NAMES: &[&str] = &[ - "CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT1", "LPT2", "LPT3", - "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9", -]; +pub fn windows_object_key_violation(object: &str) -> Option<&'static str> { + // A NUL byte is rejected on every platform as ObjectNameInvalid by + // is_valid_object_prefix; keep that answer on Windows too. + if object.contains('\0') { + return None; + } -/// Returns true when `object` contains a path segment that NTFS can store but -/// the Win32 API cannot address afterwards (issue #3449): segments ending in a -/// dot or a space, and reserved DOS device names — bare or with an extension -/// (`NUL.txt`), matching classic Win32 path resolution semantics. -#[allow( - dead_code, - reason = "live on Windows: callers sit inside the #[cfg(target_os = \"windows\")] block in check_object_name_for_length_and_slash (backlog#1823)" -)] -pub fn object_name_has_windows_incompatible_segment(object: &str) -> bool { - object.split(['/', '\\']).any(|segment| { - if segment.ends_with('.') || segment.ends_with(' ') { - return true; + let mut segment_violation = false; + for segment in object.split(['/', '\\']).filter(|segment| !segment.is_empty()) { + match rustfs_utils::check_windows_file_name(segment) { + Err(WindowsFileNameError::ReservedCharacter | WindowsFileNameError::ControlCharacter) => { + return Some(WINDOWS_RESERVED_CHARACTERS_REASON); + } + Err(WindowsFileNameError::TrailingDotOrSpace | WindowsFileNameError::ReservedDeviceName) => { + segment_violation = true; + } + // Segment length is enforced in bytes by object_key_segments_fit_on_disk, + // which is stricter than the UTF-16 limit and answers ObjectNameInvalid. + Ok(()) | Err(WindowsFileNameError::TooLong | WindowsFileNameError::Empty) => {} } - // Device names are matched on the part before the first dot, with - // trailing spaces ignored (so `NUL .txt` is also reserved). - let base = segment.split('.').next().unwrap_or(segment).trim_end_matches(' '); - WINDOWS_RESERVED_NAMES.iter().any(|name| base.eq_ignore_ascii_case(name)) - }) + } + segment_violation.then_some(WINDOWS_RESERVED_SEGMENT_REASON) } pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Result<()> { @@ -229,33 +238,33 @@ pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Res return Err(StorageError::ObjectNamePrefixAsSlash(bucket.to_owned(), object.to_owned())); } + // Reject names that NTFS would refuse, or create but the Win32 path layer + // cannot read back (os error 3), e.g. `a:b`, `baddir.` or `NUL.txt`. #[cfg(target_os = "windows")] - { - if object.contains(':') - || object.contains('*') - || object.contains('?') - || object.contains('"') - || object.contains('|') - || object.contains('<') - || object.contains('>') - // || object.contains('\\') - { - return Err(StorageError::InvalidArgument( - bucket.to_owned(), - object.to_owned(), - WINDOWS_RESERVED_CHARACTERS_REASON.to_owned(), - )); - } + if let Some(reason) = windows_object_key_violation(object) { + return Err(StorageError::InvalidArgument(bucket.to_owned(), object.to_owned(), reason.to_owned())); + } - // Reject names that NTFS would happily create but the Win32 path - // layer cannot read back (os error 3), e.g. `baddir.` or `NUL.txt`. - if object_name_has_windows_incompatible_segment(object) { - return Err(StorageError::InvalidArgument( - bucket.to_owned(), - object.to_owned(), - WINDOWS_RESERVED_SEGMENT_REASON.to_owned(), - )); - } + Ok(()) +} + +/// Admission check for every request that introduces `object` into the +/// namespace: PutObject, a CopyObject destination, and CreateMultipartUpload. +/// Sharing one entry keeps their rejections (and S3 error codes) identical. +/// +/// `data_movement` writes (decommission, rebalance) relocate keys that already +/// exist, so they skip the new-key-only rules and stay able to move old data. +/// Internal meta buckets are exempt too: their keys are produced by RustFS +/// itself and resolve to the same path either way. +pub fn check_new_object_args(bucket: &str, object: &str, data_movement: bool) -> Result<()> { + check_put_object_args(bucket, object)?; + + if cfg!(target_os = "windows") && !data_movement && !is_meta_bucketname(bucket) && object.contains('\\') { + return Err(StorageError::InvalidArgument( + bucket.to_owned(), + object.to_owned(), + WINDOWS_BACKSLASH_REASON.to_owned(), + )); } Ok(()) @@ -316,10 +325,6 @@ pub fn check_bucket_and_object_names(bucket: &str, object: &str) -> Result<()> { return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string())); } - // if cfg!(target_os = "windows") && object.contains('\\') { - // return Err(StorageError::ObjectNameInvalid(bucket.to_string(), object.to_string())); - // } - Ok(()) } @@ -586,103 +591,193 @@ mod tests { } #[test] - fn test_object_name_has_windows_incompatible_segment() { + fn test_windows_object_key_violation_segments() { + let segment = Some(WINDOWS_RESERVED_SEGMENT_REASON); + // Segments ending in a dot are writable on NTFS but unreadable via Win32 paths. - assert!(object_name_has_windows_incompatible_segment("baddir.")); - assert!(object_name_has_windows_incompatible_segment("dir1/dir2/baddir.")); - assert!(object_name_has_windows_incompatible_segment("dir1/baddir./file.txt")); - assert!(object_name_has_windows_incompatible_segment("file.txt.")); + assert_eq!(windows_object_key_violation("baddir."), segment); + assert_eq!(windows_object_key_violation("dir1/dir2/baddir."), segment); + assert_eq!(windows_object_key_violation("dir1/baddir./file.txt"), segment); + assert_eq!(windows_object_key_violation("file.txt."), segment); // Segments ending in a space. - assert!(object_name_has_windows_incompatible_segment("file.txt ")); - assert!(object_name_has_windows_incompatible_segment("dir /file.txt")); + assert_eq!(windows_object_key_violation("file.txt "), segment); + assert_eq!(windows_object_key_violation("dir /file.txt"), segment); // Reserved DOS device names, bare and case-insensitive. - assert!(object_name_has_windows_incompatible_segment("NUL")); - assert!(object_name_has_windows_incompatible_segment("nul")); - assert!(object_name_has_windows_incompatible_segment("CON")); - assert!(object_name_has_windows_incompatible_segment("prn")); - assert!(object_name_has_windows_incompatible_segment("Aux")); - assert!(object_name_has_windows_incompatible_segment("COM1")); - assert!(object_name_has_windows_incompatible_segment("com9")); - assert!(object_name_has_windows_incompatible_segment("LPT1")); - assert!(object_name_has_windows_incompatible_segment("lpt9")); - assert!(object_name_has_windows_incompatible_segment("dir/NUL/file.txt")); + for object in [ + "NUL", + "nul", + "CON", + "prn", + "Aux", + "COM1", + "com9", + "LPT1", + "lpt9", + "dir/NUL/file.txt", + // Superscript port numbers and console aliases are devices too. + "COM\u{b9}", + "dir/lpt\u{b3}", + "CONIN$", + "conout$", + ] { + assert_eq!(windows_object_key_violation(object), segment, "{object:?}"); + } // Reserved device names with an extension still resolve to the device. - assert!(object_name_has_windows_incompatible_segment("NUL.txt")); - assert!(object_name_has_windows_incompatible_segment("dir/aux.log")); - assert!(object_name_has_windows_incompatible_segment("con.tar.gz")); - assert!(object_name_has_windows_incompatible_segment("com1.dat")); - assert!(object_name_has_windows_incompatible_segment("NUL .txt")); + for object in [ + "NUL.txt", + "dir/aux.log", + "con.tar.gz", + "com1.dat", + "NUL .txt", + "COM\u{b2}.log", + ] { + assert_eq!(windows_object_key_violation(object), segment, "{object:?}"); + } // Backslash-separated segments are checked as well. - assert!(object_name_has_windows_incompatible_segment("dir\\baddir.\\file.txt")); - assert!(object_name_has_windows_incompatible_segment("dir\\nul")); + assert_eq!(windows_object_key_violation("dir\\baddir.\\file.txt"), segment); + assert_eq!(windows_object_key_violation("dir\\nul"), segment); // Valid names must not be flagged. - assert!(!object_name_has_windows_incompatible_segment("file.txt")); - assert!(!object_name_has_windows_incompatible_segment("dir.name/file")); - assert!(!object_name_has_windows_incompatible_segment("path/to/file.tar.gz")); - assert!(!object_name_has_windows_incompatible_segment("nullable")); - assert!(!object_name_has_windows_incompatible_segment("CONSOLE")); - assert!(!object_name_has_windows_incompatible_segment("com10")); - assert!(!object_name_has_windows_incompatible_segment("com0")); - assert!(!object_name_has_windows_incompatible_segment("lpt")); - assert!(!object_name_has_windows_incompatible_segment("aux-data/file")); - assert!(!object_name_has_windows_incompatible_segment("object with spaces inside")); - assert!(!object_name_has_windows_incompatible_segment("")); + for object in [ + "file.txt", + "dir.name/file", + "path/to/file.tar.gz", + "nullable", + "CONSOLE", + "com10", + "com0", + "lpt", + "CONIN", + "aux-data/file", + "object with spaces inside", + "dir/", + "", + ] { + assert_eq!(windows_object_key_violation(object), None, "{object:?}"); + } } #[test] - fn test_check_object_name_windows_incompatible_segments() { - // Rejected on Windows (would be written but unreadable, issue #3449); - // valid on non-Windows platforms. - for object in [ - "baddir.", - "dir1/dir2/baddir.", - "dir1/baddir./file.txt", - "file.txt ", - "NUL", - "nul.txt", - "COM1", - "dir/LPT9.log", - ] { - let result = check_object_name_for_length_and_slash("test-bucket", object); - if cfg!(target_os = "windows") { - assert!( - matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_SEGMENT_REASON), - "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}" - ); - } else { - assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}"); - } + fn test_windows_object_key_violation_characters() { + let characters = Some(WINDOWS_RESERVED_CHARACTERS_REASON); + for object in ["path/*sUt*mykey", "a:b", "what?", "pipe|name", "quote\"d", "lt"] { + assert_eq!(windows_object_key_violation(object), characters, "{object:?}"); } - // Valid on every platform. - for object in ["file.txt", "dir.name/file", "nullable", "CONSOLE", "com10"] { - assert!( - check_object_name_for_length_and_slash("test-bucket", object).is_ok(), - "object name must be valid on all platforms: {object:?}" + // Control characters 0x01-0x1F cannot appear in a Win32 file name. + for object in ["a\u{1}b", "tab\tname", "dir/line\nbreak", "esc\u{1b}", "unit\u{1f}"] { + assert_eq!(windows_object_key_violation(object), characters, "{object:?}"); + } + + // NUL keeps its platform-independent ObjectNameInvalid answer. + assert_eq!(windows_object_key_violation("a\0b"), None); + + // A character violation outranks a segment violation anywhere in the key. + assert_eq!(windows_object_key_violation("baddir./a:b"), characters); + } + + /// Existing `a\b` keys must stay readable, deletable and abortable, so the + /// shared name check never rejects `\` (only new writes do, on Windows). + #[test] + fn test_backslash_passes_the_shared_name_check() { + assert_eq!(windows_object_key_violation("a\\b"), None); + assert_eq!(windows_object_key_violation("dir\\sub\\file.txt"), None); + assert!(check_object_name_for_length_and_slash("test-bucket", "a\\b").is_ok()); + assert!(check_object_args("test-bucket", "a\\b").is_ok()); + assert!(check_bucket_and_object_names("test-bucket", "a\\b").is_ok()); + // Data movement relocates keys that already exist. + assert!(check_new_object_args("test-bucket", "a\\b", true).is_ok()); + assert!(check_new_object_args(RUSTFS_META_BUCKET, "config\\a", false).is_ok()); + } + + #[test] + fn test_check_new_object_args_matches_put_object_args() { + for object in [ + "file.txt", + "", + "/leading", + "a//b", + "x/../y", + &"k".repeat(1025), + &"s".repeat(256), + "baddir.", + "a:b", + "NUL.txt", + ] { + assert_eq!( + format!("{:?}", check_new_object_args("test-bucket", object, false)), + format!("{:?}", check_put_object_args("test-bucket", object)), + "{object:?}" ); } + assert!(matches!( + check_new_object_args("Bad_Bucket", "file.txt", false), + Err(StorageError::BucketNameInvalid(_)) + )); } #[test] - fn test_check_object_name_windows_reserved_characters() { - // Keys containing Win32-reserved characters are rejected on Windows - // with a descriptive reason (issue #3299); valid elsewhere. - for object in ["path/*sUt*mykey", "a:b", "what?", "pipe|name", "quote\"d", "lt"] { - let result = check_object_name_for_length_and_slash("test-bucket", object); - if cfg!(target_os = "windows") { + #[cfg(not(target_os = "windows"))] + fn test_check_new_object_args_keeps_full_key_set_off_windows() { + for object in ["a\\b", "a:b", "baddir.", "NUL", "CONIN$", "tab\tname"] { + assert!(check_new_object_args("test-bucket", object, false).is_ok(), "{object:?}"); + } + } + + /// `a\b` and `a/b` resolve to the same file on a Windows disk, so a new + /// `a\b` would overwrite (or be overwritten by) `a/b`. + #[test] + #[cfg(target_os = "windows")] + fn windows_new_object_key_rejects_backslash_alias() { + assert!(check_new_object_args("test-bucket", "a/b", false).is_ok()); + for object in ["a\\b", "dir\\", "dir/sub\\file"] { + assert!( + matches!( + &check_new_object_args("test-bucket", object, false), + Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_BACKSLASH_REASON + ), + "{object:?}" + ); + } + // Reads, deletes and in-flight multipart uploads of existing keys keep working. + assert!(check_object_args("test-bucket", "a\\b").is_ok()); + assert!(check_del_obj_args("test-bucket", "a\\b").is_ok()); + assert!(check_new_object_args("test-bucket", "a\\b", true).is_ok()); + } + + #[test] + #[cfg(target_os = "windows")] + fn windows_new_object_key_rejects_unaddressable_names() { + for (object, expected) in [ + ("baddir.", WINDOWS_RESERVED_SEGMENT_REASON), + ("dir/LPT9.log", WINDOWS_RESERVED_SEGMENT_REASON), + ("COM\u{b9}", WINDOWS_RESERVED_SEGMENT_REASON), + ("CONOUT$", WINDOWS_RESERVED_SEGMENT_REASON), + ("a:b", WINDOWS_RESERVED_CHARACTERS_REASON), + ("tab\tname", WINDOWS_RESERVED_CHARACTERS_REASON), + ("esc\u{1b}", WINDOWS_RESERVED_CHARACTERS_REASON), + ] { + for result in [ + check_object_name_for_length_and_slash("test-bucket", object), + check_put_object_args("test-bucket", object), + check_new_object_args("test-bucket", object, false), + check_object_args("test-bucket", object), + ] { assert!( - matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_CHARACTERS_REASON), - "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}" + matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == expected), + "{object:?} must be rejected with {expected:?}, got {result:?}" ); - } else { - assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}"); } } + // NUL keeps the platform-independent answer. + assert!(matches!( + check_put_object_args("test-bucket", "a\0b"), + Err(StorageError::ObjectNameInvalid(_, _)) + )); } #[test] diff --git a/crates/ecstore/src/cluster/rpc/client.rs b/crates/ecstore/src/cluster/rpc/client.rs index ca781d6be..307b6ded0 100644 --- a/crates/ecstore/src/cluster/rpc/client.rs +++ b/crates/ecstore/src/cluster/rpc/client.rs @@ -353,6 +353,15 @@ fn apply_peer_replay_response( } } +fn rename_data_grpc_stage(path: &str) -> bool { + matches!( + path, + "/node_service.NodeService/RenameData" + | "/node_service.NodeService/RenameDataAtIncarnation" + | "/node_service.NodeService/RenameDataWithQuotaFence" + ) +} + impl Service> for ReplayScopeChannel where S: Service, Response = HttpResponse>, @@ -370,6 +379,12 @@ where } fn call(&mut self, mut request: HttpRequest) -> Self::Future { + let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path()); + let request_scope_started = if observe_rename_data { + rustfs_io_metrics::put_stage_timer() + } else { + None + }; let authenticated = self.audience.as_ref().is_some_and(|_| { request .headers() @@ -404,11 +419,30 @@ where } } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE, + request_scope_started, + ); let audience = self.audience.clone(); let future = self.inner.call(request); Box::pin(async move { - let response = future.await?; + let response = if observe_rename_data { + rustfs_io_metrics::observe_put_stage_future( + future, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY, + ) + .await? + } else { + future.await? + }; + let replay_response_started = if observe_rename_data { + rustfs_io_metrics::put_stage_timer() + } else { + None + }; if let (Some(audience), Some(challenge)) = (audience, challenge) { let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers()); if let Err(error) = &response_state @@ -426,6 +460,10 @@ where } apply_peer_replay_response(audience, sent_state, response_state); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE, + replay_response_started, + ); Ok(response) }) } diff --git a/crates/ecstore/src/cluster/rpc/network_probe.rs b/crates/ecstore/src/cluster/rpc/network_probe.rs index 765cc7322..db2221cfb 100644 --- a/crates/ecstore/src/cluster/rpc/network_probe.rs +++ b/crates/ecstore/src/cluster/rpc/network_probe.rs @@ -325,6 +325,24 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError { match status.code() { Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable, Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut, + Code::Cancelled | Code::Internal => { + let mut source = std::error::Error::source(&status); + while let Some(error) = source { + if error + .downcast_ref::() + .is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR))) + { + return NetworkPeerProbeError::Unreachable; + } + source = if let Some(error) = error.downcast_ref::() { + // io::Error::source skips its immediate custom error. + error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static)) + } else { + error.source() + }; + } + NetworkPeerProbeError::ProtocolFailure + } _ => NetworkPeerProbeError::ProtocolFailure, } } @@ -335,6 +353,35 @@ mod tests { use crate::layout::endpoint::Endpoint; use crate::layout::endpoints::{Endpoints, PoolEndpoints}; + #[test] + fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() { + for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] { + let status = tonic::Status::from_error(Box::new(h2::Error::from(reason))); + assert!(matches!(status.code(), Code::Cancelled | Code::Internal)); + let code = status.code(); + assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable); + let mut wrapped = tonic::Status::new(code, "wrapped transport reset"); + wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason)))); + assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable); + } + for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] { + let status = tonic::Status::from_error(Box::new(h2::Error::from(reason))); + assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure); + } + for status in [ + tonic::Status::cancelled("peer cancelled the application request"), + tonic::Status::internal("peer rejected the probe"), + tonic::Status::unauthenticated("authentication rejected"), + tonic::Status::permission_denied("authorization rejected"), + ] { + assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure); + } + assert_eq!( + map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")), + NetworkPeerProbeError::TimedOut + ); + } + fn pacing(duration: Duration) -> Arc { NetworkPeerProbeClient::from_endpoint_pools(&topology()) .with_diagnostic_pacing(Instant::now(), duration) diff --git a/crates/ecstore/src/cluster/rpc/peer_rest_client.rs b/crates/ecstore/src/cluster/rpc/peer_rest_client.rs index eb9a023f8..8f0f01527 100644 --- a/crates/ecstore/src/cluster/rpc/peer_rest_client.rs +++ b/crates/ecstore/src/cluster/rpc/peer_rest_client.rs @@ -48,14 +48,15 @@ use rustfs_protos::proto_gen::node_service::{ GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, GetSysConfigRequest, GetSysErrorsRequest, HealControlRequest, LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest, - LocalStorageInfoRequest, Mss, PingRequest, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, - ReplacementRecoveryStatusRequest, ScannerActivityRequest, ScannerActivityResponse, ScannerDirtyUsageSnapshotRequest, - ScannerDirtyUsageSnapshotResponse, ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseRequest, - ScannerPublicationLeaseResponse, ScannerScopedDirtyUsageAckRequest, ScannerScopedDirtyUsageAckResponse, - ScannerScopedDirtyUsageEntry, ServerInfoRequest, SignalServiceRequest, SignalServiceResponse, StartDecommissionRequest, - StartProfilingRequest, StopRebalanceRequest, TierDailyStatsRequest, TierMutationAbortRequest, TierMutationCommitRequest, - TierMutationControlResponse, TierMutationFailureClass, TierMutationPeerState, TierMutationPrepareRequest, - node_service_client::NodeServiceClient, tier_mutation_control_service_client::TierMutationControlServiceClient, + LocalStorageInfoRequest, Mss, PeerDriveAdmissionRequest, PingRequest, ReloadPoolMetaRequest, + ReloadSiteReplicationConfigRequest, ReplacementRecoveryStatusRequest, ScannerActivityRequest, ScannerActivityResponse, + ScannerDirtyUsageSnapshotRequest, ScannerDirtyUsageSnapshotResponse, ScannerPublicationLeaseReleaseRequest, + ScannerPublicationLeaseRequest, ScannerPublicationLeaseResponse, ScannerScopedDirtyUsageAckRequest, + ScannerScopedDirtyUsageAckResponse, ScannerScopedDirtyUsageEntry, ServerInfoRequest, SignalServiceRequest, + SignalServiceResponse, StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest, TierDailyStatsRequest, + TierMutationAbortRequest, TierMutationCommitRequest, TierMutationControlResponse, TierMutationFailureClass, + TierMutationPeerState, TierMutationPrepareRequest, node_service_client::NodeServiceClient, + tier_mutation_control_service_client::TierMutationControlServiceClient, }; pub use rustfs_protos::{PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS}; use rustfs_protos::{TierMutationRpcPhase, evict_failed_connection}; @@ -89,6 +90,7 @@ pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2; pub const KMS_SIGNAL_SUBSYSTEM: &str = "kms"; const BACKGROUND_HEAL_STATUS_MAX_MESSAGE_SIZE: usize = 64 * 1024; const REPLACEMENT_RECOVERY_STATUS_MAX_MESSAGE_SIZE: usize = 64 * 1024; +const PEER_DRIVE_ADMISSION_MAX_MESSAGE_SIZE: usize = 256 * 1024; const HEAL_CONTROL_FINGERPRINT_MAX_SIZE: usize = 256; const HEAL_CONTROL_PAYLOAD_MAX_SIZE: usize = 64 * 1024; const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60; @@ -711,6 +713,16 @@ fn scanner_publication_lease_deadline( .ok_or_else(|| Error::other("scanner publication lease deadline overflowed")) } +/// A peer's readiness view of one drive served by another host. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PeerDriveAdmissionView { + pub pool_index: i32, + pub set_index: i32, + pub disk_index: i32, + /// True when the peer counts the drive towards its own read quorum. + pub online: bool, +} + #[derive(Clone, Debug)] pub struct PeerRestClient { pub host: XHost, @@ -864,6 +876,11 @@ impl PeerRestClient { ) } + /// Topology `host:port` this client talks to, as written in the endpoint list. + pub fn topology_member(&self) -> &str { + &self.topology_member + } + pub fn new(host: XHost, grid_host: String) -> Self { let topology_member = host.to_string(); Self { @@ -1677,6 +1694,51 @@ impl PeerRestClient { .await } + /// Ask the peer how it currently sees the drives served by `host`. + /// + /// `Ok(None)` means the peer predates the RPC and cannot answer. + pub async fn peer_drive_admission(&self, host: &str) -> Result>> { + self.finalize_result( + async { + let mut client = self + .get_client() + .await? + .max_decoding_message_size(PEER_DRIVE_ADMISSION_MAX_MESSAGE_SIZE); + let response = match client + .peer_drive_admission(Request::new(PeerDriveAdmissionRequest { host: host.to_string() })) + .await + { + Ok(response) => response.into_inner(), + Err(status) if status.code() == tonic::Code::Unimplemented => { + // RUSTFS_COMPAT_TODO(peer-drive-admission-v1): old peers cannot report how they see a restarting node, so its startup re-admission gate skips them. Remove after the minimum supported RustFS peer version implements PeerDriveAdmission. + return Ok(None); + } + Err(status) => return Err(status.into()), + }; + if !response.success { + return Err(match (response.error_code, response.error_info) { + (None, None) => Error::other("peer drive admission failed without an error"), + (error_code, error_info) => control_plane_failure("peer_drive_admission", None, error_code, error_info), + }); + } + Ok(Some( + response + .drives + .into_iter() + .map(|drive| PeerDriveAdmissionView { + pool_index: drive.pool_index, + set_index: drive.set_index, + disk_index: drive.disk_index, + online: drive.online, + }) + .collect(), + )) + } + .await, + ) + .await + } + pub async fn prepare_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Result { self.tier_mutation_control(TierMutationRpcPhase::Prepare, mutation_id, canonical_payload) .await diff --git a/crates/ecstore/src/cluster/rpc/remote_disk.rs b/crates/ecstore/src/cluster/rpc/remote_disk.rs index e067609a0..a98d6a22a 100644 --- a/crates/ecstore/src/cluster/rpc/remote_disk.rs +++ b/crates/ecstore/src/cluster/rpc/remote_disk.rs @@ -26,8 +26,9 @@ use crate::disk::{ BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, - DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions, - RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one, + DiskOption, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, GuardedFileInfoVersions, + PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, + UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one, disk_store::{ DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE, get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout, @@ -59,12 +60,12 @@ use rustfs_protos::proto_gen::node_service::RenamePartRequest; use rustfs_protos::proto_gen::node_service::{ BatchReadVersionRequest, BatchReadVersionResponse, CheckPartsRequest, CompareAndUpdateFileOutcome, CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest, - DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest, - MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest, - ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest, - RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, - SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, - WriteMetadataRequest, node_service_client::NodeServiceClient, + DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, GuardedDeleteVersionsRequest, ListDirRequest, + ListVolumesRequest, MakeVolumeRequest, MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, + ReadMetadataRequest, ReadMultipleRequest, ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, + RecoverDeleteTransactionRequest, RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, + SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, + UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, node_service_client::NodeServiceClient, }; use serde::{Serialize, de::DeserializeOwned}; use std::{ @@ -217,6 +218,24 @@ fn attach_mutation_body_digest( crate::cluster::rpc::set_tonic_rolling_canonical_body_digest(request, &canonical_body).map_err(Error::other) } +fn attach_quota_mutation_body_digest( + request: &mut Request, + canonical_body: std::result::Result, std::num::TryFromIntError>, +) -> Result<()> { + let topology = crate::services::notification_sys::configured_fleet_topology_fingerprint() + .ok_or_else(|| Error::other("quota mutation topology is not initialized or conflicts with startup"))?; + let body = canonical_body.map_err(|_| Error::other("quota request length cannot be represented"))?; + let body = rustfs_protos::canonical_quota_mutation_request_body(topology, &body) + .map_err(|_| Error::other("quota topology envelope length cannot be represented"))?; + request.metadata_mut().insert( + rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER, + topology + .parse() + .map_err(|_| Error::other("quota topology cannot be encoded"))?, + ); + crate::cluster::rpc::set_tonic_canonical_body_digest(request, &body).map_err(Error::other) +} + fn decode_volume_infos(volume_infos: Vec) -> Result> { volume_infos .into_iter() @@ -899,6 +918,63 @@ fn spawn_control_channel_prewarm(addr: String) { } impl RemoteDisk { + async fn guarded_delete_versions_rpc( + &self, + volume: &str, + payload: Vec, + probe_only: bool, + ) -> Result { + if payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(DiskError::FileCorrupt); + } + self.execute_with_timeout( + || async { + let mut client = self.get_client().await?; + let mut request = Request::new(GuardedDeleteVersionsRequest { + disk: self.endpoint.to_string(), + volume: volume.to_owned(), + payload: payload.into(), + probe_only, + }); + let body = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref()) + .map_err(|_| DiskError::FileCorrupt)?; + crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?; + // The additive method is mandatory: an old peer cannot safely delete + // a guarded request through any legacy batch method. + let response = client + .guarded_delete_versions(request) + .await + .map_err(|status| { + if status.code() == Code::Unimplemented { + DiskError::GuardedDeleteUnsupported + } else { + status.into() + } + })? + .into_inner(); + if !response.success { + return Err(response.error.ok_or(DiskError::FileCorrupt)?.into()); + } + if response.error.is_some() { + return Err(DiskError::FileCorrupt); + } + Ok(response) + }, + get_max_timeout_duration(), + ) + .await + } + + pub(crate) async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result { + match self + .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: path.to_owned() }) + .await? + { + DeleteRecoveryRpcReply::OrphanDataDir { plain } => Ok(plain), + _ => Err(Error::other("unexpected orphan data directory inspection reply")), + } + } + async fn recover_delete_transaction( &self, volume: &str, @@ -2279,6 +2355,7 @@ impl RemoteDisk { self.execute_with_timeout_for_op( "rename_data", || async { + let prepare_started = rustfs_io_metrics::put_stage_timer(); let file_info = compat_json(fi)?; let file_info_bin = encode_file_info_msgpack(fi)?; let mut client = self.get_client().await?; @@ -2299,31 +2376,83 @@ impl RemoteDisk { }); let canonical_body = rustfs_protos::canonical_rename_data_request_body(request.get_ref()); let incarnation_bound = !request.get_ref().bucket_incarnation_id.is_empty(); - if scanner_publication_lease_token.is_some() || incarnation_bound { + let quota_fenced = rustfs_utils::http::metadata_compat::contains_key_str( + &fi.metadata, + crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + ); + if quota_fenced { + attach_quota_mutation_body_digest(&mut request, canonical_body)?; + } else if scanner_publication_lease_token.is_some() || incarnation_bound { let canonical_body = canonical_body.map_err(|_| Error::other("rename_data request length cannot be represented"))?; crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?; } else { attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?; } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE, + prepare_started, + ); - let response = if incarnation_bound { - // Older peers return Unimplemented before mutation; never downgrade. - client.rename_data_at_incarnation(request).await? - } else { - client.rename_data(request).await? - } - .into_inner(); + let rpc_started = rustfs_io_metrics::put_stage_timer(); + let response = rustfs_io_metrics::observe_put_stage_future( + async { + if quota_fenced { + // The method itself proves quota-token enforcement; an old peer must fail closed. + client.rename_data_with_quota_fence(request).await + } else if incarnation_bound { + // Older peers return Unimplemented before mutation; never downgrade. + client.rename_data_at_incarnation(request).await + } else { + client.rename_data(request).await + } + }, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT, + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY, + ) + .await?; + let response_process_started = rustfs_io_metrics::put_stage_timer(); + let response_headers_started = rustfs_io_metrics::put_stage_timer(); + let _ = response.metadata().len(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS, + response_headers_started, + ); + let into_inner_started = rustfs_io_metrics::put_stage_timer(); + let response = response.into_inner(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER, + into_inner_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC, + rpc_started, + ); + let success_check_started = rustfs_io_metrics::put_stage_timer(); if !response.success { return Err(response.error.unwrap_or_default().into()); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK, + success_check_started, + ); + let decode_started = rustfs_io_metrics::put_stage_timer(); let rename_data_resp = decode_msgpack_or_json::( &response.rename_data_resp_bin, &response.rename_data_resp, "RenameDataResp", )?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE, + decode_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS, + response_process_started, + ); Ok(rename_data_resp) }, @@ -2404,6 +2533,39 @@ impl RemoteDisk { #[async_trait::async_trait] impl DiskAPI for RemoteDisk { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + let response = self.guarded_delete_versions_rpc("", Vec::new(), true).await?; + if !response.errors.is_empty() || !response.item_errors.is_empty() { + return Err(DiskError::FileCorrupt); + } + Ok(()) + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + let count = versions.len(); + let payload = GuardedDeleteVersionsPayload { versions, opts }; + if let Err(error) = payload.validate() { + return vec![Some(error); count]; + } + let encoded = match rmp_serde::to_vec_named(&payload) { + Ok(encoded) => encoded, + Err(error) => return vec![Some(error.into()); count], + }; + match self.guarded_delete_versions_rpc(volume, encoded, false).await { + Ok(response) => { + // New guarded peers must return typed errors, never lossy strings. + if response.item_errors.len() != count { + return vec![Some(DiskError::FileCorrupt); count]; + } + decode_delete_versions_errors(response, count) + } + Err(error) => vec![Some(error); count], + } + } #[tracing::instrument(level = "trace", skip_all)] fn to_string(&self) -> String { self.endpoint.to_string() @@ -2911,8 +3073,15 @@ impl DiskAPI for RemoteDisk { .map_err(|_| Error::other("snapshot lease TTL cannot be represented"))?, }); let canonical_body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()); - attach_mutation_body_digest(&mut request, canonical_body, "acquire_snapshot_lease")?; - let response = client.acquire_snapshot_lease(request).await?.into_inner(); + let quota_fence = volume == crate::disk::RUSTFS_META_BUCKET && crate::disk::is_quota_mutation_fence_path(path); + let response = if quota_fence { + attach_quota_mutation_body_digest(&mut request, canonical_body)?; + client.acquire_quota_mutation_fence(request).await? + } else { + attach_mutation_body_digest(&mut request, canonical_body, "acquire_snapshot_lease")?; + client.acquire_snapshot_lease(request).await? + } + .into_inner(); snapshot_lease_token_from_response(response) }, get_max_timeout_duration(), @@ -2953,8 +3122,15 @@ impl DiskAPI for RemoteDisk { token: token.as_bytes().to_vec().into(), }); let canonical_body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()); - attach_mutation_body_digest(&mut request, canonical_body, "release_snapshot_lease")?; - let response = client.release_snapshot_lease(request).await?.into_inner(); + let quota_fence = volume == crate::disk::RUSTFS_META_BUCKET && crate::disk::is_quota_mutation_fence_path(path); + let response = if quota_fence { + attach_quota_mutation_body_digest(&mut request, canonical_body)?; + client.release_quota_mutation_fence(request).await? + } else { + attach_mutation_body_digest(&mut request, canonical_body, "release_snapshot_lease")?; + client.release_snapshot_lease(request).await? + } + .into_inner(); if !response.success { return Err(response.error.unwrap_or_default().into()); } @@ -3149,7 +3325,7 @@ impl DiskAPI for RemoteDisk { opts_bin: opts_bin.into(), }); - crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts); if let Some(request_payload_bytes) = request_payload_bytes { crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes); } @@ -3227,6 +3403,7 @@ impl DiskAPI for RemoteDisk { record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started); let request_payload_bytes = batch_read_version_attribution_enabled .then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin)); + let request_opts = req.opts; let batch_result = self .execute_with_timeout_for_op( "batch_read_version", @@ -3240,7 +3417,7 @@ impl DiskAPI for RemoteDisk { batch_read_version_req_bin: batch_read_version_req_bin.into(), }); - crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(); + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts); if let Some(request_payload_bytes) = request_payload_bytes { crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes( request_payload_bytes.saturating_add(disk_len), @@ -3779,6 +3956,21 @@ impl DiskAPI for RemoteDisk { } } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + match self + .recover_delete_transaction( + volume, + DeleteRecoveryRpcOperation::Exists { + object: object.to_owned(), + }, + ) + .await? + { + DeleteRecoveryRpcReply::Evidence { present } => Ok(present), + _ => Err(Error::other("unexpected delete recovery existence reply")), + } + } + async fn discover_delete_transactions( &self, volume: &str, @@ -4278,6 +4470,101 @@ mod tests { static INIT: Once = Once::new(); + #[tokio::test] + async fn quota_fence_old_peer_rejects_without_legacy_fallback() { + runtime_sources::ensure_test_rpc_secret(); + crate::services::notification_sys::install_cross_pool_fence_fleet_proof_for_test(); + let _without_proof = crate::services::notification_sys::without_live_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_none()); + assert!(crate::services::notification_sys::configured_fleet_topology_fingerprint().is_some()); + let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new()) + .await + .expect("old peer must bind"); + let remote = remote_disk_for_addr(&peer.addr).await; + let fence_path = crate::disk::quota_mutation_fence_path("bucket", "object"); + assert!( + remote + .acquire_snapshot_lease(crate::disk::RUSTFS_META_BUCKET, &fence_path) + .await + .is_err() + ); + assert!( + remote + .release_snapshot_lease(crate::disk::RUSTFS_META_BUCKET, &fence_path, SnapshotLeaseToken::revoke_all()) + .await + .is_err() + ); + let mut file_info = sample_rename_data_file_info(); + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::new_v4().to_string(), + ); + assert!( + remote + .rename_data("tmp", "source", file_info, "bucket", "object") + .await + .is_err() + ); + let methods = peer + .peer + .unsupported_methods + .lock() + .expect("unsupported method list lock") + .clone(); + assert_eq!( + methods, + vec![ + "/node_service.NodeService/AcquireQuotaMutationFence", + "/node_service.NodeService/ReleaseQuotaMutationFence", + "/node_service.NodeService/RenameDataWithQuotaFence", + ], + "quota operations must never fall back to snapshot or unfenced rename methods" + ); + peer.stop().await; + } + + #[tokio::test] + async fn guarded_delete_old_peer_rejects_without_legacy_fallback() { + runtime_sources::ensure_test_rpc_secret(); + let peer = TestGrpcPeer::spawn(Bytes::new(), Bytes::new()) + .await + .expect("old peer must bind"); + let remote = remote_disk_for_addr(&peer.addr).await; + assert_eq!(remote.supports_guarded_delete_versions().await, Err(DiskError::GuardedDeleteUnsupported)); + let group = GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".into(), + versions: vec![FileInfo::default()], + ..Default::default() + }, + expected_current: crate::disk::CurrentDeleteGeneration::Absent, + }; + let errors = remote + .delete_versions_guarded_recoverable( + "bucket", + vec![group], + DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + ) + .await; + assert_eq!(errors, vec![Some(DiskError::GuardedDeleteUnsupported)]); + let methods = peer + .peer + .unsupported_methods + .lock() + .expect("unsupported method list lock") + .clone(); + assert_eq!( + methods, + vec!["/node_service.NodeService/GuardedDeleteVersions"; 2], + "neither capability failure nor guarded application may call a legacy deletion method" + ); + peer.stop().await; + } + #[test] fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() { let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2) @@ -4666,6 +4953,7 @@ mod tests { format_data: Bytes, read_all_data: Bytes, next_format_read_gate: Arc>>>, + unsupported_methods: Arc>>, } #[derive(Debug, Default)] @@ -4684,6 +4972,7 @@ mod tests { format_data, read_all_data, next_format_read_gate: Arc::default(), + unsupported_methods: Arc::default(), } } @@ -4830,13 +5119,20 @@ mod tests { Ok(grpc.unary(method, request).await) }) } - _ => Box::pin(async move { - let mut response = http::Response::new(tonic::body::Body::default()); - let headers = response.headers_mut(); - headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into()); - headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE); - Ok(response) - }), + _ => { + self.peer + .unsupported_methods + .lock() + .expect("unsupported method list lock") + .push(request.uri().path().to_owned()); + Box::pin(async move { + let mut response = http::Response::new(tonic::body::Body::default()); + let headers = response.headers_mut(); + headers.insert(tonic::Status::GRPC_STATUS, (tonic::Code::Unimplemented as i32).into()); + headers.insert(http::header::CONTENT_TYPE, tonic::metadata::GRPC_CONTENT_TYPE); + Ok(response) + }) + } } } } @@ -5116,6 +5412,8 @@ mod tests { cleanup_data_dir: Some(Uuid::new_v4()), sign: Some(vec![0x14, 0x35]), old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)), + old_current_source_checked: false, + old_current_source: None, }; let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode"); let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test(); @@ -5518,6 +5816,8 @@ mod tests { cleanup_data_dir: Some(Uuid::new_v4()), sign: Some(vec![1_u8; 32]), old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)), + old_current_source_checked: false, + old_current_source: None, }; let json = serde_json::to_vec(&response).expect("rename data response json should encode"); let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode"); diff --git a/crates/ecstore/src/cluster/rpc/runtime_sources.rs b/crates/ecstore/src/cluster/rpc/runtime_sources.rs index 4ba4ef06d..ca0968b9d 100644 --- a/crates/ecstore/src/cluster/rpc/runtime_sources.rs +++ b/crates/ecstore/src/cluster/rpc/runtime_sources.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::disk::ReadOptions; use rustfs_io_metrics::internode_metrics::{ INTERNODE_MSGPACK_CODEC_JSON, INTERNODE_MSGPACK_CODEC_MSGPACK, INTERNODE_MSGPACK_DIRECTION_RESPONSE, INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_OPERATION_GRPC_COMPARE_AND_UPDATE_FILE, @@ -19,8 +20,409 @@ use rustfs_io_metrics::internode_metrics::{ INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_TRANSPORT_BACKEND_GRPC, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics, }; +#[cfg(feature = "diagnostic-read-version-caller")] +use rustfs_io_metrics::internode_metrics::{ + ReadVersionRequestCaller, ReadVersionRequestMode, record_read_version_request_by_caller, +}; use std::time::Duration; +#[cfg(feature = "diagnostic-read-version-caller")] +fn is_path_separator(byte: u8) -> bool { + matches!(byte, b'/' | b'\\') +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn path_byte_matches(path_byte: u8, pattern_byte: u8) -> bool { + path_byte == pattern_byte || (pattern_byte == b'/' && is_path_separator(path_byte)) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn path_ends_with(path: &str, suffix: &str) -> bool { + let path = path.as_bytes(); + let suffix = suffix.as_bytes(); + let Some(start) = path.len().checked_sub(suffix.len()) else { + return false; + }; + path[start..] + .iter() + .zip(suffix) + .all(|(&path_byte, &pattern_byte)| path_byte_matches(path_byte, pattern_byte)) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn path_contains(path: &str, pattern: &str) -> bool { + let path = path.as_bytes(); + let pattern = pattern.as_bytes(); + !pattern.is_empty() + && path.len() >= pattern.len() + && path.windows(pattern.len()).any(|window| { + window + .iter() + .zip(pattern) + .all(|(&path_byte, &pattern_byte)| path_byte_matches(path_byte, pattern_byte)) + }) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) fn read_version_request_caller_from_location(file: &str) -> ReadVersionRequestCaller { + if path_ends_with(file, "/store/peer.rs") { + ReadVersionRequestCaller::Peer + } else if path_ends_with(file, "/store/init.rs") { + ReadVersionRequestCaller::Startup + } else if path_ends_with(file, "/store/heal.rs") + || path_ends_with(file, "/set_disk/ops/heal.rs") + || path_ends_with(file, "/set_disk/ops/heal_walk.rs") + { + ReadVersionRequestCaller::Heal + } else if path_ends_with(file, "/store/list_objects.rs") { + ReadVersionRequestCaller::ListObjects + } else if path_ends_with(file, "/store/object.rs") || path_ends_with(file, "/set_disk/ops/object.rs") { + ReadVersionRequestCaller::Object + } else if path_ends_with(file, "/set_disk/core/metadata_read.rs") { + ReadVersionRequestCaller::MetadataRead + } else if path_ends_with(file, "/set_disk/ops/multipart.rs") { + ReadVersionRequestCaller::Multipart + } else if path_ends_with(file, "/disk/disk_store.rs") { + ReadVersionRequestCaller::DiskStore + } else if path_ends_with(file, "/disk/mod.rs") { + ReadVersionRequestCaller::DiskDispatch + } else if path_ends_with(file, "/set_disk/read.rs") { + ReadVersionRequestCaller::SetDiskRead + } else if path_ends_with(file, "/set_disk/core/io_primitives.rs") { + ReadVersionRequestCaller::SetDiskIo + } else if path_ends_with(file, "/services/rebalance/worker.rs") { + ReadVersionRequestCaller::Rebalance + } else if path_ends_with(file, "/core/pools.rs") { + ReadVersionRequestCaller::Decommission + } else if path_ends_with(file, "/tier_free_version_recovery.rs") { + ReadVersionRequestCaller::TierRecovery + } else if path_contains(file, "/bucket/lifecycle/") { + ReadVersionRequestCaller::Lifecycle + } else if path_contains(file, "/cache_value/") { + ReadVersionRequestCaller::CacheValue + } else if path_contains(file, "/ecstore/src/") { + ReadVersionRequestCaller::Other + } else { + ReadVersionRequestCaller::Unattributed + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) fn current_read_version_request_caller() -> Option { + current_read_version_task_context().and_then(|context| context.caller) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_request_caller(caller: ReadVersionRequestCaller, future: F) -> F::Output +where + F: std::future::Future, +{ + let mut context = current_read_version_task_context().unwrap_or_default(); + context.caller = Some(caller); + with_read_version_task_context(context, future).await +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_request_caller_if_absent(caller: ReadVersionRequestCaller, future: F) -> F::Output +where + F: std::future::Future, +{ + if current_read_version_request_caller().is_some() { + future.await + } else { + with_read_version_request_caller(caller, future).await + } +} + +#[cfg(all(test, feature = "diagnostic-read-version-caller"))] +pub(crate) async fn with_optional_read_version_request_caller(caller: Option, future: F) -> F::Output +where + F: std::future::Future, +{ + match caller { + Some(caller) => with_read_version_request_caller(caller, future).await, + None => future.await, + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +use rustfs_io_metrics::internode_metrics::{ReadVersionAttribution, ReadVersionContextBoundary, ReadVersionEntrySource}; +#[cfg(feature = "diagnostic-read-version-caller")] +use rustfs_io_metrics::s3_http_metrics::{S3HttpOperationContext, current_s3_http_operation_context}; + +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +struct ReadVersionEntryContext { + source: ReadVersionEntrySource, + api_operation: Option, + caller: Option, + // Entry provenance stays fixed; the parent snapshot follows each handoff. + parent_api_operation: Option, + parent_caller: Option, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionEntryContext { + fn new( + source: ReadVersionEntrySource, + api_operation: Option, + caller: Option, + ) -> Self { + Self { + source, + api_operation, + caller, + parent_api_operation: api_operation, + parent_caller: caller, + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +struct ReadVersionTaskContext { + caller: Option, + entry: Option, +} + +/// Capture values before spawning or enqueueing; absence is part of the value. +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) struct ReadVersionDiagnosticContext { + pub(crate) api_operation: Option, + pub(crate) caller: Option, + entry: Option, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +tokio::task_local! { + static CURRENT_READ_VERSION_CONTEXT: std::cell::Cell; + static CURRENT_READ_VERSION_BATCH_CONTEXTS: Vec; +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn current_read_version_task_context() -> Option { + CURRENT_READ_VERSION_CONTEXT.try_with(std::cell::Cell::get).ok() +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn with_read_version_task_context(context: ReadVersionTaskContext, future: F) -> impl std::future::Future +where + F: std::future::Future, +{ + CURRENT_READ_VERSION_CONTEXT.scope(std::cell::Cell::new(context), future) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionDiagnosticContext { + pub(crate) fn capture() -> Self { + let task_context = current_read_version_task_context(); + Self { + api_operation: current_s3_http_operation_context(), + caller: task_context.and_then(|context| context.caller), + entry: task_context.and_then(|context| context.entry), + } + } + + pub(crate) fn capture_entry(source: ReadVersionEntrySource) -> Self { + let mut context = Self::capture(); + if context.entry.is_none() { + context.entry = Some(ReadVersionEntryContext::new(source, context.api_operation, context.caller)); + context.observe(ReadVersionContextBoundary::Entry); + } + context + } + + pub(crate) fn observe(self, boundary: ReadVersionContextBoundary) { + self.observe_with_mode(boundary, None); + } + + fn observe_with_mode(self, boundary: ReadVersionContextBoundary, mode: Option) { + rustfs_io_metrics::internode_metrics::record_read_version_context_transition( + self.entry.map_or(ReadVersionEntrySource::Unclassified, |entry| entry.source), + boundary, + self.api_operation, + current_s3_http_operation_context(), + self.caller, + current_read_version_request_caller(), + mode, + ); + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn with_read_version_entry(source: ReadVersionEntrySource, future: F) -> impl std::future::Future +where + F: std::future::Future, +{ + // Box before constructing the scope future so large storage futures do not + // inflate every caller frame. Capture task-local values only when polled. + let future = Box::pin(future); + async move { + let context = ReadVersionDiagnosticContext::capture_entry(source); + with_read_version_task_context( + ReadVersionTaskContext { + caller: context.caller, + entry: context.entry, + }, + future, + ) + .await + } +} + +/// Set a missing caller and entry source in one task-local scope for the object-read boundary. +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) fn with_read_version_caller_and_entry_if_absent( + caller: ReadVersionRequestCaller, + source: ReadVersionEntrySource, + future: F, +) -> impl std::future::Future +where + F: std::future::Future, +{ + let future = Box::pin(future); + async move { + let api_operation = current_s3_http_operation_context(); + let mut task_context = current_read_version_task_context().unwrap_or_default(); + if task_context.caller.is_none() { + task_context.caller = Some(caller); + } + let new_entry = task_context.entry.is_none(); + if new_entry { + task_context.entry = Some(ReadVersionEntryContext::new(source, api_operation, task_context.caller)); + } + let diagnostic = ReadVersionDiagnosticContext { + api_operation, + caller: task_context.caller, + entry: task_context.entry, + }; + let observed = async move { + if new_entry { + diagnostic.observe(ReadVersionContextBoundary::Entry); + } + future.await + }; + with_read_version_task_context(task_context, observed).await + } +} + +/// Capture on the caller before spawning, then restore when the child is polled. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn with_read_version_spawn_context( + source: ReadVersionEntrySource, + boundary: ReadVersionContextBoundary, + future: F, +) -> impl std::future::Future +where + F: std::future::Future, +{ + let context = ReadVersionDiagnosticContext::capture_entry(source); + let future = Box::pin(future); + with_read_version_diagnostic_context(context, boundary, future) +} + +/// Observe a named entry even when its scope retains an earlier source. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn observe_read_version_context_boundary(boundary: ReadVersionContextBoundary) { + ReadVersionDiagnosticContext::capture().observe(boundary); +} + +/// The earliest observed storage entry, for diagnostic consumers only. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn current_read_version_entry_source() -> Option { + current_read_version_task_context().and_then(|context| context.entry.map(|entry| entry.source)) +} + +/// Restore only this item's context and observe the child after restoration. +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_diagnostic_context( + context: ReadVersionDiagnosticContext, + boundary: ReadVersionContextBoundary, + future: F, +) -> F::Output +where + F: std::future::Future, +{ + let observed = async { + context.observe(boundary); + future.await + }; + let mut forwarded_entry = context.entry; + if let Some(entry) = &mut forwarded_entry { + entry.parent_api_operation = context.api_operation; + entry.parent_caller = context.caller; + } + let task_context = ReadVersionTaskContext { + caller: context.caller, + entry: forwarded_entry, + }; + let scoped = with_read_version_task_context(task_context, observed); + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(context.api_operation, scoped).await +} + +/// A batch has item contexts, not one owner's S3 task-local operation. +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_read_version_batch_contexts(contexts: Vec, future: F) -> F::Output +where + F: std::future::Future, +{ + let batch = CURRENT_READ_VERSION_BATCH_CONTEXTS.scope(contexts, future); + let scoped = with_read_version_task_context(ReadVersionTaskContext::default(), batch); + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(None, scoped).await +} + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) async fn with_batch_read_version_item_context(index: usize, item_count: usize, future: F) -> F::Output +where + F: std::future::Future, +{ + let context = CURRENT_READ_VERSION_BATCH_CONTEXTS + .try_with(|contexts| (contexts.len() == item_count).then(|| contexts.get(index).copied()).flatten()) + .ok() + .flatten(); + match context { + Some(context) => with_read_version_diagnostic_context(context, ReadVersionContextBoundary::UnaryFallback, future).await, + None => future.await, + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn common_read_version_attribution(values: impl Iterator>) -> ReadVersionAttribution { + values + .map(|value| value.map_or(ReadVersionAttribution::Absent, ReadVersionAttribution::Common)) + .reduce(|common, value| { + if common == value { + common + } else { + ReadVersionAttribution::Mixed + } + }) + .unwrap_or(ReadVersionAttribution::Absent) +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn record_read_version_entry_to_rpc_context(opts: &ReadOptions) { + let context = ReadVersionDiagnosticContext::capture(); + let mode = Some(read_version_request_mode(opts)); + if let Some(entry) = context.entry { + ReadVersionDiagnosticContext { + api_operation: entry.api_operation, + caller: entry.caller, + ..context + } + .observe_with_mode(ReadVersionContextBoundary::EntryToRpc, mode); + ReadVersionDiagnosticContext { + api_operation: entry.parent_api_operation, + caller: entry.parent_caller, + ..context + } + .observe_with_mode(ReadVersionContextBoundary::RemoteRpc, mode); + } else { + context.observe_with_mode(ReadVersionContextBoundary::RemoteRpc, mode); + } +} + #[cfg(test)] use rustfs_io_metrics::internode_metrics::InternodeMetricsSnapshot; @@ -106,7 +508,18 @@ pub(crate) fn record_remote_disk_grpc_read_all_request() { .record_outgoing_request_for_operation_and_backend(INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC); } -pub(crate) fn record_remote_disk_grpc_read_version_request() { +#[cfg(feature = "diagnostic-read-version-caller")] +fn read_version_request_mode(opts: &ReadOptions) -> ReadVersionRequestMode { + match (opts.read_data, opts.healing, opts.incl_free_versions) { + (false, false, false) => ReadVersionRequestMode::MetadataOnly, + (true, false, false) => ReadVersionRequestMode::Data, + (false, true, false) => ReadVersionRequestMode::Healing, + (false, false, true) => ReadVersionRequestMode::IncludeFreeVersions, + _ => ReadVersionRequestMode::Combined, + } +} + +pub(crate) fn record_remote_disk_grpc_read_version_request(_opts: &ReadOptions) { if !rustfs_io_metrics::get_stage_metrics_enabled() { return; } @@ -114,9 +527,19 @@ pub(crate) fn record_remote_disk_grpc_read_version_request() { INTERNODE_OPERATION_GRPC_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC, ); + #[cfg(feature = "diagnostic-read-version-caller")] + let api_operation = rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context(); + #[cfg(feature = "diagnostic-read-version-caller")] + record_read_version_request_by_caller( + api_operation, + current_read_version_request_caller().unwrap_or_default(), + read_version_request_mode(_opts), + ); + #[cfg(feature = "diagnostic-read-version-caller")] + record_read_version_entry_to_rpc_context(_opts); } -pub(crate) fn record_remote_disk_grpc_batch_read_version_request() { +pub(crate) fn record_remote_disk_grpc_batch_read_version_request(_opts: &ReadOptions) { if !rustfs_io_metrics::get_stage_metrics_enabled() { return; } @@ -124,6 +547,30 @@ pub(crate) fn record_remote_disk_grpc_batch_read_version_request() { INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, INTERNODE_TRANSPORT_BACKEND_GRPC, ); + #[cfg(feature = "diagnostic-read-version-caller")] + { + let attribution = CURRENT_READ_VERSION_BATCH_CONTEXTS.try_with(|contexts| { + ( + common_read_version_attribution(contexts.iter().map(|context| context.api_operation)), + common_read_version_attribution(contexts.iter().map(|context| context.caller)), + common_read_version_attribution(contexts.iter().map(|context| context.entry.map(|entry| entry.source))), + ) + }); + let (api, caller, source) = attribution.unwrap_or_else(|_| { + let context = ReadVersionDiagnosticContext::capture(); + ( + common_read_version_attribution(std::iter::once(context.api_operation)), + common_read_version_attribution(std::iter::once(context.caller)), + common_read_version_attribution(std::iter::once(context.entry.map(|entry| entry.source))), + ) + }); + rustfs_io_metrics::internode_metrics::record_batch_read_version_request_by_caller( + api, + caller, + source, + read_version_request_mode(_opts), + ); + } } pub(crate) fn record_remote_disk_grpc_batch_read_version_stage(stage: &'static str, duration: Duration) { @@ -303,3 +750,591 @@ pub(crate) fn internode_metrics_snapshot_for_test() -> InternodeMetricsSnapshot pub(crate) fn internode_msgpack_json_decode_error_total_for_test() -> u64 { global_internode_metrics().msgpack_json_decode_error_total_for_test() } + +#[cfg(test)] +mod tests { + #[cfg(feature = "diagnostic-read-version-caller")] + use super::*; + + #[cfg(feature = "diagnostic-read-version-caller")] + fn diagnostic_operation(operation: rustfs_s3_ops::S3Operation) -> S3HttpOperationContext { + let mut request = rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard::new("GET"); + request.in_scope(|| { + rustfs_io_metrics::record_s3_op(operation); + current_s3_http_operation_context().expect("operation should be captured") + }) + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn combined_caller_and_entry_scope_captures_once_and_restores() { + let operation = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(operation), async { + let before = ReadVersionDiagnosticContext::capture(); + let captured = with_read_version_caller_and_entry_if_absent( + ReadVersionRequestCaller::Object, + ReadVersionEntrySource::ObjectRead, + async { ReadVersionDiagnosticContext::capture() }, + ) + .await; + assert_eq!(captured.api_operation, Some(operation)); + assert_eq!(captured.caller, Some(ReadVersionRequestCaller::Object)); + assert_eq!( + captured.entry.map(|entry| (entry.source, entry.caller)), + Some((ReadVersionEntrySource::ObjectRead, Some(ReadVersionRequestCaller::Object))) + ); + assert_eq!(ReadVersionDiagnosticContext::capture(), before); + + let existing_entry = ReadVersionEntryContext::new( + ReadVersionEntrySource::ConfigRead, + Some(operation), + Some(ReadVersionRequestCaller::Peer), + ); + let existing = ReadVersionTaskContext { + caller: Some(ReadVersionRequestCaller::Peer), + entry: Some(existing_entry), + }; + let preserved = with_read_version_task_context(existing, async { + with_read_version_caller_and_entry_if_absent( + ReadVersionRequestCaller::Object, + ReadVersionEntrySource::ObjectRead, + async { ReadVersionDiagnosticContext::capture() }, + ) + .await + }) + .await; + assert_eq!(preserved.caller, Some(ReadVersionRequestCaller::Peer)); + assert_eq!( + preserved.entry.map(|entry| (entry.source, entry.caller)), + Some((ReadVersionEntrySource::ConfigRead, Some(ReadVersionRequestCaller::Peer))) + ); + }) + .await; + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_list_retains_background_entry_through_shared_workers() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + runtime.block_on(async { + tokio::spawn(with_read_version_entry(ReadVersionEntrySource::RecoveryDispositionMaintenance, async { + let entry = ReadVersionDiagnosticContext::capture(); + assert_eq!(entry.api_operation, None); + assert_eq!(entry.caller, None); + let expected_entry = entry.entry.expect("background entry is observed"); + assert_eq!(expected_entry.source.as_str(), "recovery_disposition_maintenance"); + with_read_version_entry(ReadVersionEntrySource::ListObjects, async { + with_read_version_request_caller(ReadVersionRequestCaller::ListObjects, async { + let parent = ReadVersionDiagnosticContext::capture_entry(ReadVersionEntrySource::ListObjects); + assert_eq!(parent.entry, Some(expected_entry)); + for boundary in [ + ReadVersionContextBoundary::ListProducer, + ReadVersionContextBoundary::ListGather, + ] { + tokio::spawn(with_read_version_diagnostic_context(parent, boundary, async move { + let child = ReadVersionDiagnosticContext::capture(); + assert_eq!(child.api_operation, None); + assert_eq!(child.caller, Some(ReadVersionRequestCaller::ListObjects)); + let child_entry = child.entry.expect("worker retains background provenance"); + assert_eq!(child_entry.source, expected_entry.source); + assert_eq!(child_entry.api_operation, None); + assert_eq!(child_entry.caller, None); + assert_eq!(child_entry.parent_api_operation, None); + assert_eq!(child_entry.parent_caller, Some(ReadVersionRequestCaller::ListObjects)); + })) + .await + .expect("list worker completes"); + } + }) + .await; + }) + .await; + assert_eq!(ReadVersionDiagnosticContext::capture(), entry); + })) + .await + .expect("background task completes"); + assert_eq!(ReadVersionDiagnosticContext::capture().entry, None); + }); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_spawn_scope_captures_parent_and_isolates_absence() { + struct LargeFuture([u8; 64 * 1024]); + impl std::future::Future for LargeFuture { + type Output = ReadVersionDiagnosticContext; + fn poll(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll { + assert_eq!(self.0[0], 7); + std::task::Poll::Ready(ReadVersionDiagnosticContext::capture()) + } + } + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let put = diagnostic_operation(rustfs_s3_ops::S3Operation::PutObject); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + runtime.block_on(async { + let child = rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context( + Some(put), + with_read_version_request_caller( + ReadVersionRequestCaller::Object, + with_read_version_entry(ReadVersionEntrySource::ConfigWrite, async { + let child = with_read_version_spawn_context( + ReadVersionEntrySource::AppPutOwner, + ReadVersionContextBoundary::AppPutOwner, + LargeFuture([7; 64 * 1024]), + ); + assert!(std::mem::size_of_val(&child) < 1024, "spawn scope must not embed the storage future"); + tokio::spawn(child).await.expect("child task") + }), + ), + ) + .await; + assert_eq!(child.api_operation, Some(put)); + assert_eq!(child.caller, Some(ReadVersionRequestCaller::Object)); + assert_eq!(child.entry.expect("earlier source retained").source, ReadVersionEntrySource::ConfigWrite); + let internal = with_read_version_spawn_context( + ReadVersionEntrySource::AppPutOwner, + ReadVersionContextBoundary::AppPutOwner, + LargeFuture([7; 64 * 1024]), + ); + let observed = rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context( + Some(get), + with_read_version_request_caller( + ReadVersionRequestCaller::Peer, + with_read_version_entry(ReadVersionEntrySource::ConfigRead, async { + let observed = internal.await; + assert_eq!(current_s3_http_operation_context(), Some(get)); + assert_eq!(current_read_version_entry_source(), Some(ReadVersionEntrySource::ConfigRead)); + observed + }), + ), + ) + .await; + assert_eq!(observed.api_operation, None); + assert_eq!(observed.caller, None); + assert_eq!( + observed.entry.expect("internal source retained").source, + ReadVersionEntrySource::AppPutOwner + ); + assert_eq!(current_s3_http_operation_context(), None); + assert_eq!(current_read_version_entry_source(), None); + }); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_entry_keeps_large_future_off_the_scope_frame_and_captures_when_polled() { + struct LargeFuture([u8; 64 * 1024]); + impl std::future::Future for LargeFuture { + type Output = ReadVersionDiagnosticContext; + fn poll(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll { + assert_eq!(self.0[0], 7); + std::task::Poll::Ready(ReadVersionDiagnosticContext::capture()) + } + } + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + let read = with_read_version_entry(ReadVersionEntrySource::ObjectInfoRead, LargeFuture([7; 64 * 1024])); + assert!(std::mem::size_of_val(&read) < 1024, "scope frame must not embed the storage future"); + assert_eq!(current_read_version_entry_source(), None); + let observed = runtime.block_on(rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context( + Some(get), + with_read_version_entry(ReadVersionEntrySource::ConfigRead, read), + )); + assert_eq!(observed.entry.expect("entry captured at poll").source, ReadVersionEntrySource::ConfigRead); + assert_eq!(observed.api_operation, Some(get)); + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_named_entry_is_observed_when_earlier_source_is_preserved() { + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + for earlier in [None, Some(ReadVersionEntrySource::ConfigRead)] { + let api = earlier.map(|_| get); + let entry = with_read_version_entry(ReadVersionEntrySource::DataUsageCacheRead, async { + observe_read_version_context_boundary(ReadVersionContextBoundary::ScannerCacheLoad); + assert_eq!(current_s3_http_operation_context(), api); + }); + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, entry).await, + None => entry.await, + } + }) + .await; + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + }); + }); + let rows = snapshots.snapshot().into_vec(); + for (source, api, state) in [ + ("data_usage_cache_read", "unattributed", "absent"), + ("config_read", "s3:GetObject", "preserved"), + ] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("boundary") == Some(&"scanner_cache_load") + && labels.get("entry_source") == Some(&source) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(1)) + }), + "a named cache entry remains observable with source {source}" + ); + } + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_context_records_absent_internal_entry_and_real_child_loss() { + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::s3_http_metrics::{with_isolated_s3_http_operation_context, with_s3_http_operation_context}; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + with_read_version_entry(ReadVersionEntrySource::ConfigRead, async { + let context = ReadVersionDiagnosticContext::capture(); + tokio::spawn(with_read_version_diagnostic_context( + context, + ReadVersionContextBoundary::FullWait, + async { + assert_eq!(current_s3_http_operation_context(), None); + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }, + )) + .await + .expect("internal child should complete"); + }) + .await; + with_s3_http_operation_context( + Some(get), + with_read_version_entry( + ReadVersionEntrySource::ObjectRead, + with_read_version_request_caller(ReadVersionRequestCaller::Object, async { + let context = ReadVersionDiagnosticContext::capture(); + tokio::spawn(with_read_version_diagnostic_context( + context, + ReadVersionContextBoundary::EarlyStop, + async move { + assert_eq!(current_s3_http_operation_context(), Some(get)); + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + with_isolated_s3_http_operation_context(None, async { + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }) + .await; + let mut missing_caller = + current_read_version_task_context().expect("diagnostic scope should be active"); + missing_caller.caller = None; + with_read_version_task_context(missing_caller, async { + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }) + .await; + }, + )) + .await + .expect("external child should complete"); + }), + ), + ) + .await; + assert_eq!(ReadVersionDiagnosticContext::capture().entry, None); + record_read_version_entry_to_rpc_context(&ReadOptions::default()); + }) + }); + let rows = snapshots.snapshot().into_vec(); + let observed = |source: &str, state: &str, parent: &str, child: &str| { + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&"remote_rpc") + && labels.get("api_context") == Some(&state) + && labels.get("parent_api_operation") == Some(&parent) + && labels.get("child_api_operation") == Some(&child) + && labels.get("mode") == Some(&"metadata_only") + && matches!(value, DebugValue::Counter(1)) + }) + }; + assert!(observed("config_read", "absent", "unattributed", "unattributed")); + assert!(observed("object_read", "preserved", "s3:GetObject", "s3:GetObject")); + assert!(observed("object_read", "lost", "s3:GetObject", "unattributed")); + assert!(observed("unclassified", "unobserved", "unattributed", "unattributed")); + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("boundary") == Some(&"remote_rpc") + && labels.get("parent_caller") == Some(&"object") + && labels.get("child_caller") == Some(&"unattributed") + && labels.get("caller_context") == Some(&"lost") + && matches!(value, DebugValue::Counter(1)) + }), + "RPC caller must be compared with its direct parent's seed, not an earlier unseeded entry" + ); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn diagnostic_context_restores_ambient_scope_after_pending_item_is_cancelled() { + use std::future::Future; + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + let parent = ReadVersionDiagnosticContext { + api_operation: Some(get), + caller: Some(ReadVersionRequestCaller::Object), + entry: Some(ReadVersionEntryContext::new( + ReadVersionEntrySource::ObjectRead, + Some(get), + Some(ReadVersionRequestCaller::Object), + )), + }; + let internal = ReadVersionDiagnosticContext { + api_operation: None, + caller: None, + entry: Some(ReadVersionEntryContext::new(ReadVersionEntrySource::ConfigRead, None, None)), + }; + with_read_version_diagnostic_context(parent, ReadVersionContextBoundary::Entry, async { + let mut pending = Box::pin(with_read_version_diagnostic_context( + internal, + ReadVersionContextBoundary::UnaryFallback, + async { + assert_eq!(ReadVersionDiagnosticContext::capture(), internal); + std::future::pending::<()>().await; + }, + )); + std::future::poll_fn(|cx| { + assert!(pending.as_mut().poll(cx).is_pending()); + std::task::Poll::Ready(()) + }) + .await; + assert_eq!( + ReadVersionDiagnosticContext::capture(), + parent, + "a pending item must not leak into its poller's scope" + ); + drop(pending); + assert_eq!( + ReadVersionDiagnosticContext::capture(), + parent, + "cancellation must retain the ambient request" + ); + }) + .await; + assert_eq!(ReadVersionDiagnosticContext::capture().entry, None); + assert_eq!(current_s3_http_operation_context(), None); + assert_eq!(current_read_version_request_caller(), None); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_batch_attributes_each_dimension_and_counts_one_rpc() { + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let get = diagnostic_operation(rustfs_s3_ops::S3Operation::GetObject); + let list = diagnostic_operation(rustfs_s3_ops::S3Operation::ListObjectsV2); + let context = |api, caller, source| ReadVersionDiagnosticContext { + api_operation: api, + caller, + entry: Some(ReadVersionEntryContext::new(source, api, caller)), + }; + let external = context(Some(get), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead); + let internal = context(None, None, ReadVersionEntrySource::ConfigRead); + let internal_object = context(None, Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ConfigRead); + let heal = context(Some(get), Some(ReadVersionRequestCaller::Heal), ReadVersionEntrySource::ObjectRead); + let other_api = context(Some(list), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + for contexts in [ + vec![external, external], + vec![external, internal], + vec![external, internal_object], + vec![external, heal], + vec![external, other_api], + vec![internal, internal], + ] { + with_read_version_batch_contexts(contexts, async { + assert_eq!(current_s3_http_operation_context(), None); + assert_eq!(current_read_version_request_caller(), None); + record_remote_disk_grpc_batch_read_version_request(&ReadOptions { + read_data: true, + ..Default::default() + }); + }) + .await; + } + }) + }); + let labels = snapshots + .snapshot() + .into_vec() + .into_iter() + .filter(|(key, _, _, _)| { + key.key().name() == "rustfs_system_network_internode_batch_read_version_requests_by_caller_total" + }) + .map(|(key, _, _, value)| { + assert!(matches!(value, DebugValue::Counter(1)), "batch must count one RPC, not its items"); + let labels = key + .key() + .labels() + .map(|label| (label.key().to_owned(), label.value().to_owned())) + .collect::>(); + assert_eq!(labels["mode"], "data"); + (labels["api_operation"].clone(), labels["caller"].clone(), labels["entry_source"].clone()) + }) + .collect::>(); + assert_eq!(labels.len(), 6); + for (api, caller, source) in [ + ("s3:GetObject", "object", "object_read"), + ("mixed", "mixed", "mixed"), + ("mixed", "object", "mixed"), + ("s3:GetObject", "mixed", "object_read"), + ("mixed", "object", "object_read"), + ("unattributed", "unattributed", "config_read"), + ] { + assert!(labels.contains(&(api.to_owned(), caller.to_owned(), source.to_owned()))); + } + } + + #[test] + #[cfg(feature = "diagnostic-read-version-caller")] + fn read_version_request_mode_uses_only_bounded_read_flags() { + assert_eq!(read_version_request_mode(&ReadOptions::default()), ReadVersionRequestMode::MetadataOnly); + assert_eq!( + read_version_request_mode(&ReadOptions { + read_data: true, + ..Default::default() + }), + ReadVersionRequestMode::Data + ); + assert_eq!( + read_version_request_mode(&ReadOptions { + healing: true, + ..Default::default() + }), + ReadVersionRequestMode::Healing + ); + assert_eq!( + read_version_request_mode(&ReadOptions { + incl_free_versions: true, + ..Default::default() + }), + ReadVersionRequestMode::IncludeFreeVersions + ); + assert_eq!( + read_version_request_mode(&ReadOptions { + read_data: true, + healing: true, + ..Default::default() + }), + ReadVersionRequestMode::Combined + ); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn read_version_request_caller_classification_is_separator_agnostic() { + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/store/peer.rs"), + ReadVersionRequestCaller::Peer + ); + assert_eq!( + read_version_request_caller_from_location(r"C:\repo\crates\ecstore\src\store\peer.rs"), + ReadVersionRequestCaller::Peer + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/notstore/peer.rs"), + ReadVersionRequestCaller::Other + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/store/object.rs"), + ReadVersionRequestCaller::Object + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/core/metadata_read.rs"), + ReadVersionRequestCaller::MetadataRead + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/ops/heal.rs"), + ReadVersionRequestCaller::Heal + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/set_disk/ops/multipart.rs"), + ReadVersionRequestCaller::Multipart + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/disk/disk_store.rs"), + ReadVersionRequestCaller::DiskStore + ); + assert_eq!( + read_version_request_caller_from_location(r"C:\repo\crates\ecstore\src\bucket\lifecycle\worker.rs"), + ReadVersionRequestCaller::Lifecycle + ); + assert_eq!( + read_version_request_caller_from_location("/repo/crates/ecstore/src/other/module.rs"), + ReadVersionRequestCaller::Other + ); + assert_eq!( + read_version_request_caller_from_location("/repo/external/module.rs"), + ReadVersionRequestCaller::Unattributed + ); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn read_version_request_caller_context_is_scoped_to_the_read_future() { + assert_eq!(current_read_version_request_caller(), None); + let caller = + with_read_version_request_caller(ReadVersionRequestCaller::Peer, async { current_read_version_request_caller() }) + .await; + assert_eq!(caller, Some(ReadVersionRequestCaller::Peer)); + assert_eq!(current_read_version_request_caller(), None); + } +} diff --git a/crates/ecstore/src/config/com.rs b/crates/ecstore/src/config/com.rs index b35e776da..f4235d514 100644 --- a/crates/ecstore/src/config/com.rs +++ b/crates/ecstore/src/config/com.rs @@ -574,17 +574,20 @@ where >, { let h = HeaderMap::new(); - let mut rd = api - .get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts) - .await - .map_err(|err| { - if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) { - Error::ConfigNotFound - } else { - warn!("read_config_with_metadata: err: {:?}, file: {}", err, file); - err - } - })?; + let read = api.get_object_reader(RUSTFS_META_BUCKET, file, None, h, opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigRead, + read, + ); + let mut rd = read.await.map_err(|err| { + if err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _)) { + Error::ConfigNotFound + } else { + warn!("read_config_with_metadata: err: {:?}, file: {}", err, file); + err + } + })?; let data = if let Some(max_bytes) = max_bytes { let object_size = usize::try_from(rd.object_info.size).map_err(|_| Error::CorruptedFormat)?; @@ -807,7 +810,18 @@ where >, { let mut put_data = PutObjReader::from_vec(data); - match api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await { + let write = api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let write = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ConfigWrite, + async { + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ConfigWrite, + ); + write.await + }, + ); + match write.await { Ok(object_info) => Ok(object_info), Err(err) => { if log_error { @@ -2839,7 +2853,7 @@ mod tests { use rustfs_config::{ DEFAULT_DELIMITER, ENABLE_KEY, EnableState, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_TABLE, SCANNER_CYCLE, SCANNER_IDLE_MODE, SCANNER_KEYS, SCANNER_SPEED, - SCANNER_SUB_SYS, + SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK, }; #[tokio::test] @@ -2869,6 +2883,11 @@ mod tests { #[derive(Debug, Default)] struct MetadataWriteStore { error: Option, + #[cfg(feature = "diagnostic-read-version-caller")] + observations: + std::sync::Mutex, Option<&'static str>)>>, + #[cfg(feature = "diagnostic-read-version-caller")] + pending: bool, } #[async_trait::async_trait] @@ -2899,7 +2918,20 @@ mod tests { _data: &mut Self::PutObjectReader, _opts: &Self::ObjectOptions, ) -> core::result::Result { - Err(self.error.clone().expect("test store error should be configured")) + #[cfg(feature = "diagnostic-read-version-caller")] + { + self.observations.lock().expect("observations lock").push(( + crate::cluster::rpc::runtime_sources::current_read_version_entry_source(), + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + )); + if self.pending { + std::future::pending::<()>().await; + } + } + match self.error.clone() { + Some(error) => Err(error), + None => Ok(ObjectInfo::default()), + } } } @@ -2907,6 +2939,10 @@ mod tests { async fn save_config_preserves_retryable_system_volume_errors() { let store = Arc::new(MetadataWriteStore { error: Some(Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())), + #[cfg(feature = "diagnostic-read-version-caller")] + observations: Default::default(), + #[cfg(feature = "diagnostic-read-version-caller")] + pending: false, }); let error = save_config_with_opts_inner(store, "buckets/example/.metadata.bin", Vec::new(), &ObjectOptions::default(), false) @@ -2918,6 +2954,59 @@ mod tests { Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), "buckets/example/.metadata.bin".to_string()) ); } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation() { + use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context}; + + let mut guard = S3HttpRequestGuard::new("PUT"); + let put = guard.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject); + current_s3_http_operation_context().expect("captured PutObject") + }); + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for api in [None, Some(put)] { + for case in ["success", "error", "cancelled"] { + let store = Arc::new(MetadataWriteStore { + error: (case == "error").then_some(Error::VolumeNotFound), + pending: case == "cancelled", + ..Default::default() + }); + let opts = ObjectOptions::default(); + let write = async { + let result = tokio::time::timeout( + std::time::Duration::from_secs(1), + save_config_with_opts_inner(store.clone(), "buckets/diagnostic/state", Vec::new(), &opts, false), + ) + .await; + match case { + "success" => assert!(result.expect("write completes").is_ok()), + "error" => { + assert!(matches!(result.expect("error completes"), Err(Error::InsufficientWriteQuorum(_, _)))) + } + _ => assert!(result.is_err(), "pending write is cancelled"), + } + }; + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, write).await, + None => write.await, + } + }) + .await; + assert_eq!( + *store.observations.lock().expect("observations lock"), + vec![(Some(earlier.unwrap_or(ReadVersionEntrySource::ConfigWrite)), api.map(|_| "s3:PutObject"))] + ); + assert_eq!(current_read_version_entry_source(), None, "write scope restores: {case}"); + assert_eq!(current_s3_http_operation_context(), None, "API scope restores: {case}"); + } + } + } + } use rustfs_lock::client::LockClient; use rustfs_lock::client::local::LocalClient; use rustfs_lock::{LockError, LockInfo, LockResponse, LockStats}; @@ -4112,6 +4201,8 @@ mod tests { let value = match *key { SCANNER_SPEED => "fast", SCANNER_IDLE_MODE => "false", + // "1" is this key's default, which the diff encoding omits. + SCANNER_WALKERS_PER_DISK => "2", _ => "1", }; scanner_kvs.insert((*key).to_string(), value.to_string()); @@ -4133,6 +4224,8 @@ mod tests { let expected = match *key { SCANNER_SPEED => "fast", SCANNER_IDLE_MODE => "false", + // "1" is this key's default, which the diff encoding omits. + SCANNER_WALKERS_PER_DISK => "2", _ => "1", }; assert_eq!(decoded_scanner.get(key), expected, "scanner key {key} should survive roundtrip"); diff --git a/crates/ecstore/src/config/scanner.rs b/crates/ecstore/src/config/scanner.rs index 140f7c186..a4db32ed4 100644 --- a/crates/ecstore/src/config/scanner.rs +++ b/crates/ecstore/src/config/scanner.rs @@ -15,12 +15,13 @@ use rustfs_config::server_config::{KV, KVS}; use rustfs_config::{ DEFAULT_SCANNER_BITROT_CYCLE_SECS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES, - DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE, - DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED, - DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, - SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, - SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, - SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, + DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_DISK_IO_BUDGET, + DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, + DEFAULT_SCANNER_SPEED, DEFAULT_SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, + SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, + SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, + SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, SCANNER_DISK_IO_BUDGET_IOPS, SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, + SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, }; use std::sync::LazyLock; @@ -92,6 +93,21 @@ pub static DEFAULT_KVS: LazyLock = LazyLock::new(|| { value: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(), hidden_if_empty: false, }, + KV { + key: SCANNER_WALKERS_PER_DISK.to_owned(), + value: DEFAULT_SCANNER_WALKERS_PER_DISK.to_string(), + hidden_if_empty: false, + }, + KV { + key: SCANNER_DISK_IO_BUDGET_IOPS.to_owned(), + value: DEFAULT_SCANNER_DISK_IO_BUDGET.to_owned(), + hidden_if_empty: false, + }, + KV { + key: SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC.to_owned(), + value: DEFAULT_SCANNER_DISK_IO_BUDGET.to_owned(), + hidden_if_empty: false, + }, KV { key: SCANNER_YIELD_EVERY_N_OBJECTS.to_owned(), value: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(), diff --git a/crates/ecstore/src/core/pools.rs b/crates/ecstore/src/core/pools.rs index f807eb603..a2e742ac4 100644 --- a/crates/ecstore/src/core/pools.rs +++ b/crates/ecstore/src/core/pools.rs @@ -2983,6 +2983,11 @@ fn resolve_decommission_partial_listing_entry( if let Some(entry) = entries.resolve(resolver) { return Ok(entry); } + if disk_error_count == 0 + && let Some(prefix_dir) = decommission_prefix_directory_candidate(entries.as_ref()) + { + return Ok(prefix_dir); + } let object = entries .as_ref() @@ -3005,6 +3010,24 @@ fn resolve_decommission_partial_listing_entry( }) } +/// The prefix directory a partial listing sample consists of, when every +/// candidate is one. +/// +/// The walker emits a directory with empty metadata only when it holds no +/// `xl.meta`, so such a sample is listing structure rather than erasure-coded +/// data: node-local files such as the MRF journal under +/// `.rustfs.sys/buckets/.heal/` create one on a single drive, and the +/// recursive walk lists any object beneath it as its own entry. Below the +/// directory quorum there is nothing to reconcile, so the callback receives the +/// prefix exactly as a quorum-resolved one and skips it. Objects whose metadata +/// is present but unreadable keep their metadata bytes and never match here; +/// mixed samples stay unresolved. +fn decommission_prefix_directory_candidate(candidates: &[Option]) -> Option { + let mut candidates = candidates.iter().flatten(); + let first = candidates.next()?; + (first.is_dir() && candidates.all(MetaCacheEntry::is_dir)).then(|| first.clone()) +} + fn validate_decommission_durable_ilm_copy( path: &str, source_record: &ValidatedDurableIlmRecord, @@ -3691,6 +3714,55 @@ fn activation_pool_meta_lock_error(err: rustfs_lock::LockError) -> Error { } } +fn rebalance_worker_read_lock_error(err: rustfs_lock::LockError, object: &str) -> Error { + match err { + rustfs_lock::LockError::QuorumNotReached { required, achieved } => Error::NamespaceLockQuorumUnavailable { + mode: "read", + bucket: RUSTFS_META_BUCKET.to_string(), + object: object.to_string(), + required, + achieved, + }, + other => Error::Lock(other), + } +} + +/// Read-only admission to an already durable rebalance run. Keeping this +/// separate from the activation writer prevents admission from authorizing a +/// pool or rebalance metadata write under shared namespace locks. +pub(crate) struct PoolRebalanceWorkerFence { + pool_meta_guard: rustfs_lock::NamespaceLockGuard, + rebalance_meta_guard: rustfs_lock::NamespaceLockGuard, + fleet_proof: Option, +} + +impl PoolRebalanceWorkerFence { + pub(crate) fn set_fleet_proof( + &mut self, + fleet_proof: Option, + ) { + self.fleet_proof = fleet_proof; + } + + pub(crate) fn ensure_held(&self) -> Result<()> { + if self.pool_meta_guard.is_lock_lost() + || self.pool_meta_guard.is_released() + || self.rebalance_meta_guard.is_lock_lost() + || self.rebalance_meta_guard.is_released() + { + return Err(Error::other("worker admission lock lost before rebalance activation")); + } + if self + .fleet_proof + .as_ref() + .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof)) + { + return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)); + } + Ok(()) + } +} + pub(crate) struct PoolRebalanceActivationFence { pool_meta_guard: Arc, rebalance_meta_guard: Arc, @@ -3762,6 +3834,35 @@ tokio::task_local! { pub(crate) static REBALANCE_ACTIVATION_LOCK_ATTEMPT: Arc; } +pub(crate) async fn acquire_pool_rebalance_worker_locks(pool: Arc) -> Result +where + S: crate::storage_api_contracts::namespace::NamespaceLocking< + Error = Error, + NamespaceLock = rustfs_lock::NamespaceLockWrapper, + >, +{ + // Match activation and entry admission: rebalance.bin -> pool.bin. Joining + // the same durable run must coexist with entries retaining their run fence. + let rebalance_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, REBAL_META_NAME).await?; + #[cfg(test)] + let _ = REBALANCE_ACTIVATION_LOCK_ATTEMPT.try_with(|attempted| attempted.notify_one()); + let rebalance_meta_guard = rebalance_meta_lock + .get_read_lock(get_lock_acquire_timeout()) + .await + .map_err(|err| rebalance_worker_read_lock_error(err, REBAL_META_NAME))?; + let pool_meta_lock = pool.new_ns_lock(RUSTFS_META_BUCKET, POOL_META_NAME).await?; + let pool_meta_guard = pool_meta_lock + .get_read_lock(get_lock_acquire_timeout()) + .await + .map_err(|err| rebalance_worker_read_lock_error(err, POOL_META_NAME))?; + + Ok(PoolRebalanceWorkerFence { + pool_meta_guard, + rebalance_meta_guard, + fleet_proof: None, + }) +} + pub(crate) async fn acquire_pool_rebalance_activation_locks( pool: Arc, fleet_proof: Option, @@ -6685,12 +6786,12 @@ fn pool_meta_cas_options(object: &str, token: &PoolMetaCasToken, fence: &PoolMet fence.ensure_held()?; let mut opts = ObjectOptions { max_parity: true, - write_completion: crate::object_api::WriteCompletion::TailDrained, no_lock: true, http_preconditions: Some(pool_meta_cas_preconditions(token, object)?), ..Default::default() }; fence.add_to_options(&mut opts, object); + opts.write_completion = crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair; Ok(opts) } @@ -6718,6 +6819,72 @@ async fn execute_pool_meta_object_cas( fence: &PoolMetaPersistenceFence<'_>, phase: &'static str, ) -> PoolMetaCasWriteOutcome +where + S: EcstoreObjectIO, +{ + // Bound the CAS state retained by callers, including ordinary metadata + // transactions that never enter identity repair. Large inline retry states + // otherwise propagate through startup and cross-crate storage futures. + Box::pin(async move { + if object != POOL_META_IDENTITY_NAME { + return execute_pool_meta_object_cas_once(pool, object, data, opts, fence, phase).await; + } + // Keep the exact identity owned by this transaction. A quorum read alone + // cannot discharge repair responsibility for a missing disk replica. + let identity = data; + let mut outcome = execute_pool_meta_object_cas_once(pool.clone(), object, identity.clone(), opts, fence, phase).await; + for delay_secs in [1, 2, 4] { + if !outcome.source.as_deref().is_some_and(|err| { + matches!(err, Error::Io(inner) if inner.get_ref().is_some_and(|source| source.is::())) + }) { + break; + } + // The caller's transaction arm remains live across reconnect waits, + // verification and repair. Cancellation must retain its sticky gate. + tokio::time::sleep(std::time::Duration::from_secs(delay_secs)).await; + let repair_options = async { + fence.ensure_held()?; + let (current, info) = read_config_no_lock_preserve_empty_with_metadata(pool.clone(), object).await?; + if current != identity { + return Err(Error::PreconditionFailed); + } + let token = info + .etag + .filter(|etag| !etag.trim().is_empty()) + .map(PoolMetaCasToken::Existing) + .unwrap_or(PoolMetaCasToken::Unsafe); + pool_meta_cas_options(object, &token, fence) + } + .await; + outcome = match repair_options { + Ok(opts) => execute_pool_meta_object_cas_once(pool.clone(), object, identity.clone(), opts, fence, phase).await, + Err(err) => PoolMetaCasWriteOutcome::not_dispatched(err), + }; + // A rejected repair cannot turn the preceding committed write into a + // harmless CAS conflict that the outer loop could accept on readback. + outcome.may_have_mutated = true; + outcome.result = outcome.result.map_err(|err| { + let original = outcome.source.get_or_insert_with(|| Arc::new(err)); + Error::other(pool_metadata_error( + crate::error::PoolMetadataFailure::TransactionUnknown, + phase, + Some(Arc::clone(original)), + )) + }); + } + outcome + }) + .await +} + +async fn execute_pool_meta_object_cas_once( + pool: Arc, + object: &str, + data: Vec, + opts: ObjectOptions, + fence: &PoolMetaPersistenceFence<'_>, + phase: &'static str, +) -> PoolMetaCasWriteOutcome where S: EcstoreObjectIO, { @@ -6735,7 +6902,11 @@ where "payload_sha256": rustfs_utils::crypto::hex(Sha256::digest(&data)), "if_match": opts.http_preconditions.as_ref().and_then(|p| p.if_match.as_deref()), "if_none_match": opts.http_preconditions.as_ref().and_then(|p| p.if_none_match.as_deref()), - "tail_drained": opts.write_completion == crate::object_api::WriteCompletion::TailDrained, + "tail_drained": matches!( + opts.write_completion, + crate::object_api::WriteCompletion::TailDrained + | crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair + ), "no_lock": opts.no_lock, }) }); @@ -12430,6 +12601,18 @@ impl ECStore { Ok(pool_meta) } + pub(crate) async fn load_runtime_pool_meta_under_worker_fence( + &self, + write_state: &mut PoolMetaWriteState, + worker_fence: &PoolRebalanceWorkerFence, + operation: &str, + ) -> Result { + worker_fence.ensure_held()?; + let pool_meta = self.load_runtime_pool_meta_observing(write_state, operation).await?; + worker_fence.ensure_held()?; + Ok(pool_meta) + } + pub(crate) async fn load_runtime_pool_meta_under_activation_fence( &self, write_state: &mut PoolMetaWriteState, @@ -24124,7 +24307,7 @@ mod pools_tests { }; use crate::bucket::metadata_sys; use crate::data_movement; - use crate::disk::{STORAGE_FORMAT_FILE, endpoint::Endpoint}; + use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET, STORAGE_FORMAT_FILE, endpoint::Endpoint}; use crate::error::{Error, StorageError}; use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints}; use crate::object_api::{ObjectInfo, ObjectOptions}; @@ -24216,6 +24399,335 @@ mod pools_tests { identity: StdMutex, String)>>, } + #[test] + fn pool_meta_cas_future_remains_stack_bounded() { + let pool = Arc::new(PartialPoolMetaWriteStorage::default()); + let fence = PoolMetaPersistenceFence::OfflineRecovery; + let future = super::execute_pool_meta_object_cas( + pool.clone(), + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + ObjectOptions::default(), + &fence, + "identity_cas", + ); + let original = super::execute_pool_meta_object_cas_once( + pool, + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + ObjectOptions::default(), + &fence, + "identity_cas", + ); + let future_size = std::mem::size_of_val(&future); + let original_size = std::mem::size_of_val(&original); + assert!( + future_size <= 4 * 1024, + "pool metadata CAS future must remain stack-bounded; measured {future_size} bytes versus {original_size} without repair" + ); + } + + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum IdentityRepairFault { + Healthy, + Once, + Always, + Changed, + Unreadable, + TypedReadFailure, + MissingEtag, + EmptyEtag, + Reject, + Unknown, + BlockBackoff, + BlockRead, + BlockWrite, + } + + #[derive(Debug)] + struct IdentityRepairStorage { + inner: PartialPoolMetaWriteStorage, + fault: IdentityRepairFault, + writes: AtomicUsize, + reads: AtomicUsize, + blocked: tokio::sync::Notify, + } + + impl IdentityRepairStorage { + fn new(fault: IdentityRepairFault) -> Arc { + Arc::new(Self { + inner: PartialPoolMetaWriteStorage::default(), + fault, + writes: AtomicUsize::new(0), + reads: AtomicUsize::new(0), + blocked: tokio::sync::Notify::new(), + }) + } + } + + #[async_trait::async_trait] + impl ObjectIO for IdentityRepairStorage { + type Error = Error; + type RangeSpec = HTTPRangeSpec; + type HeaderMap = http::HeaderMap; + type ObjectOptions = ObjectOptions; + type ObjectInfo = ObjectInfo; + type GetObjectReader = crate::object_api::GetObjectReader; + type PutObjectReader = crate::object_api::PutObjReader; + + async fn get_object_reader( + &self, + bucket: &str, + object: &str, + range: Option, + headers: http::HeaderMap, + opts: &ObjectOptions, + ) -> std::result::Result { + self.reads.fetch_add(1, Ordering::SeqCst); + if self.fault == IdentityRepairFault::Unreadable { + return Err(Error::ErasureReadQuorum); + } + if self.fault == IdentityRepairFault::TypedReadFailure { + return Err(Error::other(super::pool_metadata_error( + crate::error::PoolMetadataFailure::ReadUnavailable, + "identity_repair_read", + Some(Arc::new(Error::ErasureReadQuorum)), + ))); + } + if self.fault == IdentityRepairFault::BlockRead { + self.blocked.notify_one(); + std::future::pending::<()>().await; + } + let mut reader = self.inner.get_object_reader(bucket, object, range, headers, opts).await?; + match self.fault { + IdentityRepairFault::Changed => { + reader.stream = Box::new(Cursor::new(b"other-identity".to_vec())); + reader.object_info.size = 14; + } + IdentityRepairFault::MissingEtag => reader.object_info.etag = None, + IdentityRepairFault::EmptyEtag => reader.object_info.etag = Some(" ".to_owned()), + _ => {} + } + Ok(reader) + } + + async fn put_object( + &self, + bucket: &str, + object: &str, + data: &mut Self::PutObjectReader, + opts: &ObjectOptions, + ) -> std::result::Result { + let attempt = self.writes.fetch_add(1, Ordering::SeqCst); + assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair); + assert!(opts.max_parity && opts.no_lock); + if attempt > 0 { + assert_eq!( + opts.http_preconditions + .as_ref() + .and_then(HTTPPreconditions::if_none_match_value), + None + ); + assert_eq!( + opts.http_preconditions.as_ref().and_then(HTTPPreconditions::if_match_value), + self.inner + .identity + .lock() + .expect("identity lock") + .as_ref() + .map(|(_, etag)| etag.as_str()), + "repair must use the observed post-commit CAS token", + ); + if self.fault == IdentityRepairFault::Reject { + return Err(Error::PreconditionFailed); + } + if self.fault == IdentityRepairFault::BlockWrite { + self.blocked.notify_one(); + std::future::pending::<()>().await; + } + } + let info = self.inner.put_object(bucket, object, data, opts).await?; + if self.fault == IdentityRepairFault::Unknown { + return Err(Error::Timeout); + } + if self.fault != IdentityRepairFault::Healthy && (attempt == 0 || self.fault == IdentityRepairFault::Always) { + if self.fault == IdentityRepairFault::BlockBackoff { + self.blocked.notify_one(); + } + return Err(Error::other(crate::set_disk::SystemMetadataRepairRequired)); + } + Ok(info) + } + } + + #[tokio::test(start_paused = true)] + async fn pool_identity_owner_repair_preserves_cas_and_fail_closed_boundaries() { + for (fault, object, expected_writes, expected_reads, succeeds) in [ + (IdentityRepairFault::Healthy, POOL_META_IDENTITY_NAME, 1, 0, true), + (IdentityRepairFault::Once, POOL_META_IDENTITY_NAME, 2, 1, true), + (IdentityRepairFault::Always, POOL_META_IDENTITY_NAME, 4, 3, false), + (IdentityRepairFault::Changed, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::Unreadable, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::TypedReadFailure, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::MissingEtag, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::EmptyEtag, POOL_META_IDENTITY_NAME, 1, 1, false), + (IdentityRepairFault::Reject, POOL_META_IDENTITY_NAME, 2, 1, false), + (IdentityRepairFault::Unknown, POOL_META_IDENTITY_NAME, 1, 0, false), + (IdentityRepairFault::Once, POOL_META_NAME, 1, 0, false), + ] { + let pool = IdentityRepairStorage::new(fault); + let state = super::PoolMetaWriteState::default(); + let mut transaction = state.arm_transaction(); + let started = tokio::time::Instant::now(); + let result = save_pool_meta_object_cas( + pool.clone(), + object, + b"owned-identity".to_vec(), + &PoolMetaCasToken::Missing, + &PoolMetaPersistenceFence::OfflineRecovery, + "identity_cas", + &mut transaction, + ) + .await; + assert_eq!(result.is_ok(), succeeds, "{fault:?}: {result:?}"); + let original_source = result + .as_ref() + .err() + .and_then(|err| err.pool_metadata_failure()) + .and_then(|failure| failure.source.clone()); + if fault == IdentityRepairFault::TypedReadFailure { + assert_eq!( + original_source + .as_ref() + .expect("read failure source") + .pool_metadata_failure() + .expect("retain the original read context") + .kind, + crate::error::PoolMetadataFailure::ReadUnavailable + ); + } + if succeeds { + transaction.disarm(); + } else { + assert_eq!( + result + .expect_err("failed repair") + .pool_metadata_failure() + .expect("typed uncertainty") + .kind, + crate::error::PoolMetadataFailure::TransactionUnknown, + "{fault:?}" + ); + } + drop(transaction); + let status = state.ensure_write_safe("after repair"); + assert_eq!(status.is_ok(), succeeds, "{fault:?}"); + if let Err(blocked) = status { + let source = blocked + .pool_metadata_failure() + .expect("blocked transaction context") + .source + .as_ref() + .expect("blocked transaction source"); + assert!( + Arc::ptr_eq(source, original_source.as_ref().expect("caller failure source")), + "caller and latch must retain the same original cause: {fault:?}" + ); + } + assert_eq!(pool.writes.load(Ordering::SeqCst), expected_writes, "{fault:?}"); + assert_eq!(pool.reads.load(Ordering::SeqCst), expected_reads, "{fault:?}"); + if fault == IdentityRepairFault::Always { + assert_eq!(started.elapsed(), StdDuration::from_secs(7), "repair backoff must be bounded"); + } + } + } + + #[tokio::test(start_paused = true)] + async fn pool_identity_owner_repair_cancellation_keeps_the_transaction_armed() { + for fault in [ + IdentityRepairFault::BlockBackoff, + IdentityRepairFault::BlockRead, + IdentityRepairFault::BlockWrite, + ] { + let pool = IdentityRepairStorage::new(fault); + let state = Arc::new(tokio::sync::Mutex::new(super::PoolMetaWriteState::default())); + let task = tokio::spawn({ + let pool = pool.clone(); + let state = state.clone(); + async move { + let state = state.lock().await; + let mut transaction = state.arm_transaction(); + save_pool_meta_object_cas( + pool, + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + &PoolMetaCasToken::Missing, + &PoolMetaPersistenceFence::OfflineRecovery, + "identity_cas", + &mut transaction, + ) + .await + } + }); + pool.blocked.notified().await; + task.abort(); + assert!(task.await.expect_err("cancelled repair").is_cancelled()); + let state = state.lock().await; + let error = state + .ensure_write_safe("cancelled repair") + .expect_err("partial identity remains blocked"); + assert_eq!(error.pool_metadata_failure().expect("typed uncertainty").phase, "identity_cas"); + assert_eq!(state.active_transactions.load(Ordering::SeqCst), 0); + assert!(pool.inner.identity.lock().expect("identity lock").is_some()); + } + } + + #[tokio::test] + async fn pool_identity_owner_repair_stops_after_fence_loss() { + let client = Arc::new(LocalClient::with_manager(Arc::new(GlobalLockManager::new()))); + let lock = NamespaceLock::with_clients_and_quorum("identity-repair-fence".to_string(), vec![client], 1); + let request = LockRequest::new( + ObjectKey::new(super::RUSTFS_META_BUCKET, POOL_META_NAME), + LockType::Exclusive, + "identity-repair-owner", + ) + .with_acquire_timeout(StdDuration::from_secs(1)) + .with_ttl(StdDuration::from_millis(50)) + .with_refresh_interval(StdDuration::from_millis(50)); + let owner = Arc::new( + lock.acquire_guard(&request) + .await + .expect("acquire fence") + .expect("available fence"), + ); + let fence = PoolMetaPersistenceFence::Owned(owner.clone()); + let pool = IdentityRepairStorage::new(IdentityRepairFault::Once); + let state = super::PoolMetaWriteState::default(); + let mut transaction = state.arm_transaction(); + let error = save_pool_meta_object_cas( + pool.clone(), + POOL_META_IDENTITY_NAME, + b"owned-identity".to_vec(), + &PoolMetaCasToken::Missing, + &fence, + "identity_cas", + &mut transaction, + ) + .await + .expect_err("expired owner must not repair"); + assert!(owner.is_lock_lost()); + assert_eq!( + error.pool_metadata_failure().expect("typed uncertainty").kind, + crate::error::PoolMetadataFailure::TransactionUnknown + ); + assert_eq!(pool.writes.load(Ordering::SeqCst), 1); + assert_eq!(pool.reads.load(Ordering::SeqCst), 0); + drop(transaction); + state + .ensure_write_safe("lost repair fence") + .expect_err("lost fence retains the write gate"); + } + #[test] fn decommission_target_gate_retry_is_bounded() { assert!(!decommission_capacity_target_gate_retry_exhausted(0)); @@ -24692,7 +25204,7 @@ mod pools_tests { _ => panic!("phase fixture must receive valid pool metadata"), }; assert!(opts.max_parity && opts.no_lock); - assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrained); + assert_eq!(opts.write_completion, crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair); self.trace.calls.lock().expect("trace lock").push(WriteCall { pool: self.pool, phase, @@ -28293,6 +28805,122 @@ mod pools_tests { assert!(message.contains("1 disk error(s)")); } + fn decommission_listing_resolver() -> MetadataResolutionParams { + MetadataResolutionParams { + dir_quorum: 2, + obj_quorum: 2, + bucket: RUSTFS_META_BUCKET.to_string(), + ..Default::default() + } + } + + fn node_local_prefix_directory_entry() -> MetaCacheEntry { + MetaCacheEntry { + name: "buckets/.heal/".to_string(), + ..Default::default() + } + } + + fn single_version_object_entry(name: &str) -> MetaCacheEntry { + let mut meta = rustfs_filemeta::FileMeta::new(); + meta.add_version(FileInfo { + volume: RUSTFS_META_BUCKET.to_string(), + name: name.to_string(), + size: 1, + mod_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }) + .expect("test file metadata should accept object version"); + MetaCacheEntry { + name: name.to_string(), + metadata: meta.marshal_msg().expect("test file metadata should marshal"), + ..Default::default() + } + } + + #[test] + fn test_resolve_decommission_partial_listing_entry_passes_prefix_directory_below_dir_quorum() { + let entry = resolve_decommission_partial_listing_entry( + MetaCacheEntries(vec![Some(node_local_prefix_directory_entry()), None, None]), + decommission_listing_resolver(), + RUSTFS_META_BUCKET, + BUCKET_META_PREFIX, + 0, + 1, + 0, + OffsetDateTime::now_utc(), + ) + .expect("a prefix directory on one healthy drive is listing structure, not unresolved metadata"); + + assert_eq!(entry.name, "buckets/.heal/"); + assert!(entry.is_dir()); + } + + #[test] + fn test_resolve_decommission_partial_listing_entry_keeps_prefix_directory_fail_closed_on_disk_error() { + let unresolved_entry = resolve_decommission_partial_listing_entry( + MetaCacheEntries(vec![Some(node_local_prefix_directory_entry()), None, None]), + decommission_listing_resolver(), + RUSTFS_META_BUCKET, + BUCKET_META_PREFIX, + 1, + 1, + 0, + OffsetDateTime::now_utc(), + ) + .expect_err("a prefix directory sampled beside a drive error must stay fail-closed"); + + assert_eq!(unresolved_entry.bucket, RUSTFS_META_BUCKET); + assert_eq!(unresolved_entry.object, "buckets/.heal/"); + assert_eq!(unresolved_entry.candidate_count, 1); + assert_eq!(unresolved_entry.disk_error_count, 1); + assert_eq!(unresolved_entry.reason, "metadata_resolution_failed"); + } + + #[test] + fn test_resolve_decommission_partial_listing_entry_rejects_object_below_obj_quorum() { + let object = "buckets/photos/.metadata.bin"; + let unresolved_entry = resolve_decommission_partial_listing_entry( + MetaCacheEntries(vec![Some(single_version_object_entry(object)), None, None]), + decommission_listing_resolver(), + RUSTFS_META_BUCKET, + BUCKET_META_PREFIX, + 0, + 1, + 0, + OffsetDateTime::now_utc(), + ) + .expect_err("an object sampled below the metadata quorum must stay unresolved"); + + assert_eq!(unresolved_entry.object, object); + assert_eq!(unresolved_entry.candidate_count, 1); + assert_eq!(unresolved_entry.disk_error_count, 0); + assert_eq!(unresolved_entry.reason, "metadata_resolution_failed"); + } + + #[test] + fn test_resolve_decommission_partial_listing_entry_rejects_mixed_prefix_and_directory_object_sample() { + let unresolved_entry = resolve_decommission_partial_listing_entry( + MetaCacheEntries(vec![ + Some(node_local_prefix_directory_entry()), + Some(single_version_object_entry("buckets/.heal/")), + None, + ]), + decommission_listing_resolver(), + RUSTFS_META_BUCKET, + BUCKET_META_PREFIX, + 0, + 1, + 0, + OffsetDateTime::now_utc(), + ) + .expect_err("a prefix directory beside a same-name directory object must stay unresolved"); + + assert_eq!(unresolved_entry.object, "buckets/.heal/"); + assert_eq!(unresolved_entry.candidate_count, 2); + assert_eq!(unresolved_entry.disk_error_count, 0); + } + #[test] fn unresolved_entry_ledger_requires_individual_verification_before_completion() { let generation = OffsetDateTime::now_utc(); @@ -28458,6 +29086,100 @@ mod pools_tests { ); } + #[tokio::test] + async fn decommission_listing_skips_node_local_heal_journal_prefix() { + let (dirs, store) = metadata_sys::test_support::isolated_store_over_temp_disks().await; + metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + + let generation = OffsetDateTime::now_utc(); + { + let mut pool_meta = store.pool_meta.write().await; + pool_meta.dont_save = false; + pool_meta.pools[0].decommission = Some(PoolDecommissionInfo { + start_time: Some(generation), + ..Default::default() + }); + } + store + .save_current_pool_meta_for_test(&[0]) + .await + .expect("active decommission metadata should be persisted before listing"); + + // The MRF queue writes its journals with raw local-disk writes on the + // node that owns them, so one drive ends up with a plain directory tree + // that holds no xl.meta anywhere. + let journal_dir = dirs[0] + .path() + .join(RUSTFS_META_BUCKET) + .join(BUCKET_META_PREFIX) + .join(".heal") + .join("mrf"); + tokio::fs::create_dir_all(&journal_dir) + .await + .expect("node-local journal directory should be created"); + tokio::fs::write(journal_dir.join("journal.bin"), b"") + .await + .expect("legacy journal should be written"); + tokio::fs::write(journal_dir.join("journal-scoped.bin"), b"scoped-journal") + .await + .expect("scoped journal should be written"); + + let listed = Arc::new(tokio::sync::Mutex::new(Vec::::new())); + let listed_cb = listed.clone(); + let callback: ListCallback = Arc::new(move |entry: MetaCacheEntry| { + let listed = listed_cb.clone(); + Box::pin(async move { + listed.lock().await.push(entry); + }) + }); + let entry_error = Arc::new(tokio::sync::Mutex::new(None::)); + + store.pools[0].disk_set[0] + .list_objects_to_decommission( + store.clone(), + CancellationToken::new(), + DecomBucketInfo { + name: RUSTFS_META_BUCKET.to_string(), + prefix: BUCKET_META_PREFIX.to_string(), + }, + callback, + entry_error.clone(), + 0, + 0, + generation, + false, + ) + .await + .expect("a node-local journal directory must not fail the decommission listing"); + + assert!(entry_error.lock().await.is_none(), "{:?}", entry_error.lock().await); + assert!( + store.pool_meta.read().await.pools[0] + .decommission + .as_ref() + .expect("decommission should remain active") + .unresolved_entries + .is_empty(), + "a node-local prefix must not be recorded as unresolved object metadata" + ); + + let listed = listed.lock().await; + let heal_entries = listed + .iter() + .filter(|entry| entry.name.starts_with("buckets/.heal/")) + .collect::>(); + assert!( + heal_entries + .iter() + .any(|entry| entry.name == "buckets/.heal/" && entry.is_dir()), + "the journal prefix should reach the callback as a skippable directory: {heal_entries:?}" + ); + assert!( + heal_entries.iter().all(|entry| !entry.is_object()), + "raw journal files must never surface as objects: {heal_entries:?}" + ); + } + #[tokio::test] async fn decommission_metadata_saves_stay_monotonic_across_clock_rollback_before_terminal_restart() { let (_dirs, store) = metadata_sys::test_support::isolated_store_over_temp_disks().await; diff --git a/crates/ecstore/src/data_movement/mod.rs b/crates/ecstore/src/data_movement/mod.rs index 6370da0eb..c2f888937 100644 --- a/crates/ecstore/src/data_movement/mod.rs +++ b/crates/ecstore/src/data_movement/mod.rs @@ -893,6 +893,8 @@ fn is_data_movement_rewritten_metadata(object_info: &ObjectInfo, key: &str, norm SUFFIX_ACTUAL_SIZE, SUFFIX_CRC, SUFFIX_PART_CHECKSUMS, + // Inline placement is local to each physical copy, like its data directory. + rustfs_utils::http::SUFFIX_INLINE_DATA, crate::object_api::ENCRYPTED_PART_LAYOUT_QUORUM_SUFFIX, ] .iter() @@ -3532,6 +3534,23 @@ mod tests { assert!(overwrite_resume_for_target(&source, source.clone())); } + #[test] + fn test_data_movement_identity_accepts_inline_layout_change_only() { + let source = overwrite_equivalence_source(); + let mut target = source.clone(); + rustfs_utils::http::insert_str( + Arc::make_mut(&mut target.user_defined), + rustfs_utils::http::SUFFIX_INLINE_DATA, + "true".to_string(), + ); + assert!(is_equivalent_data_movement_object(&source, &target)); + assert!(is_equivalent_data_movement_object(&target, &source)); + + Arc::make_mut(&mut target.user_defined).insert("x-amz-meta-inline-data".to_string(), "user value".to_string()); + assert!(!is_equivalent_data_movement_object(&source, &target)); + assert!(!is_equivalent_data_movement_object(&target, &source)); + } + #[test] fn test_data_movement_overwrite_resume_accepts_destination_marker() { let source = overwrite_equivalence_source(); diff --git a/crates/ecstore/src/data_movement/scanner_backlog.rs b/crates/ecstore/src/data_movement/scanner_backlog.rs index 22c4a925c..6f7a1d058 100644 --- a/crates/ecstore/src/data_movement/scanner_backlog.rs +++ b/crates/ecstore/src/data_movement/scanner_backlog.rs @@ -110,19 +110,17 @@ async fn read_replica(set: Arc) -> Result reader, Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => { return Ok(ScannerPauseBacklogRetirementRead { replica, etag: None }); diff --git a/crates/ecstore/src/data_usage/mod.rs b/crates/ecstore/src/data_usage/mod.rs index af803a3a2..75aaaa601 100644 --- a/crates/ecstore/src/data_usage/mod.rs +++ b/crates/ecstore/src/data_usage/mod.rs @@ -1325,6 +1325,18 @@ fn select_admin_data_usage_snapshot( && authoritative.is_complete_bucket_usage_snapshot() && observed_data_usage_is_newer(&observed, &authoritative) => { + // A bucket can span several sets. Without a current or retained + // generation for every live set, its observed sum is only a subset, + // not a replacement for the last complete bucket total. Keep the + // baseline's timestamp and provenance until coverage is available. + if authoritative_format + && observed + .usage_snapshot_set_states + .iter() + .any(|state| !state.tombstone && state.scanner_cycle.is_none()) + { + return (authoritative, authoritative_format); + } (merge_partial_observation_for_admin(authoritative, observed), true) } Some(observed) if observed_data_usage_is_newer(&observed, &authoritative) => (observed, true), @@ -2466,7 +2478,13 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu } continue; } - if !cached.stale_snapshot_pending && persisted_update.is_some_and(|persisted| cached.usage_updated_at <= persisted) { + // An observed scan can finish after mutations it never enumerated. Its + // completion time cannot supersede unreconciled single-process counts. + let observed_dirty_usage = data_usage_info.usage_snapshot_converged == Some(false) && cached.dirty; + if !cached.stale_snapshot_pending + && !observed_dirty_usage + && persisted_update.is_some_and(|persisted| cached.usage_updated_at <= persisted) + { continue; } @@ -2529,19 +2547,17 @@ where use crate::object_api::ObjectOptions; use http::HeaderMap; - match store - .get_object_reader( - RUSTFS_META_BUCKET, - key, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let read = store.get_object_reader(RUSTFS_META_BUCKET, key, None, HeaderMap::new(), &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::DataUsageCacheRead, + read, + ); + match read.await { // A cache object that fails to decode is treated as absent rather than // as an error: a corrupt cache should not stall the caller, and the // next scanner pass rewrites it. @@ -3064,6 +3080,8 @@ mod tests { transient_failures: Mutex, reads: Mutex>, terminal_error: Mutex>, + #[cfg(feature = "diagnostic-read-version-caller")] + record_context: bool, } impl UsageCacheReadStore { @@ -3072,6 +3090,8 @@ mod tests { transient_failures: Mutex::new(n), reads: Mutex::new(Vec::new()), terminal_error: Mutex::new(None), + #[cfg(feature = "diagnostic-read-version-caller")] + record_context: false, } } @@ -3080,6 +3100,8 @@ mod tests { transient_failures: Mutex::new(0), reads: Mutex::new(Vec::new()), terminal_error: Mutex::new(Some(error)), + #[cfg(feature = "diagnostic-read-version-caller")] + record_context: false, } } @@ -3106,6 +3128,12 @@ mod tests { _h: Self::HeaderMap, _opts: &Self::ObjectOptions, ) -> Result { + #[cfg(feature = "diagnostic-read-version-caller")] + if self.record_context { + crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request( + &crate::disk::ReadOptions::default(), + ); + } self.reads.lock().await.push(object.to_string()); if let Some(error) = self.terminal_error.lock().await.clone() { return Err(error); @@ -3205,6 +3233,89 @@ mod tests { ); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_usage_cache_read_preserves_api_and_earlier_entry() { + use crate::cluster::rpc::runtime_sources::with_read_version_entry; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context, + }; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("diagnostic usage cache runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut request = S3HttpRequestGuard::new("GET"); + let get = request.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context().expect("dispatched GET operation") + }); + request.response(200); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + let ambient = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(); + for (operation, source) in [ + (None, None), + (Some(get), None), + (None, Some(ReadVersionEntrySource::ServerInfoPools)), + (Some(get), Some(ReadVersionEntrySource::StoreObjectReader)), + ] { + let store = UsageCacheReadStore { + record_context: true, + ..Default::default() + }; + with_s3_http_operation_context(operation, async { + let load = load_data_usage_cache(&store, "usage-cache"); + let cache = match source { + Some(source) => with_read_version_entry(source, load).await, + None => load.await, + } + .expect("missing diagnostic cache is harmless"); + assert!(cache.cache.is_empty()); + assert_eq!(store.read_keys().await.len(), 2, "both prefixed and legacy cache reads are observed"); + }) + .await; + assert_eq!( + crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(), + ambient, + "cache reads must restore the caller's complete diagnostic scope" + ); + } + assert_eq!(current_s3_http_operation_context(), None); + }); + }); + let rows = snapshots.snapshot().into_vec(); + for (source, operation, state) in [ + ("data_usage_cache_read", "unattributed", "absent"), + ("data_usage_cache_read", "s3:GetObject", "preserved"), + ("server_info_pools", "unattributed", "absent"), + ("store_object_reader", "s3:GetObject", "preserved"), + ] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&"remote_rpc") + && labels.get("parent_api_operation") == Some(&operation) + && labels.get("child_api_operation") == Some(&operation) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(2)) + }), + "cache reads must retain API and the earliest observed entry: {source}/{operation}" + ); + } + } + #[tokio::test] async fn data_usage_removal_surfaces_missing_system_volume_as_read_quorum() { for cause in [Error::BucketNotFound(RUSTFS_META_BUCKET.to_string()), Error::VolumeNotFound] { @@ -3718,6 +3829,169 @@ mod tests { assert!(selected.unknown_tier_stats.is_none()); } + fn usage_snapshots_with_missing_observed_set() -> (DataUsageInfo, DataUsageInfo) { + let baseline_time = SystemTime::UNIX_EPOCH + Duration::from_secs(10); + let mut authoritative = data_usage_info_for_test("large-bucket", 510_000, 580_000_000_000, baseline_time); + authoritative.scanner_epoch = Some(4); + authoritative.scanner_cycle = Some(10); + authoritative.usage_snapshot_converged = Some(true); + + let mut observed = data_usage_info_for_test("large-bucket", 3_000, 3_000_000_000, baseline_time + Duration::from_secs(1)); + observed.scanner_epoch = Some(4); + observed.scanner_cycle = Some(11); + observed.usage_snapshot_complete = false; + observed.usage_snapshot_partial = true; + observed.usage_snapshot_converged = Some(false); + observed.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity()); + observed.usage_snapshot_set_states = vec![ + rustfs_data_usage::DataUsageSnapshotSetState { + pool_index: 0, + set_index: 0, + scanner_epoch: Some(4), + scanner_cycle: Some(11), + scan_plan_digest: Some([1; 32]), + complete: true, + ..Default::default() + }, + rustfs_data_usage::DataUsageSnapshotSetState { + pool_index: 1, + set_index: 0, + scan_plan_digest: Some([1; 32]), + ..Default::default() + }, + ]; + assert!( + observed.is_valid_partial_snapshot(), + "a missing set is valid diagnostic progress, not a bucket total" + ); + (authoritative, observed) + } + + #[test] + fn partial_admin_observation_does_not_replace_totals_with_a_missing_set() { + let (authoritative, observed) = usage_snapshots_with_missing_observed_set(); + let identity = authoritative.snapshot_identity(); + + let (selected, _) = select_admin_data_usage_snapshot(authoritative, true, Some(observed)); + + assert_eq!(selected.buckets_usage["large-bucket"].objects_count, 510_000); + assert_eq!(selected.buckets_usage["large-bucket"].size, 580_000_000_000); + assert_eq!(selected.objects_total_count, 510_000); + assert_eq!(selected.objects_total_size, 580_000_000_000); + assert_eq!( + selected.snapshot_identity(), + identity, + "retained counts must not gain a newer timestamp or generation" + ); + assert!(selected.is_complete_bucket_usage_snapshot()); + } + + #[tokio::test] + async fn partial_admin_observation_does_not_promote_a_legacy_baseline() { + let (legacy, observed) = usage_snapshots_with_missing_observed_set(); + let (selected, current_format) = select_admin_data_usage_snapshot(legacy, false, Some(observed)); + let (normalized, _) = normalize_loaded_data_usage(selected, current_format).await; + + assert!(normalized.is_valid_partial_snapshot()); + assert!(!normalized.is_complete_bucket_usage_snapshot()); + assert_eq!(normalized.usage_snapshot_converged, Some(false)); + assert_eq!(normalized.objects_total_count, 3_000); + assert_eq!(normalized.objects_total_size, 3_000_000_000); + } + + #[test] + fn partial_admin_observation_accepts_decreases_when_every_set_is_covered() { + let (authoritative, mut observed) = usage_snapshots_with_missing_observed_set(); + // A compatible retained generation is useful even when it is not a + // completed scan of the current cycle. A tombstone explicitly removes + // a set; neither case is the same as an unavailable live set. + for tombstone in [false, true] { + observed.usage_snapshot_set_states[1].tombstone = tombstone; + observed.usage_snapshot_set_states[1].scanner_epoch = (!tombstone).then_some(4); + observed.usage_snapshot_set_states[1].scanner_cycle = (!tombstone).then_some(10); + let (selected, _) = select_admin_data_usage_snapshot(authoritative.clone(), true, Some(observed.clone())); + + assert_eq!(selected.objects_total_count, 3_000); + assert_eq!(selected.objects_total_size, 3_000_000_000); + assert!(selected.is_valid_partial_snapshot()); + assert!(!selected.is_complete_bucket_usage_snapshot()); + assert_eq!(selected.usage_snapshot_converged, Some(false)); + } + } + + #[tokio::test] + #[serial] + async fn partial_admin_observation_preserves_persisted_totals_across_cache_reload() { + use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks; + use crate::config::com::save_config; + + let (_dirs, store) = isolated_store_over_temp_disks().await; + let (authoritative, observed) = usage_snapshots_with_missing_observed_set(); + let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage"); + let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation"); + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone()) + .await + .expect("persist complete usage"); + save_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(), observed_bytes.clone()) + .await + .expect("persist observation with a missing set"); + clear_usage_memory_cache_for_test().await; + invalidate_data_usage_snapshot_cache().await; + + for reload in [false, false, true] { + if reload { + // Recreate process-local usage state, not an OS process restart. + clear_usage_memory_cache_for_test().await; + invalidate_data_usage_snapshot_cache().await; + } + let mut displayed = load_admin_data_usage_from_backend_cached(store.clone()) + .await + .expect("load admin usage through the real snapshot cache"); + apply_bucket_usage_memory_overlay(&mut displayed).await; + assert_eq!(displayed.objects_total_count, 510_000); + assert_eq!(displayed.objects_total_size, 580_000_000_000); + assert_eq!(displayed.bucket_sizes["large-bucket"], 580_000_000_000); + assert_eq!(displayed.snapshot_identity(), authoritative.snapshot_identity()); + } + let quota = load_data_usage_from_backend_cached(store.clone()) + .await + .expect("load quota authority"); + assert_eq!(quota.objects_total_size, 580_000_000_000); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("read primary"), + primary_bytes + ); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + .await + .expect("read diagnostic observation"), + observed_bytes, + "admin reads must not rewrite either persisted snapshot" + ); + + let mut reconciled = data_usage_info_for_test("large-bucket", 2, 42, SystemTime::UNIX_EPOCH + Duration::from_secs(12)); + reconciled.scanner_epoch = Some(4); + reconciled.scanner_cycle = Some(12); + reconciled.usage_snapshot_converged = Some(true); + save_config( + store.clone(), + DATA_USAGE_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&reconciled).expect("encode confirmed deletion"), + ) + .await + .expect("persist a later complete scan after deletion"); + invalidate_data_usage_snapshot_cache().await; + let displayed = load_admin_data_usage_from_backend_cached(store) + .await + .expect("load confirmed decrease"); + assert_eq!((displayed.objects_total_count, displayed.objects_total_size), (2, 42)); + assert_eq!(displayed.snapshot_identity(), reconciled.snapshot_identity()); + clear_usage_memory_cache_for_test().await; + invalidate_data_usage_snapshot_cache().await; + } + #[tokio::test] async fn authoritative_save_cleanup_removes_observed_snapshot_best_effort() { let store = UsageCasStore::default(); @@ -6522,6 +6796,117 @@ mod tests { ); } + #[tokio::test] + #[serial] + async fn observed_admin_snapshot_preserves_unreconciled_mutations() { + let bucket = "observed-mutations"; + for partial in [false, true] { + for delete in [false, true] { + for timestamp_offset in [0, 10] { + clear_usage_memory_cache_for_test().await; + let mut baseline = data_usage_info_for_test(bucket, 2, 20, SystemTime::now() - Duration::from_secs(10)); + baseline.scanner_epoch = Some(4); + baseline.scanner_cycle = Some(10); + baseline.usage_snapshot_converged = Some(true); + replace_bucket_usage_memory_from_info(&baseline).await; + let (expected_count, expected_size) = if delete { + record_bucket_object_delete_memory(bucket, 10, true).await; + (1, 10) + } else { + record_bucket_object_write_memory(bucket, None, 10).await; + (3, 30) + }; + let mutation_time = memory_cache().read().await[bucket].usage_updated_at; + let mut observed = baseline.clone(); + observed.last_update = Some(mutation_time + Duration::from_secs(timestamp_offset)); + observed.scanner_cycle = Some(11); + observed.usage_snapshot_complete = !partial; + observed.usage_snapshot_partial = partial; + observed.usage_snapshot_converged = Some(false); + observed.usage_snapshot_authoritative_baseline = Some(baseline.snapshot_identity()); + if partial { + observed.usage_snapshot_set_states = vec![rustfs_data_usage::DataUsageSnapshotSetState { + pool_index: 0, + set_index: 0, + scanner_cycle: Some(11), + scanner_epoch: Some(4), + scan_plan_digest: Some([1; 32]), + complete: false, + tombstone: false, + segment_invalidation_proof: None, + }]; + assert!(observed.is_valid_partial_snapshot()); + } + + // Observational saves invalidate the admin cache without reconciling + // memory. Selecting that newer view must not erase local mutations. + let (selected, current_format) = select_admin_data_usage_snapshot(baseline, true, Some(observed)); + assert_eq!(selected.scanner_cycle, Some(11)); + let (selected, _) = normalize_loaded_data_usage(selected, current_format).await; + let mut distributed = selected.clone(); + apply_bucket_usage_memory_overlay_if_authoritative(&mut distributed, false).await; + assert_eq!((distributed.objects_total_count, distributed.objects_total_size), (2, 20)); + + let mut response = selected; + apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await; + assert_eq!( + (response.objects_total_count, response.objects_total_size), + (expected_count, expected_size), + "partial={partial}, delete={delete}, timestamp_offset={timestamp_offset}" + ); + assert_eq!(response.versions_total_count, expected_count); + assert_eq!(response.bucket_sizes[bucket], expected_size); + assert_eq!(response.buckets_usage[bucket].objects_count, expected_count); + assert_eq!(response.usage_snapshot_converged, Some(false)); + assert_eq!(response.usage_snapshot_partial, partial); + let cache = memory_cache().read().await; + assert!(cache[bucket].dirty, "an admin read cannot acknowledge mutations"); + assert!(!cache[bucket].stale_snapshot_pending); + assert_eq!(cache[bucket].pending_scanner_position, None); + drop(cache); + + // A later converged scan can legitimately correct the count downward; + // the display fix must not retain dirty counts or a historical maximum. + let mut converged = data_usage_info_for_test(bucket, 0, 0, mutation_time + Duration::from_secs(20)); + converged.scanner_epoch = Some(4); + converged.scanner_cycle = Some(12); + converged.usage_snapshot_converged = Some(true); + replace_bucket_usage_memory_from_info(&converged).await; + apply_bucket_usage_memory_overlay_if_authoritative(&mut converged, true).await; + assert_eq!((converged.objects_total_count, converged.objects_total_size), (0, 0)); + assert!(!memory_cache().read().await[bucket].dirty); + } + } + } + } + + #[tokio::test] + #[serial] + async fn observed_admin_snapshot_does_not_overlay_clean_or_unknown_usage() { + let bucket = "observed-baseline"; + for unknown in [false, true] { + clear_usage_memory_cache_for_test().await; + let baseline = data_usage_info_for_test(bucket, 2, 20, SystemTime::now() - Duration::from_secs(10)); + replace_bucket_usage_memory_from_info(&baseline).await; + if unknown { + memory_cache().write().await.insert( + bucket.to_string(), + cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false), + ); + record_bucket_object_write_memory(bucket, None, 10).await; + } + let mutation_time = memory_cache().read().await[bucket].usage_updated_at; + let mut observed = data_usage_info_for_test(bucket, 100, 1_000, mutation_time + Duration::from_secs(10)); + observed.usage_snapshot_converged = Some(false); + observed.usage_snapshot_authoritative_baseline = Some(baseline.snapshot_identity()); + let (mut response, _) = select_admin_data_usage_snapshot(baseline, true, Some(observed)); + apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await; + assert_eq!((response.objects_total_count, response.objects_total_size), (100, 1_000)); + assert_eq!(response.usage_snapshot_complete, !unknown); + assert_eq!(response.usage_snapshot_converged, Some(false)); + } + } + #[tokio::test] #[serial] async fn scanner_sync_preserves_dirty_memory_update_with_later_partial_snapshot() { diff --git a/crates/ecstore/src/diagnostics/admin_server_info.rs b/crates/ecstore/src/diagnostics/admin_server_info.rs index 3bbf7cf05..41b6e8647 100644 --- a/crates/ecstore/src/diagnostics/admin_server_info.rs +++ b/crates/ecstore/src/diagnostics/admin_server_info.rs @@ -15,7 +15,9 @@ use crate::cluster::rpc::{ ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, }; -use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached}; +use crate::data_usage::{ + DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot, +}; use crate::error::{Error, Result}; use crate::{ disk::endpoint::{Endpoint, EndpointType}, @@ -37,12 +39,16 @@ use rustfs_protos::{ }; use std::{ collections::{HashMap, HashSet}, - time::Duration, + time::{Duration, Instant}, }; -use time::OffsetDateTime; use tokio::time::timeout; use tonic::Request; -use tracing::warn; +use tracing::{error, trace, warn}; + +const EVENT_SERVER_PING: &str = "server_ping"; +const EVENT_SERVER_INFO: &str = "server_info"; +const LOG_COMPONENT_ECSTORE: &str = "ecstore"; +const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics"; use shadow_rs::shadow; @@ -157,11 +163,22 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> { let response: PingResponse = client.ping(request).await?.into_inner(); - let ping_response_body = flatbuffers::root::(&response.body); - if let Err(e) = ping_response_body { - eprintln!("{e}"); + if flatbuffers::root::(&response.body).is_err() { + error!( + event = EVENT_SERVER_PING, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DIAGNOSTICS, + result = "invalid_response", + "Server ping response decoding failed" + ); } else { - println!("ping_resp:body(flatbuffer): {ping_response_body:?}"); + trace!( + event = EVENT_SERVER_PING, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DIAGNOSTICS, + result = "success", + "Server ping completed" + ); } Ok(()) @@ -259,16 +276,10 @@ fn collect_runtime_server_stats() -> (MemStats, u64, u64) { } pub async fn get_server_info(get_pools: bool) -> InfoMessage { - let nowt: OffsetDateTime = OffsetDateTime::now_utc(); - - warn!("get_server_info start {:?}", nowt); + let started = tracing::enabled!(tracing::Level::TRACE).then(Instant::now); let local = get_local_server_property().await; - let after1 = OffsetDateTime::now_utc(); - - warn!("get_local_server_property end {:?}", after1 - nowt); - let mut servers = { if let Some(sys) = runtime_sources::notification_sys() { sys.server_info().await @@ -277,9 +288,6 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { } }; - let after2 = OffsetDateTime::now_utc(); - - warn!("server_info end {:?}", after2 - after1); servers.push(local); let mut buckets = rustfs_madmin::Buckets::default(); @@ -294,7 +302,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { if let Some(store) = runtime_sources::object_store_handle() { mode = ITEM_ONLINE; apply_data_usage_result( - load_data_usage_from_backend_cached(store.clone()).await, + load_server_info_usage_snapshot(store.clone()).await, &mut buckets, &mut objects, &mut versions, @@ -314,15 +322,8 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { ); } - let after3 = OffsetDateTime::now_utc(); - - warn!("load_data_usage_from_backend end {:?}", after3 - after2); - let backend_info = StorageAdminApi::backend_info(store.as_ref()).await; - let after4 = OffsetDateTime::now_utc(); - - warn!("backend_info end {:?}", after4 - after3); if let Some(endpoints) = runtime_sources::endpoint_pools() { let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints); if added > 0 || !report.is_complete() { @@ -344,9 +345,6 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { } let (online_disks, offline_disks, unknown_disks) = get_online_offline_disks_stats(&all_disks); - let after5 = OffsetDateTime::now_utc(); - - warn!("get_online_offline_disks_stats end {:?}", after5 - after4); backend = rustfs_madmin::ErasureBackend { backend_type: rustfs_madmin::BackendType::ErasureType, online_disks: online_disks.sum(), @@ -359,14 +357,24 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage { }; if get_pools { pools = get_pools_info(&all_disks).await.unwrap_or_default(); - let after6 = OffsetDateTime::now_utc(); - - warn!("get_pools_info end {:?}", after6 - after5); } } let services = rustfs_madmin::Services::default(); + if let Some(started) = started { + trace!( + event = EVENT_SERVER_INFO, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DIAGNOSTICS, + state = "completed", + mode, + include_pools = get_pools, + duration_ms = started.elapsed().as_secs_f64() * 1000.0, + "Server info collected" + ); + } + InfoMessage { mode: Some(mode.to_string()), domain: None, @@ -678,12 +686,16 @@ async fn get_pools_info(all_disks: &[Disk]) -> Result apply_erasure_set_usage(&cache, erasure_set), Err(_) => erasure_set.usage_error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string()), } @@ -967,6 +979,93 @@ mod tests { assert_eq!(usage.size, 6); } + #[tokio::test] + #[serial] + async fn server_info_reads_complete_observation_without_promoting_authority() { + use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks; + use crate::config::com::{read_config, save_config}; + use crate::data_usage::{ + DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache, + load_data_usage_from_backend_cached, + }; + use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo}; + use std::time::{Duration, SystemTime}; + + let (_dirs, store) = isolated_store_over_temp_disks().await; + let primary = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)), + scanner_epoch: Some(1), + usage_snapshot_bootstrap_pending: true, + usage_snapshot_converged: Some(false), + ..Default::default() + }; + let observed = DataUsageInfo { + last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)), + scanner_epoch: Some(1), + scanner_cycle: Some(1), + buckets_count: 1, + objects_total_count: 3, + versions_total_count: 4, + delete_markers_total_count: 1, + objects_total_size: 6, + buckets_usage: [( + "bucket".to_string(), + BucketUsageInfo { + objects_count: 3, + versions_count: 4, + delete_markers_count: 1, + size: 6, + ..Default::default() + }, + )] + .into_iter() + .collect(), + usage_snapshot_complete: true, + usage_snapshot_converged: Some(false), + usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()), + ..Default::default() + }; + let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode"); + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone()) + .await + .expect("bootstrap should persist"); + save_config( + store.clone(), + DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&observed).expect("observation should encode"), + ) + .await + .expect("observation should persist"); + invalidate_data_usage_snapshot_cache().await; + + let displayed = super::load_server_info_usage_snapshot(store.clone()) + .await + .expect("server info should load the complete observation"); + assert_eq!(displayed.usage_snapshot_converged, Some(false)); + let mut buckets = rustfs_madmin::Buckets::default(); + let mut objects = rustfs_madmin::Objects::default(); + let mut versions = rustfs_madmin::Versions::default(); + let mut delete_markers = rustfs_madmin::DeleteMarkers::default(); + let mut usage = rustfs_madmin::Usage::default(); + apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage); + assert!(objects.error.is_none()); + assert_eq!( + (buckets.count, objects.count, versions.count, delete_markers.count, usage.size), + (1, 3, 4, 1, 6) + ); + let quota = load_data_usage_from_backend_cached(store.clone()) + .await + .expect("quota should retain its bootstrap state"); + assert!(!quota.is_complete_bucket_usage_snapshot()); + assert_eq!( + read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("primary should remain readable"), + primary_bytes + ); + invalidate_data_usage_snapshot_cache().await; + } + #[test] fn incomplete_data_usage_is_unavailable_in_server_info() { let mut buckets = rustfs_madmin::Buckets::default(); diff --git a/crates/ecstore/src/disk/cleanup_runtime.rs b/crates/ecstore/src/disk/cleanup_runtime.rs new file mode 100644 index 000000000..8293e577a --- /dev/null +++ b/crates/ecstore/src/disk/cleanup_runtime.rs @@ -0,0 +1,1292 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Admission and execution isolation for PUT, local/RPC disk cleanup and GC. +//! PUT admission precedes staging; receiver and GC disk work share a separate +//! budget so coordinators waiting for peers cannot exhaust receiver capacity. + +use crate::disk::{LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_DISK}; +use std::collections::BTreeSet; +use std::future::Future; +use std::io; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock}; +use std::time::Instant; +use tokio::runtime::{Handle, Runtime}; +use tokio::sync::{OwnedSemaphorePermit, Semaphore}; + +const ENABLE: &str = "RUSTFS_CLEANUP_ISOLATE_ENABLE"; +const BUDGET: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING"; +const BLOCKING_THREADS: &str = "RUSTFS_CLEANUP_BLOCKING_THREADS"; +const ASYNC_THREADS: &str = "RUSTFS_CLEANUP_ASYNC_THREADS"; +const CPUS: &str = "RUSTFS_CLEANUP_CPUS"; +const WORKERS: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS"; +const RENAME_TAIL_CLEANUP_LANE_ENABLE: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ENABLE"; +const DISK_PENDING: &str = "RUSTFS_CLEANUP_DISK_MAX_PENDING"; +const DISK_WORKERS: &str = "RUSTFS_CLEANUP_DISK_WORKERS"; +const GC_WORKERS: &str = "RUSTFS_CLEANUP_GC_WORKERS"; +const EVENT_CLEANUP_RUNTIME: &str = "disk_cleanup_runtime"; +pub(crate) const OLD_DATA_CLEANUP_RECEIPT_FILE: &str = ".rustfs-old-data-cleanup-receipt.json"; + +#[derive(Clone, Copy)] +enum DiskAdmission { + Request, + Guarded, +} + +#[derive(Clone)] +pub(crate) struct Reservation { + _admission: Arc, + execution: Arc, + handle: Handle, +} + +struct Admission { + _permit: OwnedSemaphorePermit, + metrics_enabled: bool, + metric: &'static str, +} + +impl Admission { + fn new(permit: OwnedSemaphorePermit, metric: &'static str) -> Self { + let metrics_enabled = rustfs_io_metrics::put_stage_metrics_enabled(); + if metrics_enabled { + metrics::gauge!(metric).increment(1.0); + } + Self { + _permit: permit, + metrics_enabled, + metric, + } + } +} + +impl Drop for Admission { + fn drop(&mut self) { + if self.metrics_enabled { + metrics::gauge!(self.metric).decrement(1.0); + } + } +} + +/// A lease release cannot queue behind work that might need the releasing +/// reader's locks. It relinquishes the lease first, then tries this disk budget. +pub(crate) struct DiskExecution { + _active: Admission, + _pending: Admission, +} + +/// Created only inside an owned scan. Steps can borrow the scan's directory +/// iterator and paths: cancelling its waiter cannot cancel a step's syscalls. +#[cfg_attr(test, derive(Default))] +pub(crate) struct GcBudget { + gates: Option<(Arc, Arc)>, +} + +impl GcBudget { + pub(crate) async fn step(&self, future: F) -> super::error::Result + where + F: Future>, + { + let Some((pending, execution)) = &self.gates else { + return future.await; + }; + // Scan producers are bounded and hold no object locks here. Unlike + // requests, they can wait for admission without an unbounded backlog. + let started = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now); + let pending = pending.clone().acquire_owned().await.map_err(io::Error::other)?; + execute_disk( + Admission::new(pending, "rustfs_cleanup_disk_pending"), + None, + execution.clone(), + started, + future, + ) + .await + } +} + +fn try_disk_permit(semaphore: &Arc) -> io::Result { + semaphore.clone().try_acquire_owned().map_err(|err| { + if rustfs_io_metrics::put_stage_metrics_enabled() { + metrics::counter!("rustfs_cleanup_disk_admission_rejected_total").increment(1); + } + io::Error::new(io::ErrorKind::WouldBlock, err) + }) +} + +async fn execute_disk( + pending: Admission, + active: Option, + execution: Arc, + started: Option, + future: F, +) -> super::error::Result +where + F: Future>, +{ + let _pending = pending; + let active = match active { + Some(active) => active, + None => execution.acquire_owned().await.map_err(super::error::DiskError::other)?, + }; + let _active = Admission::new(active, "rustfs_cleanup_disk_active"); + if let Some(started) = started { + metrics::histogram!("rustfs_cleanup_disk_execution_wait_seconds").record(started.elapsed().as_secs_f64()); + } + future.await +} + +struct Config { + pending: usize, + workers: usize, + async_threads: usize, + blocking_threads: usize, + cpus: Vec, + disk_pending: usize, + disk_workers: usize, + gc_workers: usize, +} + +fn positive_env(name: &str, default: usize) -> io::Result { + match std::env::var(name) { + Ok(value) => value + .parse::() + .ok() + .filter(|v| *v > 0 && *v <= Semaphore::MAX_PERMITS) + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, format!("{name} must be a positive supported count"))), + Err(std::env::VarError::NotPresent) => Ok(default), + Err(err) => Err(io::Error::new(io::ErrorKind::InvalidInput, err)), + } +} + +// Keep the guard's accepted values aligned with the lane's existing parser. +fn legacy_rename_tail_cleanup_lane_enabled(value: Option<&str>) -> bool { + matches!(value, Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON")) +} + +fn parse_cpus(value: &str) -> io::Result> { + let invalid = || { + io::Error::new( + io::ErrorKind::InvalidInput, + "cleanup CPUs must be comma-separated CPU IDs or ascending ranges", + ) + }; + if value.trim().is_empty() { + return Err(invalid()); + } + let mut cpus = BTreeSet::new(); + for part in value.split(',') { + let mut ends = part.trim().split('-'); + let first = ends.next().and_then(|s| s.parse::().ok()).ok_or_else(invalid)?; + let last = match ends.next() { + Some(s) => s.parse::().map_err(|_| invalid())?, + None => first, + }; + if ends.next().is_some() || first > last || last >= 1024 { + return Err(invalid()); + } + cpus.extend(first..=last); + } + Ok(cpus.into_iter().collect()) +} + +impl Config { + fn from_env() -> io::Result { + for name in [ + "RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP", + "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER", + "RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP", + ] { + if rustfs_utils::get_env_bool(name, false) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("{name} cannot be combined with cleanup isolation"), + )); + } + } + if legacy_rename_tail_cleanup_lane_enabled(std::env::var(RENAME_TAIL_CLEANUP_LANE_ENABLE).ok().as_deref()) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + format!("{RENAME_TAIL_CLEANUP_LANE_ENABLE} cannot be combined with cleanup isolation"), + )); + } + let workers = positive_env(WORKERS, 2)?; + let pending = positive_env(BUDGET, 1024)?; + let blocking_threads = positive_env(BLOCKING_THREADS, 4)?; + let cpus = match std::env::var(CPUS) { + Ok(value) if value.trim().eq_ignore_ascii_case("none") => Vec::new(), + Ok(value) => parse_cpus(&value)?, + Err(std::env::VarError::NotPresent) if cfg!(target_os = "linux") => vec![1, 2], + Err(std::env::VarError::NotPresent) => Vec::new(), + Err(err) => return Err(io::Error::new(io::ErrorKind::InvalidInput, err)), + }; + let available = std::thread::available_parallelism().map_or(1, usize::from); + let async_threads = positive_env(ASYNC_THREADS, available.min(if cpus.is_empty() { 2 } else { cpus.len().min(2) }))?; + let disk_workers = positive_env(DISK_WORKERS, blocking_threads)?; + Ok(Self { + pending, + workers, + async_threads, + blocking_threads, + cpus, + // Bound queueing to sixteen service waves by default. Increasing + // async coordinator concurrency does not create more OS threads. + disk_pending: positive_env(DISK_PENDING, disk_workers.saturating_mul(16).min(Semaphore::MAX_PERMITS))?, + disk_workers, + gc_workers: positive_env(GC_WORKERS, 1)?, + }) + } + + fn validate(&self) -> io::Result<()> { + if self.pending == 0 + || self.pending > Semaphore::MAX_PERMITS + || self.workers == 0 + || self.workers > self.pending + || self.async_threads == 0 + || self.async_threads > 1024 + || self.blocking_threads == 0 + || self.blocking_threads > 1024 + || self.disk_pending == 0 + || self.disk_pending > Semaphore::MAX_PERMITS + || self.disk_workers == 0 + || self.disk_workers > self.disk_pending + || self.gc_workers == 0 + || self.gc_workers > self.disk_workers + { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cleanup counts must be positive: PUT workers <= PUT pending, GC workers <= disk workers <= disk pending, async/blocking threads <= 1024", + )); + } + validate_cpus(&self.cpus) + } +} + +#[cfg(target_os = "linux")] +fn validate_cpus(cpus: &[usize]) -> io::Result<()> { + if cpus.is_empty() { + return Ok(()); + } + let allowed = rustix::thread::sched_getaffinity(None)?; + if cpus + .iter() + .any(|cpu| *cpu >= rustix::thread::CpuSet::MAX_CPU || !allowed.is_set(*cpu)) + { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "cleanup CPUs must be a subset of the caller's allowed CPU set", + )); + } + Ok(()) +} + +#[cfg(not(target_os = "linux"))] +fn validate_cpus(cpus: &[usize]) -> io::Result<()> { + if cpus.is_empty() { + Ok(()) + } else { + Err(io::Error::new(io::ErrorKind::Unsupported, "cleanup CPU affinity requires Linux")) + } +} + +fn bind_thread(cpus: &[usize]) -> io::Result<()> { + if cpus.is_empty() { + return Ok(()); + } + #[cfg(target_os = "linux")] + { + let mut set = rustix::thread::CpuSet::new(); + for cpu in cpus { + set.set(*cpu); + } + rustix::thread::sched_setaffinity(None, &set)?; + if rustix::thread::sched_getaffinity(None)? != set { + return Err(io::Error::other("cleanup CPU affinity changed during thread startup")); + } + Ok(()) + } + #[cfg(not(target_os = "linux"))] + { + validate_cpus(cpus) + } +} + +struct CleanupRuntime { + runtime: Option, + admission: Arc, + execution: Arc, + affinity_failed: Arc, + disk_admission: Arc, + disk_execution: Arc, + gc_execution: Arc, +} + +impl CleanupRuntime { + fn new(config: Config) -> io::Result { + config.validate()?; + let affinity_failed = Arc::new(AtomicBool::new(false)); + let failed = affinity_failed.clone(); + let cpus = config.cpus; + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(config.async_threads) + .max_blocking_threads(config.blocking_threads) + .thread_name("rustfs-cleanup") + .enable_all() + .on_thread_start(move || { + // Explicit fsync dispatch must not escape into the shared + // fsync runtime. This callback also runs on blocking threads. + crate::disk::os::use_current_runtime_for_fsync(); + if let Err(err) = bind_thread(&cpus) + && !failed.swap(true, Ordering::AcqRel) + { + tracing::error!(event = EVENT_CLEANUP_RUNTIME, component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK, state = "affinity_failed", error = %err, + "Cleanup thread affinity failed"); + } + }) + .build()?; + Ok(Self { + runtime: Some(runtime), + admission: Arc::new(Semaphore::new(config.pending)), + execution: Arc::new(Semaphore::new(config.workers)), + affinity_failed, + disk_admission: Arc::new(Semaphore::new(config.disk_pending)), + disk_execution: Arc::new(Semaphore::new(config.disk_workers)), + gc_execution: Arc::new(Semaphore::new(config.gc_workers)), + }) + } + + fn reserve(&self) -> io::Result { + if self.affinity_failed.load(Ordering::Acquire) { + return Err(io::Error::other("cleanup thread affinity failed; refusing new cleanup admission")); + } + let metrics_enabled = rustfs_io_metrics::put_stage_metrics_enabled(); + // Callers may already own source-reader or data-movement locks. Do + // not wait here behind a PUT whose completion requires those locks. + let permit = self.admission.clone().try_acquire_owned().map_err(|err| { + if metrics_enabled { + metrics::counter!("rustfs_cleanup_admission_rejected_total").increment(1); + } + io::Error::new(io::ErrorKind::WouldBlock, err) + })?; + let runtime = self + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))?; + Ok(Reservation { + _admission: Arc::new(Admission::new(permit, "rustfs_cleanup_admitted_puts")), + execution: self.execution.clone(), + handle: runtime.handle().clone(), + }) + } + + async fn disk_operation(&self, future: F, mode: DiskAdmission) -> super::error::Result + where + F: Future> + Send + 'static, + T: Send + 'static, + { + if self.affinity_failed.load(Ordering::Acquire) && !super::os::is_isolated_io_thread() { + return Err(io::Error::other("cleanup thread affinity failed; refusing new disk cleanup admission").into()); + } + let started = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now); + let pending = try_disk_permit(&self.disk_admission)?; + let pending = Admission::new(pending, "rustfs_cleanup_disk_pending"); + let runtime = self + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))?; + let execution = self.disk_execution.clone(); + // An externally guarded mutation may already hold a namespace lock + // needed by an active disk job. Never queue it behind that job. + let active = if matches!(mode, DiskAdmission::Guarded) { + try_disk_permit(&execution)? + } else { + // Keep queued work in the caller's existing future. A waiter + // timeout must discard an unstarted delete, not leave a task that + // can later delete a reused path after its caller released locks. + execution.clone().acquire_owned().await.map_err(io::Error::other)? + }; + // A permit does not mean the worker has polled this operation. Use + // the result receiver's lifetime to discard work cancelled during + // runtime scheduling, before its first poll can start filesystem I/O. + let (result_tx, result_rx) = tokio::sync::oneshot::channel(); + let task = runtime.spawn(async move { + if result_tx.is_closed() { + return; + } + // After the first poll, the owned job must retain both permits + // through I/O even if the waiter cancels and closes result_rx. + let result = execute_disk(pending, Some(active), execution, started, future).await; + let _ = result_tx.send(result); + }); + match result_rx.await { + Ok(result) => result, + Err(err) => { + // Preserve a panic's JoinError rather than erasing its cause + // into a closed-channel error. + task.await.map_err(super::error::DiskError::other)?; + Err(super::error::DiskError::other(err)) + } + } + } + + fn try_disk_execution(&self) -> io::Result { + let pending = try_disk_permit(&self.disk_admission)?; + let active = try_disk_permit(&self.disk_execution)?; + Ok(DiskExecution { + _pending: Admission::new(pending, "rustfs_cleanup_disk_pending"), + _active: Admission::new(active, "rustfs_cleanup_disk_active"), + }) + } + + async fn gc_operation(&self, scan: F) -> super::error::Result<()> + where + F: FnOnce(GcBudget) -> Fut + Send + 'static, + Fut: Future> + Send + 'static, + { + if self.affinity_failed.load(Ordering::Acquire) { + return Err(io::Error::other("cleanup thread affinity failed; refusing new GC scan").into()); + } + let handle = self + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))? + .handle(); + let execution = self.gc_execution.clone(); + let budget = GcBudget { + gates: Some((self.disk_admission.clone(), self.disk_execution.clone())), + }; + handle + .spawn(async move { + let active = execution.acquire_owned().await.map_err(super::error::DiskError::other)?; + let _active = Admission::new(active, "rustfs_cleanup_gc_active"); + scan(budget).await + }) + .await + .map_err(super::error::DiskError::other)? + } +} + +impl Drop for CleanupRuntime { + fn drop(&mut self) { + if let Some(runtime) = self.runtime.take() { + runtime.shutdown_background(); + } + } +} + +fn instance() -> io::Result> { + static INSTANCE: OnceLock, String>> = OnceLock::new(); + match INSTANCE.get_or_init(|| { + if !rustfs_utils::get_env_bool(ENABLE, false) { + return Ok(None); + } + Config::from_env() + .and_then(CleanupRuntime::new) + .map(Some) + .map_err(|err| err.to_string()) + }) { + Ok(runtime) => Ok(runtime.as_ref()), + Err(err) => Err(io::Error::other(err.clone())), + } +} + +pub(crate) fn reserve() -> io::Result> { + match instance()? { + Some(runtime) => runtime.reserve().map(Some), + None => Ok(None), + } +} + +pub(crate) fn enabled() -> io::Result { + Ok(instance()?.is_some()) +} + +/// Local cleanup entry point, also used by remote RPC receivers. This budget +/// is distinct from coordinator execution: coordinators can wait for peers +/// without exhausting the peers' capacity to service disk cleanup. +pub(crate) async fn disk_operation(future: F) -> super::error::Result +where + F: Future> + Send + 'static, + T: Send + 'static, +{ + match instance()? { + Some(runtime) => runtime.disk_operation(future, DiskAdmission::Request).await, + None => future.await, + } +} + +pub(crate) async fn guarded_disk_operation(future: F) -> super::error::Result +where + F: Future> + Send + 'static, + T: Send + 'static, +{ + match instance()? { + Some(runtime) => runtime.disk_operation(future, DiskAdmission::Guarded).await, + None => future.await, + } +} + +/// Lease release must run even under cleanup saturation, or reader tokens +/// would leak. Only its optional deferred deletion consumes disk permits. +pub(crate) async fn release_lease(future: F) -> super::error::Result<()> +where + F: Future> + Send + 'static, +{ + match instance()? { + Some(runtime) => runtime + .runtime + .as_ref() + .ok_or_else(|| io::Error::other("cleanup runtime stopped"))? + .spawn(future) + .await + .map_err(super::error::DiskError::other)?, + None => future.await, + } +} + +pub(crate) fn try_disk_execution() -> io::Result> { + instance()?.map(CleanupRuntime::try_disk_execution).transpose() +} + +/// At most gc_workers scans can produce work. Each scan yields the shared +/// disk execution budget between entries without spawning a task per entry. +pub(crate) async fn run_gc(scan: F) -> super::error::Result<()> +where + F: FnOnce(GcBudget) -> Fut + Send + 'static, + Fut: Future> + Send + 'static, +{ + match instance()? { + Some(runtime) => runtime.gc_operation(scan).await, + None => scan(GcBudget { gates: None }).await, + } +} + +/// Dispatch only already-admitted work. Keeping the runtime handle in the +/// reservation prevents configuration errors from dropping committed cleanup. +pub(crate) fn spawn(reservation: Option, future: F) -> tokio::task::JoinHandle +where + F: Future + Send + 'static, + F::Output: Send + 'static, +{ + let started = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now); + match reservation { + Some(reservation) => reservation.handle.clone().spawn(async move { + // This semaphore is never closed. Both queued work and overflow + // use this gate, and cancellation of the waiter cannot release it. + let _permit = reservation.execution.clone().acquire_owned().await; + if let Some(started) = started { + metrics::histogram!("rustfs_cleanup_execution_wait_seconds").record(started.elapsed().as_secs_f64()); + } + let result = future.await; + drop(reservation); + result + }), + None => tokio::spawn(future), + } +} + +/// Queue workers, queue overflow and synchronous failure cleanup share a gate. +/// With isolation disabled, retain the original inline polling behavior. +pub(crate) async fn run(reservation: Option, future: F) +where + F: Future + Send + 'static, +{ + if reservation.is_none() { + future.await; + } else if let Err(err) = spawn(reservation, future).await { + tracing::error!(event = EVENT_CLEANUP_RUNTIME, component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK, state = "task_failed", error = %err, + "Cleanup task failed"); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::AtomicUsize; + use std::time::Duration; + use tokio::sync::oneshot; + + fn config(pending: usize, workers: usize, blocking_threads: usize) -> Config { + Config { + pending, + workers, + async_threads: workers.min(2), + blocking_threads, + cpus: Vec::new(), + disk_pending: 4, + disk_workers: 2, + gc_workers: 1, + } + } + + async fn deadline(future: F) -> F::Output { + tokio::time::timeout(Duration::from_secs(20), future) + .await + .expect("cleanup test timed out") + } + + #[test] + fn cpu_list_rejects_invalid_or_unbounded_ranges() { + assert_eq!(parse_cpus("3, 1-3,6").unwrap(), vec![1, 2, 3, 6]); + for value in ["", " ", "1,", "-1", "2-1", "1-2-3", "no", "1024", "0-18446744073709551615"] { + assert_eq!(parse_cpus(value).unwrap_err().kind(), io::ErrorKind::InvalidInput, "{value}"); + } + } + + #[test] + fn cleanup_lane_guard_matches_legacy_boolean_values() { + for (value, expected) in [ + (Some("1"), true), + (Some("true"), true), + (Some("TRUE"), true), + (Some("yes"), true), + (Some("YES"), true), + (Some("on"), true), + (Some("ON"), true), + (Some("True"), false), + (Some("enabled"), false), + (None, false), + ] { + assert_eq!(legacy_rename_tail_cleanup_lane_enabled(value), expected, "{value:?}"); + } + } + + #[test] + fn invalid_environment_is_rejected_before_admission() { + const CHILD: &str = "RUSTFS_CLEANUP_CONFIG_TEST_CHILD"; + if std::env::var_os(CHILD).is_some() { + assert!(reserve().is_err()); + return; + } + // Each configuration needs a fresh process: production intentionally + // caches configuration, and mutating process env races parallel tests. + for (name, value) in [ + (WORKERS, "0"), + (WORKERS, "5"), + (BUDGET, "invalid"), + (BLOCKING_THREADS, "0"), + (BLOCKING_THREADS, "1025"), + (ASYNC_THREADS, "0"), + (ASYNC_THREADS, "1025"), + (DISK_PENDING, "0"), + (DISK_WORKERS, "3"), + (GC_WORKERS, "3"), + (CPUS, "1-0"), + ("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP", "true"), + ("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER", "true"), + ("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP", "true"), + (RENAME_TAIL_CLEANUP_LANE_ENABLE, "true"), + ] { + let result = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "disk::cleanup_runtime::tests::invalid_environment_is_rejected_before_admission", + "--nocapture", + ]) + .env(CHILD, "1") + .env(ENABLE, "true") + .env(WORKERS, "1") + .env(BUDGET, "4") + .env(BLOCKING_THREADS, "1") + .env(ASYNC_THREADS, "1") + .env(DISK_PENDING, "2") + .env(DISK_WORKERS, "2") + .env(GC_WORKERS, "1") + .env_remove(CPUS) + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .env_remove(RENAME_TAIL_CLEANUP_LANE_ENABLE) + .env(name, value) + .output() + .unwrap(); + assert!(result.status.success(), "{name}={value}: {}", String::from_utf8_lossy(&result.stderr)); + assert!( + String::from_utf8_lossy(&result.stdout).contains("1 passed"), + "child must execute the admission assertion" + ); + } + } + + #[test] + fn invalid_budgets_fail_before_building_threads() { + for config in [config(0, 1, 1), config(1, 0, 1), config(1, 2, 1), config(1, 1, 0)] { + assert!(matches!(CleanupRuntime::new(config), Err(err) if err.kind() == io::ErrorKind::InvalidInput)); + } + } + + #[tokio::test] + async fn disk_work_keeps_budget_after_rpc_waiter_cancellation() { + let mut settings = config(1, 1, 1); + settings.disk_pending = 1; + settings.disk_workers = 1; + let runtime = Arc::new(CleanupRuntime::new(settings).unwrap()); + let receiver = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let rpc = tokio::spawn(async move { + receiver + .disk_operation( + async move { + tokio::task::spawn_blocking(move || { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + entered.send(()).unwrap(); + released.recv_timeout(Duration::from_secs(20)).unwrap(); + }) + .await + .unwrap(); + Ok(()) + }, + DiskAdmission::Request, + ) + .await + }); + deadline(entry).await.unwrap(); + rpc.abort(); + assert!(rpc.await.unwrap_err().is_cancelled()); + assert_eq!(runtime.disk_admission.available_permits(), 0); + assert_eq!(runtime.disk_execution.available_permits(), 0); + let ran = Arc::new(AtomicBool::new(false)); + let invoked = ran.clone(); + let rejected = runtime + .disk_operation( + async move { + invoked.store(true, Ordering::SeqCst); + Ok(()) + }, + DiskAdmission::Request, + ) + .await; + assert!(matches!(rejected, Err(super::super::error::DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert!(!ran.load(Ordering::SeqCst), "rejected work must not execute"); + release.send(()).unwrap(); + deadline(async { + while runtime.disk_admission.available_permits() == 0 { + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!(runtime.disk_execution.available_permits(), 1); + } + + #[tokio::test] + async fn cancelling_queued_disk_work_discards_it_before_execution() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime")); + let occupied = runtime + .disk_execution + .clone() + .acquire_many_owned(2) + .await + .expect("occupy all execution slots"); + let receiver = runtime.clone(); + let ran = Arc::new(AtomicBool::new(false)); + let invoked = ran.clone(); + let waiter = tokio::spawn(async move { + receiver + .disk_operation( + async move { + invoked.store(true, Ordering::SeqCst); + Ok(()) + }, + DiskAdmission::Request, + ) + .await + }); + deadline(async { + while runtime.disk_admission.available_permits() == 4 { + tokio::task::yield_now().await; + } + }) + .await; + waiter.abort(); + assert!(waiter.await.expect_err("cancel queued waiter").is_cancelled()); + assert_eq!( + runtime.disk_admission.available_permits(), + 4, + "unstarted work must release pending capacity immediately" + ); + drop(occupied); + deadline(runtime.disk_operation(async { Ok(()) }, DiskAdmission::Request)) + .await + .expect("later work must progress"); + assert!(!ran.load(Ordering::SeqCst), "cancelled queued deletion must never execute later"); + } + + #[tokio::test] + async fn cancelling_disk_work_before_worker_poll_preserves_reused_path() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime")); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let stalled = runtime.runtime.as_ref().expect("live runtime").spawn(async move { + entered.send(()).expect("worker entered"); + released + .recv_timeout(Duration::from_secs(30)) + .expect("release stalled worker"); + }); + deadline(entry).await.expect("worker must stall before dispatch"); + let dir = tempfile::tempdir().expect("temporary disk"); + let backup = dir.path().join("xl.meta.bkp"); + std::fs::write(&backup, b"previous transaction").expect("old backup"); + { + let path = backup.clone(); + let deletion = runtime.disk_operation( + async move { + tokio::fs::remove_file(path).await?; + Ok(()) + }, + DiskAdmission::Request, + ); + tokio::pin!(deletion); + assert!(futures::poll!(&mut deletion).is_pending()); + assert_eq!(runtime.disk_execution.available_permits(), 1, "execution was admitted but not polled"); + } + std::fs::write(&backup, b"next transaction").expect("reuse path after cancelled waiter"); + release.send(()).expect("resume worker"); + deadline(stalled).await.expect("worker resumed"); + deadline(async { + while runtime.disk_admission.available_permits() != 4 { + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!( + std::fs::read(backup).expect("unstarted deletion must not execute after cancellation"), + b"next transaction" + ); + assert_eq!(runtime.disk_execution.available_permits(), 2); + } + + #[tokio::test] + async fn disk_task_panic_preserves_cause_and_releases_budget() { + async fn panicking_job() -> super::super::error::Result<()> { + panic!("injected cleanup panic"); + } + let runtime = CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime"); + let error = deadline(runtime.disk_operation(panicking_job(), DiskAdmission::Request)) + .await + .expect_err("worker panic must reach its caller"); + let super::super::error::DiskError::Io(error) = error else { + panic!("worker panic must retain its I/O error source"); + }; + let cause = error + .get_ref() + .expect("panic source") + .downcast_ref::() + .expect("preserved JoinError"); + assert!(cause.is_panic()); + assert_eq!(runtime.disk_admission.available_permits(), 4); + assert_eq!(runtime.disk_execution.available_permits(), 2); + } + + #[tokio::test] + async fn coordinator_waiting_for_disk_does_not_consume_receiver_capacity() { + let mut settings = config(1, 1, 1); + settings.disk_pending = 1; + settings.disk_workers = 1; + let runtime = Arc::new(CleanupRuntime::new(settings).unwrap()); + let reservation = runtime.reserve().unwrap(); + let receiver = runtime.clone(); + let result = deadline(spawn(Some(reservation), async move { + receiver + .disk_operation(async { Ok(42) }, DiskAdmission::Request) + .await + .unwrap() + })) + .await + .unwrap(); + assert_eq!(result, 42); + } + + #[tokio::test] + async fn gc_shares_disk_budget_and_guarded_requests_do_not_wait_under_locks() { + let mut settings = config(1, 1, 1); + settings.disk_pending = 1; + settings.disk_workers = 1; + let runtime = Arc::new(CleanupRuntime::new(settings).unwrap()); + let receiver = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = oneshot::channel(); + let rpc = tokio::spawn(async move { + receiver + .disk_operation( + async move { + entered.send(()).unwrap(); + released.await.unwrap(); + Ok(()) + }, + DiskAdmission::Request, + ) + .await + }); + deadline(entry).await.unwrap(); + let guarded = runtime.disk_operation(async { Ok(()) }, DiskAdmission::Guarded).await; + assert!(matches!(guarded, Err(super::super::error::DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + let budget = GcBudget { + gates: Some((runtime.disk_admission.clone(), runtime.disk_execution.clone())), + }; + let gc = budget.step(async { Ok(7) }); + tokio::pin!(gc); + assert!( + futures::poll!(&mut gc).is_pending(), + "GC must join the shared budget rather than bypass it" + ); + release.send(()).unwrap(); + deadline(rpc).await.unwrap().unwrap(); + assert_eq!(deadline(gc).await.unwrap(), 7); + + // Also reject a guarded mutation when pending capacity is free but + // execution is occupied. This prevents namespace -> budget inversion. + let execution = runtime.disk_execution.clone().acquire_owned().await.unwrap(); + let guarded = runtime.disk_operation(async { Ok(()) }, DiskAdmission::Guarded).await; + assert!(matches!(guarded, Err(super::super::error::DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(runtime.disk_admission.available_permits(), 1); + assert!(matches!(runtime.try_disk_execution(), Err(err) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(runtime.disk_admission.available_permits(), 1); + drop(execution); + let lease_cleanup = runtime.try_disk_execution().unwrap(); + assert_eq!(runtime.disk_admission.available_permits(), 0); + assert_eq!(runtime.disk_execution.available_permits(), 0); + drop(lease_cleanup); + assert_eq!(runtime.disk_execution.available_permits(), 1); + } + + #[tokio::test] + async fn gc_scan_limit_leaves_disk_slots_available_between_entries() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).unwrap()); + let first_runtime = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = oneshot::channel(); + let first = tokio::spawn(async move { + first_runtime + .gc_operation(move |_| async move { + entered.send(()).unwrap(); + released.await.unwrap(); + Ok(()) + }) + .await + }); + deadline(entry).await.unwrap(); + assert_eq!(runtime.gc_execution.available_permits(), 0); + assert_eq!( + deadline(runtime.disk_operation(async { Ok(42) }, DiskAdmission::Request)) + .await + .unwrap(), + 42 + ); + let second = runtime.gc_operation(|_| async { Ok(()) }); + tokio::pin!(second); + assert!(futures::poll!(&mut second).is_pending()); + release.send(()).unwrap(); + deadline(first).await.unwrap().unwrap(); + deadline(second).await.unwrap(); + } + + #[tokio::test] + async fn gc_steps_borrow_scan_state_and_keep_budget_after_waiter_cancellation() { + let runtime = Arc::new(CleanupRuntime::new(config(1, 1, 1)).expect("cleanup runtime")); + let receiver = runtime.clone(); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let waiter = tokio::spawn(async move { + receiver + .gc_operation(move |budget| async move { + let scan_task = tokio::task::id(); + let mut count = 0; + for _ in 0..32 { + budget + .step(async { + assert_eq!(tokio::task::id(), scan_task, "each entry must stay on the owned scan task"); + count += 1; + Ok(()) + }) + .await?; + } + assert_eq!(count, 32); + budget + .step(async move { + tokio::task::spawn_blocking(move || { + entered.send(()).expect("notify started syscall"); + released.recv_timeout(Duration::from_secs(20)).expect("release syscall"); + }) + .await + .expect("blocking cleanup"); + Ok(()) + }) + .await + }) + .await + }); + deadline(entry).await.expect("GC must start"); + waiter.abort(); + assert!(waiter.await.expect_err("cancelled waiter").is_cancelled()); + assert_eq!(runtime.gc_execution.available_permits(), 0); + assert_eq!(runtime.disk_admission.available_permits(), 3); + assert_eq!(runtime.disk_execution.available_permits(), 1); + release.send(()).expect("release blocking cleanup"); + deadline(async { + while runtime.gc_execution.available_permits() == 0 { + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!(runtime.disk_admission.available_permits(), 4); + assert_eq!(runtime.disk_execution.available_permits(), 2); + } + + #[test] + fn defaults_bound_queue_waves_and_separate_threads_from_jobs() { + const CHILD: &str = "RUSTFS_CLEANUP_DEFAULTS_TEST_CHILD"; + if std::env::var_os(CHILD).is_some() { + let mut config = Config::from_env().expect("default configuration"); + let expected_cpus = match std::env::var(CPUS).as_deref() { + Ok("none") => Vec::new(), + Ok("3-4") => vec![3, 4], + _ if cfg!(target_os = "linux") => vec![1, 2], + _ => Vec::new(), + }; + assert_eq!(config.cpus, expected_cpus); + assert_eq!(config.workers, 64); + assert!(config.async_threads <= 2); + assert_eq!(config.blocking_threads, 4); + assert_eq!(config.disk_workers, 4); + assert_eq!(config.disk_pending, 64); + // Queue/thread defaults are independent of the host's CPU mask. + // The integration test exercises actual default/custom affinity. + config.cpus.clear(); + config.validate().expect("defaults must be internally consistent"); + let threads = config.async_threads; + let runtime = CleanupRuntime::new(config).expect("runtime with many logical jobs"); + assert_eq!(runtime.runtime.as_ref().expect("live runtime").metrics().num_workers(), threads); + return; + } + for cpus in [None, Some("3-4"), Some("none")] { + let mut command = std::process::Command::new(std::env::current_exe().expect("test executable")); + command + .args([ + "--exact", + "disk::cleanup_runtime::tests::defaults_bound_queue_waves_and_separate_threads_from_jobs", + ]) + .env(CHILD, "1") + .env(WORKERS, "64") + .env(BUDGET, "1024") + .env(GC_WORKERS, "1") + .env_remove(ASYNC_THREADS) + .env_remove(BLOCKING_THREADS) + .env_remove(DISK_WORKERS) + .env_remove(DISK_PENDING) + .env_remove(CPUS) + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP"); + if let Some(cpus) = cpus { + command.env(CPUS, cpus); + } + let output = command.output().expect("run defaults test in a fresh process"); + assert!(output.status.success(), "{}", String::from_utf8_lossy(&output.stderr)); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + } + } + + #[tokio::test] + async fn affinity_failure_rejects_new_admission_without_dropping_admitted_work() { + let runtime = CleanupRuntime::new(config(2, 1, 1)).unwrap(); + let reservation = runtime.reserve().unwrap(); + runtime.affinity_failed.store(true, Ordering::Release); + assert!(runtime.reserve().is_err()); + assert_eq!(deadline(spawn(Some(reservation), async { 42 })).await.unwrap(), 42); + assert_eq!(runtime.admission.available_permits(), 2); + } + + #[tokio::test] + async fn admission_is_retained_by_every_commit_and_cleanup_owner() { + let runtime = CleanupRuntime::new(config(1, 1, 1)).unwrap(); + let request = runtime.reserve().unwrap(); + let commit = request.clone(); + let (release, released) = oneshot::channel(); + let cleanup = spawn(Some(commit.clone()), async move { + released.await.unwrap(); + }); + drop(request); + drop(commit); + assert!( + matches!(runtime.reserve(), Err(err) if err.kind() == io::ErrorKind::WouldBlock), + "queued and running cleanup must retain admission without waiting on external locks" + ); + release.send(()).unwrap(); + deadline(cleanup).await.unwrap(); + let next = runtime.reserve().unwrap(); + assert_eq!(runtime.admission.available_permits(), 0); + drop(next); + assert_eq!(runtime.admission.available_permits(), 1); + } + + #[tokio::test] + async fn cancelling_cleanup_waiter_does_not_cancel_work_or_release_budget() { + let runtime = CleanupRuntime::new(config(1, 1, 1)).unwrap(); + let reservation = runtime.reserve().unwrap(); + let (entered, entry) = oneshot::channel(); + let (release, released) = oneshot::channel(); + let waiter = tokio::spawn(run(Some(reservation), async move { + entered.send(()).unwrap(); + released.await.unwrap(); + })); + deadline(entry).await.unwrap(); + waiter.abort(); + assert!(waiter.await.unwrap_err().is_cancelled()); + assert_eq!(runtime.admission.available_permits(), 0); + assert_eq!(runtime.execution.available_permits(), 0); + release.send(()).unwrap(); + deadline(async { + loop { + if let Ok(reservation) = runtime.reserve() { + drop(reservation); + break; + } + tokio::task::yield_now().await; + } + }) + .await; + assert_eq!(runtime.admission.available_permits(), 1); + } + + #[tokio::test] + async fn worker_and_overflow_dispatch_share_one_execution_budget() { + let runtime = CleanupRuntime::new(config(8, 2, 1)).unwrap(); + let active = Arc::new(AtomicUsize::new(0)); + let peak = Arc::new(AtomicUsize::new(0)); + let gate = Arc::new(Semaphore::new(0)); + let (entered, mut entries) = tokio::sync::mpsc::unbounded_channel(); + let mut jobs = Vec::new(); + for index in 0..8 { + let reservation = runtime.reserve().unwrap(); + let active = active.clone(); + let peak = peak.clone(); + let gate = gate.clone(); + let entered = entered.clone(); + let work = async move { + let running = active.fetch_add(1, Ordering::SeqCst) + 1; + peak.fetch_max(running, Ordering::SeqCst); + entered.send(()).unwrap(); + gate.acquire().await.unwrap().forget(); + active.fetch_sub(1, Ordering::SeqCst); + }; + // Both a dequeued job and a queue-full continuation enter `run`; + // no-tail temporary cleanup uses direct owned dispatch. + jobs.push(if index % 2 == 0 { + tokio::spawn(run(Some(reservation), work)) + } else { + spawn(Some(reservation), work) + }); + } + deadline(entries.recv()).await.unwrap(); + deadline(entries.recv()).await.unwrap(); + assert_eq!(runtime.execution.available_permits(), 0); + assert!(entries.try_recv().is_err()); + assert_eq!(runtime.admission.available_permits(), 0); + gate.add_permits(8); + for job in jobs { + deadline(job).await.unwrap(); + } + assert_eq!(peak.load(Ordering::SeqCst), 2); + assert_eq!(runtime.admission.available_permits(), 8); + } + + #[tokio::test] + async fn cleanup_children_and_blocking_work_use_dedicated_threads() { + let runtime = CleanupRuntime::new(config(1, 1, 1)).unwrap(); + let task = spawn(Some(runtime.reserve().unwrap()), async move { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + tokio::spawn(async { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + tokio::task::spawn_blocking(|| { + assert_eq!(std::thread::current().name(), Some("rustfs-cleanup")); + }) + .await + .unwrap(); + }) + .await + .unwrap(); + }); + deadline(task).await.unwrap(); + assert_ne!(std::thread::current().name(), Some("rustfs-cleanup")); + } + + #[tokio::test] + async fn blocking_pool_saturation_does_not_block_foreground_pool() { + let runtime = CleanupRuntime::new(config(2, 2, 1)).unwrap(); + let (entered, entry) = oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let first = spawn(Some(runtime.reserve().unwrap()), async move { + tokio::task::spawn_blocking(move || { + entered.send(()).unwrap(); + released.recv_timeout(Duration::from_secs(20)).unwrap(); + }) + .await + .unwrap(); + }); + deadline(entry).await.unwrap(); + let (submitted, submission) = oneshot::channel(); + let (second_entered, mut second_entry) = oneshot::channel(); + let second = spawn(Some(runtime.reserve().unwrap()), async move { + let task = tokio::task::spawn_blocking(move || { + second_entered.send(()).unwrap(); + }); + submitted.send(()).unwrap(); + task.await.unwrap(); + }); + deadline(submission).await.unwrap(); + assert!(matches!(second_entry.try_recv(), Err(oneshot::error::TryRecvError::Empty))); + assert_eq!(deadline(tokio::task::spawn_blocking(|| 42)).await.unwrap(), 42); + release.send(()).unwrap(); + deadline(first).await.unwrap(); + deadline(second).await.unwrap(); + deadline(second_entry).await.unwrap(); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn affinity_covers_async_and_blocking_threads_without_changing_caller() { + let caller = rustix::thread::sched_getaffinity(None).unwrap(); + let cpu = (0..rustix::thread::CpuSet::MAX_CPU).find(|cpu| caller.is_set(*cpu)).unwrap(); + let mut config = config(1, 1, 1); + config.cpus = vec![cpu]; + let runtime = CleanupRuntime::new(config).unwrap(); + deadline(spawn(Some(runtime.reserve().unwrap()), async move { + let expected = { + let mut set = rustix::thread::CpuSet::new(); + set.set(cpu); + set + }; + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), expected); + let actual = tokio::task::spawn_blocking(|| rustix::thread::sched_getaffinity(None).unwrap()) + .await + .unwrap(); + assert_eq!(actual, expected); + })) + .await + .unwrap(); + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), caller); + assert!(validate_cpus(&[rustix::thread::CpuSet::MAX_CPU]).is_err()); + } + + #[cfg(not(target_os = "linux"))] + #[test] + fn explicit_cpu_affinity_is_rejected_on_unsupported_platforms() { + assert_eq!(validate_cpus(&[0]).unwrap_err().kind(), io::ErrorKind::Unsupported); + validate_cpus(&[]).unwrap(); + } +} diff --git a/crates/ecstore/src/disk/disk_store.rs b/crates/ecstore/src/disk/disk_store.rs index 01091325b..6ae2975b8 100644 --- a/crates/ecstore/src/disk/disk_store.rs +++ b/crates/ecstore/src/disk/disk_store.rs @@ -12,14 +12,15 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::disk::cleanup_runtime; use crate::disk::io_admission::{ AdmissionReader, BoxedChunkReader, DiskIoAdmission, DiskIoClass, DiskIoContext, current_disk_io_context, with_disk_io_context, with_disk_io_permit, }; use crate::disk::{ CheckPartsResp, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskError, DiskInfo, DiskInfoOptions, DiskLocation, Endpoint, - Error, FileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, Result, - SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, + Error, FileInfoVersions, GuardedFileInfoVersions, MmapCopyStageMetrics, ReadMultipleReq, ReadMultipleResp, ReadOptions, + RenameDataResp, Result, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, health_state::{ RuntimeDriveHealthState, classify_drive_recovery, get_drive_returning_probe_interval, get_drive_returning_success_threshold, get_drive_suspect_failure_threshold, record_drive_offline_duration, @@ -484,10 +485,10 @@ impl LocalDiskWrapper { opts: DeleteOptions, namespace_owner: Option>, ) -> Result<()> { - self.track_disk_health_mutation( + self.track_cleanup_mutation( "delete", DiskMetricMutation::Delete, - || async { Box::pin(self.disk.delete_with_namespace_owner(volume, path, opts, namespace_owner)).await }, + || async { Box::pin(self.cleanup_delete(volume, path, opts, namespace_owner)).await }, get_max_timeout_duration(), ) .await @@ -1475,6 +1476,28 @@ impl LocalDiskWrapper { ) } + async fn cleanup_delete( + &self, + volume: &str, + path: &str, + options: DeleteOptions, + owner: Option>, + ) -> Result<()> { + if !cleanup_runtime::enabled()? { + return self.disk.delete_with_namespace_owner(volume, path, options, owner).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + let guarded = owner.is_some(); + let operation = async move { disk.delete_with_namespace_owner(&volume, &path, options, owner).await }; + if guarded { + cleanup_runtime::guarded_disk_operation(operation).await + } else { + cleanup_runtime::disk_operation(operation).await + } + } + /// Run a delete under an owned coordinator task when a publication guard /// is present. This keeps the guard alive if the RPC waiter is cancelled /// while the local namespace mutation is still in progress. @@ -1493,12 +1516,13 @@ impl LocalDiskWrapper { } else { get_max_timeout_duration() }; + let cleanup_owner = external_guard.clone(); run_owned_mutation(external_guard, move || async move { operation - .track_disk_health_mutation( + .track_cleanup_mutation( "delete", DiskMetricMutation::Delete, - || async { operation.disk.delete(&volume, &path, options).await }, + || async { operation.cleanup_delete(&volume, &path, options, cleanup_owner).await }, timeout_duration, ) .await @@ -1805,21 +1829,7 @@ impl LocalDiskWrapper { } // Clean up - disk.delete( - test_bucket, - test_filename, - DeleteOptions { - recursive: false, - immediate: false, - undo_write: false, - undo_delete: false, - old_data_dir: None, - expected_delete_marker: None, - recoverable_delete: false, - delete_recovery_bucket_incarnation: None, - }, - ) - .await?; + disk.delete(test_bucket, test_filename, DeleteOptions::default()).await?; Ok(()) }) @@ -2054,6 +2064,58 @@ impl LocalDiskWrapper { result } + pub(crate) async fn delete_versions_guarded_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + let version_count = versions.len(); + self.metrics.record_operation_call("delete_versions"); + // Check if disk is faulty before proceeding + if self.health.is_faulty() { + self.metrics.record_availability_error(); + return vec![Some(DiskError::FaultyDisk); versions.len()]; + } + + // Check if disk is stale + if let Err(e) = self.check_disk_stale().await { + self.metrics.record_availability_error(); + return vec![Some(e); versions.len()]; + } + + let _io_permit = match self.acquire_disk_io_permit().await { + Ok(permit) => permit, + Err(error) => return vec![Some(error); version_count], + }; + + // Record operation start + self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed); + let waiting_guard = self.health.waiting_guard(); + let metric_waiting_guard = self.metrics.waiting_guard(); + let started = Instant::now(); + + // Execute the operation + let result = self + .disk + .delete_versions_guarded_with_namespace_owner(volume, versions, opts, namespace_owner) + .await; + self.metrics.record_operation_latency("delete_versions", started.elapsed()); + self.record_batch_delete_error_metrics(&result); + + drop(waiting_guard); + drop(metric_waiting_guard); + let has_err = result.iter().any(|e| e.is_some()); + if !has_err { + // Log success and decrement waiting counter + self.health.record_operation_success(&self.endpoint(), "operation_success"); + self.metrics.record_mutation_success(DiskMetricMutation::Delete); + } + + result + } + pub async fn get_current_disk_id(&self) -> Option { *self.disk_id.read().await } @@ -2092,6 +2154,31 @@ impl LocalDiskWrapper { self.track_disk_health_with_op("unknown", operation, timeout_duration).await } + /// Cleanup latency includes waiting behind a node-wide execution budget, + /// so a waiter deadline alone is not evidence that this disk is faulty. + /// The active disk health probe retains its original timeout policy. + async fn track_cleanup_mutation( + &self, + op: &'static str, + mutation: DiskMetricMutation, + operation: F, + timeout_duration: Duration, + ) -> Result + where + F: FnOnce() -> Fut, + Fut: std::future::Future>, + { + // Runtime initialization must not prevent control-plane release of + // reader tokens. Disk I/O entries still propagate configuration errors. + let action = if cleanup_runtime::enabled().unwrap_or(false) { + TimeoutHealthAction::IgnoreFailure + } else { + TimeoutHealthAction::MarkFailure + }; + self.track_disk_health_with_op_timeout_action_and_mutation(op, operation, timeout_duration, action, mutation) + .await + } + async fn track_disk_health_mutation( &self, op: &'static str, @@ -2272,6 +2359,21 @@ impl LocalDiskWrapper { #[async_trait::async_trait] impl DiskAPI for LocalDiskWrapper { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + // The local implementation shares this binary's capability. Availability + // and disk identity remain mutation checks and write-quorum inputs. + self.disk.supports_guarded_delete_versions().await + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + self.delete_versions_guarded_with_namespace_owner(volume, versions, opts, None) + .await + } + async fn delete_version_recoverable( &self, volume: &str, @@ -2302,6 +2404,14 @@ impl DiskAPI for LocalDiskWrapper { opts.recoverable_delete = true; self.delete_versions(volume, versions, opts).await } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + self.track_disk_health_with_op( + "has_delete_transaction_evidence", + || async { self.disk.has_delete_transaction_evidence(volume, object).await }, + get_max_timeout_duration(), + ) + .await + } async fn discover_delete_transactions( &self, volume: &str, @@ -2589,9 +2699,37 @@ impl DiskAPI for LocalDiskWrapper { } async fn release_snapshot_lease(&self, volume: &str, path: &str, token: SnapshotLeaseToken) -> Result<()> { - self.track_disk_health_with_op( + // Quota fence release is control-plane work and may wait for in-flight + // mutations. Keep its original waiter lifecycle instead of creating + // detached cleanup tasks for repeated fence-release RPCs. + if volume == super::RUSTFS_META_BUCKET && super::is_quota_mutation_fence_path(path) { + return self + .track_disk_health_with_op( + "release_snapshot_lease", + || async { self.disk.release_snapshot_lease(volume, path, token).await }, + get_max_timeout_duration(), + ) + .await; + } + self.track_cleanup_mutation( "release_snapshot_lease", - || async { self.disk.release_snapshot_lease(volume, path, token).await }, + DiskMetricMutation::None, + || async { + if self.disk.release_snapshot_lease_without_cleanup(volume, path, token).await { + return Ok(()); + } + if !cleanup_runtime::enabled().unwrap_or(false) { + // The raw release removes the token before trying the + // reclaim budget. Initialization failure therefore keeps + // the cleanup intent and data, without leaking a reader + // or falling back to unbudgeted filesystem deletion. + return self.disk.release_snapshot_lease(volume, path, token).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + cleanup_runtime::release_lease(async move { disk.release_snapshot_lease(&volume, &path, token).await }).await + }, get_max_timeout_duration(), ) .await @@ -2616,10 +2754,22 @@ impl DiskAPI for LocalDiskWrapper { } else { get_max_timeout_duration() }; - self.track_disk_health_mutation( + self.track_cleanup_mutation( "delete_data_dir", DiskMetricMutation::Delete, - || async { self.disk.delete_data_dir(volume, path, opts).await }, + || async { + if !cleanup_runtime::enabled()? { + return self.disk.delete_data_dir(volume, path, opts).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + cleanup_runtime::disk_operation(async move { + let _scope = scope; + disk.delete_data_dir(&volume, &path, opts).await + }) + .await + }, timeout, ) .await @@ -2654,6 +2804,7 @@ impl DiskAPI for LocalDiskWrapper { .await } + #[track_caller] async fn read_version( &self, org_volume: &str, @@ -2662,7 +2813,7 @@ impl DiskAPI for LocalDiskWrapper { version_id: &str, opts: &ReadOptions, ) -> Result { - self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit( + let operation = self.track_disk_health_with_op_timeout_action_and_mutation_with_io_permit( "read_version", |io_permit| async move { with_disk_io_permit(io_permit, self.disk.read_version(org_volume, volume, path, version_id, opts)).await @@ -2670,8 +2821,20 @@ impl DiskAPI for LocalDiskWrapper { get_max_timeout_duration(), TimeoutHealthAction::MarkFailure, DiskMetricMutation::None, - ) - .await + ); + #[cfg(feature = "diagnostic-read-version-caller")] + { + if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_some() { + operation.await + } else { + let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location( + std::panic::Location::caller().file(), + ); + crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await + } + } + #[cfg(not(feature = "diagnostic-read-version-caller"))] + operation.await } async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result { @@ -2866,10 +3029,10 @@ impl DiskAPI for LocalDiskWrapper { } async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()> { - self.track_disk_health_mutation( + self.track_cleanup_mutation( "delete", DiskMetricMutation::Delete, - || async { self.disk.delete(volume, path, opt).await }, + || async { self.cleanup_delete(volume, path, opt, None).await }, get_max_timeout_duration(), ) .await @@ -2902,11 +3065,25 @@ impl DiskAPI for LocalDiskWrapper { } async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()> { - self.track_disk_health_mutation( + let cleanup = path.rsplit('/').next() == Some(cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE); + self.track_disk_health_with_op_timeout_action_and_mutation( "write_all", - DiskMetricMutation::Write, - || async { self.disk.write_all(volume, path, data).await }, + || async { + if !cleanup || !cleanup_runtime::enabled()? { + return self.disk.write_all(volume, path, data).await; + } + let disk = self.disk.clone(); + let volume = volume.to_owned(); + let path = path.to_owned(); + cleanup_runtime::disk_operation(async move { disk.write_all(&volume, &path, data).await }).await + }, get_max_timeout_duration(), + if cleanup && cleanup_runtime::enabled()? { + TimeoutHealthAction::IgnoreFailure + } else { + TimeoutHealthAction::MarkFailure + }, + DiskMetricMutation::Write, ) .await } @@ -3299,6 +3476,256 @@ mod tests { let _ = walk.await; } + #[tokio::test] + async fn invalid_cleanup_config_does_not_leak_snapshot_readers() { + const CHILD: &str = "RUSTFS_CLEANUP_INVALID_LEASE_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().expect("test executable")) + .args([ + "--exact", + "disk::disk_store::tests::invalid_cleanup_config_does_not_leak_snapshot_readers", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_CLEANUP_ASYNC_THREADS", "0") + .output() + .expect("run with invalid cleanup configuration") + }) + .await + .expect("child process task"); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + let dir = tempfile::tempdir().expect("temporary disk"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 path")).expect("endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + let wrapper = LocalDiskWrapper::new(disk.clone(), false); + let volume = "invalid-cleanup-config"; + let path = "object/data"; + wrapper.make_volume(volume).await.expect("bucket"); + wrapper + .write_all(volume, "object/data/part.1", Bytes::from_static(b"live shard")) + .await + .expect("write shard"); + let token = wrapper.acquire_snapshot_lease(volume, path).await.expect("GET lease"); + wrapper + .release_snapshot_lease(volume, path, token) + .await + .expect("control-only release must not depend on cleanup initialization"); + assert!(matches!( + wrapper.renew_snapshot_lease(volume, path, token).await, + Err(DiskError::FileNotFound) + )); + + let token = wrapper + .acquire_snapshot_lease(volume, path) + .await + .expect("reader of old data"); + // Register reclaim intent directly; this only updates the lease + // registry while a reader is present and performs no filesystem I/O. + assert_eq!( + disk.delete_data_dir( + volume, + path, + DeleteOptions { + recursive: true, + ..Default::default() + } + ) + .await + .expect("defer reclaim"), + DataDirDeleteStatus::Deferred + ); + assert!( + wrapper.release_snapshot_lease(volume, path, token).await.is_err(), + "invalid configuration must still reject reclamation" + ); + assert!( + matches!(wrapper.renew_snapshot_lease(volume, path, token).await, Err(DiskError::FileNotFound)), + "failed reclamation must not strand a reader token" + ); + assert_eq!( + wrapper.read_all(volume, "object/data/part.1").await.expect("retained data"), + Bytes::from_static(b"live shard"), + "configuration failure must not fall back to unbudgeted deletion" + ); + } + + #[tokio::test] + async fn rpc_cleanup_entries_share_disk_budget() { + const CHILD: &str = "RUSTFS_CLEANUP_RPC_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "disk::disk_store::tests::rpc_cleanup_entries_share_disk_budget", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_CLEANUP_ASYNC_THREADS", "1") + .env("RUSTFS_CLEANUP_DISK_MAX_PENDING", "1") + .env("RUSTFS_CLEANUP_DISK_WORKERS", "1") + .env("RUSTFS_CLEANUP_GC_WORKERS", "1") + .env("RUSTFS_CLEANUP_CPUS", "none") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .output() + .unwrap() + }) + .await + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + let dir = tempfile::tempdir().unwrap(); + let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap(); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.unwrap()); + let wrapper = LocalDiskWrapper::new(disk, false); + let volume = "cleanup-receiver"; + let path = "object/old-data"; + let body_path = format!("{path}/part.1"); + let receipt_path = format!("{path}/{}", cleanup_runtime::OLD_DATA_CLEANUP_RECEIPT_FILE); + wrapper.make_volume(volume).await.unwrap(); + wrapper + .write_all(volume, &body_path, Bytes::from_static(b"old body")) + .await + .unwrap(); + let read_lease = wrapper + .acquire_snapshot_lease(volume, path) + .await + .expect("ordinary GET lease"); + let (entered, entry) = tokio::sync::oneshot::channel(); + let (release, released) = std::sync::mpsc::channel(); + let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move { + entered.send(()).unwrap(); + // Deliberately stall the only cleanup scheduler to prove that + // ordinary GET completion does not dispatch to it. + released + .recv_timeout(Duration::from_secs(30)) + .expect("release stalled scheduler"); + Ok(()) + })); + tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap(); + tokio::time::timeout(Duration::from_secs(5), wrapper.release_snapshot_lease(volume, path, read_lease)) + .await + .expect("ordinary GET release must progress while the cleanup scheduler is stalled") + .expect("release without reclamation"); + let fence_path = super::super::quota_mutation_fence_path(volume, "object"); + let fence = wrapper + .acquire_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path) + .await + .unwrap(); + wrapper + .release_snapshot_lease(super::super::RUSTFS_META_BUCKET, &fence_path, fence) + .await + .unwrap(); + let timeout = wrapper + .track_cleanup_mutation( + "delete", + DiskMetricMutation::Delete, + std::future::pending::>, + Duration::from_millis(10), + ) + .await; + assert!(matches!(timeout, Err(DiskError::Timeout))); + assert!(!wrapper.health.is_faulty(), "cleanup queue delay must not mark a healthy disk faulty"); + let options = DeleteOptions { + recursive: true, + ..Default::default() + }; + let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + // This is the same entry point used after RPC authentication and + // scanner publication-lease validation by handle_delete. + let result = wrapper + .delete_with_publication_guard(volume, path, options.clone(), Some(Arc::new(DropProbe(drops.clone())))) + .await; + assert!(matches!(result, Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert!(matches!(wrapper.delete_data_dir(volume, path, options.clone()).await, + Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert!(matches!(wrapper.write_all(volume, &receipt_path, Bytes::from_static(b"receipt")).await, + Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"old body")); + release.send(()).unwrap(); + occupied.await.unwrap().unwrap(); + wrapper + .write_all(volume, &receipt_path, Bytes::from_static(b"receipt")) + .await + .unwrap(); + wrapper + .delete_with_publication_guard(volume, path, options, None) + .await + .unwrap(); + assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound))); + // Snapshot readers defer old-data cleanup. Saturation must release + // their token while retaining reclaim intent, rather than leaking a + // live token or performing an unbudgeted delete on the RPC runtime. + wrapper + .write_all(volume, &body_path, Bytes::from_static(b"leased body")) + .await + .unwrap(); + let lease = wrapper.acquire_snapshot_lease(volume, path).await.unwrap(); + assert_eq!( + wrapper + .delete_data_dir( + volume, + path, + DeleteOptions { + recursive: true, + ..Default::default() + } + ) + .await + .unwrap(), + DataDirDeleteStatus::Deferred + ); + let (entered, entry) = tokio::sync::oneshot::channel(); + let (release, released) = tokio::sync::oneshot::channel(); + let occupied = tokio::spawn(cleanup_runtime::disk_operation(async move { + entered.send(()).unwrap(); + released.await.unwrap(); + Ok(()) + })); + tokio::time::timeout(Duration::from_secs(20), entry).await.unwrap().unwrap(); + assert!(matches!(wrapper.release_snapshot_lease(volume, path, lease).await, + Err(DiskError::Io(err)) if err.kind() == io::ErrorKind::WouldBlock)); + assert_eq!(wrapper.read_all(volume, &body_path).await.unwrap(), Bytes::from_static(b"leased body")); + release.send(()).unwrap(); + occupied.await.unwrap().unwrap(); + assert_eq!( + wrapper + .delete_data_dir( + volume, + path, + DeleteOptions { + recursive: true, + ..Default::default() + } + ) + .await + .unwrap(), + DataDirDeleteStatus::Deleted + ); + assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound))); + } + #[tokio::test] async fn owned_mutation_keeps_publication_guard_after_waiter_cancellation() { let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -3568,6 +3995,97 @@ mod tests { assert_eq!(snapshot.total_errors_availability, 1); } + #[tokio::test] + async fn local_guarded_delete_capability_is_independent_of_faulty_or_stale_mutation_rejection() { + for stale in [false, true] { + let dir = tempfile::tempdir().expect("create local disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse local endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("create local disk")); + let volume = "guarded-capability"; + let object = "object"; + disk.make_volume(volume).await.expect("create object volume"); + let mut info = FileInfo::new(object, 1, 0); + info.name = object.to_owned(); + info.version_id = Some(Uuid::nil()); + info.data_dir = Some(Uuid::new_v4()); + info.mod_time = Some(::time::OffsetDateTime::now_utc()); + info.erasure.index = 1; + info.size = 8; + info.parts = vec![ObjectPartInfo { + number: 1, + size: 8, + actual_size: 8, + ..Default::default() + }]; + info.data = Some(Bytes::from_static(b"original")); + info.metadata.insert("etag".into(), "etag-a".into()); + info.set_inline_data(); + disk.write_metadata(volume, volume, object, info.clone()) + .await + .expect("publish guarded fixture metadata"); + let generation = crate::disk::CurrentDeleteGeneration::from_file_info(&info).expect("capture object generation"); + let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false); + let expected_error = if stale { + let mut format_info = disk.format_info.write().await; + format_info.id = Some(Uuid::new_v4()); + format_info.file_info = Some(tokio::fs::metadata(dir.path()).await.expect("local disk directory metadata")); + format_info.last_check = Some(::time::OffsetDateTime::now_utc()); + drop(format_info); + wrapper.set_disk_id_state(Some(Uuid::new_v4())).await; + DiskError::DiskNotFound + } else { + wrapper.force_offline_for_test(); + DiskError::FaultyDisk + }; + let object_dir = dir.path().join(volume).join(object); + let metadata_path = object_dir.join(crate::disk::STORAGE_FORMAT_FILE); + let before = tokio::fs::read(&metadata_path).await.expect("read original metadata"); + wrapper + .supports_guarded_delete_versions() + .await + .expect("local binary capability is known even when the drive is unavailable"); + assert_eq!( + wrapper.metrics_snapshot().total_errors_availability, + 0, + "capability does not probe drive availability" + ); + let transaction = Uuid::new_v4(); + let result = wrapper + .delete_versions_guarded_recoverable( + volume, + vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: object.to_owned(), + versions: vec![FileInfo { + name: object.to_owned(), + version_id: info.version_id, + ..Default::default() + }], + ..Default::default() + }, + expected_current: generation, + }], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!(result, vec![Some(expected_error)], "mutation availability checks remain enforced"); + assert_eq!( + tokio::fs::read(&metadata_path) + .await + .expect("metadata survives rejected mutation"), + before + ); + assert!( + !object_dir.join(transaction.to_string()).exists(), + "rejected mutation creates no recovery record" + ); + assert_eq!(wrapper.metrics_snapshot().total_errors_availability, 1); + } + } + #[tokio::test] async fn local_disk_health_wrapper_counts_faulty_precheck_rejections() { let dir = tempfile::tempdir().expect("temp dir should be created"); diff --git a/crates/ecstore/src/disk/error.rs b/crates/ecstore/src/disk/error.rs index 9c31377c0..ade7c9dbd 100644 --- a/crates/ecstore/src/disk/error.rs +++ b/crates/ecstore/src/disk/error.rs @@ -167,6 +167,12 @@ pub enum DiskError { #[error("method not allowed")] MethodNotAllowed, + #[error("conditional delete object generation changed")] + ConditionalDeleteConflict, + + #[error("guarded batch deletion is not supported by this disk")] + GuardedDeleteUnsupported, + #[error("erasure write quorum")] ErasureWriteQuorum, @@ -762,6 +768,8 @@ impl Clone for DiskError { DiskError::PartMissingOrCorrupt => DiskError::PartMissingOrCorrupt, DiskError::NoHealRequired => DiskError::NoHealRequired, DiskError::MethodNotAllowed => DiskError::MethodNotAllowed, + DiskError::ConditionalDeleteConflict => DiskError::ConditionalDeleteConflict, + DiskError::GuardedDeleteUnsupported => DiskError::GuardedDeleteUnsupported, DiskError::ErasureWriteQuorum => DiskError::ErasureWriteQuorum, DiskError::ErasureReadQuorum => DiskError::ErasureReadQuorum, DiskError::ShortWrite => DiskError::ShortWrite, @@ -819,6 +827,8 @@ impl DiskError { DiskError::Timeout => 0x29, DiskError::InvalidPath => 0x2A, DiskError::RemoteClientUnavailable(_) => 0x2B, + DiskError::ConditionalDeleteConflict => 0x2C, + DiskError::GuardedDeleteUnsupported => 0x2D, } } @@ -867,6 +877,8 @@ impl DiskError { 0x29 => Some(DiskError::Timeout), 0x2A => Some(DiskError::InvalidPath), 0x2B => Some(DiskError::RemoteClientUnavailable(String::new())), + 0x2C => Some(DiskError::ConditionalDeleteConflict), + 0x2D => Some(DiskError::GuardedDeleteUnsupported), _ => None, } } @@ -1645,6 +1657,21 @@ mod tests { assert!(rendered.contains("transport error: connection refused")); } + #[test] + fn guarded_delete_errors_preserve_wire_and_io_types() { + for (error, code) in [ + (DiskError::ConditionalDeleteConflict, 0x2C), + (DiskError::GuardedDeleteUnsupported, 0x2D), + ] { + assert_eq!(error.to_u32(), code); + assert_eq!(DiskError::from_u32(code), Some(error.clone())); + let wire: rustfs_protos::proto_gen::node_service::Error = error.clone().into(); + assert_eq!(DiskError::from(wire), error); + let io_error: std::io::Error = error.clone().into(); + assert_eq!(DiskError::from(io_error), error); + } + } + #[test] fn remote_client_unavailable_wire_roundtrip_keeps_variant_and_detail() { let original = DiskError::RemoteClientUnavailable("dial tcp: connection refused".to_string()); diff --git a/crates/ecstore/src/disk/fs.rs b/crates/ecstore/src/disk/fs.rs index 7dbbaf14d..15a52ef7b 100644 --- a/crates/ecstore/src/disk/fs.rs +++ b/crates/ecstore/src/disk/fs.rs @@ -223,12 +223,6 @@ pub fn rename_std(from: impl AsRef, to: impl AsRef) -> io::Result<() std::fs::rename(from, to) } -#[cfg(any(not(windows), test))] -#[tracing::instrument(level = "debug", skip_all)] -pub async fn read_file(path: impl AsRef) -> io::Result> { - fs::read(path.as_ref()).await -} - // Bucket existence cache - reduces statx syscalls for repeated bucket checks /// Cache for bucket directory existence checks. @@ -614,28 +608,6 @@ mod tests { assert_eq!(content, "test content"); } - #[tokio::test] - async fn test_read_file() { - let temp_dir = TempDir::new().unwrap(); - let file_path = temp_dir.path().join("test_read.txt"); - - let test_content = b"This is test content for reading"; - tokio::fs::write(&file_path, test_content).await.unwrap(); - - // Read file - let read_content = read_file(&file_path).await.unwrap(); - assert_eq!(read_content, test_content); - } - - #[tokio::test] - async fn test_read_file_nonexistent() { - let temp_dir = TempDir::new().unwrap(); - let file_path = temp_dir.path().join("nonexistent.txt"); - - // Should fail for non-existent file - assert!(read_file(&file_path).await.is_err()); - } - #[tokio::test] async fn test_same_file() { let temp_dir = TempDir::new().unwrap(); diff --git a/crates/ecstore/src/disk/lifecycle_test_io.rs b/crates/ecstore/src/disk/lifecycle_test_io.rs new file mode 100644 index 000000000..e331b8869 --- /dev/null +++ b/crates/ecstore/src/disk/lifecycle_test_io.rs @@ -0,0 +1,72 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bucket-scoped observations distinguish exact metadata reads from walks. +use std::collections::HashMap; +use std::sync::{Arc, Mutex, OnceLock}; + +#[derive(Default, Clone)] +pub(crate) struct Counts { + pub walks: usize, + pub reads: Vec, +} + +fn observers() -> &'static Mutex>>> { + static OBSERVERS: OnceLock>>>> = OnceLock::new(); + OBSERVERS.get_or_init(Mutex::default) +} + +pub(crate) struct Observer { + bucket: String, + counts: Arc>, +} + +impl Observer { + pub fn new(bucket: &str) -> Self { + let counts = Arc::new(Mutex::new(Counts::default())); + assert!( + observers() + .lock() + .expect("I/O observers lock") + .insert(bucket.to_owned(), counts.clone()) + .is_none(), + "each test must use its own bucket" + ); + Self { + bucket: bucket.to_owned(), + counts, + } + } + + pub fn snapshot(&self) -> Counts { + self.counts.lock().expect("I/O counts lock").clone() + } +} + +impl Drop for Observer { + fn drop(&mut self) { + observers().lock().expect("I/O observers lock").remove(&self.bucket); + } +} + +pub(crate) fn observe(bucket: &str, object: Option<&str>) { + let observer = observers().lock().expect("I/O observers lock").get(bucket).cloned(); + if let Some(observer) = observer { + let mut counts = observer.lock().expect("I/O counts lock"); + match object { + Some(object) => counts.reads.push(object.to_owned()), + None => counts.walks += 1, + } + } +} diff --git a/crates/ecstore/src/disk/local.rs b/crates/ecstore/src/disk/local.rs index 66cc3e10b..82d21b39d 100644 --- a/crates/ecstore/src/disk/local.rs +++ b/crates/ecstore/src/disk/local.rs @@ -38,6 +38,7 @@ mod uring_read_chunks; #[path = "uring_result_budget.rs"] mod uring_result_budget; pub use replacement_lease::ReplacementExecutionLease; +pub(crate) use replacement_lease::is_blank_target as is_blank_replacement_target; use crate::crash_inject::{self, CrashPoint}; use crate::data_usage::local_snapshot::ensure_data_usage_layout; @@ -52,12 +53,12 @@ use crate::disk::disk_store::{get_drive_walkdir_stall_timeout, get_object_disk_r use crate::disk::io_admission::{DiskIoPermit, spawn_blocking_with_disk_io_permit}; use crate::disk::{ BUCKET_META_PREFIX, CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN, - CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, ConditionalFileUpdate, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskInfo, - DiskInfoOptions, DiskLocation, DiskMetrics, FileInfoVersions, FileReader, FileWriter, MmapCopyStageMetrics, OldCurrentSize, - PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, PART_TRANSACTION_ROLLBACK, PartTransactionAction, RUSTFS_META_BUCKET, - RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, - STORAGE_FORMAT_FILE, STORAGE_FORMAT_FILE_BACKUP, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, - acquire_context_disk_io_permit, conv_part_err_to_int, + CHECK_PART_VOLUME_NOT_FOUND, CheckPartsResp, ConditionalFileUpdate, CurrentDeleteGeneration, DataDirDeleteStatus, + DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskMetrics, FileInfoVersions, FileReader, FileWriter, + GuardedFileInfoVersions, MmapCopyStageMetrics, OldCurrentSize, PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META, + PART_TRANSACTION_ROLLBACK, PartTransactionAction, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, RUSTFS_META_TMP_DELETED_BUCKET, + ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, STORAGE_FORMAT_FILE_BACKUP, + SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, acquire_context_disk_io_permit, conv_part_err_to_int, endpoint::Endpoint, error::{DiskError, Error, FileAccessDeniedWithContext, Result}, error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error}, @@ -174,6 +175,24 @@ fn delete_recovery_part_file(entry: &str) -> bool { }) } +/// Classify the entire leaf before unlinking anything. UUID-shaped S3 prefixes +/// and transaction witnesses share this namespace with erasure data dirs. +fn orphan_data_dir_parts(path: &Path) -> std::io::Result> { + let protected = || std::io::Error::new(ErrorKind::InvalidData, "orphan data directory contains protected entries"); + if !std::fs::symlink_metadata(path)?.file_type().is_dir() { + return Err(protected()); + } + let mut parts = Vec::new(); + for entry in std::fs::read_dir(path)? { + let entry = entry?; + if !entry.file_type()?.is_file() || !entry.file_name().to_str().is_some_and(delete_recovery_part_file) { + return Err(protected()); + } + parts.push(entry.path()); + } + Ok(parts) +} + fn delete_transaction_marker_id(entry: &str, prefix: &str) -> Option { entry .strip_prefix(prefix) @@ -221,8 +240,10 @@ struct ListingMetadataRead { has_namespace_child_candidate: bool, } -fn read_all_data_std(path: &Path) -> core::result::Result<(Vec, Option), ReadAllError> { - let mut file = std::fs::File::open(path).map_err(ReadAllError::Open)?; +fn read_all_data_std( + opened: std::io::Result, +) -> core::result::Result<(Vec, Option), ReadAllError> { + let mut file = opened.map_err(ReadAllError::Open)?; let metadata = file.metadata().map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?; if metadata.is_dir() { @@ -315,7 +336,7 @@ async fn write_delete_rollback_file( volume: &str, object_dir: &Path, rollback_dir: Uuid, - name: &str, + name: &'static str, data: &[u8], namespace_owner: Option>, ) -> Result<()> { @@ -333,17 +354,33 @@ async fn write_delete_rollback_file( std::fs::create_dir_all(&staging_root)?; #[cfg(test)] run_owned_file_write_before_open(&path); + // tempfile's Windows `persist` hands both the staged and the final path + // straight to `MoveFileExW`, bypassing std's long-path handling, so a + // deep object key failed every delete at MAX_PATH (rustfs#8545). + let staging_root = rustfs_utils::extended_length_path(&staging_root)?; + let destination = rustfs_utils::extended_length_path(&path)?; // Private directory staging is reclaimed by existing tmp cleanup after // a crash and cannot leave a regular artifact in the user's bucket. let staging_dir = tempfile::Builder::new() .prefix(".delete-rollback-") .tempdir_in(&staging_root)?; - let mut staged = tempfile::NamedTempFile::new_in(staging_dir.path())?; + let staged_path = staging_dir.path().join(name); + let mut staged_options = std::fs::OpenOptions::new(); + staged_options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + staged_options.mode(0o600); + } + let mut staged = staged_options.open(&staged_path)?; std::io::Write::write_all(&mut staged, &data)?; if sync { - staged.as_file().sync_all()?; + staged.sync_all()?; } - staged.persist(path).map_err(|err| err.error)?; + // Close the writer before publishing via std::fs::rename, which applies + // verbatim long-path normalization and POSIX-semantics replacement on Windows. + drop(staged); + std::fs::rename(&staged_path, &destination)?; if sync { os::fsync_dir_std(&backup_dir)?; if let Some(parent) = backup_dir.parent() { @@ -5353,7 +5390,7 @@ pub struct LocalDisk { /// pathname is later covered by another mount. io_root: PathBuf, #[cfg(target_os = "linux")] - mount_lease: std::fs::File, + mount_lease: Arc, #[cfg(target_os = "linux")] mount_lease_mount_id: Option, /// Public path for callers that need the configured disk layout. Internal @@ -5610,24 +5647,6 @@ fn mount_id_from_mountinfo_contents(mountinfo: &str, path: &Path) -> Option } impl LocalDisk { - #[cfg(target_os = "linux")] - fn open_mount_lease(root: &Path) -> Result<(std::fs::File, PathBuf, Option)> { - use rustix::fs::{Mode, OFlags, open}; - use std::os::fd::AsRawFd as _; - - let fd = open( - root, - OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, - Mode::empty(), - ) - .map_err(std::io::Error::from) - .map_err(DiskError::from)?; - let lease = std::fs::File::from(fd); - let io_root = PathBuf::from(format!("/proc/self/fd/{}/.", lease.as_raw_fd())); - let mount_id = mount_id_for_fd(&lease); - Ok((lease, io_root, mount_id)) - } - #[cfg(not(target_os = "linux"))] fn open_mount_lease(root: &Path) -> Result { Ok(root.to_path_buf()) @@ -5711,7 +5730,11 @@ impl LocalDisk { let root = publication_root.path().to_path_buf(); #[cfg(target_os = "linux")] - let (mount_lease, io_root, mount_lease_mount_id) = Self::open_mount_lease(&root)?; + let (mount_lease, io_root, mount_lease_mount_id) = ( + publication_root.directory().clone(), + publication_root.io_path().to_path_buf(), + mount_id_for_fd(publication_root.directory()), + ); #[cfg(not(target_os = "linux"))] let io_root = Self::open_mount_lease(&root)?; @@ -5759,9 +5782,11 @@ impl LocalDisk { state = "format_path_resolved", "Local disk format path resolved" ); - let (format_data, format_meta) = read_file_exists(&io_format_path).await.inspect_err(|err| { - log_startup_disk_error("read_format_json", &io_format_path, err); - })?; + let (format_data, format_meta) = read_file_exists(&io_format_path, &publication_root) + .await + .inspect_err(|err| { + log_startup_disk_error("read_format_json", &io_format_path, err); + })?; let mut id = None; // let mut format_legacy = false; @@ -5794,8 +5819,9 @@ impl LocalDisk { Box::pin(async move { match get_disk_info(root.clone()).await { Ok((info, is_root_disk)) => { - let physical_device_ids = match rustfs_utils::os::get_physical_device_ids(root.to_string_lossy().as_ref()) - { + let physical_device_ids = match win32_volume_path(&root).and_then(|volume_path| { + rustfs_utils::os::get_physical_device_ids(volume_path.to_string_lossy().as_ref()) + }) { Ok(ids) => ids, Err(err) => { warn!( @@ -5907,7 +5933,13 @@ impl LocalDisk { let io_root = disk.io_root.clone(); let publication_root = disk.publication_root.clone(); - tokio::spawn(Self::cleanup_deleted_objects_loop(io_root, publication_root, exit_rx)); + tokio::spawn(Self::cleanup_deleted_objects_loop( + io_root, + publication_root, + #[cfg(target_os = "linux")] + disk.mount_lease.clone(), + exit_rx, + )); debug!( event = EVENT_DISK_LOCAL_STARTUP_CLEANUP, component = LOG_COMPONENT_ECSTORE, @@ -5923,30 +5955,53 @@ impl LocalDisk { async fn cleanup_deleted_objects_loop( root: PathBuf, publication_root: os::PublicationRoot, + #[cfg(target_os = "linux")] mount_lease: Arc, mut exit_rx: tokio::sync::broadcast::Receiver<()>, ) { let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL; let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); loop { tokio::select! { _ = interval.tick() => { - if let Err(err) = Self::cleanup_deleted_objects(root.clone()).await { + let root = root.clone(); + let publication_root = publication_root.clone(); + #[cfg(target_os = "linux")] + let mount_lease = mount_lease.clone(); + if let Err(err) = super::cleanup_runtime::run_gc(move |budget| async move { + // Queued/detached scans must keep /proc/self/fd paths + // attached to the original mount after LocalDisk drops. + #[cfg(target_os = "linux")] + let _mount_lease = mount_lease; + if let Err(err) = Self::cleanup_deleted_objects(root.clone(), &budget).await { + error!( + event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK_LOCAL, + task = "deleted_objects", + state = "failed", + error = ?err, + "Disk local background cleanup failed" + ); + } + if let Err(err) = Self::cleanup_stale_tmp_objects(root.clone(), &publication_root, &budget).await { + error!( + event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DISK_LOCAL, + task = "stale_tmp_objects", + state = "failed", + error = ?err, + "Disk local background cleanup failed" + ); + } + Ok(()) + }).await { error!( event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, component = LOG_COMPONENT_ECSTORE, subsystem = LOG_SUBSYSTEM_DISK_LOCAL, - task = "deleted_objects", - state = "failed", - error = ?err, - "Disk local background cleanup failed" - ); - } - if let Err(err) = Self::cleanup_stale_tmp_objects(root.clone(), &publication_root).await { - error!( - event = EVENT_DISK_LOCAL_BACKGROUND_CLEANUP, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_DISK_LOCAL, - task = "stale_tmp_objects", + task = "cleanup_dispatch", state = "failed", error = ?err, "Disk local background cleanup failed" @@ -6037,95 +6092,148 @@ impl LocalDisk { } } - async fn cleanup_stale_tmp_objects(root: PathBuf, publication_root: &os::PublicationRoot) -> Result<()> { - Self::cleanup_stale_tmp_objects_with_expiry(root, publication_root, STALE_TMP_OBJECT_EXPIRY).await + async fn cleanup_stale_tmp_objects( + root: PathBuf, + publication_root: &os::PublicationRoot, + budget: &super::cleanup_runtime::GcBudget, + ) -> Result<()> { + Self::cleanup_stale_tmp_objects_with_expiry(root, publication_root, STALE_TMP_OBJECT_EXPIRY, budget).await } async fn cleanup_stale_tmp_objects_with_expiry( root: PathBuf, publication_root: &os::PublicationRoot, expiry: Duration, + budget: &super::cleanup_runtime::GcBudget, ) -> Result<()> { let tmp_path = Self::meta_path(&root, RUSTFS_META_TMP_BUCKET); - let mut entries = match fs::read_dir(&tmp_path).await { + let mut entries = match budget.step(async { Ok(fs::read_dir(&tmp_path).await?) }).await { Ok(entries) => entries, Err(e) => { - if e.kind() == ErrorKind::NotFound { + if matches!(&e, DiskError::Io(err) if err.kind() == ErrorKind::NotFound) { return Ok(()); } - return Err(e.into()); + return Err(e); } }; - while let Some(entry) = entries.next_entry().await? { - let name = entry.file_name().to_string_lossy().to_string(); - if name.is_empty() || name == "." || name == ".." || name == ".trash" { - continue; + loop { + let more = budget + .step(async { + let Some(entry) = entries.next_entry().await? else { + return Ok(false); + }; + Self::cleanup_stale_tmp_entry(entry, &root, publication_root, expiry).await?; + Ok(true) + }) + .await?; + if !more { + break; } - - let file_type = entry.file_type().await?; - if !file_type.is_dir() { - continue; - } - - let Some(age) = entry - .metadata() - .await? - .modified() - .ok() - .and_then(|modified| modified.elapsed().ok()) - else { - continue; - }; - if age <= expiry { - continue; - } - - let target_path = Self::meta_path(&root, RUSTFS_META_TMP_DELETED_BUCKET).join(Uuid::new_v4().to_string()); - rename_all(entry.path(), target_path, Self::meta_path(&root, RUSTFS_META_BUCKET), publication_root).await?; } Ok(()) } - async fn cleanup_deleted_objects(root: PathBuf) -> Result<()> { + async fn cleanup_stale_tmp_entry( + entry: fs::DirEntry, + root: &Path, + publication_root: &os::PublicationRoot, + expiry: Duration, + ) -> Result<()> { + let name = entry.file_name().to_string_lossy().to_string(); + if name.is_empty() || name == "." || name == ".." || name == ".trash" || !entry.file_type().await?.is_dir() { + return Ok(()); + } + let Some(age) = entry + .metadata() + .await? + .modified() + .ok() + .and_then(|modified| modified.elapsed().ok()) + else { + return Ok(()); + }; + if age <= expiry { + return Ok(()); + } + let staging = Self::meta_path(publication_root.path(), RUSTFS_META_TMP_BUCKET).join(entry.file_name()); + let owner = os::acquire_namespace_mutation_lease_with_owner(&staging, None).await; + // An active publisher may have consumed or refreshed this entry while + // GC waited. Recheck the actual path before moving the whole prefix. + let metadata = match fs::symlink_metadata(entry.path()).await { + Ok(metadata) => metadata, + Err(error) if error.kind() == ErrorKind::NotFound => return Ok(()), + Err(error) => return Err(error.into()), + }; + if !metadata.is_dir() + || metadata + .modified() + .ok() + .and_then(|modified| modified.elapsed().ok()) + .is_none_or(|age| age <= expiry) + { + return Ok(()); + } + let target_path = Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET).join(Uuid::new_v4().to_string()); + os::rename_all_with_owner( + entry.path(), + target_path, + Self::meta_path(root, RUSTFS_META_BUCKET), + publication_root, + Some(owner), + ) + .await?; + Ok(()) + } + + async fn cleanup_deleted_objects(root: PathBuf, budget: &super::cleanup_runtime::GcBudget) -> Result<()> { let trash = Self::meta_path(&root, RUSTFS_META_TMP_DELETED_BUCKET); - let mut entries = match fs::read_dir(&trash).await { + let mut entries = match budget.step(async { Ok(fs::read_dir(&trash).await?) }).await { Ok(entries) => entries, Err(e) => { - if e.kind() == ErrorKind::NotFound { + if matches!(&e, DiskError::Io(err) if err.kind() == ErrorKind::NotFound) { return Ok(()); } - return Err(e.into()); + return Err(e); } }; - while let Some(entry) = entries.next_entry().await? { - let name = entry.file_name().to_string_lossy().to_string(); - if name.is_empty() || name == "." || name == ".." { - continue; - } - - let file_type = entry.file_type().await?; - - let path = trash.join(name); - - if file_type.is_dir() { - if let Err(e) = tokio::fs::remove_dir_all(path).await - && e.kind() != ErrorKind::NotFound - { - return Err(e.into()); - } - } else if let Err(e) = tokio::fs::remove_file(path).await - && e.kind() != ErrorKind::NotFound - { - return Err(e.into()); + loop { + let more = budget + .step(async { + let Some(entry) = entries.next_entry().await? else { + return Ok(false); + }; + Self::cleanup_trash_entry(entry, &trash).await?; + Ok(true) + }) + .await?; + if !more { + break; } } Ok(()) } + async fn cleanup_trash_entry(entry: fs::DirEntry, trash: &Path) -> Result<()> { + let name = entry.file_name().to_string_lossy().to_string(); + if name.is_empty() || name == "." || name == ".." { + return Ok(()); + } + let path = trash.join(name); + let result = if entry.file_type().await?.is_dir() { + fs::remove_dir_all(path).await + } else { + fs::remove_file(path).await + }; + match result { + Err(err) if err.kind() != ErrorKind::NotFound => Err(err.into()), + _ => Ok(()), + } + } + fn is_valid_volname(volname: &str) -> bool { if volname.len() < 3 { return false; @@ -6252,6 +6360,19 @@ impl LocalDisk { self.local_disk_object_path(self.io_root(), bucket, key) } + /// Only for operations that open through PublicationRoot or a commit guard: + /// containment and symlink checks happen with the actual open, off the runtime. + fn io_get_object_open_path(&self, bucket: &str, key: &str) -> Result { + #[cfg(target_os = "linux")] + { + let (bucket_path, path) = build_local_disk_object_path(self.io_root(), bucket, key); + check_local_disk_object_path_components(self.io_root(), &bucket_path, &path)?; + Ok(path) + } + #[cfg(not(target_os = "linux"))] + self.io_get_object_path(bucket, key) + } + fn io_get_bucket_path(&self, bucket: &str) -> Result { self.local_disk_bucket_path(self.io_root(), bucket) } @@ -6410,7 +6531,7 @@ impl LocalDisk { let mut meta = FileMeta::new(); if !fi.fresh { - let (buf, _) = read_file_exists(&p).await?; + let (buf, _) = read_file_exists(&p, &self.publication_root).await?; if !buf.is_empty() { let _ = meta.unmarshal_msg(&buf).map_err(|_| { meta = FileMeta::new(); @@ -6436,6 +6557,36 @@ impl LocalDisk { Ok(()) } + /// Keep the common GET completion on its caller. Only a last-reader + /// release with pending reclamation needs an owned cleanup task. Leave + /// that token untouched so the slow path owns the complete transition. + pub(in crate::disk) async fn release_snapshot_lease_without_cleanup( + &self, + volume: &str, + path: &str, + token: SnapshotLeaseToken, + ) -> bool { + let key = SnapshotLeaseKey { + volume: volume.to_owned(), + path: path.to_owned(), + }; + let mut registry = self.snapshot_leases.lock(); + let Some(entry) = registry.entries.get_mut(&key) else { + return true; + }; + if !entry.deleting + && entry.pending_delete.is_some() + && (entry.tokens.is_empty() || (entry.tokens.len() == 1 && entry.tokens.contains(&token))) + { + return false; + } + entry.tokens.remove(&token); + if entry.tokens.is_empty() && !entry.deleting { + registry.entries.remove(&key); + } + true + } + async fn delete_data_dir_with_namespace_owner( &self, volume: &str, @@ -6474,12 +6625,24 @@ impl LocalDisk { let claim = SnapshotLeaseDeleteClaim::new(&self.snapshot_leases, key); let result = self - .delete_unleased_with_namespace_owner(volume, path, &opts, namespace_owner) + .delete_unleased_with_namespace_owner(volume, path, &opts, namespace_owner, None) .await; claim.finish(result.is_ok()); result.map(|()| DataDirDeleteStatus::Deleted) } + pub(crate) async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result { + let path = self.io_get_object_path(volume, path)?; + let permit = acquire_context_disk_io_permit().await.map_err(DiskError::Io)?; + spawn_blocking_with_disk_io_permit(permit, move || match orphan_data_dir_parts(&path) { + Ok(_) => Ok(true), + Err(err) if matches!(err.kind(), ErrorKind::NotFound | ErrorKind::InvalidData) => Ok(false), + Err(err) => Err(to_file_error(err).into()), + }) + .await + .map_err(DiskError::from)? + } + pub(crate) async fn delete_versions_with_namespace_owner( &self, volume: &str, @@ -6506,6 +6669,37 @@ impl LocalDisk { errs } + pub(crate) async fn delete_versions_guarded_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + let payload = super::GuardedDeleteVersionsPayload { versions, opts }; + if let Err(error) = payload.validate() { + return vec![Some(error); payload.versions.len()]; + } + let super::GuardedDeleteVersionsPayload { versions, mut opts } = payload; + opts.recoverable_delete = true; + let mut errors = Vec::with_capacity(versions.len()); + for version in &versions { + errors.push( + self.delete_versions_guarded_internal( + volume, + &version.group.name, + &version.group.versions, + &opts, + namespace_owner.clone(), + Some(&version.expected_current), + ) + .await + .err(), + ); + } + errors + } + async fn delete_version_inner(&self, volume: &str, path: &str, fi: FileInfo, mutation: DeleteVersionMutation) -> Result<()> { let DeleteVersionMutation { force_del_marker, @@ -6846,7 +7040,7 @@ impl LocalDisk { ) .await } else { - self.delete_file_with_namespace_owner(&volume_dir, &xl_path, true, false, namespace_owner.clone()) + self.delete_file_with_namespace_owner(&volume_dir, &xl_path, true, false, namespace_owner.clone(), None) .await }; @@ -7082,6 +7276,15 @@ impl LocalDisk { namespace_owner: Option>, ) -> Result<()> { crate::hp_guard!("LocalDisk::delete"); + if opt.orphan_data_dir { + return self + .delete_data_dir_with_namespace_owner(volume, path, opt, namespace_owner) + .await + .and_then(|status| match status { + DataDirDeleteStatus::Deleted => Ok(()), + DataDirDeleteStatus::Deferred => Err(DiskError::DiskOngoingReq), + }); + } let mut opt = opt; let mut namespace_owner = namespace_owner; let handled_version_delete = if opt.recursive @@ -7108,8 +7311,19 @@ impl LocalDisk { } else { false }; + let staging_cleanup = if volume == RUSTFS_META_TMP_BUCKET && opt.recursive { + let staging = tmp_staging_mutation_path(&self.root, path)?; + let lease = os::acquire_namespace_mutation_lease_with_owner(&staging, namespace_owner.take()).await; + namespace_owner = Some(lease.clone()); + Some(StagingCleanupLease { + io_prefix: tmp_staging_mutation_path(self.io_root(), path)?, + lease, + }) + } else { + None + }; match self - .delete_unleased_with_namespace_owner(volume, path, &opt, namespace_owner) + .delete_unleased_with_namespace_owner(volume, path, &opt, namespace_owner, staging_cleanup.as_ref()) .await { Err(DiskError::FileNotFound) if handled_version_delete => Ok(()), @@ -7216,7 +7430,7 @@ impl LocalDisk { } async fn delete_unleased(&self, volume: &str, path: &str, opt: &DeleteOptions) -> Result<()> { - self.delete_unleased_with_namespace_owner(volume, path, opt, None).await + self.delete_unleased_with_namespace_owner(volume, path, opt, None, None).await } async fn delete_unleased_with_namespace_owner( @@ -7225,6 +7439,7 @@ impl LocalDisk { path: &str, opt: &DeleteOptions, namespace_owner: Option>, + staging_cleanup: Option<&StagingCleanupLease>, ) -> Result<()> { let volume_dir = self.io_get_bucket_path(volume)?; if !skip_access_checks(volume) @@ -7235,6 +7450,69 @@ impl LocalDisk { let file_path = self.io_get_object_path(volume, path)?; check_path_length(file_path.to_string_lossy().as_ref())?; + if opt.orphan_data_dir { + if opt.recursive || opt.old_data_dir.is_some() || opt.recoverable_delete { + return Err(DiskError::FileCorrupt); + } + let (object, data_dir) = path.rsplit_once('/').ok_or(DiskError::InvalidPath)?; + let data_dir = Uuid::parse_str(data_dir).map_err(|_| DiskError::InvalidPath)?; + if data_dir.is_nil() { + return Err(DiskError::InvalidPath); + } + // A snapshot may postpone this mutation until after the caller's + // object lock is released. Recheck metadata and recovery obligations + // under the same metadata -> namespace order as normal publication. + let owner = os::acquire_metadata_mutation_lease(&self.get_object_path(volume, object)?, namespace_owner).await; + if self.has_delete_transaction_evidence(volume, object).await? { + return Err(DiskError::DiskOngoingReq); + } + let metadata_path = self.io_get_object_path(volume, &format!("{object}/{STORAGE_FORMAT_FILE}"))?; + let lease = os::acquire_namespace_mutation_lease_with_owner(&file_path, Some(owner)).await; + let sync = effective_durability(volume).syncs_commit_metadata(); + let result = os::run_blocking_namespace_operation(lease, move || { + let bytes = std::fs::read(metadata_path)?; + let metadata = FileMeta::load(&bytes).map_err(|_| std::io::Error::from(ErrorKind::InvalidData))?; + if metadata + .get_data_dirs() + .map_err(|_| std::io::Error::from(ErrorKind::InvalidData))? + .into_iter() + .flatten() + .any(|referenced| referenced == data_dir) + { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + let parts = orphan_data_dir_parts(&file_path)?; + for part in parts { + #[cfg(all(any(test, feature = "test-util"), not(windows)))] + os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &part); + // A descendant may be published after preflight. Unlinking + // only the classified files and using rmdir never removes it. + match std::fs::remove_file(part) { + Ok(()) => {} + Err(err) if err.kind() == ErrorKind::NotFound => {} + Err(err) => return Err(err), + } + } + if sync { + os::fsync_dir_std(&file_path)?; + } + std::fs::remove_dir(&file_path)?; + if sync && let Some(parent) = file_path.parent() { + os::fsync_dir_std(parent)?; + } + Ok(()) + }) + .await + .map_err(|err| { + if err.kind() == ErrorKind::InvalidData { + DiskError::FileCorrupt + } else { + to_file_error(err).into() + } + }); + self.io_backend.invalidate_cached_fds_under(volume, path); + return result; + } let mut recursive = opt.recursive; if recursive && let Some(transaction) = opt.old_data_dir @@ -7270,7 +7548,13 @@ impl LocalDisk { if !entry.file_type()?.is_file() { return Err(corrupt()); } - if delete_recovery_part_file(name) { + if delete_recovery_part_file(name) + || (name != reserved && is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX)) + { + // An interrupted legacy delete can reserve a still-live + // data dir. Only this transaction's commit marker below + // authorizes reclaiming it; remove the old reservation + // with the payload before dropping that proof. parts.push(entry.path()); } else if name == committed || name == recovery_committed || name == reserved { has_committed_marker |= name == committed || name == recovery_committed; @@ -7322,8 +7606,15 @@ impl LocalDisk { })?; recursive = false; } - self.delete_file_with_namespace_owner(&volume_dir, &file_path, recursive, opt.immediate, namespace_owner) - .await?; + self.delete_file_with_namespace_owner( + &volume_dir, + &file_path, + recursive, + opt.immediate, + namespace_owner, + staging_cleanup, + ) + .await?; // A deleted shard must not remain readable through the io_uring fd cache. self.io_backend.invalidate_cached_fds_under(volume, path); Ok(()) @@ -7336,7 +7627,7 @@ impl LocalDisk { recursive: bool, immediate_purge: bool, ) -> Result<()> { - self.delete_file_with_namespace_owner(base_path, delete_path, recursive, immediate_purge, None) + self.delete_file_with_namespace_owner(base_path, delete_path, recursive, immediate_purge, None, None) .await } @@ -7348,6 +7639,7 @@ impl LocalDisk { recursive: bool, immediate_purge: bool, namespace_owner: Option>, + staging_cleanup: Option<&'a StagingCleanupLease>, ) -> BoxFuture<'a, Result<()>> { Box::pin(async move { // debug!("delete_file {:?}\n base_path:{:?}", &delete_path, &base_path); @@ -7365,9 +7657,15 @@ impl LocalDisk { if recursive { self.move_to_trash_with_namespace_owner(delete_path, recursive, immediate_purge, namespace_owner.clone()) .await?; - } else if delete_path.is_dir() { + } else if staging_cleanup.is_some_and(|staging| staging.io_prefix == *delete_path) || delete_path.is_dir() { // debug!("delete_file remove_dir {:?}", &delete_path); - if let Err(err) = os::remove_dir_with_owner(delete_path, namespace_owner.clone()).await { + // Pruning can reach the TMP prefix already held by this cleanup. + // Reuse its lease in the executor rather than locking it again. + let removal = match staging_cleanup.filter(|staging| staging.io_prefix == *delete_path) { + Some(staging) => os::remove_dir_with_lease(delete_path, staging.lease.clone()).await, + None => os::remove_dir_with_owner(delete_path, namespace_owner.clone()).await, + }; + if let Err(err) = removal { // debug!("remove_dir err {:?} when {:?}", &err, &delete_path); // A missing or still-populated directory is benign here; see // is_benign_object_rmdir_error (handles the illumos/Solaris EEXIST @@ -7416,8 +7714,15 @@ impl LocalDisk { } if let Some(dir_path) = delete_path.parent() { - self.delete_file_with_namespace_owner(base_path, &PathBuf::from(dir_path), false, false, namespace_owner.clone()) - .await?; + self.delete_file_with_namespace_owner( + base_path, + &PathBuf::from(dir_path), + false, + false, + namespace_owner.clone(), + staging_cleanup, + ) + .await?; } // debug!("delete_file done {:?}", &delete_path); @@ -7489,10 +7794,11 @@ impl LocalDisk { // - metadata failure -> to_file_error // - parse failure -> propagated verbatim from read_xl_meta_no_data_sync (`?`) let path = file_path.as_ref().to_path_buf(); + let root = self.publication_root.clone(); let (data, modtime) = spawn_blocking_with_disk_io_permit(io_permit, move || -> Result<(Vec, Option)> { // Read-only open, equivalent to O_RDONLY (get_readonly_options only sets read(true)). - let mut f = std::fs::File::open(&path).map_err(to_file_error)?; + let mut f = root.open_readonly(&path).map_err(to_file_error)?; let meta = f.metadata().map_err(to_file_error)?; @@ -7546,6 +7852,7 @@ impl LocalDisk { let metadata_path = self.io_get_object_path(bucket, path.to_string_lossy().as_ref()); // A part's existence check, metadata read and decode share one dispatch. // Keep open errors unmapped for the existing missing-volume fallback. + let root = self.publication_root.clone(); let result = spawn_blocking_with_disk_io_permit(io_permit, move || -> Result<_> { let part_error = |error: String| ObjectPartInfo { number: num, @@ -7558,7 +7865,7 @@ impl LocalDisk { // Invalid metadata paths remain request errors, but missing data wins // first, as it does in the serial reader. let metadata_path = metadata_path?; - Ok(read_all_data_std(&metadata_path) + Ok(read_all_data_std(root.open_readonly(&metadata_path)) .map(|(data, _)| ObjectPartInfo::unmarshal(&data).unwrap_or_else(|err| part_error(err.to_string())))) }) .await; @@ -7578,8 +7885,9 @@ impl LocalDisk { let object_dir = self.io_get_object_path(volume, object_name)?; let metadata_path = object_dir.join(STORAGE_FORMAT_FILE); let volume_dir = self.io_get_bucket_path(volume)?; + let root = self.publication_root.clone(); let result = spawn_blocking_with_disk_io_permit(io_permit, move || { - let (bytes, _) = read_all_data_std(&metadata_path)?; + let (bytes, _) = read_all_data_std(root.open_readonly(&metadata_path))?; let file_meta = FileMeta::load(&bytes).ok(); let data_dirs: HashSet = file_meta .as_ref() @@ -7677,7 +7985,8 @@ impl LocalDisk { // gating the volume fallback on the open error alone is equivalent to // the original code, where the fallback lived solely in the open match arm. let path = file_path.as_ref().to_path_buf(); - let res = spawn_blocking_with_disk_io_permit(io_permit, move || read_all_data_std(&path)) + let root = self.publication_root.clone(); + let res = spawn_blocking_with_disk_io_permit(io_permit, move || read_all_data_std(root.open_readonly(&path))) .await .map_err(DiskError::from)?; @@ -7784,6 +8093,19 @@ impl LocalDisk { fis: &[FileInfo], opts: &DeleteOptions, namespace_owner: Option>, + ) -> Result<()> { + self.delete_versions_guarded_internal(volume, path, fis, opts, namespace_owner, None) + .await + } + + async fn delete_versions_guarded_internal( + &self, + volume: &str, + path: &str, + fis: &[FileInfo], + opts: &DeleteOptions, + namespace_owner: Option>, + expected_current: Option<&CurrentDeleteGeneration>, ) -> Result<()> { let volume_dir = self.io_get_bucket_path(volume)?; let object_path = self.get_object_path(volume, path)?; @@ -7835,6 +8157,9 @@ impl LocalDisk { { Ok(data) => data, Err(DiskError::FileNotFound) => { + if expected_current.is_some_and(|expected| *expected != CurrentDeleteGeneration::Absent) { + return Err(DiskError::ConditionalDeleteConflict); + } self.finish_internal_delete_residue(volume, path, namespace_owner.clone()) .await; // `deleted` alone can be an explicit marker purge; only @@ -7856,6 +8181,20 @@ impl LocalDisk { let mut fm = FileMeta::default(); fm.unmarshal_msg(&data)?; + if let Some(expected) = expected_current { + let actual = match fm.into_fileinfo(volume, path, "", false, false, false) { + Ok(fi) => CurrentDeleteGeneration::from_file_info(&fi)?, + Err(rustfs_filemeta::Error::FileNotFound) => CurrentDeleteGeneration::Absent, + Err(error) => return Err(error.into()), + }; + // Metadata mutation ownership is held from this comparison through + // recovery preparation and publication, so a stale request cannot + // stage a replacement generation for deletion. + if &actual != expected { + return Err(DiskError::ConditionalDeleteConflict); + } + } + let data_dirs = fm.delete_versions(fis)?; let rollback_dir = opts.old_data_dir; let mut reserved_version_delete = false; let mut recovery_target = None; @@ -7889,15 +8228,10 @@ impl LocalDisk { } } - for fi in fis.iter() { - let data_dir = match fm.delete_version(fi) { - Ok(res) => res, + for dir in data_dirs.into_iter().flatten() { + let dir_path = match self.io_get_object_path(volume, format!("{path}/{dir}").as_str()) { + Ok(dir_path) => dir_path, Err(err) => { - let err: DiskError = err.into(); - if !fi.deleted && (err == DiskError::FileNotFound || err == DiskError::FileVersionNotFound) { - continue; - } - if reserved_version_delete && let Some(rollback_dir) = rollback_dir { return Err(self .abort_reserved_version_delete_with_failure( @@ -7906,7 +8240,7 @@ impl LocalDisk { volume, path, DeleteRollbackFailure { - stage: "delete_versions_metadata_update", + stage: "delete_versions_data_path", error: err, namespace_owner: namespace_owner.clone(), }, @@ -7920,7 +8254,7 @@ impl LocalDisk { volume, path, DeleteRollbackFailure { - stage: "delete_versions_metadata_update", + stage: "delete_versions_data_path", error: err, namespace_owner: namespace_owner.clone(), }, @@ -7929,177 +8263,137 @@ impl LocalDisk { .await); } }; - - if let Some(dir) = data_dir { - let vid = fi.version_id.unwrap_or_default(); - let _ = fm.data.remove(vec![vid, dir]); - - let dir_path = match self.io_get_object_path(volume, format!("{path}/{dir}").as_str()) { - Ok(dir_path) => dir_path, - Err(err) => { - if reserved_version_delete && let Some(rollback_dir) = rollback_dir { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_data_path", - error: err, - namespace_owner: namespace_owner.clone(), - }, - ) - .await); - } - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_data_path", - error: err, - namespace_owner: namespace_owner.clone(), - }, - &self.publication_root, - ) - .await); + if let Some(rollback_dir) = rollback_dir { + let rollback_path = object_dir.join(rollback_dir.to_string()); + if let Err(err) = fs::create_dir_all(&rollback_path).await { + let err: DiskError = to_file_error(err).into(); + if reserved_version_delete { + return Err(self + .abort_reserved_version_delete_with_failure( + object_dir, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_rollback_dir", + error: err, + namespace_owner: namespace_owner.clone(), + }, + ) + .await); } - }; - if let Some(rollback_dir) = rollback_dir { - let rollback_path = object_dir.join(rollback_dir.to_string()); - if let Err(err) = fs::create_dir_all(&rollback_path).await { - let err: DiskError = to_file_error(err).into(); - if reserved_version_delete { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_rollback_dir", - error: err, - namespace_owner: namespace_owner.clone(), - }, - ) - .await); - } - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - Some(rollback_dir), - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_rollback_dir", - error: err, - namespace_owner: namespace_owner.clone(), - }, - &self.publication_root, - ) - .await); - } - let reserved = match self - .reserve_version_delete_with_namespace_owner(volume, path, dir, rollback_dir, namespace_owner.clone()) - .await - { - Ok(reserved) => reserved, - Err(err) => { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_reserve_data", - error: err, - namespace_owner: namespace_owner.clone(), - }, - ) - .await); - } - }; - reserved_version_delete |= reserved; - let rollback_data_path = rollback_path.join(dir.to_string()); - if !reserved - && let Err(err) = os::rename_all_ignore_missing_source_with_owner( - &dir_path, - &rollback_data_path, - &rollback_path, - &self.publication_root, - namespace_owner.clone(), - ) - .await - { - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - Some(rollback_dir), - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_stage_data", - error: err, - namespace_owner: namespace_owner.clone(), - }, - &self.publication_root, - ) - .await); - } - if should_fail_after_delete_data_staged(path) { - if reserved_version_delete { - return Err(self - .abort_reserved_version_delete_with_failure( - object_dir, - rollback_dir, - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_test_after_stage", - error: DiskError::Unexpected, - namespace_owner: namespace_owner.clone(), - }, - ) - .await); - } - return Err(restore_delete_rollback_after_error( - object_dir, - &xlpath, - Some(rollback_dir), - volume, - path, - DeleteRollbackFailure { - stage: "delete_versions_test_after_stage", - error: DiskError::Unexpected, - namespace_owner: namespace_owner.clone(), - }, - &self.publication_root, - ) - .await); - } - } else if let Err(err) = self - .move_to_trash_with_namespace_owner(&dir_path, true, false, namespace_owner.clone()) + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + Some(rollback_dir), + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_rollback_dir", + error: err, + namespace_owner: namespace_owner.clone(), + }, + &self.publication_root, + ) + .await); + } + let reserved = match self + .reserve_version_delete_with_namespace_owner(volume, path, dir, rollback_dir, namespace_owner.clone()) .await - && !(err == DiskError::FileNotFound || err == DiskError::VolumeNotFound) { - return Err(err); + Ok(reserved) => reserved, + Err(err) => { + return Err(self + .abort_reserved_version_delete_with_failure( + object_dir, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_reserve_data", + error: err, + namespace_owner: namespace_owner.clone(), + }, + ) + .await); + } }; + reserved_version_delete |= reserved; + let rollback_data_path = rollback_path.join(dir.to_string()); + if !reserved + && let Err(err) = os::rename_all_ignore_missing_source_with_owner( + &dir_path, + &rollback_data_path, + &rollback_path, + &self.publication_root, + namespace_owner.clone(), + ) + .await + { + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + Some(rollback_dir), + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_stage_data", + error: err, + namespace_owner: namespace_owner.clone(), + }, + &self.publication_root, + ) + .await); + } + if should_fail_after_delete_data_staged(path) { + if reserved_version_delete { + return Err(self + .abort_reserved_version_delete_with_failure( + object_dir, + rollback_dir, + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_test_after_stage", + error: DiskError::Unexpected, + namespace_owner: namespace_owner.clone(), + }, + ) + .await); + } + return Err(restore_delete_rollback_after_error( + object_dir, + &xlpath, + Some(rollback_dir), + volume, + path, + DeleteRollbackFailure { + stage: "delete_versions_test_after_stage", + error: DiskError::Unexpected, + namespace_owner: namespace_owner.clone(), + }, + &self.publication_root, + ) + .await); + } + } else if let Err(err) = self + .move_to_trash_with_namespace_owner(&dir_path, true, false, namespace_owner.clone()) + .await + && !(err == DiskError::FileNotFound || err == DiskError::VolumeNotFound) + { + return Err(err); + }; - // The version's data dir was staged or trashed; drop any cached - // io_uring descriptors under it so a deleted `part.N` cannot keep - // answering reads (rustfs/backlog#1175). - self.io_backend.invalidate_cached_fds_under(volume, &format!("{path}/{dir}")); - } + // The version's data dir was staged or trashed; drop any cached + // io_uring descriptors under it so a deleted `part.N` cannot keep + // answering reads (rustfs/backlog#1175). + self.io_backend.invalidate_cached_fds_under(volume, &format!("{path}/{dir}")); } // Remove xl.meta when no versions remain if fm.versions.is_empty() { if let Err(err) = self - .delete_file_with_namespace_owner(&volume_dir, &xlpath, true, false, namespace_owner.clone()) + .delete_file_with_namespace_owner(&volume_dir, &xlpath, true, false, namespace_owner.clone(), None) .await { if reserved_version_delete && let Some(rollback_dir) = rollback_dir { @@ -8505,16 +8799,16 @@ impl LocalDisk { os::make_dir_all(parent, skip_parent).await?; } + let root = self.publication_root.clone(); tokio::task::spawn_blocking(move || { #[cfg(test)] run_owned_file_write_before_open(&path); - let mut file = std::fs::OpenOptions::new() - .create(true) - .write(true) - .truncate(true) - .open(&path) - .map_err(to_file_error)?; + let (mut file, parent_handle) = if sync == SyncMode::FileAndDir { + root.create_truncate_with_parent(&path).map_err(to_file_error)? + } else { + (root.create_truncate(&path).map_err(to_file_error)?, None) + }; std::io::Write::write_all(&mut file, buf.as_ref()).map_err(to_file_error)?; if sync != SyncMode::None { file.sync_data().map_err(to_file_error)?; @@ -8523,7 +8817,7 @@ impl LocalDisk { if sync == SyncMode::FileAndDir && let Some(parent) = path.parent() { - os::fsync_dir_std(parent).map_err(to_file_error)?; + os::fsync_directory_handle_std(parent, parent_handle.as_ref()).map_err(to_file_error)?; } } Ok::<_, std::io::Error>(()) @@ -8548,7 +8842,18 @@ impl LocalDisk { os::make_dir_all(parent, skip_parent).await?; } - let f = super::fs::open_file(path.as_ref(), mode).await.map_err(to_file_error)?; + let f = if mode == O_CREATE | O_WRONLY | O_TRUNC { + let path = path.as_ref().to_path_buf(); + let root = self.publication_root.clone(); + File::from_std( + tokio::task::spawn_blocking(move || root.create_truncate(&path)) + .await + .map_err(DiskError::from)? + .map_err(to_file_error)?, + ) + } else { + super::fs::open_file(path.as_ref(), mode).await.map_err(to_file_error)? + }; Ok(f) } @@ -9424,9 +9729,9 @@ fn file_meta_counts_toward_limit(meta: &FileMeta) -> bool { } // Filter std::io::ErrorKind::NotFound -async fn read_file_exists(path: impl AsRef) -> Result<(Bytes, Option)> { +async fn read_file_exists(path: impl AsRef, root: &os::PublicationRoot) -> Result<(Bytes, Option)> { let p = path.as_ref(); - let (data, meta) = match read_file_all(&p).await { + let (data, meta) = match read_file_all(&p, root).await { Ok((data, meta)) => (data, Some(meta)), Err(e) => { if e == Error::FileNotFound { @@ -9445,32 +9750,28 @@ async fn read_file_exists(path: impl AsRef) -> Result<(Bytes, Option) -> Result<(Bytes, Metadata)> { - let p = path.as_ref(); - let meta = read_file_metadata(&path).await?; - - let data = fs::read(&p) - .await - .inspect_err(|err| { - log_startup_disk_io_error("read_file_all", p, err); - }) - .map_err(to_file_error)?; - - Ok((data.into(), meta)) -} - -async fn read_file_metadata(p: impl AsRef) -> Result { - let path = p.as_ref(); - let meta = fs::metadata(path) - .await - .inspect_err(|err| { - if err.kind() != ErrorKind::NotFound { - log_startup_disk_io_error("read_file_metadata", path, err); - } - }) - .map_err(to_file_error)?; - - Ok(meta) +async fn read_file_all(path: impl AsRef, root: &os::PublicationRoot) -> Result<(Bytes, Metadata)> { + let path = path.as_ref().to_path_buf(); + let root = root.clone(); + tokio::task::spawn_blocking(move || { + let result = (|| { + let mut file = root.open_readonly(&path)?; + let metadata = file.metadata()?; + let mut data = Vec::new(); + std::io::Read::read_to_end(&mut file, &mut data)?; + Ok::<_, std::io::Error>((Bytes::from(data), metadata)) + })(); + result + .inspect_err(|err| { + if err.kind() != ErrorKind::NotFound { + log_startup_disk_io_error("read_file_all", &path, err); + } + }) + .map_err(to_file_error) + .map_err(DiskError::from) + }) + .await + .map_err(DiskError::from)? } fn skip_access_checks(p: impl AsRef) -> bool { @@ -9490,6 +9791,23 @@ fn skip_access_checks(p: impl AsRef) -> bool { false } +struct StagingCleanupLease { + io_prefix: PathBuf, + lease: Arc, +} + +// Use the configured root rather than per-handle /proc fd paths. A nested +// publication and cleanup of its top-level TMP entry must share one lease. +fn tmp_staging_mutation_path(root: &Path, key: &str) -> Result { + let (temporary, object) = build_local_disk_object_path(root, RUSTFS_META_TMP_BUCKET, key); + let object = normalize_path_components(object); + let relative = object.strip_prefix(&temporary).map_err(|_| DiskError::InvalidPath)?; + Ok(relative + .components() + .next() + .map_or_else(|| temporary.clone(), |first| temporary.join(first.as_os_str()))) +} + fn local_disk_object_path(root: &Path, bucket: &str, key: &str) -> Result { let (bucket_path, path) = build_local_disk_object_path(root, bucket, key); check_local_disk_valid_object_path(root, &bucket_path, &path)?; @@ -9552,13 +9870,19 @@ fn check_local_disk_valid_path(root: &Path, path: impl AsRef) -> Result<() #[cfg(target_os = "linux")] fn check_local_disk_valid_object_path_at(root: &Path, root_fd: &std::fs::File, bucket_path: &Path, path: &Path) -> Result<()> { + check_local_disk_object_path_components(root, bucket_path, path)?; + reject_local_disk_symlink_components_at(root, root_fd, &normalize_path_components(path)) +} + +#[cfg(target_os = "linux")] +fn check_local_disk_object_path_components(root: &Path, bucket_path: &Path, path: &Path) -> Result<()> { let bucket_path = normalize_path_components(bucket_path); let path = normalize_path_components(path); if !bucket_path.starts_with(root) || !path.starts_with(&bucket_path) { return Err(DiskError::InvalidPath); } - reject_local_disk_symlink_components_at(root, root_fd, &path) + Ok(()) } #[cfg(target_os = "linux")] @@ -9877,6 +10201,7 @@ impl LocalDisk { ..opts.clone() }, namespace_owner.clone(), + None, ) .await } @@ -9888,7 +10213,7 @@ impl LocalDisk { // Preserve the legacy marker-only cleanup policy without // upgrading disk-local evidence into a quorum receipt. match self - .delete_unleased_with_namespace_owner(volume, &transaction_path, &opts, namespace_owner.clone()) + .delete_unleased_with_namespace_owner(volume, &transaction_path, &opts, namespace_owner.clone(), None) .await { Ok(()) | Err(DiskError::FileNotFound) => self @@ -10199,6 +10524,18 @@ pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize, #[async_trait::async_trait] impl DiskAPI for LocalDisk { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + Ok(()) + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + self.delete_versions_guarded_with_namespace_owner(volume, versions, opts, None) + .await + } async fn delete_version_recoverable( &self, volume: &str, @@ -10229,6 +10566,9 @@ impl DiskAPI for LocalDisk { opts.recoverable_delete = true; self.delete_versions(volume, versions, opts).await } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + self.delete_recovery_has_evidence(volume, object).await + } async fn discover_delete_transactions( &self, volume: &str, @@ -10407,9 +10747,9 @@ impl DiskAPI for LocalDisk { } } - let p = self.io_get_object_path(volume, path)?; + let p = self.io_get_object_open_path(volume, path)?; - let (data, _) = read_file_all(&p).await?; + let (data, _) = read_file_all(&p, &self.publication_root).await?; Ok(data) } @@ -11366,6 +11706,8 @@ impl DiskAPI for LocalDisk { // TODO(backlog): support io.writer cancellation and early termination in walk_dir #[tracing::instrument(level = "trace", skip_all)] async fn walk_dir(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> { + #[cfg(test)] + super::lifecycle_test_io::observe(&opts.bucket, None); self.wait_for_startup_cleanup().await; // Callers that do not pin a stall budget inherit the configured default, so @@ -11701,8 +12043,17 @@ impl DiskAPI for LocalDisk { entry.deleting = true; opts }; + // Release the reader token even under saturation. A rejected reclaim + // retains pending_delete (and its on-disk receipt) for a later retry. + // Do not wait for execution while a releasing reader may own locks. let claim = SnapshotLeaseDeleteClaim::new(&self.snapshot_leases, key); - let result = self.delete_unleased(volume, path, &opts).await; + let result = match super::cleanup_runtime::try_disk_execution() { + Ok(permit) => { + let _permit = permit; + self.delete_unleased(volume, path, &opts).await + } + Err(err) => Err(err.into()), + }; claim.finish(result.is_ok()); result } @@ -11926,6 +12277,8 @@ impl DiskAPI for LocalDisk { #[tracing::instrument(level = "trace", skip_all)] async fn read_xl(&self, volume: &str, path: &str, read_data: bool) -> Result { + #[cfg(test)] + super::lifecycle_test_io::observe(volume, Some(path)); crate::hp_guard!("LocalDisk::read_xl"); let file_path = self.io_get_object_path(volume, path)?; let file_dir = self.io_get_bucket_path(volume)?; @@ -11976,7 +12329,7 @@ impl DiskAPI for LocalDisk { }; // if req.metadata_only {} - match read_file_all(&fpath).await { + match read_file_all(&fpath, &self.publication_root).await { Ok((data, meta)) => { found += 1; @@ -12201,13 +12554,28 @@ async fn wait_for_startup_cleanup_signal( } #[tracing::instrument(level = "trace", skip_all)] +/// The disk root stays non-verbatim for `/`-joined IO, but the Win32 volume +/// APIs behind `rustfs_utils::os` only accept a path past the legacy limit in +/// extended-length form. Shorter roots are passed unchanged so the volume +/// identities those APIs report keep their conventional `C:\` form. +fn win32_volume_path(path: &Path) -> std::io::Result> { + // CreateDirectoryW's 248-unit limit, the lowest of the legacy path limits. + const LEGACY_MAX_PATH: usize = 248; + if path.to_string_lossy().encode_utf16().count() < LEGACY_MAX_PATH { + return Ok(std::borrow::Cow::Borrowed(path)); + } + rustfs_utils::extended_length_path(path) +} + async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInfo, bool)> { let drive_path = drive_path.to_string_lossy().to_string(); check_path_length(&drive_path)?; - let disk_info = get_info(&drive_path).inspect_err(|err| { - log_startup_disk_io_error("get_disk_info_stat", Path::new(&drive_path), err); - })?; + let disk_info = win32_volume_path(Path::new(&drive_path)) + .and_then(|volume_path| get_info(&volume_path)) + .inspect_err(|err| { + log_startup_disk_io_error("get_disk_info_stat", Path::new(&drive_path), err); + })?; let root_drive = if let Some(root_disk_threshold) = runtime_sources::root_disk_threshold_for_erasure_disk().await { if root_disk_threshold > 0 { disk_info.total <= root_disk_threshold @@ -12375,6 +12743,366 @@ mod test { assert!(!should_read_legacy_inline_part(&ec12, &config)); } + fn guarded_delete_group(fi: FileInfo, expected_current: CurrentDeleteGeneration) -> GuardedFileInfoVersions { + GuardedFileInfoVersions { + group: FileInfoVersions { + name: fi.name.clone(), + versions: vec![fi], + ..Default::default() + }, + expected_current, + } + } + + #[tokio::test] + async fn guarded_delete_queued_before_replacement_rejects_stale_generation() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let object = "guarded-object"; + let object_dir = dir.path().join("bucket").join(object); + fs::create_dir_all(&object_dir).await.expect("create object directory"); + let mut original = test_file_info(object, Uuid::nil(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"original"))); + original.metadata.insert("etag".into(), "etag-a".into()); + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture original generation"); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), test_meta(original.clone())) + .await + .expect("write original metadata"); + let transaction = Uuid::new_v4(); + let owner = + os::acquire_metadata_mutation_lease(&disk.get_object_path("bucket", object).expect("object path"), None).await; + let mut deletion = Box::pin(disk.delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(original.clone(), expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + )); + assert!( + tokio::time::timeout(Duration::from_millis(50), deletion.as_mut()) + .await + .is_err(), + "guarded delete waits for metadata ownership" + ); + let mut replacement = original; + replacement.data_dir = Some(Uuid::new_v4()); + replacement.mod_time = replacement.mod_time.map(|time| time + time::Duration::seconds(1)); + replacement.metadata.insert("etag".into(), "etag-b".into()); + let newer = test_meta(replacement); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &newer) + .await + .expect("writer commits replacement while owning metadata"); + drop(owner); + assert_eq!(deletion.await, vec![Some(DiskError::ConditionalDeleteConflict)]); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("replacement metadata remains"), + newer + ); + assert!( + !object_dir.join(transaction.to_string()).exists(), + "rejected generation creates no recovery record" + ); + } + + #[tokio::test] + async fn guarded_delete_remote_only_matching_generation_accepts_missing_and_nil_data_directory() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + for (case, data_dir, remote_version, remote_version_id) in [ + ("absent", None, None, None), + ("empty", Some(Uuid::nil()), Some(String::new()), None), + ("nil-version", None, None, Some(Uuid::nil())), + ] { + let object = format!("guarded-remote-only-{case}"); + let object_dir = dir.path().join("bucket").join(&object); + fs::create_dir_all(&object_dir) + .await + .expect("create remote-only object directory"); + let mut original = test_file_info(&object, Uuid::nil(), data_dir, None); + original.metadata.insert("etag".into(), "etag-remote".into()); + original.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + original.transition_tier = "WARM".into(); + original.transitioned_objname = "remote/object".into(); + original.transition_version = remote_version; + original.transition_version_id = remote_version_id; + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture legacy remote-only generation"); + let before = test_meta(original.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &before) + .await + .expect("write remote-only metadata"); + let persisted = FileMeta::load(&before) + .expect("decode remote-only metadata") + .into_fileinfo("bucket", &object, "", false, false, false) + .expect("read remote-only generation"); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&persisted).expect("capture persisted remote generation"), + expected + ); + // Observed transition state belongs to the guard; a version-removal + // descriptor must not request another transition commit. + let mut deletion = FileInfo { + name: original.name, + version_id: original.version_id, + mod_time: original.mod_time, + ..Default::default() + }; + let free_version_id = Uuid::new_v4(); + deletion.set_tier_free_version_id(&free_version_id.to_string()); + let transaction = Uuid::new_v4(); + assert_eq!( + disk.delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(deletion, expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await, + vec![None], + "matching remote-only guard succeeds: {case}" + ); + let after = fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("durable tier cleanup metadata"); + let remaining = FileMeta::load(&after).expect("decode tier cleanup metadata"); + assert_eq!(remaining.versions.len(), 1, "only the tier cleanup owner remains: {case}"); + let listed = remaining + .get_file_info_versions("bucket", &object, false) + .expect("list live and free versions"); + assert!(listed.versions.is_empty(), "matching remote-only object version is removed: {case}"); + assert_eq!( + listed.free_versions.len(), + 1, + "deletion retains exactly one remote cleanup record: {case}" + ); + let cleanup = &listed.free_versions[0]; + assert!( + cleanup.deleted && cleanup.tier_free_version(), + "remaining metadata is a canonical tier cleanup marker: {case}" + ); + assert_eq!(cleanup.version_id, Some(free_version_id)); + assert_eq!(cleanup.transition_tier, "WARM"); + assert_eq!(cleanup.transitioned_objname, "remote/object"); + assert_eq!(cleanup.transition_version, None); + assert_eq!(cleanup.transition_version_id, None); + assert_eq!(cleanup.transition_version_state, rustfs_filemeta::TransitionVersionState::Unknown); + assert!( + matches!( + remaining.into_fileinfo("bucket", &object, "", false, false, false), + Err(rustfs_filemeta::Error::FileNotFound) + ), + "normal object reads no longer find the deleted version: {case}" + ); + assert_eq!( + fs::read(object_dir.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("recoverable metadata backup"), + before + ); + } + } + + #[tokio::test] + async fn guarded_delete_remote_tuple_changes_preserve_metadata_data_and_recovery_state() { + use rustfs_utils::http::metadata_compat::{SUFFIX_TRANSITION_TIER_DESTINATION_ID, insert_str}; + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + for changed_field in [ + "backend", + "tier", + "object", + "version", + "version-state", + "status", + "local-to-remote", + ] { + let object = format!("guarded-remote-{changed_field}"); + let data_dir = Uuid::new_v4(); + let object_dir = dir.path().join("bucket").join(&object); + let data_path = object_dir.join(data_dir.to_string()).join("part.1"); + fs::create_dir_all(data_path.parent().expect("data directory")) + .await + .expect("create data directory"); + fs::write(&data_path, b"preserved-payload") + .await + .expect("write local data that must survive"); + let mut original = test_file_info(&object, Uuid::nil(), Some(data_dir), None); + original.size = 17; + original.parts[0].size = 17; + original.parts[0].actual_size = 17; + original.metadata.insert("etag".into(), "etag-same".into()); + original.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + original.transition_tier = "WARM".into(); + original.transitioned_objname = "remote/object-a".into(); + original.transition_version = Some("opaque/version-a==".into()); + original.transition_version_state = rustfs_filemeta::TransitionVersionState::Exact; + insert_str(&mut original.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32)); + let mut current = original.clone(); + match changed_field { + "backend" => insert_str(&mut current.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "cd".repeat(32)), + "tier" => current.transition_tier = "COLD".into(), + "object" => current.transitioned_objname = "remote/object-b".into(), + "version" => current.transition_version = Some("opaque/version-b==".into()), + "version-state" => current.transition_version_state = rustfs_filemeta::TransitionVersionState::Unknown, + "status" => current.transition_status = "pending".into(), + "local-to-remote" => { + original.transition_status.clear(); + original.transition_tier.clear(); + original.transitioned_objname.clear(); + original.transition_version = None; + original.transition_version_state = rustfs_filemeta::TransitionVersionState::Unknown; + rustfs_utils::http::metadata_compat::remove_str( + &mut original.metadata, + SUFFIX_TRANSITION_TIER_DESTINATION_ID, + ); + } + _ => unreachable!("test controls changed fields"), + } + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture original remote generation"); + let actual = CurrentDeleteGeneration::from_file_info(¤t).expect("capture changed remote generation"); + assert_ne!(actual, expected, "remote identity change must alter the guard: {changed_field}"); + assert_eq!(current.version_id, original.version_id); + assert_eq!(current.data_dir, original.data_dir); + assert_eq!(current.mod_time, original.mod_time); + assert_eq!(current.get_etag(), original.get_etag()); + let current_metadata = test_meta(current); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), ¤t_metadata) + .await + .expect("publish changed remote identity"); + let transaction = Uuid::new_v4(); + let errors = disk + .delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(original, expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!( + errors, + vec![Some(DiskError::ConditionalDeleteConflict)], + "stale remote tuple is rejected: {changed_field}" + ); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("current metadata remains"), + current_metadata + ); + assert_eq!(fs::read(&data_path).await.expect("payload remains"), b"preserved-payload"); + assert!( + !object_dir.join(transaction.to_string()).exists(), + "stale remote tuple creates no recovery artifacts: {changed_field}" + ); + } + } + + #[tokio::test] + async fn guarded_delete_absence_guard_preserves_new_object() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let object = "guarded-object"; + let object_dir = dir.path().join("bucket").join(object); + fs::create_dir_all(&object_dir).await.expect("create object directory"); + let mut current = test_file_info(object, Uuid::nil(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"current"))); + current.metadata.insert("etag".into(), "etag-current".into()); + let newer = test_meta(current.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &newer) + .await + .expect("write current metadata"); + let transaction = Uuid::new_v4(); + let errors = disk + .delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(current, CurrentDeleteGeneration::Absent)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!(errors, vec![Some(DiskError::ConditionalDeleteConflict)]); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("current metadata remains"), + newer + ); + assert!(!object_dir.join(transaction.to_string()).exists()); + } + + #[tokio::test] + async fn guarded_delete_recovery_rollback_preserves_replacement() { + let dir = tempfile::tempdir().expect("create disk directory"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 disk path")).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk"); + ensure_test_volume(&disk, "bucket").await; + let object = "guarded-object"; + let object_dir = dir.path().join("bucket").join(object); + fs::create_dir_all(&object_dir).await.expect("create object directory"); + let mut original = test_file_info(object, Uuid::nil(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"original"))); + original.metadata.insert("etag".into(), "etag-a".into()); + let expected = CurrentDeleteGeneration::from_file_info(&original).expect("capture original generation"); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), test_meta(original.clone())) + .await + .expect("write original metadata"); + let transaction = Uuid::new_v4(); + assert_eq!( + disk.delete_versions_guarded_recoverable( + "bucket", + vec![guarded_delete_group(original.clone(), expected)], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await, + vec![None] + ); + let applied = disk + .inspect_delete_transaction("bucket", object, transaction) + .await + .expect("inspect prepared delete"); + let mut replacement = original; + replacement.data_dir = Some(Uuid::new_v4()); + replacement.mod_time = replacement.mod_time.map(|time| time + time::Duration::seconds(1)); + replacement.metadata.insert("etag".into(), "etag-b".into()); + let newer = test_meta(replacement); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &newer) + .await + .expect("commit replacement after delete preparation"); + let error = disk + .settle_delete_transaction( + "bucket", + object, + transaction, + applied.fingerprint, + super::super::DeleteRecoveryAction::Rollback, + ) + .await + .expect_err("stale rollback cannot overwrite replacement"); + assert_eq!(error, DiskError::OutdatedXLMeta); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("replacement remains after rejected rollback"), + newer + ); + } + pub(super) fn test_meta(fi: FileInfo) -> Vec { let mut meta = FileMeta::default(); meta.add_version(fi).expect("test metadata should accept file info"); @@ -13572,10 +14300,15 @@ mod test { disk.wait_for_startup_cleanup().await; assert_eq!(disk.startup_cleanup_ready.load(Ordering::Acquire), 1); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().join("missing-root"), &publication_root, Duration::ZERO) - .await - .expect("missing tmp path should be a cleanup no-op"); - LocalDisk::cleanup_deleted_objects(dir.path().join("missing-root")) + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().join("missing-root"), + &publication_root, + Duration::ZERO, + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("missing tmp path should be a cleanup no-op"); + LocalDisk::cleanup_deleted_objects(dir.path().join("missing-root"), &crate::disk::cleanup_runtime::GcBudget::default()) .await .expect("missing trash path should be a cleanup no-op"); @@ -13588,9 +14321,14 @@ mod test { fs::create_dir_all(&trash_root).await.expect("trash dir should be created"); backdate_mtime(&stale_dir, Duration::from_secs(10)); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().to_path_buf(), &publication_root, Duration::ZERO) - .await - .expect("stale tmp directory should move to trash"); + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().to_path_buf(), + &publication_root, + Duration::ZERO, + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("stale tmp directory should move to trash"); assert!(!stale_dir.exists(), "stale tmp directory should be moved away"); assert!(live_file.exists(), "plain tmp files should be ignored by stale dir cleanup"); @@ -13600,7 +14338,7 @@ mod test { fs::create_dir_all(trash_root.join("trash-dir")) .await .expect("trash dir should be created"); - LocalDisk::cleanup_deleted_objects(dir.path().to_path_buf()) + LocalDisk::cleanup_deleted_objects(dir.path().to_path_buf(), &crate::disk::cleanup_runtime::GcBudget::default()) .await .expect("trash cleanup should remove files and directories"); assert!( @@ -17650,6 +18388,306 @@ mod test { } } + #[cfg(unix)] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn cancelled_tmp_parent_pruning_retains_staging_lease_in_executor() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + let root = tempfile::tempdir().expect("cancelled parent pruning fixture"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + disk.write_all(RUSTFS_META_TMP_BUCKET, "stage/child/payload", Bytes::from_static(b"old staging bytes")) + .await + .expect("seed nested staging prefix"); + let parent = disk + .io_get_object_path(RUSTFS_META_TMP_BUCKET, "stage") + .expect("staging parent path"); + let staging_key = tmp_staging_mutation_path(&disk.root, "stage").expect("shared staging key"); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + let _hook = hooks::install_at(hooks::Stage::Remove, &parent, move || { + let _ = entered_tx.send(()); + let _ = release_rx.recv_timeout(Duration::from_secs(30)); + }); + let cleanup_disk = disk.clone(); + let mut cleanup = tokio::spawn(async move { + cleanup_disk + .delete( + RUSTFS_META_TMP_BUCKET, + "stage/child", + DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("physical parent rmdir pause"), + result = &mut cleanup => panic!("cleanup returned before parent pruning: {result:?}"), + } + }) + .await + .expect("cleanup must reach its already-leased parent"); + cleanup.abort(); + assert!(cleanup.await.expect_err("cleanup waiter cancelled").is_cancelled()); + let mut next_owner = + tokio::spawn(async move { os::acquire_namespace_mutation_lease_with_owner(&staging_key, None).await }); + assert!( + tokio::time::timeout(Duration::from_secs(1), &mut next_owner).await.is_err(), + "the physical rmdir must retain the staging lease after cancellation" + ); + drop(release_tx); + let _next_owner = tokio::time::timeout(Duration::from_secs(10), next_owner) + .await + .expect("staging lease released after physical rmdir") + .expect("next staging owner"); + assert!(!parent.exists(), "the empty staging parent must actually be pruned"); + assert!(disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET).expect("TMP path").is_dir()); + } + + #[test] + fn tmp_staging_keys_share_top_level_prefix_and_reject_escape() { + let root = tempfile::tempdir().expect("staging key fixture"); + let expected = root.path().join(RUSTFS_META_TMP_BUCKET).join("stage"); + for path in ["stage", "stage/child", "stage/./child", "stage/child/../next"] { + assert_eq!(tmp_staging_mutation_path(root.path(), path).expect("contained staging key"), expected); + } + assert!(matches!(tmp_staging_mutation_path(root.path(), "../escape"), Err(DiskError::InvalidPath))); + } + + #[cfg(unix)] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial_test::serial(dst_dir_fsync_group_commit)] + async fn inline_heal_owned_receiver_restores_metadata_when_tmp_cleanup_races_rollback() { + use crate::disk::{disk_store::LocalDiskWrapper, os::prepared_publication_test_hooks as hooks}; + + let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(true); + let _mode = durability_mode_override::set(DurabilityMode::Strict); + for stale_gc in [false, true] { + let root = tempfile::tempdir().expect("owned receiver fixture directory"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk")); + let bucket = "inline-heal-owned-receiver"; + let object = "inline-heal-owned-receiver-tmp-cleanup-rollback-object"; + let staging = "inline-heal-owned-receiver-tmp-cleanup-stage"; + ensure_test_volume(&disk, bucket).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + let object_dir = disk.io_get_object_path(bucket, object).expect("physical object path"); + let metadata_path = object_dir.join(STORAGE_FORMAT_FILE); + fs::create_dir_all(&object_dir).await.expect("object directory"); + let version = Uuid::new_v4(); + let mut old = test_file_info(object, version, None, Some(Bytes::from_static(b"old owned receiver payload"))); + old.set_inline_data(); + let old_metadata = test_meta(old); + fs::write(&metadata_path, &old_metadata) + .await + .expect("seed acknowledged metadata"); + let staging_dir = disk + .io_get_object_path(RUSTFS_META_TMP_BUCKET, staging) + .expect("physical staging prefix"); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); + // Group fsync hooks use canonical paths, including on Linux mount-FD IO paths. + let fsync_hook_path = object_dir.canonicalize().expect("canonical group fsync path"); + let _hook = hooks::install_at(hooks::Stage::DirFsync, &fsync_hook_path, move || { + let _ = entered_tx.send(()); + // Dropping the sender also releases this executor when an assertion fails. + let _ = release_rx.recv_timeout(Duration::from_secs(30)); + }); + os::fsync_dir_recorder::set_grouped_failure(&object_dir, io::ErrorKind::PermissionDenied); + let mut repaired = test_file_info(object, version, None, Some(Bytes::from_static(b"new owned receiver payload"))); + repaired.set_inline_data(); + repaired.set_healing(); + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let owner = ctx.begin_namespace_commit(); + let owner_probe = Arc::downgrade(&owner); + let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false); + let mut receiver = tokio::spawn(async move { + wrapper + .rename_data_observed_with_guards( + RUSTFS_META_TMP_BUCKET, + staging, + &repaired, + bucket, + object, + crate::disk::RenameDataGuards { + external_guard: Some(owner), + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + entered = entered_rx => entered.expect("physical fsync pause"), + result = &mut receiver => match result { + Ok(observation) => panic!("receiver completed before the fsync pause; error: {:?}", observation.result.err()), + Err(error) => panic!("receiver task failed before the fsync pause: {error}"), + }, + } + }) + .await + .expect("owned receiver must reach post-publication fsync"); + assert_ne!(fs::read(&metadata_path).await.expect("published repaired metadata"), old_metadata); + assert_eq!( + fs::read(staging_dir.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("active staging rollback backup"), + old_metadata + ); + // An incarnation-bound receiver drains even after its RPC waiter disappears. + receiver.abort(); + assert!(receiver.await.err().expect("RPC waiter cancellation").is_cancelled()); + assert!(ctx.namespace_commits_pending(), "the physical receiver must remain active"); + let cleanup_disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("reconnected disk handle")); + if stale_gc { + std::fs::File::open(&staging_dir) + .expect("staging directory") + .set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::UNIX_EPOCH)) + .expect("make the active prefix eligible for stale GC"); + } + let mut cleanup = tokio::spawn(async move { + if stale_gc { + let mut entries = fs::read_dir(cleanup_disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?).await?; + while let Some(entry) = entries.next_entry().await? { + if entry.file_name() == staging { + return LocalDisk::cleanup_stale_tmp_entry( + entry, + cleanup_disk.io_root(), + &cleanup_disk.publication_root, + Duration::from_secs(1), + ) + .await; + } + } + panic!("active staging prefix must be discoverable by GC"); + } + cleanup_disk + .delete( + RUSTFS_META_TMP_BUCKET, + staging, + DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + }); + assert!( + tokio::time::timeout(Duration::from_secs(1), &mut cleanup).await.is_err(), + "request cleanup and stale GC must wait for the active receiver" + ); + drop(release_tx); + tokio::time::timeout(Duration::from_secs(10), async { + while owner_probe.upgrade().is_some() || ctx.namespace_commits_pending() { + tokio::task::yield_now().await; + } + }) + .await + .expect("physical receiver must finish its failed-fsync rollback"); + tokio::time::timeout(Duration::from_secs(10), cleanup) + .await + .expect("tmp cleanup must finish after physical rollback") + .expect("tmp cleanup task") + .expect("deferred tmp cleanup"); + assert_eq!( + fs::read(&metadata_path).await.expect("metadata after failed-fsync rollback"), + old_metadata, + "temporary-prefix cleanup must not destroy an active receiver's rollback source" + ); + } + } + + #[cfg(unix)] + #[tokio::test] + #[serial_test::serial(dst_dir_fsync_group_commit)] + async fn inline_heal_staged_backup_restores_metadata_after_commit_failure() { + let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(true); + for mode in [DurabilityMode::None, DurabilityMode::Relaxed, DurabilityMode::Strict] { + let _mode = durability_mode_override::set(mode); + for version in [Uuid::nil(), Uuid::new_v4()] { + for old_inline in [true, false] { + let root = tempfile::tempdir().expect("rollback fixture directory"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk"); + let bucket = "inline-heal-rollback"; + let object = "inline-heal-staged-backup-rollback-object"; + ensure_test_volume(&disk, bucket).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + let object_dir = disk.io_get_object_path(bucket, object).expect("physical object path"); + fs::create_dir_all(&object_dir).await.expect("object directory"); + let old_dir = Uuid::new_v4(); + let old = if old_inline { + test_file_info(object, version, None, Some(Bytes::from_static(b"old inline bytes"))) + } else { + fs::create_dir_all(object_dir.join(old_dir.to_string())) + .await + .expect("external directory"); + fs::write(object_dir.join(old_dir.to_string()).join("part.1"), b"x") + .await + .expect("old external shard"); + test_file_info(object, version, Some(old_dir), None) + }; + let old_metadata = test_meta(old); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &old_metadata) + .await + .expect("seed old metadata"); + if mode == DurabilityMode::Strict { + os::fsync_dir_recorder::set_grouped_failure(&object_dir, io::ErrorKind::PermissionDenied); + } else { + set_rename_data_fail_after_metadata_commit(object); + } + let mut new = test_file_info(object, version, None, Some(Bytes::from_static(b"repaired inline bytes"))); + new.set_healing(); + let error = disk + .rename_data(RUSTFS_META_TMP_BUCKET, "heal", new, bucket, object) + .await + .expect_err("injected post-publication failure"); + let expected_kind = if mode == DurabilityMode::Strict { + io::ErrorKind::PermissionDenied + } else { + io::ErrorKind::Other + }; + assert!( + matches!(error, DiskError::Io(ref error) if error.kind() == expected_kind), + "{mode:?}: {error:?}" + ); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("restored metadata"), + old_metadata + ); + if !old_inline { + assert_eq!( + fs::read(object_dir.join(old_dir.to_string()).join("part.1")) + .await + .expect("old shard"), + b"x" + ); + } + let staging = disk.io_get_object_path(RUSTFS_META_TMP_BUCKET, "heal").expect("staging path"); + assert!( + !staging.join(STORAGE_FORMAT_FILE_BACKUP).exists(), + "rollback must consume the staging backup" + ); + for entry in std::fs::read_dir(&object_dir).expect("object entries") { + let entry = entry.expect("object entry"); + if entry.file_type().expect("entry type").is_dir() { + assert!( + !entry.path().join(STORAGE_FORMAT_FILE_BACKUP).exists(), + "heal must not publish a live rollback backup" + ); + } + } + } + } + } + } + #[tokio::test] async fn windows_and_unix_rename_data_inline_post_commit_error_restores_old_metadata() { use tempfile::tempdir; @@ -18743,6 +19781,117 @@ mod test { assert!(!logs.contents().contains("reliable_rename failed")); } + // rustfs#8545: the rollback backup is staged through tempfile, whose Windows + // `persist` calls `MoveFileExW` with the raw path; past MAX_PATH every + // delete of such a key failed with 500 while PUT and GET still worked. + #[tokio::test(flavor = "current_thread")] + async fn windows_and_unix_delete_backs_up_metadata_beyond_max_path() { + use tempfile::tempdir; + + let dir = tempdir().expect("temp dir should be created"); + let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); + let bucket = "bucket"; + ensure_test_volume(&disk, bucket).await; + + let deep_prefix = format!("dir/{}", "a".repeat(200)); + let version_id = Uuid::parse_str("37373737-1111-2222-3333-444444444444").expect("version id should parse"); + let data_dir = Uuid::parse_str("38383838-1111-2222-3333-444444444444").expect("data dir should parse"); + let rollback_dir = Uuid::parse_str("39393939-1111-2222-3333-444444444444").expect("rollback dir should parse"); + let opts = DeleteOptions { + old_data_dir: Some(rollback_dir), + ..Default::default() + }; + + for batch in [false, true] { + let object = format!("{deep_prefix}/{}", if batch { "batch" } else { "single" }); + let object_dir = dir.path().join(bucket).join(&object); + let backup = object_dir.join(rollback_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP); + assert!( + backup.to_str().expect("temp dir should be utf8").encode_utf16().count() > 260, + "regression path must cross the traditional Windows MAX_PATH boundary: {backup:?}" + ); + fs::create_dir_all(&object_dir).await.expect("object dir should be created"); + let fi = test_file_info(&object, version_id, Some(data_dir), Some(Bytes::from_static(b"inline"))); + let meta = test_meta(fi.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &meta) + .await + .expect("inline metadata should be written"); + + if batch { + disk.delete_versions_internal(bucket, &object, &[fi], &opts, None) + .await + .expect("batch delete of a deep key should succeed"); + } else { + disk.delete_version(bucket, &object, fi, false, opts.clone()) + .await + .expect("delete of a deep key should succeed"); + } + + assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists(), "{object}: metadata should be deleted"); + assert_eq!( + fs::read(&backup).await.expect("rollback backup should be published"), + meta, + "{object}: backup should hold the pre-delete metadata" + ); + } + } + + /// Opens a disk at `root` and drives metadata and data reads and writes + /// through it: they extend the root by joining `/`-separated relative paths + /// as strings, so they all depend on the root staying non-verbatim. + async fn assert_local_disk_serves_objects_at(root: &Path) { + let endpoint = Endpoint::try_from(root.to_str().expect("disk root should be utf8")).expect("endpoint should parse"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); + assert!( + !disk.root.to_string_lossy().starts_with(r"\\?\"), + "the disk root must not be verbatim: {:?}", + disk.root + ); + let bucket = "bucket"; + ensure_test_volume(&disk, bucket).await; + + let object = "dir/object"; + let version_id = Uuid::parse_str("41414141-1111-2222-3333-444444444444").expect("version id should parse"); + let data_dir = Uuid::parse_str("42424242-1111-2222-3333-444444444444").expect("data dir should parse"); + let fi = test_file_info(object, version_id, Some(data_dir), Some(Bytes::from_static(b"inline"))); + let meta = test_meta(fi); + disk.write_all(bucket, &format!("{object}/{STORAGE_FORMAT_FILE}"), Bytes::from(meta.clone())) + .await + .expect("metadata should be written below the disk root"); + let raw = disk + .read_xl(bucket, object, true) + .await + .expect("metadata should be read back"); + assert_eq!(raw.buf, meta); + + let payload = Bytes::from_static(b"part payload"); + disk.write_all(bucket, "dir/other/part.1", payload.clone()) + .await + .expect("data should be written below the disk root"); + assert_eq!( + disk.read_all(bucket, "dir/other/part.1") + .await + .expect("data should be read back"), + payload + ); + } + + // A disk root past MAX_PATH resolved to `\\?\C:\...`, where `/` is not a + // separator, so the first `/`-joined metadata read (pool.bin.identity) + // failed with ERROR_INVALID_NAME and startup never completed. + #[tokio::test] + async fn windows_and_unix_local_disk_root_beyond_max_path_serves_objects() { + let dir = tempfile::tempdir().expect("temp dir should be created"); + let mut root = dir.path().to_path_buf(); + while root.to_str().expect("temp dir should be utf8").encode_utf16().count() <= 270 { + root.push("r".repeat(100)); + } + fs::create_dir_all(&root).await.expect("long disk root should be created"); + + assert_local_disk_serves_objects_at(&root).await; + } + #[tokio::test(flavor = "current_thread")] async fn test_ignore_missing_source_helper_still_warns_on_real_error() { use tempfile::tempdir; @@ -20466,6 +21615,220 @@ mod test { assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists()); } + #[tokio::test] + async fn test_windows_delete_version_long_path_crosses_max_path_boundary() { + use super::super::DeleteRecoveryAction; + use tempfile::tempdir; + + let dir = tempdir().expect("temp dir should be created"); + let base_len = dir.path().as_os_str().len().saturating_add(1); + let pad_len = 150usize.saturating_sub(base_len).max(16); + let root = dir.path().join("x".repeat(pad_len)); + fs::create_dir_all(&root).await.expect("long disk root should be created"); + let endpoint = Endpoint::try_from(root.to_str().expect("disk root should be utf8")).expect("endpoint should parse"); + let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created"); + + let bucket = "test-bucket"; + ensure_test_volume(&disk, bucket).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await; + ensure_test_volume(&disk, RUSTFS_META_TMP_DELETED_BUCKET).await; + + let path_code_units = |path: &Path| -> usize { + #[cfg(windows)] + { + use std::os::windows::ffi::OsStrExt; + path.as_os_str().encode_wide().count() + } + #[cfg(not(windows))] + { + path.as_os_str().len() + } + }; + let probe_rollback_dir = Uuid::new_v4(); + let single_char_backup_len = path_code_units( + &disk + .get_object_path(bucket, "dir/a") + .expect("probe object path should resolve") + .join(probe_rollback_dir.to_string()) + .join(STORAGE_FORMAT_FILE_BACKUP), + ); + + for target_backup_len in [259usize, 260, 261, 300] { + let key_suffix_len = target_backup_len + .checked_sub(single_char_backup_len) + .and_then(|extra| extra.checked_add(1)) + .expect("resolved disk root must leave room for the target backup path length"); + let object = format!("dir/{}", "a".repeat(key_suffix_len)); + let physical_object_dir = disk + .get_object_path(bucket, &object) + .expect("physical long object path should resolve"); + let object_dir = disk + .io_get_object_path(bucket, &object) + .expect("io long object path should resolve"); + let rollback_dir = Uuid::new_v4(); + let physical_backup_file_path = physical_object_dir + .join(rollback_dir.to_string()) + .join(STORAGE_FORMAT_FILE_BACKUP); + assert_eq!( + path_code_units(&physical_backup_file_path), + target_backup_len, + "physical backup path must hit the exact target length: {physical_backup_file_path:?}" + ); + let backup_file_path = object_dir.join(rollback_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP); + + let data_dir = Uuid::new_v4(); + let data_path = object_dir.join(data_dir.to_string()); + fs::create_dir_all(&data_path).await.expect("long data dir should be created"); + fs::write(data_path.join("part.1"), b"x") + .await + .expect("part data should be written"); + let fi = test_file_info(&object, Uuid::nil(), Some(data_dir), None); + let meta = test_meta(fi.clone()); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &meta) + .await + .expect("metadata should be written"); + + let read_fi = disk + .read_version( + bucket, + bucket, + &object, + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read_version should succeed on long path"); + assert_eq!(read_fi.name, object); + + disk.delete_version( + bucket, + &object, + fi.clone(), + false, + DeleteOptions { + old_data_dir: Some(rollback_dir), + ..Default::default() + }, + ) + .await + .expect("delete_version should stage rollback backup across MAX_PATH"); + assert!(backup_file_path.is_file(), "rollback backup must exist at {backup_file_path:?}"); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = std::fs::metadata(&backup_file_path) + .expect("rollback backup metadata should be readable") + .permissions() + .mode() + & 0o777; + assert_eq!(mode, 0o600, "rollback backup must retain owner-only 0o600 permissions"); + } + + disk.undo_write_with_namespace_owner( + bucket, + &object, + fi.clone(), + DeleteOptions { + undo_write: true, + undo_delete: true, + old_data_dir: Some(rollback_dir), + ..Default::default() + }, + None, + ) + .await + .expect("undo_write should restore metadata and data across MAX_PATH"); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("restored metadata should be readable"), + meta + ); + assert!(!object_dir.join(rollback_dir.to_string()).exists()); + + let commit_rollback_dir = Uuid::new_v4(); + disk.delete_version( + bucket, + &object, + fi.clone(), + false, + DeleteOptions { + old_data_dir: Some(commit_rollback_dir), + ..Default::default() + }, + ) + .await + .expect("second delete_version should succeed across MAX_PATH"); + disk.delete_with_namespace_owner( + bucket, + &format!("{object}/{commit_rollback_dir}"), + DeleteOptions { + recursive: true, + immediate: true, + old_data_dir: Some(commit_rollback_dir), + ..Default::default() + }, + None, + ) + .await + .expect("cleanup of committed delete transaction should succeed across MAX_PATH"); + assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists()); + assert!(!object_dir.join(commit_rollback_dir.to_string()).exists()); + + let mut inline_fi = test_file_info(&object, Uuid::nil(), None, Some(Bytes::from_static(b"x"))); + inline_fi.set_inline_data(); + let inline_meta = test_meta(inline_fi.clone()); + fs::create_dir_all(&object_dir) + .await + .expect("object dir should be recreated for recoverable delete"); + fs::write(object_dir.join(STORAGE_FORMAT_FILE), &inline_meta) + .await + .expect("inline metadata should be written"); + + let tx = Uuid::new_v4(); + let mut delete_req = FileInfo { + name: object.clone(), + ..Default::default() + }; + delete_req.set_tier_free_version_id(&Uuid::new_v4().to_string()); + disk.delete_version_with_namespace_owner( + bucket, + &object, + delete_req, + false, + DeleteOptions { + old_data_dir: Some(tx), + recoverable_delete: true, + ..Default::default() + }, + None, + ) + .await + .expect("recoverable delete_version should prepare across MAX_PATH"); + + let inspected = disk + .delete_recovery_inspect(bucket, &object, tx) + .await + .expect("delete_recovery_inspect should succeed across MAX_PATH"); + let published = disk + .delete_recovery_settle(bucket, &object, tx, inspected.fingerprint, DeleteRecoveryAction::PublishCommit, None) + .await + .expect("PublishCommit should write quorum receipt across MAX_PATH"); + let finished = disk + .delete_recovery_settle(bucket, &object, tx, published.fingerprint, DeleteRecoveryAction::FinishCommit, None) + .await + .expect("FinishCommit should write finished markers across MAX_PATH"); + disk.delete_recovery_settle(bucket, &object, tx, finished.fingerprint, DeleteRecoveryAction::FinalizeCommit, None) + .await + .expect("FinalizeCommit should remove transaction directory across MAX_PATH"); + assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists()); + assert!(!object_dir.join(tx.to_string()).exists()); + } + } + #[tokio::test] async fn test_rename_data_failure_before_metadata_commit_preserves_old_metadata() { use tempfile::tempdir; @@ -20705,9 +22068,14 @@ mod test { // Backdate after the write above: creating stale/data refreshes the // scanned tmp/stale directory's mtime. backdate_mtime(&tmp.join("stale"), Duration::from_secs(10)); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().to_path_buf(), &publication_root, Duration::ZERO) - .await - .expect("operation should succeed"); + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().to_path_buf(), + &publication_root, + Duration::ZERO, + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("operation should succeed"); assert!(!tmp.join("stale").exists()); assert!(trash.exists()); @@ -20734,9 +22102,14 @@ mod test { fs::write(&fresh_dir, b"temporary").await.expect("operation should succeed"); fs::write(®ular_file, b"keep").await.expect("operation should succeed"); - LocalDisk::cleanup_stale_tmp_objects_with_expiry(dir.path().to_path_buf(), &publication_root, Duration::from_secs(60)) - .await - .expect("operation should succeed"); + LocalDisk::cleanup_stale_tmp_objects_with_expiry( + dir.path().to_path_buf(), + &publication_root, + Duration::from_secs(60), + &crate::disk::cleanup_runtime::GcBudget::default(), + ) + .await + .expect("operation should succeed"); assert!(tmp.join("fresh").exists()); assert!(regular_file.exists()); @@ -20745,6 +22118,44 @@ mod test { assert!(entries.next_entry().await.expect("operation should succeed").is_none()); } + #[cfg(target_os = "linux")] + #[tokio::test] + async fn background_cleanup_keeps_mount_lease_after_disk_drop() { + let dir = tempfile::tempdir().unwrap(); + let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap(); + let disk = LocalDisk::new(&endpoint, false).await.unwrap(); + let root = disk.io_root.clone(); + let trash_entry = LocalDisk::meta_path(&root, RUSTFS_META_TMP_DELETED_BUCKET).join("detached-gc"); + fs::create_dir_all(&trash_entry).await.unwrap(); + fs::write(trash_entry.join("part.1"), b"garbage").await.unwrap(); + let live_file = root.join("keep"); + fs::write(&live_file, b"live").await.unwrap(); + let (exit, exited) = tokio::sync::broadcast::channel(1); + tokio::time::pause(); + let cleanup = tokio::spawn(LocalDisk::cleanup_deleted_objects_loop( + root.clone(), + disk.publication_root.clone(), + disk.mount_lease.clone(), + exited, + )); + tokio::task::yield_now().await; + drop(disk); + tokio::task::yield_now().await; + assert!(live_file.exists(), "the GC owner must retain the original /proc/self/fd root"); + tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL + Duration::from_secs(1)).await; + tokio::time::resume(); + timeout(Duration::from_secs(20), async { + while trash_entry.exists() { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .expect("detached cleanup must finish against the pinned mount"); + assert_eq!(fs::read(&live_file).await.unwrap(), b"live"); + exit.send(()).unwrap(); + timeout(Duration::from_secs(20), cleanup).await.unwrap().unwrap(); + } + #[tokio::test(start_paused = true)] async fn cleanup_loop_interval_does_not_tick_immediately() { let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL; @@ -23484,6 +24895,7 @@ mod test { expected_delete_marker: None, delete_recovery_bucket_incarnation: None, recoverable_delete: false, + orphan_data_dir: false, }; disk.delete("test-volume", "test-file.txt", delete_opts) .await @@ -23887,10 +25299,12 @@ mod test { #[tokio::test] async fn test_read_file_exists() { - let test_file = "./test_read_exists.txt"; + let directory = tempfile::tempdir().expect("test directory"); + let root = os::PublicationRoot::new(directory.path()).expect("publication root"); + let test_file = &directory.path().join("read_exists"); // Test non-existent file - let (data, metadata) = read_file_exists(test_file).await.expect("operation should succeed"); + let (data, metadata) = read_file_exists(test_file, &root).await.expect("operation should succeed"); assert!(data.is_empty()); assert!(metadata.is_none()); @@ -23898,7 +25312,7 @@ mod test { fs::write(test_file, b"test content").await.expect("operation should succeed"); // Test existing file - let (data, metadata) = read_file_exists(test_file).await.expect("operation should succeed"); + let (data, metadata) = read_file_exists(test_file, &root).await.expect("operation should succeed"); assert_eq!(data.as_ref(), b"test content"); assert!(metadata.is_some()); @@ -23908,14 +25322,16 @@ mod test { #[tokio::test] async fn test_read_file_all() { - let test_file = "./test_read_all.txt"; + let directory = tempfile::tempdir().expect("test directory"); + let root = os::PublicationRoot::new(directory.path()).expect("publication root"); + let test_file = &directory.path().join("read_all"); let test_content = b"test content for read_all"; // Create test file fs::write(test_file, test_content).await.expect("operation should succeed"); // Test reading file - let (data, metadata) = read_file_all(test_file).await.expect("operation should succeed"); + let (data, metadata) = read_file_all(test_file, &root).await.expect("operation should succeed"); assert_eq!(data.as_ref(), test_content); assert!(metadata.is_file()); assert_eq!(metadata.len(), test_content.len() as u64); @@ -23924,22 +25340,6 @@ mod test { let _ = fs::remove_file(test_file).await; } - #[tokio::test] - async fn test_read_file_metadata() { - let test_file = "./test_metadata.txt"; - - // Create test file - fs::write(test_file, b"test").await.expect("operation should succeed"); - - // Test reading metadata - let metadata = read_file_metadata(test_file).await.expect("operation should succeed"); - assert!(metadata.is_file()); - assert_eq!(metadata.len(), 4); // "test" is 4 bytes - - // Clean up - let _ = fs::remove_file(test_file).await; - } - #[test] fn test_is_root_path() { // Unix root path @@ -24303,6 +25703,22 @@ mod test { assert!(debug.contains("endpoint")); } + #[cfg(windows)] + #[tokio::test] + async fn windows_local_disk_opens_url_style_drive_endpoint() { + use tempfile::tempdir; + + let dir = tempdir().expect("temp dir should be created"); + let drive_path = dir.path().to_string_lossy().replace('\\', "/"); + let endpoint = + Endpoint::try_from(format!("http://127.0.0.1:9000/{drive_path}").as_str()).expect("URL endpoint should parse"); + assert_eq!(Path::new(&endpoint.get_file_path()), dir.path()); + + LocalDisk::new(&endpoint, false) + .await + .expect("a URL-style drive endpoint must open as a local disk"); + } + #[tokio::test] async fn std_backend_truncate_append_stream_and_full_read_restore_bytes() { use tempfile::tempdir; @@ -24475,6 +25891,303 @@ mod test { assert!(construction_source.is::()); } + async fn write_orphan_test_parent_metadata(disk: &LocalDisk, data_dir: Uuid) { + let mut info = FileInfo::new("object", 1, 1); + info.data_dir = Some(data_dir); + info.mod_time = Some(OffsetDateTime::now_utc()); + info.size = 1; + let mut metadata = FileMeta::default(); + metadata.add_version(info).expect("add parent version"); + disk.write_all("orphan", "object/xl.meta", metadata.marshal_msg().expect("encode parent metadata").into()) + .await + .expect("publish parent metadata"); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_protected_entries() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + for protected in [ + "xl.meta".to_owned(), + "unknown".to_owned(), + STORAGE_FORMAT_FILE_BACKUP.to_owned(), + DELETE_QUORUM_COMMITTED_FILE.to_owned(), + delete_recovery::PREPARED.to_owned(), + format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4()), + ] { + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"payload")) + .await + .expect("write payload"); + disk.write_all("orphan", &format!("{path}/{protected}"), Bytes::from_static(b"witness")) + .await + .expect("write protected entry"); + assert!(!disk.is_plain_orphan_data_dir("orphan", &path).await.expect("probe leaf")); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!( + disk.read_all("orphan", &format!("{path}/part.1")) + .await + .expect("read payload"), + b"payload"[..] + ); + assert_eq!( + disk.read_all("orphan", &format!("{path}/{protected}")) + .await + .expect("read witness"), + b"witness"[..] + ); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn orphan_data_dir_cleanup_rejects_symlinks() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"payload")) + .await + .expect("write payload"); + let directory = disk.io_get_object_path("orphan", &path).expect("resolve leaf"); + std::os::unix::fs::symlink(directory.join("part.1"), directory.join("part.2")).expect("create symlink"); + assert!(!disk.is_plain_orphan_data_dir("orphan", &path).await.expect("probe leaf")); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!(std::fs::read(directory.join("part.1")).expect("read payload"), b"payload"); + assert!( + std::fs::symlink_metadata(directory.join("part.2")) + .expect("stat symlink") + .is_symlink() + ); + } + + #[cfg(not(windows))] + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_child_created_after_preflight() { + use crate::disk::os::prepared_publication_test_hooks as hooks; + + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"stale")) + .await + .expect("write orphan payload"); + let directory = disk.io_get_object_path("orphan", &path).expect("resolve leaf"); + let child = directory.join("child"); + let published_child = child.clone(); + let _hook = hooks::install_at(hooks::Stage::Remove, &directory.join("part.1"), move || { + std::fs::create_dir_all(&published_child).expect("publish concurrent child"); + std::fs::write(published_child.join("xl.meta"), b"child metadata").expect("publish child metadata"); + std::fs::write(published_child.join("part.1"), b"child payload").expect("publish child payload"); + }); + assert!(disk.is_plain_orphan_data_dir("orphan", &path).await.expect("probe leaf")); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err() + ); + assert_eq!(std::fs::read(child.join("xl.meta")).expect("read child metadata"), b"child metadata"); + assert_eq!(std::fs::read(child.join("part.1")).expect("read child payload"), b"child payload"); + assert!(!directory.join("part.1").exists(), "classified orphan part was removed without recursing"); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_rechecks_snapshot_deferred_leaf() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease("orphan", &path).await.expect("acquire snapshot"); + assert_eq!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .expect("defer cleanup"), + DataDirDeleteStatus::Deferred + ); + disk.write_all("orphan", &format!("{path}/child/xl.meta"), Bytes::from_static(b"child metadata")) + .await + .expect("publish child while snapshot is retained"); + assert!(disk.release_snapshot_lease("orphan", &path, token).await.is_err()); + assert_eq!( + disk.read_all("orphan", &format!("{path}/part.1")) + .await + .expect("read protected part"), + b"payload"[..] + ); + assert_eq!( + disk.read_all("orphan", &format!("{path}/child/xl.meta")) + .await + .expect("read child metadata"), + b"child metadata"[..] + ); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_referenced_payload_after_snapshot_deferral() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let data_dir = Uuid::new_v4(); + let path = format!("object/{data_dir}"); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"restored payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease("orphan", &path).await.expect("acquire snapshot"); + assert_eq!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .expect("defer cleanup"), + DataDirDeleteStatus::Deferred + ); + write_orphan_test_parent_metadata(&disk, data_dir).await; + assert!(disk.release_snapshot_lease("orphan", &path, token).await.is_err()); + assert_eq!( + disk.read_all("orphan", &format!("{path}/part.1")) + .await + .expect("read newly referenced payload"), + b"restored payload"[..] + ); + } + + #[tokio::test] + async fn orphan_data_dir_cleanup_preserves_journal_created_after_snapshot_deferral() { + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + let part = format!("{path}/part.1"); + disk.write_all("orphan", &part, Bytes::from_static(b"rollback payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease("orphan", &path).await.expect("acquire snapshot"); + assert_eq!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + }, + ) + .await + .expect("defer cleanup"), + DataDirDeleteStatus::Deferred + ); + let record = format!("object/{}/{}", Uuid::new_v4(), delete_recovery::PREPARED); + disk.write_all("orphan", &record, Bytes::from_static(b"malformed prepared record")) + .await + .expect("publish recovery record after cleanup was deferred"); + assert_eq!(disk.release_snapshot_lease("orphan", &path, token).await, Err(DiskError::DiskOngoingReq)); + assert_eq!( + disk.read_all("orphan", &part).await.expect("read retained rollback payload"), + b"rollback payload"[..] + ); + assert_eq!( + disk.read_all("orphan", &record).await.expect("read retained recovery record"), + b"malformed prepared record"[..] + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn orphan_data_dir_cleanup_propagates_parent_directory_sync_failure() { + let _mode = durability_mode_override::set(DurabilityMode::Strict); + let root = tempfile::tempdir().expect("create disk root"); + let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("parse endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("create disk"); + ensure_test_volume(&disk, "orphan").await; + write_orphan_test_parent_metadata(&disk, Uuid::new_v4()).await; + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all("orphan", &format!("{path}/part.1"), Bytes::from_static(b"stale")) + .await + .expect("write orphan payload"); + let parent = disk.io_get_object_path("orphan", "object").expect("resolve parent directory"); + os::fsync_dir_recorder::set_failure(&parent, ErrorKind::Other); + assert!( + disk.delete_data_dir( + "orphan", + &path, + DeleteOptions { + orphan_data_dir: true, + ..Default::default() + } + ) + .await + .is_err(), + "a failed parent sync cannot report durable reclamation" + ); + assert!( + !disk + .io_get_object_path("orphan", &path) + .expect("resolve data directory") + .exists() + ); + assert!( + os::fsync_dir_recorder::take_failure(&parent).is_none(), + "the rmdir must attempt parent synchronization" + ); + } + #[tokio::test] async fn snapshot_leases_defer_data_dir_cleanup_until_last_release() { use tempfile::tempdir; @@ -24528,9 +26241,14 @@ mod test { Bytes::from_static(b"later") ); - disk.release_snapshot_lease(volume, &data_dir, renewed) - .await - .expect("renewed lease release should succeed"); + assert!( + disk.release_snapshot_lease_without_cleanup(volume, &data_dir, renewed).await, + "a non-final reader release must not dispatch cleanup" + ); + assert!( + !disk.release_snapshot_lease_without_cleanup(volume, &data_dir, second).await, + "the final reader must retain its token for owned deferred reclamation" + ); assert!( disk.read_all(volume, &first_part).await.is_ok(), "one remaining lease must keep the data directory" @@ -24617,6 +26335,44 @@ mod test { )); } + #[tokio::test] + async fn quota_mutation_fence_token_is_bound_to_resource_and_disk_process() { + let root_dir = tempfile::tempdir().expect("create quota fence disk"); + let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("parse disk endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("open initial disk process"); + let bucket = "quota-fence-volume"; + let object = "object"; + let fence_path = quota_mutation_fence_path(bucket, object); + let token = disk + .acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence_path) + .await + .expect("acquire initial quota token"); + drop( + disk.claim_quota_mutation_fence(bucket, object, token) + .await + .expect("claim matching token"), + ); + assert!(disk.claim_quota_mutation_fence(bucket, "other", token).await.is_err()); + assert!(disk.claim_quota_mutation_fence("other-bucket", object, token).await.is_err()); + drop(disk); + + let reopened = LocalDisk::new(&endpoint, false).await.expect("reopen the same disk"); + let successor = reopened + .acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence_path) + .await + .expect("acquire successor process token"); + assert!( + reopened.claim_quota_mutation_fence(bucket, object, token).await.is_err(), + "a token issued before restart must not authorize a later process" + ); + drop( + reopened + .claim_quota_mutation_fence(bucket, object, successor) + .await + .expect("claim successor token"), + ); + } + #[tokio::test] async fn quota_mutation_fence_revoke_waits_for_active_claim_and_rejects_late_claims() { use tempfile::tempdir; diff --git a/crates/ecstore/src/disk/local/commit.rs b/crates/ecstore/src/disk/local/commit.rs index 1fa6b8803..a4a01ace6 100644 --- a/crates/ecstore/src/disk/local/commit.rs +++ b/crates/ecstore/src/disk/local/commit.rs @@ -24,7 +24,7 @@ use super::{ LOG_SUBSYSTEM_DISK_LOCAL, LocalDisk, SyncMode, effective_durability, inline_metadata_rollback_dir, observe_old_current_size, remove_dir_all_if_exists, remove_dst_base_before_commit, remove_file_if_exists, rename_data_versions_signature, run_inline_preparation_before_backup, should_fail_after_metadata_commit, should_fail_before_old_metadata_backup, - should_fail_commit_rename, should_remove_staged_meta_before_commit, skip_access_checks, + should_fail_commit_rename, should_remove_staged_meta_before_commit, skip_access_checks, tmp_staging_mutation_path, }; #[cfg(test)] use super::{run_inline_before_file_sync_admission, run_owned_file_write_before_open, run_rename_data_after_first_publication}; @@ -47,6 +47,33 @@ use tokio::fs; use tracing::{info, warn}; use uuid::Uuid; +fn transitioned_old_current_source( + dst_meta_existed: bool, + dst_meta_unparsable: bool, + xlmeta: &FileMeta, + volume: &str, + path: &str, + version_id: Uuid, +) -> (bool, Option) { + if dst_meta_unparsable { + return (false, None); + } + if !dst_meta_existed { + return (true, None); + } + let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else { + return (false, None); + }; + if old_current.version_id.unwrap_or_default() != version_id { + return (true, None); + } + if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE { + (true, Some(old_current)) + } else { + (true, None) + } +} + /// Hold later repair publications after admitting one baseline object. The /// fixture arms this on one replacement disk before rejoining the cluster. #[cfg(feature = "e2e-test-hooks")] @@ -214,7 +241,7 @@ pub(super) async fn lock_rename_commit_directories( publication_root: &os::PublicationRoot, mutation_lease: Arc, ) -> Result { - #[cfg(windows)] + #[cfg(any(windows, target_os = "linux"))] let result = { let source_parent = source_parent.to_path_buf(); let destination_parent = destination_parent.to_path_buf(); @@ -222,7 +249,7 @@ pub(super) async fn lock_rename_commit_directories( let publication_root = publication_root.clone(); os::run_blocking_namespace_operation(mutation_lease, move || { let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root); - #[cfg(test)] + #[cfg(all(test, windows))] if result.is_ok() { run_destination_commit_directory_preparation(&destination_parent); } @@ -230,7 +257,7 @@ pub(super) async fn lock_rename_commit_directories( }) .await }; - #[cfg(not(windows))] + #[cfg(not(any(windows, target_os = "linux")))] let result = { let _ = mutation_lease; os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root) @@ -244,12 +271,12 @@ pub(super) async fn lock_rename_commit_directories( result.map_err(to_file_error).map_err(DiskError::from) } -async fn read_rename_destination_metadata( +async fn prepare_rename_destination_metadata( file_path: &Path, rename_commit_guard: &os::RenameCommitGuard, mutation_lease: Arc, ) -> Result> { - #[cfg(windows)] + #[cfg(any(windows, target_os = "linux"))] let result = { let file_path = file_path.to_path_buf(); let rename_commit_guard = rename_commit_guard.clone(); @@ -258,10 +285,10 @@ async fn read_rename_destination_metadata( }) .await }; - #[cfg(not(windows))] + #[cfg(not(any(windows, target_os = "linux")))] let _ = (rename_commit_guard, mutation_lease); - #[cfg(not(windows))] - let result = match super::super::fs::read_file(file_path).await { + #[cfg(not(any(windows, target_os = "linux")))] + let result = match fs::read(file_path).await { Ok(data) => Ok(Some(data)), Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), Err(err) => Err(err), @@ -327,12 +354,20 @@ impl LocalDisk { None => None, }; // Quota admission -> metadata RMW -> namespace/volume publication. + let no_inline = fi.data.is_none() && fi.size > 0; + let fi_healing = fi.is_healing(); + let staging_prefix = if fi_healing && !no_inline && src_volume == super::super::RUSTFS_META_TMP_BUCKET { + Some(tmp_staging_mutation_path(&self.root, src_path)?) + } else { + None + }; let metadata_lease = os::acquire_metadata_mutation_lease(&self.get_object_path(dst_volume, dst_path)?, state.namespace_owner.take()).await; let mutation_lease = os::acquire_rename_data_mutation_lease_with_owner( &self.root, dst_volume, &destination_object_path, + staging_prefix.as_deref(), Some(metadata_lease), ) .await; @@ -398,7 +433,7 @@ impl LocalDisk { // xl.meta path let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?; - let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?; + let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?; // data_dir path let has_data_dir_path = { @@ -430,11 +465,6 @@ impl LocalDisk { check_path_length(src_file_path.to_string_lossy().to_string().as_str())?; check_path_length(dst_file_path.to_string_lossy().to_string().as_str())?; - let no_inline = fi.data.is_none() && fi.size > 0; - // Captured before `fi` is consumed by add_version; gates the stale - // destination purge below. - let fi_healing = fi.is_healing(); - // Resolved once for the whole commit so a concurrent configuration // change can never leave a single rename_data half-synced. The tier is // keyed on the destination volume: user data staged in scratch @@ -453,8 +483,7 @@ impl LocalDisk { fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?; } // Acquire the common trees before reading destination metadata. On - // Windows this pins the object directory identity across metadata - // preparation, data publication, rollback backup, and final commit. + // Windows this pins the object directory identity across publication. let rename_commit_guard = lock_rename_commit_directories( src_file_parent, dst_file_parent, @@ -472,9 +501,12 @@ impl LocalDisk { source_volume_dir: &src_volume_dir, data_paths: has_data_dir_path.as_ref(), }; - let has_dst_buf = - read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone()) - .await?; + let has_dst_buf = prepare_rename_destination_metadata( + commit.destination_metadata, + &commit.directory_guard, + commit.mutation_lease.clone(), + ) + .await?; if no_inline { // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta @@ -508,6 +540,14 @@ impl LocalDisk { } let version_id = fi.version_id.unwrap_or_default(); + let (old_current_source_checked, old_current_source) = transitioned_old_current_source( + has_dst_buf.is_some(), + dst_meta_unparsable, + &xlmeta, + dst_volume, + dst_path, + version_id, + ); let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id)); let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok(); let rollback_data_dir = has_old_data_dir.or_else(|| { @@ -798,7 +838,9 @@ impl LocalDisk { && let Some(parent) = dst_file_path.parent() { let fsync_started = rustfs_io_metrics::put_stage_timer(); - if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await { + if let Err(err) = + os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await + { rustfs_io_metrics::record_put_object_stage_duration_from( rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC, fsync_started, @@ -897,6 +939,8 @@ impl LocalDisk { cleanup_data_dir: has_old_data_dir, sign: version_signature, old_current_size, + old_current_source_checked, + old_current_source, }) } else { // Inline metadata preparation is blocking. The transaction lease is @@ -910,6 +954,8 @@ impl LocalDisk { None }; let dst_path_for_failpoint = dst_path.to_string(); + let dst_volume_for_source = dst_volume.to_string(); + let dst_path_for_source = dst_path.to_string(); #[cfg(windows)] let source_parent = src_file_parent.to_path_buf(); let rename_commit_guard_for_preparation = commit.directory_guard.clone(); @@ -954,15 +1000,30 @@ impl LocalDisk { }; let version_id = fi.version_id.unwrap_or_default(); + let (old_current_source_checked, old_current_source) = transitioned_old_current_source( + has_dst_buf.is_some(), + dst_meta_unparsable, + &xlmeta, + &dst_volume_for_source, + &dst_path_for_source, + version_id, + ); let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id)); - let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok(); - let rollback_data_dir = old_data_dir.or_else(|| { - if old_version_exists && has_dst_buf.is_some() { - Some(inline_metadata_rollback_dir(version_id, &xlmeta)) - } else { - None - } - }); + // Healing commits never run set-level post-return rollback; keep + // their intra-commit rollback backup in the staging directory so + // `//xl.meta.bkp` is never stranded. + let rollback_data_dir = if fi_healing { + None + } else { + let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok(); + old_data_dir.or_else(|| { + if old_version_exists && has_dst_buf.is_some() { + Some(inline_metadata_rollback_dir(version_id, &xlmeta)) + } else { + None + } + }) + }; let mut staged_rollback_path = None; if let Some(d) = old_data_dir.as_ref() { let _ = xlmeta.data.remove_two(version_id, *d); @@ -978,7 +1039,7 @@ impl LocalDisk { prepared_metadata_source.write_all(&new_buf, sync)?; run_inline_preparation_before_backup(&dst_path_for_failpoint); if let Some(ref old_metadata) = has_dst_buf - && (rollback_data_dir.is_some() || sync || cfg!(test)) + && (rollback_data_dir.is_some() || fi_healing || sync || cfg!(test)) { #[cfg(windows)] let backup_path = { @@ -1003,6 +1064,8 @@ impl LocalDisk { old_data_dir, version_signature, old_current_size, + old_current_source_checked, + old_current_source, staged_rollback_path, has_dst_buf.is_none(), prepared_metadata_source, @@ -1022,6 +1085,8 @@ impl LocalDisk { cleanup_data_dir, version_signature, old_current_size, + old_current_source_checked, + old_current_source, mut local_rollback_path, destination_was_absent, prepared_metadata_source, @@ -1147,10 +1212,11 @@ impl LocalDisk { && let Some(dst_parent) = dst_file_path.parent() { let fsync_started = rustfs_io_metrics::put_stage_timer(); - if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit( + if let Err(err) = os::fsync_commit_directory( dst_parent, + &commit.directory_guard, commit.mutation_lease.clone(), - admission, + Some(admission), ) .await { @@ -1262,6 +1328,8 @@ impl LocalDisk { cleanup_data_dir, sign: version_signature, old_current_size, + old_current_source_checked, + old_current_source, }) } } diff --git a/crates/ecstore/src/disk/local/delete_recovery.rs b/crates/ecstore/src/disk/local/delete_recovery.rs index f8f4e3485..b64f6fbee 100644 --- a/crates/ecstore/src/disk/local/delete_recovery.rs +++ b/crates/ecstore/src/disk/local/delete_recovery.rs @@ -19,6 +19,152 @@ struct DiscoverySession { touched: Instant, } +impl DiscoverySession { + /// Consume one bounded step, returning ownership only when traversal remains. + fn advance(mut self, limit: u32) -> std::io::Result<(Vec, Option)> { + if self.frames.is_empty() { + match std::fs::read_dir(&self.root) { + Ok(entries) => self.frames.push((String::new(), entries)), + Err(err) if err.kind() == ErrorKind::NotFound => { + return Ok((Vec::new(), None)); + } + Err(err) => return Err(err), + } + } + let mut transactions = BTreeSet::new(); + let mut visited = 0usize; + while !self.frames.is_empty() + && transactions.len() < usize::try_from(limit).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))? + && visited < 4096 + { + let (parent, entries) = self + .frames + .last_mut() + .ok_or_else(|| std::io::Error::from(ErrorKind::InvalidData))?; + let entry = match entries.next() { + Some(entry) => entry?, + None => { + self.frames.pop(); + continue; + } + }; + visited += 1; + let name = entry + .file_name() + .into_string() + .map_err(|_| std::io::Error::from(ErrorKind::InvalidData))?; + if !safe_path(&name, false) || name.contains('/') { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + let relative = if parent.is_empty() { + name.clone() + } else { + format!("{parent}/{name}") + }; + let path = self.root.join(&relative); + reject_local_disk_symlink_components(&self.root, &path).map_err(std::io::Error::from)?; + let kind = match std::fs::symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(err) if err.kind() == ErrorKind::NotFound => continue, + Err(err) => return Err(err), + }; + if !kind.is_dir() && !kind.is_file() { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + if kind.is_file() { + let bound = match self.object.as_deref() { + Some(object) if !parent.is_empty() && !parent.contains('/') => Some((object, parent.as_str())), + Some(_) => None, + None => parent.rsplit_once('/'), + }; + if let Some((object, id)) = bound { + let transaction = + if [PREPARED, STORAGE_FORMAT_FILE_BACKUP, DELETE_QUORUM_COMMITTED_FILE].contains(&name.as_str()) { + Uuid::parse_str(id).ok().filter(|id| !id.is_nil()) + } else { + [ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] + .iter() + .find_map(|prefix| delete_transaction_marker_id(&name, prefix)) + }; + if let Some(transaction) = transaction { + let physical = match self.object.as_ref() { + Some(_) => self.root.clone(), + None => self.root.join(object), + }; + if name == STORAGE_FORMAT_FILE_BACKUP + && live_write_backup(&physical, transaction).map_err(std::io::Error::from)? + { + continue; + } + let proof_root = physical.join(transaction.to_string()); + let prepared = regular_bytes(&proof_root.join(PREPARED)).map_err(std::io::Error::from)?; + let receipt = + regular_bytes(&proof_root.join(DELETE_QUORUM_COMMITTED_FILE)).map_err(std::io::Error::from)?; + // Only a fully bound journal can name a logical + // key. Other records remain discoverable by their + // physical key for canonical inspection to reject. + let record = match prepared { + Some(bytes) => serde_json::from_slice::(&bytes).ok(), + None => receipt + .filter(|bytes| !bytes.is_empty()) + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) + .map(|receipt| receipt.journal), + }; + let logical = record + .filter(|journal| { + journal.protocol_version == DELETE_RECOVERY_PROTOCOL_VERSION + && journal.volume == self.volume + && journal.transaction == transaction + && safe_path(&journal.object, false) + && match self.object.as_ref() { + Some(scope) => scope == &journal.object, + None => build_local_disk_object_path(&self.root, "", &journal.object).1 == physical, + } + }) + .map_or_else(|| object.to_owned(), |journal| journal.object); + transactions.insert((logical, transaction)); + } + } + } else { + if self.object.is_some() && (!parent.is_empty() || !Uuid::parse_str(&name).is_ok_and(|id| !id.is_nil())) { + continue; + } + // A valid 1024-byte key can exceed 256 one-byte path + // components; allow its transaction/data directories too. + if self.frames.len() >= 1024 { + return Err(std::io::Error::from(ErrorKind::InvalidData)); + } + self.frames.push((relative, std::fs::read_dir(path)?)); + } + } + + let transactions = transactions + .into_iter() + .map(|(object, transaction)| DeleteTransactionId { object, transaction }) + .collect(); + let next = (!self.frames.is_empty()).then_some(self); + Ok((transactions, next)) + } + + async fn has_evidence(mut self, disk: &LocalDisk, path: &Path) -> Result { + loop { + let (transactions, next) = disk.advance_delete_discovery(path, self, 1).await?; + if !transactions.is_empty() { + return Ok(true); + } + let Some(mut next) = next else { + return Ok(false); + }; + next.touched = Instant::now(); + self = next; + } + } +} + static DISCOVERY_SESSIONS: std::sync::LazyLock>> = std::sync::LazyLock::new(|| ParkingLotMutex::new(HashMap::new())); @@ -173,7 +319,12 @@ fn live_write_backup(root: &Path, transaction: Uuid) -> Result { let Some(backup) = regular_metadata_bytes(&rollback.join(STORAGE_FORMAT_FILE_BACKUP))? else { return Ok(false); }; - live_write_backup_matches(&FileMeta::load(¤t)?, &FileMeta::load(&backup)?, transaction) + // Corrupt decoded metadata cannot prove that this is a live write backup. + // Keep it discoverable; filesystem and structural errors above still fail. + match (|| live_write_backup_matches(&FileMeta::load(¤t)?, &FileMeta::load(&backup)?, transaction))() { + Err(DiskError::FileCorrupt) => Ok(false), + result => result, + } } fn live_write_backup_matches(current: &FileMeta, backup: &FileMeta, transaction: Uuid) -> Result { @@ -522,13 +673,15 @@ fn inspection(root: &Path, volume: &str, object: &str, transaction: Uuid) -> Res .as_deref() .filter(|bytes| !bytes.is_empty()) .map(serde_json::from_slice) - .transpose()?; + .transpose() + .map_err(|_| DiskError::FileCorrupt)?; let committed = receipt.is_some(); let finished = regular_marker(&rollback.join(FINISHED))? || receipt.as_ref().is_some_and(|receipt| receipt.finished); let journal: Option = prepared .as_deref() .map(serde_json::from_slice) - .transpose()? + .transpose() + .map_err(|_| DiskError::FileCorrupt)? .or_else(|| receipt.as_ref().map(|receipt| receipt.journal.clone())); if receipt .as_ref() @@ -625,12 +778,7 @@ fn inspection(root: &Path, volume: &str, object: &str, transaction: Uuid) -> Res }); if name == transaction.to_string() && !(kind.is_dir() && known_staged) - && !(journal.is_none() - && kind.is_file() - && (delete_recovery_part_file(&child_name) - || [DELETE_DATA_DIR_MARKER_PREFIX, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX] - .iter() - .any(|prefix| child_name == format!("{prefix}{transaction}")))) + && !(journal.is_none() && kind.is_file() && is_metadata_less_data_dir_entry(&child_name)) && (kind.is_dir() || ![ PREPARED, @@ -834,10 +982,15 @@ impl LocalDisk { let receipt = regular_bytes(&entry.path().join(DELETE_QUORUM_COMMITTED_FILE))?; let mut records = Vec::new(); if let Some(bytes) = prepared { - records.push(serde_json::from_slice::(&bytes)?); + records + .push(serde_json::from_slice::(&bytes).map_err(|_| DiskError::FileCorrupt)?); } if let Some(bytes) = receipt.filter(|bytes| !bytes.is_empty()) { - records.push(serde_json::from_slice::(&bytes)?.journal); + records.push( + serde_json::from_slice::(&bytes) + .map_err(|_| DiskError::FileCorrupt)? + .journal, + ); } for record in records { if record.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION @@ -860,6 +1013,107 @@ impl LocalDisk { .await .map_err(to_file_error)? } + /// Local maintenance may pause a page behind recovery backpressure. It + /// retains only its own bounded set of cursors; remote callers still have + /// the ordinary idle deadline and every resumed page rechecks mount identity. + pub(crate) fn renew_delete_recovery_cursor(&self, volume: &str, cursor: &DeleteRecoveryCursor) -> bool { + let mut sessions = DISCOVERY_SESSIONS.lock(); + let Some(session) = sessions.get_mut(&cursor.token) else { + return false; + }; + if session.disk_root != self.io_root() + || session.volume != volume + || session.object.is_some() + || session.touched.elapsed() > Duration::from_secs(300) + { + return false; + } + session.touched = Instant::now(); + true + } + + fn delete_discovery_path(&self, volume: &str, object: Option<&str>) -> Result { + if !safe_path(volume, false) || volume.contains('/') || object.is_some_and(|object| !safe_path(object, false)) { + return Err(DiskError::InvalidPath); + } + match object { + Some(object) => self.io_get_object_path(volume, object), + None => self.io_get_bucket_path(volume), + } + } + + fn new_delete_discovery_session(&self, volume: &str, object: Option<&str>, path: &Path) -> Result { + #[cfg(target_os = "linux")] + let (mount, root) = { + use std::os::fd::AsRawFd; + let mount = self.mount_lease.try_clone().map_err(to_file_error)?; + let root = PathBuf::from(format!("/proc/self/fd/{}/.", mount.as_raw_fd())).join(volume); + let root = match object { + Some(object) => root.join(object), + None => root, + }; + let _ = path; + (mount, root) + }; + #[cfg(windows)] + let (mount, root) = { + use std::os::windows::fs::OpenOptionsExt; + use windows_sys::Win32::Storage::FileSystem::{ + FILE_FLAG_BACKUP_SEMANTICS, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, FILE_SHARE_WRITE, FILE_TRAVERSE, + }; + let mount = std::fs::OpenOptions::new() + .access_mode(FILE_TRAVERSE | FILE_READ_ATTRIBUTES) + .share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE) + .custom_flags(FILE_FLAG_BACKUP_SEMANTICS) + .open(self.io_root()) + .map_err(to_file_error)?; + (mount, path.to_owned()) + }; + #[cfg(all(not(target_os = "linux"), not(windows)))] + let (mount, root) = (std::fs::File::open(self.io_root()).map_err(to_file_error)?, path.to_owned()); + Ok(DiscoverySession { + disk_root: self.io_root().to_path_buf(), + volume: volume.to_owned(), + object: object.map(str::to_owned), + root, + _mount: mount, + frames: Vec::new(), + touched: Instant::now(), + }) + } + + fn validate_delete_discovery_owner(&self, session: &DiscoverySession) -> Result<()> { + if session.disk_root != self.io_root() || session.touched.elapsed() > Duration::from_secs(300) { + return Err(DiskError::InvalidPath); + } + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + let before = session._mount.metadata().map_err(to_file_error)?; + #[cfg(target_os = "linux")] + let current = self.mount_lease.metadata().map_err(to_file_error)?; + #[cfg(not(target_os = "linux"))] + let current = std::fs::metadata(self.io_root()).map_err(to_file_error)?; + if before.dev() != current.dev() || before.ino() != current.ino() { + return Err(DiskError::InvalidPath); + } + } + Ok(()) + } + + async fn advance_delete_discovery( + &self, + path: &Path, + session: DiscoverySession, + limit: u32, + ) -> Result<(Vec, Option)> { + let lease = os::acquire_namespace_mutation_lease_with_owner(path, None).await; + self.validate_delete_discovery_owner(&session)?; + os::run_blocking_namespace_operation(lease, move || session.advance(limit)) + .await + .map_err(|error| to_file_error(error).into()) + } + pub(super) async fn delete_recovery_discover( &self, volume: &str, @@ -867,237 +1121,44 @@ impl LocalDisk { cursor: Option, limit: u32, ) -> Result { - if !(1..=256).contains(&limit) - || !safe_path(volume, false) - || volume.contains('/') - || object.is_some_and(|object| !safe_path(object, false)) - { + if !(1..=256).contains(&limit) { return Err(DiskError::InvalidPath); } - let bucket_path = match object { - Some(object) => self.io_get_object_path(volume, object)?, - None => self.io_get_bucket_path(volume)?, - }; - let disk_root = self.io_root().to_path_buf(); - let (token, mut session) = if let Some(cursor) = cursor { + let path = self.delete_discovery_path(volume, object)?; + let (token, session) = if let Some(cursor) = cursor { let session = DISCOVERY_SESSIONS .lock() .remove(&cursor.token) .ok_or(DiskError::FileNotFound)?; - if session.disk_root != disk_root - || session.volume != volume - || session.object.as_deref() != object - || session.touched.elapsed() > Duration::from_secs(300) - { + if session.volume != volume || session.object.as_deref() != object { return Err(DiskError::InvalidPath); } - #[cfg(unix)] - { - use std::os::unix::fs::MetadataExt; - let before = session._mount.metadata().map_err(to_file_error)?; - #[cfg(target_os = "linux")] - let current = self.mount_lease.metadata().map_err(to_file_error)?; - #[cfg(not(target_os = "linux"))] - let current = std::fs::metadata(self.io_root()).map_err(to_file_error)?; - if before.dev() != current.dev() || before.ino() != current.ino() { - return Err(DiskError::InvalidPath); - } - } (cursor.token, session) } else { - #[cfg(target_os = "linux")] - let (mount, root) = { - use std::os::fd::AsRawFd; - let mount = self.mount_lease.try_clone().map_err(to_file_error)?; - let root = PathBuf::from(format!("/proc/self/fd/{}/.", mount.as_raw_fd())).join(volume); - let root = match object { - Some(object) => root.join(object), - None => root, - }; - (mount, root) - }; - #[cfg(windows)] - let (mount, root) = { - use std::os::windows::fs::OpenOptionsExt; - use windows_sys::Win32::Storage::FileSystem::{ - FILE_FLAG_BACKUP_SEMANTICS, FILE_READ_ATTRIBUTES, FILE_SHARE_READ, FILE_SHARE_WRITE, FILE_TRAVERSE, - }; - // Match the existing publication-root directory open contract. - let mount = std::fs::OpenOptions::new() - .access_mode(FILE_TRAVERSE | FILE_READ_ATTRIBUTES) - .share_mode(FILE_SHARE_READ | FILE_SHARE_WRITE) - .custom_flags(FILE_FLAG_BACKUP_SEMANTICS) - .open(self.io_root()) - .map_err(to_file_error)?; - (mount, bucket_path.clone()) - }; - #[cfg(all(not(target_os = "linux"), not(windows)))] - let (mount, root) = (std::fs::File::open(self.io_root()).map_err(to_file_error)?, bucket_path.clone()); - ( - Uuid::new_v4(), - DiscoverySession { - disk_root, - volume: volume.to_owned(), - object: object.map(str::to_owned), - root, - _mount: mount, - frames: Vec::new(), - touched: Instant::now(), - }, - ) + (Uuid::new_v4(), self.new_delete_discovery_session(volume, object, &path)?) }; - let lease = os::acquire_namespace_mutation_lease_with_owner(&bucket_path, None).await; - os::run_blocking_namespace_operation(lease, move || { - if session.frames.is_empty() { - match std::fs::read_dir(&session.root) { - Ok(entries) => session.frames.push((String::new(), entries)), - Err(err) if err.kind() == ErrorKind::NotFound => { - return Ok(DeleteRecoveryPage { - transactions: Vec::new(), - next: None, - }); - } - Err(err) => return Err(err), - } + let (transactions, session) = self.advance_delete_discovery(&path, session, limit).await?; + let next = if let Some(mut session) = session { + session.touched = Instant::now(); + let mut sessions = DISCOVERY_SESSIONS.lock(); + sessions.retain(|_, session| session.touched.elapsed() <= Duration::from_secs(300)); + if sessions.len() >= 128 { + return Err(DiskError::Io(std::io::Error::from(ErrorKind::WouldBlock))); } - let mut transactions = BTreeSet::new(); - let mut visited = 0usize; - while !session.frames.is_empty() - && transactions.len() < usize::try_from(limit).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))? - && visited < 4096 - { - let (parent, entries) = session - .frames - .last_mut() - .ok_or_else(|| std::io::Error::from(ErrorKind::InvalidData))?; - let entry = match entries.next() { - Some(entry) => entry?, - None => { - session.frames.pop(); - continue; - } - }; - let name = entry - .file_name() - .into_string() - .map_err(|_| std::io::Error::from(ErrorKind::InvalidData))?; - if !safe_path(&name, false) || name.contains('/') { - return Err(std::io::Error::from(ErrorKind::InvalidData)); - } - let relative = if parent.is_empty() { - name.clone() - } else { - format!("{parent}/{name}") - }; - let path = session.root.join(&relative); - reject_local_disk_symlink_components(&session.root, &path).map_err(std::io::Error::from)?; - let kind = match std::fs::symlink_metadata(&path) { - Ok(metadata) => metadata, - Err(err) if err.kind() == ErrorKind::NotFound => continue, - Err(err) => return Err(err), - }; - if !kind.is_dir() && !kind.is_file() { - return Err(std::io::Error::from(ErrorKind::InvalidData)); - } - visited += 1; - if kind.is_file() { - let bound = match session.object.as_deref() { - Some(object) if !parent.is_empty() && !parent.contains('/') => Some((object, parent.as_str())), - Some(_) => None, - None => parent.rsplit_once('/'), - }; - if let Some((object, id)) = bound { - let transaction = - if [PREPARED, STORAGE_FORMAT_FILE_BACKUP, DELETE_QUORUM_COMMITTED_FILE].contains(&name.as_str()) { - Uuid::parse_str(id).ok().filter(|id| !id.is_nil()) - } else { - [ - DELETE_DATA_DIR_MARKER_PREFIX, - RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, - RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, - ] - .iter() - .find_map(|prefix| delete_transaction_marker_id(&name, prefix)) - }; - if let Some(transaction) = transaction { - let physical = match session.object.as_ref() { - Some(_) => session.root.clone(), - None => session.root.join(object), - }; - if name == STORAGE_FORMAT_FILE_BACKUP - && live_write_backup(&physical, transaction).map_err(std::io::Error::from)? - { - continue; - } - let proof_root = physical.join(transaction.to_string()); - let prepared = regular_bytes(&proof_root.join(PREPARED)).map_err(std::io::Error::from)?; - let receipt = - regular_bytes(&proof_root.join(DELETE_QUORUM_COMMITTED_FILE)).map_err(std::io::Error::from)?; - let record = match prepared { - Some(bytes) => { - Some(serde_json::from_slice::(&bytes).map_err(std::io::Error::other)?) - } - None => receipt - .filter(|bytes| !bytes.is_empty()) - .map(|bytes| serde_json::from_slice::(&bytes).map(|receipt| receipt.journal)) - .transpose() - .map_err(std::io::Error::other)?, - }; - let logical = if let Some(journal) = record { - let bound_path = match session.object.as_ref() { - Some(scope) if scope == &journal.object => physical.clone(), - Some(_) => return Err(std::io::Error::from(ErrorKind::InvalidData)), - None => build_local_disk_object_path(&session.root, "", &journal.object).1, - }; - if journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION - || journal.volume != session.volume - || journal.transaction != transaction - || !safe_path(&journal.object, false) - || bound_path != physical - { - return Err(std::io::Error::from(ErrorKind::InvalidData)); - } - journal.object - } else { - object.to_owned() - }; - transactions.insert((logical, transaction)); - } - } - } else { - if session.object.is_some() && (!parent.is_empty() || !Uuid::parse_str(&name).is_ok_and(|id| !id.is_nil())) { - continue; - } - // A valid 1024-byte key can exceed 256 one-byte path - // components; allow its transaction/data directories too. - if session.frames.len() >= 1024 { - return Err(std::io::Error::from(ErrorKind::InvalidData)); - } - session.frames.push((relative, std::fs::read_dir(path)?)); - } - } - let next = if session.frames.is_empty() { - None - } else { - session.touched = Instant::now(); - let mut sessions = DISCOVERY_SESSIONS.lock(); - sessions.retain(|_, session| session.touched.elapsed() <= Duration::from_secs(300)); - if sessions.len() >= 128 { - return Err(std::io::Error::from(ErrorKind::WouldBlock)); - } - sessions.insert(token, session); - Some(DeleteRecoveryCursor { token }) - }; - Ok(DeleteRecoveryPage { - transactions: transactions - .into_iter() - .map(|(object, transaction)| DeleteTransactionId { object, transaction }) - .collect(), - next, - }) - }) - .await - .map_err(|err| to_file_error(err).into()) + sessions.insert(token, session); + Some(DeleteRecoveryCursor { token }) + } else { + None + }; + Ok(DeleteRecoveryPage { transactions, next }) + } + + /// Existence probes retain their traversal privately, including across cancellation. + pub(super) async fn delete_recovery_has_evidence(&self, volume: &str, object: &str) -> Result { + let path = self.delete_discovery_path(volume, Some(object))?; + self.new_delete_discovery_session(volume, Some(object), &path)? + .has_evidence(self, &path) + .await } pub(super) async fn delete_recovery_inspect( @@ -1244,26 +1305,14 @@ impl LocalDisk { for version in &plan.versions { original_versions.push(generation(version.into_fileinfo(volume, object, true)?)?); } - let mut data_dirs = Vec::with_capacity(requests.len()); - for request in requests { - if original.is_none() { + let data_dirs = if original.is_none() { + for request in requests { plan.add_version(request.clone())?; - continue; } - match plan.delete_version(request) { - Ok(Some(dir)) => { - let _ = plan.data.remove(vec![request.version_id.unwrap_or_default(), dir]); - data_dirs.push(dir); - } - Ok(None) => {} - Err(err) => { - let err: DiskError = err.into(); - if request.deleted || (err != DiskError::FileVersionNotFound && err != DiskError::FileNotFound) { - return Err(err); - } - } - } - } + Vec::new() + } else { + plan.delete_versions(requests)?.into_iter().flatten().collect::>() + }; if data_dirs.iter().any(|dir| *dir == transaction || dir.is_nil()) { return Err(DiskError::FileCorrupt); } @@ -1371,7 +1420,7 @@ impl LocalDisk { action: DeleteRecoveryAction, owner: Option>, ) -> Result { - let observed = self + let mut observed = self .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) .await?; if observed.fingerprint != expected_fingerprint { @@ -1498,13 +1547,13 @@ impl LocalDisk { }, ) .await?; + observed = self + .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) + .await?; } else { return Ok(observed); } } - let observed = self - .delete_recovery_inspect_owned(volume, object, transaction, owner.clone()) - .await?; if action == DeleteRecoveryAction::DiscardLegacy { if observed.state != DeleteRecoveryState::LegacyAmbiguous || !observed.explicit_delete_safe { return Err(DiskError::FileCorrupt); @@ -1586,6 +1635,7 @@ impl LocalDisk { ..Default::default() }, owner.clone(), + None, ) .await?; return self.delete_recovery_inspect_owned(volume, object, transaction, owner).await; @@ -1648,6 +1698,7 @@ impl LocalDisk { false, false, owner.clone(), + None, ) .await?; if effective_durability(volume).syncs_commit_metadata() { @@ -1765,6 +1816,7 @@ impl LocalDisk { ..Default::default() }, owner.clone(), + None, ) .await?; } @@ -1861,6 +1913,7 @@ impl LocalDisk { ..Default::default() }, owner.clone(), + None, ) .await?; } else { @@ -1953,10 +2006,7 @@ async fn barrier(volume: &str, object: &str, transaction: Uuid, stage: &str, dri .map_err(to_file_error)? == b"io_error" { - return Err(DiskError::Io(std::io::Error::new( - ErrorKind::Other, - "injected delete recovery barrier failure", - ))); + return Err(DiskError::Io(std::io::Error::other("injected delete recovery barrier failure"))); } Ok(()) } @@ -2214,6 +2264,149 @@ mod tests { (temp, disk, info, bytes) } + #[tokio::test] + async fn delete_recovery_absent_publish_refreshes_prepared_witness() { + let (_temp, disk, _info, _original) = fixture().await; + let root = disk.io_get_object_path("recovery", "object").expect("object path"); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)).await.expect("absent object"); + let transaction = Uuid::new_v4(); + let absent = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("inspect absent transaction"); + assert_eq!(absent.state, DeleteRecoveryState::Absent); + let mut observed = absent; + for (action, expected) in [ + (DeleteRecoveryAction::PublishCommit, DeleteRecoveryState::QuorumCommitted), + (DeleteRecoveryAction::PublishCommit, DeleteRecoveryState::QuorumCommitted), + (DeleteRecoveryAction::FinishCommit, DeleteRecoveryState::CommitFinished), + (DeleteRecoveryAction::FinalizeCommit, DeleteRecoveryState::Absent), + ] { + observed = disk + .settle_delete_transaction("recovery", "object", transaction, observed.fingerprint, action) + .await + .expect("settle absent-key witness"); + assert_eq!(observed.state, expected); + assert!(observed.current_digest.is_none()); + assert!(!root.join(STORAGE_FORMAT_FILE).exists(), "the witness cannot resurrect metadata"); + if let Some(journal) = &observed.journal { + assert_eq!(journal.transaction, transaction); + assert!(journal.original_digest.is_none() && journal.target_digest.is_none()); + assert!(journal.data_dirs.is_empty()); + } else { + assert_eq!(expected, DeleteRecoveryState::Absent); + } + } + assert!(!root.join(transaction.to_string()).exists(), "finalization retires the witness"); + } + + #[tokio::test] + async fn delete_recovery_inspection_keeps_foreign_data_markers_inert() { + for prefix in [ + DELETE_DATA_DIR_MARKER_PREFIX, + RESERVED_DELETE_DATA_DIR_MARKER_PREFIX, + RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX, + ] { + for backup in [false, true] { + let (_temp, disk, _info, original) = fixture().await; + let root = disk.io_get_object_path("recovery", "object").expect("object path"); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)) + .await + .expect("hide current metadata"); + let candidate = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let data = root.join(candidate.to_string()); + fs::create_dir(&data).await.expect("coincident data directory"); + fs::write(data.join("part.1"), b"retained shard").await.expect("retain data"); + let marker = data.join(format!("{prefix}{owner}")); + fs::write(&marker, []).await.expect("foreign transaction marker"); + if backup { + fs::write(data.join(STORAGE_FORMAT_FILE_BACKUP), &original) + .await + .expect("retain ambiguous backup"); + } + let before = fs::read(&marker).await.expect("marker before inspection"); + let state = disk + .inspect_delete_transaction("recovery", "object", candidate) + .await + .expect("foreign data marker is not corrupt transaction evidence"); + assert_eq!( + state.state, + if backup { + DeleteRecoveryState::LegacyAmbiguous + } else { + DeleteRecoveryState::Absent + } + ); + assert!(!state.explicit_delete_safe, "foreign ownership cannot authorize legacy removal"); + assert_eq!(fs::read(&marker).await.expect("preserved marker"), before); + assert_eq!(fs::read(data.join("part.1")).await.expect("preserved shard"), b"retained shard"); + if backup { + assert_eq!( + fs::read(data.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("preserved backup"), + original + ); + } + } + } + } + + #[tokio::test] + async fn delete_recovery_inspection_rejects_unsafe_foreign_data_entries() { + for shape in ["malformed", "nil", "directory", "symlink", "child", "metadata", "unknown"] { + let (_temp, disk, _info, _original) = fixture().await; + let root = disk.io_get_object_path("recovery", "object").expect("object path"); + fs::remove_file(root.join(STORAGE_FORMAT_FILE)) + .await + .expect("hide current metadata"); + let candidate = Uuid::new_v4(); + let data = root.join(candidate.to_string()); + fs::create_dir(&data).await.expect("coincident data directory"); + let marker = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4()); + match shape { + "malformed" => fs::write(data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}invalid")), []) + .await + .expect("malformed marker"), + "nil" => fs::write(data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::nil())), []) + .await + .expect("nil marker"), + "directory" => fs::create_dir(data.join(&marker)).await.expect("marker directory"), + "symlink" => { + #[cfg(unix)] + std::os::unix::fs::symlink(root.join(STORAGE_FORMAT_FILE), data.join(&marker)).expect("marker symlink"); + #[cfg(not(unix))] + continue; + } + "child" => fs::create_dir(data.join(Uuid::new_v4().to_string())) + .await + .expect("child key directory"), + "metadata" => fs::write(data.join(STORAGE_FORMAT_FILE), b"child metadata") + .await + .expect("child metadata"), + _ => fs::write(data.join("unknown"), []).await.expect("unknown file"), + } + assert!( + matches!( + disk.inspect_delete_transaction("recovery", "object", candidate).await, + Err(DiskError::FileCorrupt | DiskError::InvalidPath) + ), + "{shape}" + ); + assert!( + fs::read_dir(&data) + .await + .expect("unsafe evidence preserved") + .next_entry() + .await + .expect("read evidence") + .is_some(), + "{shape}" + ); + } + } + #[tokio::test] async fn delete_recovery_live_write_backup_preserves_delete_evidence() { let (_temp, disk, info, original) = fixture().await; @@ -2621,6 +2814,120 @@ mod tests { ); } + #[tokio::test] + async fn delete_recovery_batch_inline_matches_sequential_and_survives_settlement() { + for commit in [false, true] { + let (_temp, disk, _, _) = fixture().await; + let mut original = FileMeta::new(); + let mut requests = Vec::new(); + for id in 1_u128..=64 { + let mut info = test_file_info( + "object", + Uuid::from_u128(id), + Some(Uuid::from_u128(1000 + id)), + Some(Bytes::from(vec![u8::try_from(id).expect("small id"); 9 * 1024])), + ); + info.set_inline_data(); + original.add_version(info).expect("seed inline version"); + if id <= 48 { + requests.push(FileInfo { + name: "object".into(), + version_id: Some(Uuid::from_u128(id)), + ..Default::default() + }); + } + } + let original_bytes = original.marshal_msg().expect("encode original"); + let meta_path = disk.io_get_object_path("recovery", "object/xl.meta").expect("metadata path"); + fs::write(&meta_path, &original_bytes).await.expect("persist version chain"); + let mut expected = original; + for request in &requests { + if let Some(dir) = expected.delete_version(request).expect("sequential delete") { + expected + .data + .remove(vec![request.version_id.expect("explicit version"), dir]) + .expect("sequential inline removal"); + } + } + let transaction = Uuid::new_v4(); + let errors = disk + .delete_versions_recoverable( + "recovery", + vec![FileInfoVersions { + name: "object".into(), + volume: "recovery".into(), + versions: requests, + ..Default::default() + }], + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!(errors, vec![None]); + let applied = disk.read_all("recovery", "object/xl.meta").await.expect("applied metadata"); + assert_eq!(applied.as_ref(), expected.marshal_msg().expect("expected metadata")); + let target = disk + .read_all("recovery", &format!("object/{transaction}/{TARGET}")) + .await + .expect("prepared target"); + assert_eq!(applied, target, "publication must use the exact prepared target"); + let current = FileMeta::load(&applied).expect("load survivors"); + assert_eq!(current.versions.len(), 16); + for id in 49_u128..=64 { + let fi = current + .into_fileinfo("recovery", "object", &Uuid::from_u128(id).to_string(), true, false, true) + .expect("surviving version"); + assert_eq!( + fi.data.expect("surviving inline body").as_ref(), + vec![u8::try_from(id).expect("small id"); 9 * 1024] + ); + } + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("inspect prepared batch"); + if commit { + let committed = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("publish commit"); + let finished = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + committed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("finish cleanup"); + assert_eq!(finished.state, DeleteRecoveryState::CommitFinished); + assert_eq!(fs::read(&meta_path).await.expect("committed survivors"), applied.as_ref()); + } else { + let restored = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::Rollback, + ) + .await + .expect("rollback batch"); + assert_eq!(restored.state, DeleteRecoveryState::Absent); + assert_eq!(fs::read(&meta_path).await.expect("restored metadata"), original_bytes); + } + } + } + #[tokio::test] async fn delete_recovery_metadata_updates_publish_exact_prepared_target_bytes() { for batch in [false, true] { @@ -3028,6 +3335,226 @@ mod tests { assert!(!disk.io_get_object_path("recovery", "object").expect("object path").exists()); } + #[tokio::test] + async fn delete_recovery_reclaims_legacy_reservation_after_commit_and_snapshot_release() { + for batch in [false, true] { + for snapshot in [false, true] { + let (temp, disk, _, _) = fixture().await; + let data_dir = Uuid::new_v4(); + let transaction = Uuid::new_v4(); + let root = disk.io_get_object_path("recovery", "object").expect("object path"); + let data = root.join(data_dir.to_string()); + fs::create_dir(&data).await.expect("external data directory"); + fs::write(data.join("part.1"), b"x").await.expect("external payload"); + let reservation = data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())); + fs::write(&reservation, []).await.expect("interrupted legacy reservation"); + fs::write( + root.join(STORAGE_FORMAT_FILE), + test_meta(test_file_info("object", Uuid::nil(), Some(data_dir), None)), + ) + .await + .expect("live metadata still owns reserved data"); + let path = format!("object/{data_dir}"); + let lease = if snapshot { + Some(disk.acquire_snapshot_lease("recovery", &path).await.expect("active reader")) + } else { + None + }; + let options = DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }; + if batch { + let errors = disk + .delete_versions_recoverable( + "recovery", + vec![FileInfoVersions { + name: "object".into(), + volume: "recovery".into(), + versions: vec![request()], + ..Default::default() + }], + options, + ) + .await; + assert_eq!(errors, vec![None]); + } else { + disk.delete_version_recoverable("recovery", "object", request(), false, options) + .await + .expect("apply single delete"); + } + let applied = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("applied"); + let committed = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + applied.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("publish certified decision"); + let finish = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + committed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await; + if let Some(lease) = lease { + assert_eq!(finish.expect_err("snapshot retains payload"), DiskError::DiskOngoingReq); + assert_eq!(fs::read(data.join("part.1")).await.expect("snapshot payload"), b"x"); + assert!(reservation.is_file()); + disk.release_snapshot_lease("recovery", &path, lease) + .await + .expect("release replays safe cleanup"); + } else { + assert_eq!( + finish.expect("old reservation must not block commit").state, + DeleteRecoveryState::CommitFinished + ); + } + assert!(!data.exists(), "batch={batch}, snapshot={snapshot}"); + drop(disk); + let endpoint = Endpoint::try_from(temp.path().to_str().expect("UTF-8 disk path")).expect("endpoint"); + let disk = LocalDisk::new(&endpoint, false).await.expect("reopen after cleanup"); + let observed = disk + .inspect_delete_transaction("recovery", "object", transaction) + .await + .expect("replay inspection"); + let finished = disk + .settle_delete_transaction( + "recovery", + "object", + transaction, + observed.fingerprint, + DeleteRecoveryAction::FinishCommit, + ) + .await + .expect("idempotent finish after restart"); + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + finished.fingerprint, + DeleteRecoveryAction::FinalizeCommit, + ) + .await + .expect("finalize converged transaction"); + assert!(!root.exists(), "committed deletion leaves no physical residue"); + } + } + } + + #[tokio::test] + async fn delete_recovery_legacy_reservation_never_authorizes_unrelated_data() { + for shape in [ + "no-proof", + "foreign-commit", + "nil", + "invalid", + "directory", + "child", + "metadata", + ] { + let (_temp, disk, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + let data_dir = Uuid::new_v4(); + let path = format!("object/{data_dir}"); + let data = disk.io_get_object_path("recovery", &path).expect("data path"); + fs::create_dir(&data).await.expect("data directory"); + fs::write(data.join("part.1"), b"preserve").await.expect("payload"); + let old = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4()); + fs::write(data.join(&old), []).await.expect("old reservation"); + if shape != "no-proof" { + fs::write(data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")), []) + .await + .expect("current proof"); + } + match shape { + "foreign-commit" => fs::write(data.join(format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())), []) + .await + .expect("foreign proof"), + "nil" => fs::write(data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::nil())), []) + .await + .expect("nil reservation"), + "invalid" => fs::write(data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}invalid")), []) + .await + .expect("malformed reservation"), + "directory" => { + fs::remove_file(data.join(&old)).await.expect("replace marker"); + fs::create_dir(data.join(&old)).await.expect("marker is not a regular file"); + } + "child" => fs::create_dir(data.join("child")).await.expect("UUID-prefix descendant"), + "metadata" => fs::write(data.join(STORAGE_FORMAT_FILE), b"child metadata") + .await + .expect("UUID-prefix object"), + _ => {} + } + let error = disk + .delete_unleased( + "recovery", + &path, + &DeleteOptions { + recursive: true, + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect_err("unproven or non-regular entries must prevent all cleanup"); + assert_eq!(error, DiskError::FileCorrupt, "shape={shape}"); + assert_eq!( + fs::read(data.join("part.1")).await.expect("payload survives"), + b"preserve", + "shape={shape}" + ); + assert!(data.join(old).exists(), "reservation survives failed preflight"); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_legacy_reservation_rejects_symlink() { + let (temp, disk, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + let path = format!("object/{}", Uuid::new_v4()); + let data = disk.io_get_object_path("recovery", &path).expect("data path"); + fs::create_dir(&data).await.expect("data directory"); + let external = temp.path().join("external"); + fs::write(&external, b"external").await.expect("external payload"); + fs::write(data.join("part.1"), b"preserve").await.expect("payload"); + fs::write(data.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}")), []) + .await + .expect("current proof"); + std::os::unix::fs::symlink( + &external, + data.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())), + ) + .expect("symlink reservation"); + assert_eq!( + disk.delete_unleased( + "recovery", + &path, + &DeleteOptions { + recursive: true, + old_data_dir: Some(transaction), + ..Default::default() + } + ) + .await + .expect_err("non-regular reservation must fail closed"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(data.join("part.1")).await.expect("payload survives"), b"preserve"); + assert_eq!(fs::read(external).await.expect("symlink target survives"), b"external"); + } + #[tokio::test] async fn delete_recovery_fingerprint_rejects_new_generation() { let (_temp, disk, mut info, _original) = fixture().await; @@ -3246,6 +3773,431 @@ mod tests { assert!(witness.original_digest.is_none() && witness.target_digest.is_none() && witness.data_dirs.is_empty()); } + #[tokio::test] + async fn delete_recovery_raw_discovery_keeps_malformed_records_and_later_keys() { + let (temp, disk, _, _) = fixture().await; + let transaction = Uuid::new_v4(); + apply(&disk, transaction).await; + let bad = temp + .path() + .join("recovery/object") + .join(transaction.to_string()) + .join(PREPARED); + fs::write(&bad, b"{invalid journal").await.expect("corrupt prepared record"); + let legacy = Uuid::new_v4(); + let later = temp.path().join("recovery/later").join(legacy.to_string()); + fs::create_dir_all(&later).await.expect("another hidden key"); + fs::write(later.join(STORAGE_FORMAT_FILE_BACKUP), b"legacy backup") + .await + .expect("later transaction"); + let mut cursor = None; + let mut found = BTreeSet::new(); + loop { + let page = disk + .discover_delete_transactions("recovery", None, cursor, 1) + .await + .expect("bounded discovery continues"); + found.extend(page.transactions.into_iter().map(|id| (id.object, id.transaction))); + cursor = page.next; + if let Some(cursor) = cursor.as_ref() { + assert!(!disk.renew_delete_recovery_cursor("different-bucket", cursor)); + assert!(disk.renew_delete_recovery_cursor("recovery", cursor)); + } + if cursor.is_none() { + break; + } + } + assert!(found.contains(&("object".to_owned(), transaction))); + assert!(found.contains(&("later".to_owned(), legacy))); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .expect_err("inspection still rejects corruption"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(bad).await.expect("corrupt evidence retained"), b"{invalid journal"); + assert_eq!( + fs::read(later.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("legacy evidence retained"), + b"legacy backup" + ); + } + + async fn prepare_discovery_key( + disk: &LocalDisk, + object: &str, + transaction: Uuid, + metadata: &[u8], + ) -> DeleteRecoveryInspection { + let root = disk.io_get_object_path("recovery", object).expect("discovery fixture path"); + fs::create_dir_all(&root).await.expect("discovery fixture directory"); + fs::write(root.join(STORAGE_FORMAT_FILE), metadata) + .await + .expect("discovery fixture metadata"); + let mut request = request(); + request.name = object.to_owned(); + disk.delete_version_recoverable( + "recovery", + object, + request, + false, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await + .expect("prepare discoverable transaction"); + disk.inspect_delete_transaction("recovery", object, transaction) + .await + .expect("healthy prepared transaction") + } + + async fn discover_all_keys(disk: &LocalDisk, object: Option<&str>) -> BTreeSet<(String, Uuid)> { + let mut cursor = None; + let mut found = BTreeSet::new(); + for _ in 0..64 { + let page = disk + .discover_delete_transactions("recovery", object, cursor, 1) + .await + .expect("record corruption must not stop bounded discovery"); + found.extend(page.transactions.into_iter().map(|id| (id.object, id.transaction))); + cursor = page.next; + if cursor.is_none() { + return found; + } + } + panic!("discovery did not finish the small fixture within its page bound"); + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_isolates_corrupt_record_bindings() { + for record_name in [PREPARED, DELETE_QUORUM_COMMITTED_FILE] { + for damage in ["protocol", "volume", "transaction", "object", "unsafe_object", "invalid_json"] { + let (temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let prepared = apply(&disk, transaction).await; + let root = temp.path().join("recovery/object").join(transaction.to_string()); + if record_name == DELETE_QUORUM_COMMITTED_FILE { + disk.settle_delete_transaction( + "recovery", + "object", + transaction, + prepared.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("write receipt fixture"); + fs::remove_file(root.join(PREPARED)) + .await + .expect("discover the receipt without its prepared journal"); + } + let later = Uuid::new_v4(); + let later_before = prepare_discovery_key(&disk, "later", later, &original).await; + let path = root.join(record_name); + let mut record: serde_json::Value = + serde_json::from_slice(&fs::read(&path).await.expect("original record")).expect("valid fixture record"); + let journal = if record_name == PREPARED { + &mut record + } else { + record.get_mut("journal").expect("receipt journal") + }; + match damage { + "protocol" => journal["protocol_version"] = serde_json::json!(DELETE_RECOVERY_PROTOCOL_VERSION + 1), + "volume" => journal["volume"] = serde_json::json!("another-bucket"), + "transaction" => journal["transaction"] = serde_json::json!(Uuid::new_v4()), + "object" => journal["object"] = serde_json::json!("another-key"), + "unsafe_object" => journal["object"] = serde_json::json!("../another-key"), + "invalid_json" => {} + _ => unreachable!("fixture damage is exhaustive"), + } + let bytes = if damage == "invalid_json" { + b"{invalid record".to_vec() + } else { + serde_json::to_vec(&record).expect("encode damaged record") + }; + fs::write(&path, &bytes).await.expect("write corrupt record"); + + let found = discover_all_keys(&disk, None).await; + assert_eq!( + found, + BTreeSet::from([("object".to_owned(), transaction), ("later".to_owned(), later)]), + "{record_name}: {damage}" + ); + assert_eq!( + discover_all_keys(&disk, Some("object")).await, + BTreeSet::from([("object".to_owned(), transaction)]) + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .expect_err("canonical inspection must reject corrupt record bindings"), + DiskError::FileCorrupt, + "{record_name}: {damage}" + ); + assert_eq!(fs::read(path).await.expect("retained corrupt evidence"), bytes); + assert_eq!( + disk.inspect_delete_transaction("recovery", "later", later) + .await + .expect("later transaction remains healthy") + .fingerprint, + later_before.fingerprint, + "discovery does not mutate a later healthy transaction" + ); + } + } + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_keeps_corrupt_write_backups() { + for corrupt_current in [false, true] { + let (temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = temp.path().join("recovery/object"); + let backup = root.join(transaction.to_string()).join(STORAGE_FORMAT_FILE_BACKUP); + fs::create_dir_all(backup.parent().expect("rollback directory")) + .await + .expect("rollback directory"); + fs::write(&backup, &original).await.expect("valid write backup"); + let damaged = if corrupt_current { + root.join(STORAGE_FORMAT_FILE) + } else { + backup + }; + fs::write(&damaged, b"corrupt metadata") + .await + .expect("record-local metadata corruption"); + let later = Uuid::new_v4(); + prepare_discovery_key(&disk, "later", later, &original).await; + + assert!(!live_write_backup(&root, transaction).expect("corrupt decoded metadata is not a proven live backup")); + assert_eq!( + discover_all_keys(&disk, None).await, + BTreeSet::from([("object".to_owned(), transaction), ("later".to_owned(), later)]) + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "object", transaction) + .await + .expect_err("inspection rejects corrupt backup metadata"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(damaged).await.expect("corrupt metadata remains untouched"), b"corrupt metadata"); + } + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_preserves_bound_directory_keys_and_exact_scope() { + let (_temp, disk, _, original) = fixture().await; + let directory_transaction = Uuid::new_v4(); + prepare_discovery_key(&disk, "directory/", directory_transaction, &original).await; + let literal_transaction = Uuid::new_v4(); + prepare_discovery_key(&disk, "literal__XLDIR__", literal_transaction, &original).await; + + assert_eq!( + discover_all_keys(&disk, None).await, + BTreeSet::from([ + ("directory/".to_owned(), directory_transaction), + ("literal__XLDIR__".to_owned(), literal_transaction), + ]) + ); + assert_eq!( + discover_all_keys(&disk, Some("directory/")).await, + BTreeSet::from([("directory/".to_owned(), directory_transaction)]) + ); + assert_eq!( + discover_all_keys(&disk, Some("directory")).await, + BTreeSet::from([("directory".to_owned(), directory_transaction)]), + "a journal cannot redirect an exact object scope to a trailing-slash alias" + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "directory", directory_transaction) + .await + .expect_err("the alternate scope is not journal authority"), + DiskError::FileCorrupt + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "directory/", directory_transaction) + .await + .expect("the bound trailing-slash object remains valid") + .state, + DeleteRecoveryState::NewPrepared + ); + + let literal_record = disk + .io_get_object_path("recovery", "literal__XLDIR__") + .expect("literal key path") + .join(literal_transaction.to_string()) + .join(PREPARED); + let mut record: DeleteRecoveryJournal = + serde_json::from_slice(&fs::read(&literal_record).await.expect("literal journal")).expect("valid literal journal"); + record.object = "literal/".to_owned(); + let damaged = serde_json::to_vec(&record).expect("encode invalid literal binding"); + fs::write(&literal_record, &damaged) + .await + .expect("inject invalid literal binding"); + let found = discover_all_keys(&disk, None).await; + assert!(found.contains(&("literal__XLDIR__".to_owned(), literal_transaction))); + assert!( + !found.contains(&("literal/".to_owned(), literal_transaction)), + "physical suffixes are never decoded into authority" + ); + assert_eq!( + disk.inspect_delete_transaction("recovery", "literal__XLDIR__", literal_transaction) + .await + .expect_err("the physical fallback cannot certify an invalid logical binding"), + DiskError::FileCorrupt + ); + assert_eq!(fs::read(literal_record).await.expect("retained invalid literal binding"), damaged); + } + + #[cfg(unix)] + #[tokio::test] + async fn delete_recovery_raw_discovery_keeps_structural_backup_errors_fatal() { + let (temp, disk, _, original) = fixture().await; + let transaction = Uuid::new_v4(); + let root = temp.path().join("recovery/object"); + let rollback = root.join(transaction.to_string()); + fs::create_dir_all(&rollback).await.expect("backup directory"); + fs::write(rollback.join(STORAGE_FORMAT_FILE_BACKUP), &original) + .await + .expect("valid backup metadata"); + let outside = tempfile::tempdir().expect("external target"); + let sentinel = outside.path().join("sentinel"); + fs::write(&sentinel, b"external bytes").await.expect("external sentinel"); + std::os::unix::fs::symlink(&sentinel, rollback.join(TARGET)).expect("structurally invalid delete target"); + + assert!( + live_write_backup(&root, transaction).is_err(), + "only decoded record corruption is isolated" + ); + assert!( + disk.discover_delete_transactions("recovery", Some("object"), None, 128) + .await + .is_err(), + "a structural error must terminate discovery" + ); + assert_eq!(fs::read(sentinel).await.expect("unchanged external sentinel"), b"external bytes"); + assert_eq!( + fs::read(rollback.join(STORAGE_FORMAT_FILE_BACKUP)) + .await + .expect("unchanged backup"), + original + ); + } + + #[tokio::test] + async fn delete_recovery_evidence_probes_do_not_register_public_cursors() { + let (_temp, disk, _, _) = fixture().await; + apply(&disk, Uuid::new_v4()).await; + for _ in 0..256 { + assert!( + disk.delete_recovery_has_evidence("recovery", "object") + .await + .expect("positive private probe") + ); + } + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()), + "early positive probes must not consume any public cursor slots" + ); + } + + #[tokio::test] + async fn delete_recovery_evidence_probe_advances_empty_bounded_pages() { + let (_temp, disk, _, _) = fixture().await; + let path = disk.delete_discovery_path("recovery", Some("object")).expect("probe scope"); + let filler_id = Uuid::new_v4().to_string(); + let filler = path.join(&filler_id); + fs::create_dir_all(&filler).await.expect("non-evidence directory"); + for index in 0..5000 { + fs::write(filler.join(format!("part-{index:04}")), []) + .await + .expect("non-evidence entry"); + } + let session = disk + .new_delete_discovery_session("recovery", Some("object"), &path) + .expect("private session"); + let (first, next) = disk + .advance_delete_discovery(&path, session, 1) + .await + .expect("bounded first step"); + assert!(first.is_empty()); + assert!(next.is_some(), "an empty 4096-entry step must retain real continuation state"); + drop(next); + assert!( + !disk + .delete_recovery_has_evidence("recovery", "object") + .await + .expect("negative probe reaches EOF") + ); + + let transaction = Uuid::new_v4(); + let proof = path.join(transaction.to_string()); + fs::create_dir_all(&proof).await.expect("later evidence directory"); + fs::write(proof.join(PREPARED), b"{invalid journal") + .await + .expect("discoverable record-local corruption"); + let mut session = disk + .new_delete_discovery_session("recovery", Some("object"), &path) + .expect("private late-evidence session"); + // Seed a valid DFS continuation so this assertion is independent of + // filesystem directory ordering: the large empty frame runs first. + session.frames = vec![ + (transaction.to_string(), std::fs::read_dir(&proof).expect("later evidence frame")), + (filler_id, std::fs::read_dir(&filler).expect("empty intermediate frame")), + ]; + assert!( + session + .has_evidence(&disk, &path) + .await + .expect("late evidence survives an empty intermediate page") + ); + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()), + "private continuation never enters the public cursor table" + ); + } + + #[tokio::test] + async fn delete_recovery_cancelled_evidence_probe_drops_private_session() { + let (_temp, disk, _, _) = fixture().await; + apply(&disk, Uuid::new_v4()).await; + let path = disk.delete_discovery_path("recovery", Some("object")).expect("probe scope"); + let lease = os::acquire_namespace_mutation_lease_with_owner(&path, None).await; + let mut probe = Box::pin(disk.delete_recovery_has_evidence("recovery", "object")); + assert!( + futures::poll!(probe.as_mut()).is_pending(), + "the private session waits for the held namespace lease" + ); + drop(probe); + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()), + "cancelling before a page runs must not retain a global cursor" + ); + drop(lease); + assert!( + disk.delete_recovery_has_evidence("recovery", "object") + .await + .expect("later probes remain live") + ); + assert!( + DISCOVERY_SESSIONS + .lock() + .values() + .all(|session| session.disk_root != disk.io_root()) + ); + } + #[tokio::test] async fn delete_recovery_raw_discovery_paginates_and_binds_exact_key_scope() { let (temp, disk, _info, _original) = fixture().await; diff --git a/crates/ecstore/src/disk/local/replacement_lease.rs b/crates/ecstore/src/disk/local/replacement_lease.rs index 2a94654d7..3864c490f 100644 --- a/crates/ecstore/src/disk/local/replacement_lease.rs +++ b/crates/ecstore/src/disk/local/replacement_lease.rs @@ -26,6 +26,42 @@ pub struct ReplacementExecutionLease { // file while it is held would allow a second owner to lock a different inode. const EXECUTION_LOCK_FILE: &str = ".rustfs-replacement.lock"; +/// A compatibility retry may only admit a disk that has not received data. +/// The caller retains the mount descriptor and the replacement executor lease. +pub(crate) async fn is_blank_target(root: PathBuf) -> Result { + tokio::task::spawn_blocking(move || { + let mut directories = vec![(root, 0usize)]; + let mut entries_left = 64usize; + while let Some((directory, depth)) = directories.pop() { + for entry in std::fs::read_dir(directory)? { + let entry = entry?; + let Some(remaining) = entries_left.checked_sub(1) else { + return Ok(false); + }; + entries_left = remaining; + let kind = entry.file_type()?; + let name = entry.file_name(); + if depth == 0 && name == EXECUTION_LOCK_FILE && kind.is_file() { + if entry.metadata()?.len() != 0 { + return Ok(false); + } + } else if depth == 1 && name == crate::disk::HEALING_MARKER_PATH && kind.is_file() { + if entry.metadata()?.len() > 256 { + return Ok(false); + } + } else if kind.is_dir() && depth < 4 && (depth > 0 || name == crate::disk::RUSTFS_META_BUCKET) { + directories.push((entry.path(), depth + 1)); + } else { + return Ok(false); + } + } + } + Ok(true) + }) + .await + .map_err(DiskError::from)? +} + pub(super) async fn acquire(root: PathBuf) -> Result> { #[cfg(unix)] { @@ -56,6 +92,53 @@ pub(super) async fn acquire(root: PathBuf) -> Result String { use sha2::{Digest, Sha256}; @@ -130,7 +133,7 @@ pub(crate) fn quota_mutation_fence_path(bucket: &str, object: &str) -> String { ) } -pub(crate) fn is_quota_mutation_fence_path(path: &str) -> bool { +pub fn is_quota_mutation_fence_path(path: &str) -> bool { path.strip_prefix(QUOTA_MUTATION_FENCE_PREFIX) .is_some_and(|digest| digest.len() == 64 && digest.bytes().all(|byte| byte.is_ascii_hexdigit())) } @@ -230,6 +233,14 @@ pub enum Disk { } impl Disk { + /// A read-only probe; the physical delete repeats the same leaf checks. + pub async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result { + match self { + Self::Local(disk) => disk.get_disk().is_plain_orphan_data_dir(volume, path).await, + Self::Remote(disk) => disk.is_plain_orphan_data_dir(volume, path).await, + } + } + pub(crate) async fn delete_versions_with_namespace_owner( &self, volume: &str, @@ -346,6 +357,25 @@ impl Disk { } } + pub(crate) async fn delete_versions_guarded_recoverable_with_namespace_owner( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + namespace_owner: Option>, + ) -> Vec> { + match self { + Self::Local(disk) => { + disk.delete_versions_guarded_with_namespace_owner(volume, versions, opts, namespace_owner) + .await + } + Self::Remote(disk) => { + let _owner = namespace_owner; + disk.delete_versions_guarded_recoverable(volume, versions, opts).await + } + } + } + pub(crate) async fn cached_disk_id(&self) -> Option { match self { Disk::Local(local_disk) => local_disk.get_current_disk_id().await, @@ -356,6 +386,21 @@ impl Disk { #[async_trait::async_trait] impl DiskAPI for Disk { + async fn supports_guarded_delete_versions(&self) -> Result<()> { + match self { + Self::Local(disk) => disk.supports_guarded_delete_versions().await, + Self::Remote(disk) => disk.supports_guarded_delete_versions().await, + } + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + self.delete_versions_guarded_recoverable_with_namespace_owner(volume, versions, opts, None) + .await + } async fn delete_version_recoverable( &self, volume: &str, @@ -378,6 +423,12 @@ impl DiskAPI for Disk { Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await, } } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> Result { + match self { + Self::Local(disk) => disk.has_delete_transaction_evidence(volume, object).await, + Self::Remote(disk) => disk.has_delete_transaction_evidence(volume, object).await, + } + } async fn discover_delete_transactions( &self, volume: &str, @@ -621,6 +672,7 @@ impl DiskAPI for Disk { } #[tracing::instrument(level = "trace", skip_all)] + #[track_caller] async fn read_version( &self, _org_volume: &str, @@ -629,10 +681,20 @@ impl DiskAPI for Disk { version_id: &str, opts: &ReadOptions, ) -> Result { - match self { - Disk::Local(local_disk) => local_disk.read_version(_org_volume, volume, path, version_id, opts).await, - Disk::Remote(remote_disk) => remote_disk.read_version(_org_volume, volume, path, version_id, opts).await, + let operation = async { + match self { + Disk::Local(local_disk) => local_disk.read_version(_org_volume, volume, path, version_id, opts).await, + Disk::Remote(remote_disk) => remote_disk.read_version(_org_volume, volume, path, version_id, opts).await, + } + }; + #[cfg(feature = "diagnostic-read-version-caller")] + if crate::cluster::rpc::runtime_sources::current_read_version_request_caller().is_none() { + let caller = crate::cluster::rpc::runtime_sources::read_version_request_caller_from_location( + std::panic::Location::caller().file(), + ); + return crate::cluster::rpc::runtime_sources::with_read_version_request_caller(caller, operation).await; } + operation.await } #[tracing::instrument(level = "trace", skip_all)] @@ -1239,6 +1301,14 @@ impl Disk { Self::Remote(_) => Err(DiskError::other("replacement execution requires a local target")), } } + + pub(crate) async fn is_blank_replacement_target(&self) -> Result { + let Some(root) = self.replacement_mount_lease_root() else { + return Ok(false); + }; + let blank = local::is_blank_replacement_target(root).await?; + Ok(blank && self.has_replacement_mount_lease()) + } } pub async fn new_disk(ep: &Endpoint, opt: &DiskOption) -> Result { @@ -1324,6 +1394,7 @@ pub trait DiskAPI: Debug + Send + Sync + 'static { } async fn write_metadata(&self, org_volume: &str, volume: &str, path: &str, fi: FileInfo) -> Result<()>; async fn update_metadata(&self, volume: &str, path: &str, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()>; + #[track_caller] async fn read_version( &self, org_volume: &str, @@ -1409,6 +1480,11 @@ pub trait DiskAPI: Debug + Send + Sync + 'static { Err(DiskError::MethodNotAllowed) } async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>; + /// Exact-key existence owns its traversal state; unsupported peers cannot + /// establish the absence of recovery obligations. + async fn has_delete_transaction_evidence(&self, _volume: &str, _object: &str) -> Result { + Err(DiskError::MethodNotAllowed) + } async fn discover_delete_transactions( &self, _volume: &str, @@ -1418,6 +1494,18 @@ pub trait DiskAPI: Debug + Send + Sync + 'static { ) -> Result { Err(DiskError::MethodNotAllowed) } + /// A read-only capability check; unsupported implementations must fail closed. + async fn supports_guarded_delete_versions(&self) -> Result<()> { + Err(DiskError::GuardedDeleteUnsupported) + } + async fn delete_versions_guarded_recoverable( + &self, + _volume: &str, + versions: Vec, + _opts: DeleteOptions, + ) -> Vec> { + vec![Some(DiskError::GuardedDeleteUnsupported); versions.len()] + } async fn delete_version_recoverable( &self, _volume: &str, @@ -1492,13 +1580,15 @@ where { validate_batch_read_version_item_count(req.items.len())?; let parallelism = batch_read_version_server_parallelism(); + #[cfg(feature = "diagnostic-read-version-caller")] + let item_count = req.items.len(); let mut responses = stream::iter(req.items.into_iter().enumerate()) .map(|(index, item)| async move { - match disk - .read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts) - .await - { + let read = disk.read_version(&item.org_volume, &item.volume, &item.path, &item.version_id, &req.opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_batch_read_version_item_context(index, item_count, read); + match read.await { Ok(file_info) => BatchReadVersionResp { index, path: item.path, @@ -1648,6 +1738,117 @@ pub struct FileInfoVersions { pub free_versions: Vec, } +/// The durable remote location and restore state of the observed generation. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CurrentDeleteTransition { + pub status: String, + pub tier: String, + pub remote_object: String, + pub remote_version: Option, + pub version_state: rustfs_filemeta::TransitionVersionState, + pub destination_id: Option<[u8; 32]>, + pub remote_only: bool, +} + +/// The current object generation observed before a guarded batch delete. +/// This transient value is never persisted in object metadata. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub enum CurrentDeleteGeneration { + Absent, + Present { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + #[serde(default)] + transition: Option>, + }, +} + +impl CurrentDeleteGeneration { + pub fn from_file_info(fi: &FileInfo) -> Result { + let mod_time = fi.mod_time.ok_or(DiskError::FileCorrupt)?; + let etag = fi.get_etag().unwrap_or_default(); + let data_dir = fi.data_dir.filter(|id| !id.is_nil()); + let remote_only = fi.is_remote(); + if !fi.deleted && (etag.is_empty() || (data_dir.is_none() && !remote_only)) { + return Err(DiskError::FileCorrupt); + } + if remote_only && (fi.transition_tier.is_empty() || fi.transitioned_objname.is_empty()) { + return Err(DiskError::FileCorrupt); + } + let destination_id = + crate::services::tier::tier::tier_destination_id_from_metadata(&fi.metadata).map_err(|_| DiskError::FileCorrupt)?; + let remote_version = match fi.transition_version.as_deref() { + Some(value) => (!value.is_empty() && !Uuid::parse_str(value).is_ok_and(|id| id.is_nil())).then(|| value.to_owned()), + None => fi.transition_version_id.filter(|id| !id.is_nil()).map(|id| id.to_string()), + }; + let transition = (!fi.transition_status.is_empty() + || !fi.transition_tier.is_empty() + || !fi.transitioned_objname.is_empty() + || remote_version.is_some() + || fi.transition_version_state != rustfs_filemeta::TransitionVersionState::Unknown + || destination_id.is_some()) + .then(|| { + Box::new(CurrentDeleteTransition { + status: fi.transition_status.clone(), + tier: fi.transition_tier.clone(), + remote_object: fi.transitioned_objname.clone(), + remote_version, + version_state: fi.transition_version_state, + destination_id, + remote_only, + }) + }); + Ok(Self::Present { + version_id: fi.version_id.filter(|id| !id.is_nil()), + data_dir, + mod_time, + delete_marker: fi.deleted, + etag, + transition, + }) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GuardedFileInfoVersions { + pub group: FileInfoVersions, + pub expected_current: CurrentDeleteGeneration, +} + +#[derive(Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct GuardedDeleteVersionsPayload { + pub versions: Vec, + pub opts: DeleteOptions, +} + +impl GuardedDeleteVersionsPayload { + pub fn validate(&self) -> Result<()> { + validate_delete_recovery_options(&self.opts)?; + if self.versions.is_empty() + || self.versions.len() > 1000 + || self + .versions + .iter() + .try_fold(0_usize, |count, version| count.checked_add(version.group.versions.len())) + .is_none_or(|count| count > 1000) + || self + .versions + .iter() + .any(|version| version.group.name.starts_with('/') || version.group.versions.is_empty()) + { + return Err(DiskError::InvalidPath); + } + Ok(()) + } +} + impl FileInfoVersions { pub fn find_version_index(&self, v: &str) -> Option { if v.is_empty() { @@ -1776,6 +1977,14 @@ pub struct RenameDataResp { /// "cannot vote", never as `Absent`. #[serde(default)] pub old_current_size: Option, + /// Whether this disk decoded enough of the overwritten current version to + /// decide if a transitioned source has to be returned. + #[serde(default)] + pub old_current_source_checked: bool, + /// Transitioned old-current source used by PUT free-version cleanup receipt. + /// Empty when the old current is not transitioned or no source was known. + #[serde(default)] + pub old_current_source: Option, } #[derive(Debug, Clone, Default, Serialize, Deserialize)] @@ -1794,6 +2003,10 @@ pub struct DeleteOptions { pub delete_recovery_bucket_incarnation: Option, #[serde(default)] pub recoverable_delete: bool, + /// Reclaim only plain erasure-part leaves, preserving every other entry. + /// Keep `recursive` false so an older peer cannot widen this operation. + #[serde(default)] + pub orphan_data_dir: bool, } pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1; @@ -1824,6 +2037,7 @@ pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<( || opts.undo_delete || opts.recursive || opts.immediate + || opts.orphan_data_dir || opts.expected_delete_marker.is_some() || opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil()) { @@ -2003,6 +2217,9 @@ pub enum DeleteRecoveryAction { #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub enum DeleteRecoveryRpcOperation { + InspectOrphanDataDir { + path: String, + }, Discover { object: Option, cursor: Option, @@ -2028,6 +2245,9 @@ pub enum DeleteRecoveryRpcOperation { versions: Vec, opts: DeleteOptions, }, + Exists { + object: String, + }, } impl DeleteRecoveryRpcOperation { @@ -2048,11 +2268,13 @@ impl DeleteRecoveryRpcOperation { #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub enum DeleteRecoveryRpcReply { + OrphanDataDir { plain: bool }, Discovered(DeleteRecoveryPage), Inspected(DeleteRecoveryInspection), Settled(DeleteRecoveryInspection), AppliedVersion, AppliedVersions(Vec>), + Evidence { present: bool }, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -2168,6 +2390,243 @@ pub fn count_part_not_success(part_errs: &[usize]) -> usize { #[cfg(test)] mod tests { use super::*; + + #[test] + fn guarded_delete_generation_boxing_preserves_legacy_wire_bytes() { + #[derive(Serialize)] + struct LegacyPresent { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + transition: Option, + } + + assert!(std::mem::size_of::() < std::mem::size_of::()); + let mut fi = FileInfo { + mod_time: Some(OffsetDateTime::UNIX_EPOCH), + data_dir: Some(Uuid::from_u128(1)), + ..Default::default() + }; + fi.metadata.insert("etag".into(), "opaque-etag".into()); + let plain = CurrentDeleteGeneration::from_file_info(&fi).expect("local generation"); + fi.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + fi.transition_tier = "WARM".into(); + fi.transitioned_objname = "remote/object".into(); + fi.transition_version = Some("remote-version".into()); + let remote = CurrentDeleteGeneration::from_file_info(&fi).expect("transition generation"); + for current in [CurrentDeleteGeneration::Absent, plain, remote] { + let (legacy_messagepack, legacy_json) = match ¤t { + CurrentDeleteGeneration::Absent => ( + rmp_serde::to_vec_named("Absent").expect("legacy absent MessagePack"), + serde_json::to_vec("Absent").expect("legacy absent JSON"), + ), + CurrentDeleteGeneration::Present { + version_id, + data_dir, + mod_time, + delete_marker, + etag, + transition, + } => { + let legacy = std::collections::BTreeMap::from([( + "Present", + LegacyPresent { + version_id: *version_id, + data_dir: *data_dir, + mod_time: *mod_time, + delete_marker: *delete_marker, + etag: etag.clone(), + transition: transition.as_deref().cloned(), + }, + )]); + ( + rmp_serde::to_vec_named(&legacy).expect("legacy inline MessagePack"), + serde_json::to_vec(&legacy).expect("legacy inline JSON"), + ) + } + }; + assert_eq!(rmp_serde::to_vec_named(¤t).expect("current MessagePack"), legacy_messagepack); + assert_eq!(serde_json::to_vec(¤t).expect("current JSON"), legacy_json); + assert_eq!( + rustfs_protos::decode_delete_recovery_payload::(&legacy_messagepack) + .expect("existing payload remains readable"), + current + ); + } + } + + #[test] + fn guarded_delete_generation_requires_reliable_current_identity() { + let mut fi = FileInfo::default(); + assert_eq!(CurrentDeleteGeneration::from_file_info(&fi), Err(DiskError::FileCorrupt)); + fi.mod_time = Some(OffsetDateTime::UNIX_EPOCH); + fi.data_dir = Some(Uuid::new_v4()); + assert_eq!(CurrentDeleteGeneration::from_file_info(&fi), Err(DiskError::FileCorrupt)); + fi.metadata.insert("etag".into(), "opaque-etag".into()); + fi.version_id = Some(Uuid::nil()); + assert!(matches!( + CurrentDeleteGeneration::from_file_info(&fi), + Ok(CurrentDeleteGeneration::Present { version_id: None, .. }) + )); + fi.data_dir = Some(Uuid::nil()); + assert_eq!(CurrentDeleteGeneration::from_file_info(&fi), Err(DiskError::FileCorrupt)); + } + + #[test] + fn guarded_delete_generation_accepts_remote_only_and_legacy_unversioned_identity() { + let mut fi = FileInfo { + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)), + transition_status: rustfs_filemeta::TRANSITION_COMPLETE.to_owned(), + transition_tier: "WARM".into(), + transitioned_objname: "remote/object".into(), + ..Default::default() + }; + fi.metadata.insert("etag".into(), "opaque-etag".into()); + for version in [None, Some(String::new()), Some(Uuid::nil().to_string())] { + fi.transition_version = version; + for data_dir in [None, Some(Uuid::nil())] { + fi.data_dir = data_dir; + assert!( + matches!( + CurrentDeleteGeneration::from_file_info(&fi), + Ok(CurrentDeleteGeneration::Present { + data_dir: None, + transition: Some(transition), + .. + }) if transition.remote_version.is_none() + && transition.version_state == rustfs_filemeta::TransitionVersionState::Unknown + && transition.destination_id.is_none() + && transition.remote_only + ), + "legacy unversioned remote metadata remains usable" + ); + } + } + fi.transition_version = None; + fi.transition_version_id = Some(Uuid::nil()); + CurrentDeleteGeneration::from_file_info(&fi).expect("nil legacy remote UUID means no remote version"); + fi.transition_status.clear(); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "local payload metadata still requires a data directory" + ); + fi.transition_status = rustfs_filemeta::TRANSITION_COMPLETE.to_owned(); + fi.transition_tier.clear(); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "remote-only identity requires its tier" + ); + fi.transition_tier = "WARM".into(); + fi.transitioned_objname.clear(); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "remote-only identity requires its remote object" + ); + } + + #[test] + fn guarded_delete_remote_transition_tuple_survives_rpc_codec() { + use rustfs_utils::http::metadata_compat::{SUFFIX_TRANSITION_TIER_DESTINATION_ID, insert_str}; + let mut fi = FileInfo { + name: "object".into(), + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)), + transition_status: rustfs_filemeta::TRANSITION_COMPLETE.to_owned(), + transition_tier: "WARM".into(), + transitioned_objname: "opaque/object+name".into(), + transition_version: Some("opaque/version+id==".into()), + transition_version_state: rustfs_filemeta::TransitionVersionState::Exact, + ..Default::default() + }; + fi.metadata.insert("etag".into(), "opaque-etag".into()); + insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32)); + let generation = CurrentDeleteGeneration::from_file_info(&fi).expect("capture complete remote identity"); + let payload = GuardedDeleteVersionsPayload { + versions: vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".into(), + versions: vec![fi.clone()], + ..Default::default() + }, + expected_current: generation.clone(), + }], + opts: DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + }; + let encoded = rmp_serde::to_vec_named(&payload).expect("encode remote guarded request"); + let decoded: GuardedDeleteVersionsPayload = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode remote guarded request"); + assert_eq!(decoded.versions[0].expected_current, generation); + assert!( + matches!(&generation, CurrentDeleteGeneration::Present { transition: Some(transition), .. } + if transition.status == rustfs_filemeta::TRANSITION_COMPLETE + && transition.tier == "WARM" + && transition.remote_object == "opaque/object+name" + && transition.remote_version.as_deref() == Some("opaque/version+id==") + && transition.version_state == rustfs_filemeta::TransitionVersionState::Exact + && transition.destination_id == Some([0xab; 32]) + && transition.remote_only) + ); + insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, format!("a{}", "€".repeat(21))); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "64-byte non-ASCII backend identities are rejected without panicking" + ); + insert_str(&mut fi.metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID, "ab".repeat(32)); + fi.metadata + .insert(format!("x-minio-internal-{SUFFIX_TRANSITION_TIER_DESTINATION_ID}"), "cd".repeat(32)); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&fi), + Err(DiskError::FileCorrupt), + "conflicting backend identities fail closed" + ); + } + + #[test] + fn guarded_delete_payload_rejects_malformed_or_unbounded_batches() { + let group = GuardedFileInfoVersions { + group: FileInfoVersions { + name: "object".into(), + versions: vec![FileInfo::default()], + ..Default::default() + }, + expected_current: CurrentDeleteGeneration::Absent, + }; + let mut payload = GuardedDeleteVersionsPayload { + versions: vec![group], + opts: DeleteOptions { + old_data_dir: Some(Uuid::new_v4()), + ..Default::default() + }, + }; + payload.validate().expect("valid guarded batch"); + let encoded = rmp_serde::to_vec_named(&payload).expect("serialize guarded batch"); + let decoded: GuardedDeleteVersionsPayload = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode guarded batch"); + decoded.validate().expect("round-trip guarded batch remains valid"); + let mut trailing = encoded; + trailing.push(0xc0); + assert!(rustfs_protos::decode_delete_recovery_payload::(&trailing).is_err()); + let mut unknown = serde_json::to_value(&payload).expect("serialize guarded batch to map"); + unknown + .as_object_mut() + .expect("payload is a map") + .insert("unrecognized".into(), true.into()); + assert!(serde_json::from_value::(unknown).is_err()); + payload.versions[0].group.versions.resize_with(1001, FileInfo::default); + assert_eq!(payload.validate(), Err(DiskError::InvalidPath)); + payload.versions[0].group.versions.truncate(1); + payload.opts.undo_delete = true; + assert_eq!(payload.validate(), Err(DiskError::FileCorrupt)); + } + use endpoint::Endpoint; use local::LocalDisk; use tokio::fs; @@ -2345,6 +2804,7 @@ mod tests { expected_delete_marker: None, delete_recovery_bucket_incarnation: None, recoverable_delete: false, + orphan_data_dir: false, }; assert!(opts.recursive); @@ -2517,6 +2977,8 @@ mod tests { cleanup_data_dir: Some(uuid), sign: Some(signature.clone()), old_current_size: Some(OldCurrentSize::Present(42)), + old_current_source_checked: false, + old_current_source: None, }; assert_eq!(resp.old_data_dir, Some(uuid)); @@ -2537,6 +2999,8 @@ mod tests { cleanup_data_dir: Some(Uuid::new_v4()), sign: Some(vec![0x01, 0x02, 0x03]), old_current_size, + old_current_source_checked: false, + old_current_source: None, }; let encoded = rmp_serde::encode::to_vec_named(&resp).expect("named msgpack should encode"); @@ -2574,6 +3038,8 @@ mod tests { assert_eq!(decoded.cleanup_data_dir, None); assert_eq!(decoded.sign, legacy.sign); assert_eq!(decoded.old_current_size, None); + assert!(!decoded.old_current_source_checked); + assert!(decoded.old_current_source.is_none()); } /// Test constants @@ -2792,6 +3258,88 @@ mod delete_recovery_timeout_tests { mod delete_recovery_wire_limit_tests { use super::*; + #[test] + fn boxed_current_delete_transition_preserves_existing_wire_payload() { + #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] + #[serde(deny_unknown_fields)] + enum PriorGeneration { + Present { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + #[serde(default)] + transition: Option, + }, + } + + let version_id = Some(Uuid::from_u128(1)); + let data_dir = Some(Uuid::from_u128(2)); + let mod_time = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1); + for transition in [ + None, + Some(CurrentDeleteTransition { + status: rustfs_filemeta::TRANSITION_COMPLETE.to_owned(), + tier: "WARM".into(), + remote_object: "opaque/object+name".into(), + remote_version: Some("opaque/version+id==".into()), + version_state: rustfs_filemeta::TransitionVersionState::Exact, + destination_id: Some([0xab; 32]), + remote_only: true, + }), + ] { + let prior = PriorGeneration::Present { + version_id, + data_dir, + mod_time, + delete_marker: false, + etag: "opaque-etag".into(), + transition: transition.clone(), + }; + let generation = CurrentDeleteGeneration::Present { + version_id, + data_dir, + mod_time, + delete_marker: false, + etag: "opaque-etag".into(), + transition: transition.map(Box::new), + }; + let prior_encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed generation shape"); + let encoded = rmp_serde::to_vec_named(&generation).expect("encode boxed transition generation"); + assert_eq!(encoded, prior_encoded); + let decoded: CurrentDeleteGeneration = + rustfs_protos::decode_delete_recovery_payload(&prior_encoded).expect("decode the previous generation payload"); + assert_eq!(decoded, generation); + let prior_decoded: PriorGeneration = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode boxed transition with the previous shape"); + assert_eq!(prior_decoded, prior); + if matches!(generation, CurrentDeleteGeneration::Present { transition: None, .. }) { + #[derive(Serialize)] + enum GenerationWithoutTransition { + Present { + version_id: Option, + data_dir: Option, + mod_time: OffsetDateTime, + delete_marker: bool, + etag: String, + }, + } + let legacy = GenerationWithoutTransition::Present { + version_id, + data_dir, + mod_time, + delete_marker: false, + etag: "opaque-etag".into(), + }; + let legacy_encoded = rmp_serde::to_vec_named(&legacy).expect("encode a generation without the transition field"); + let legacy_decoded: CurrentDeleteGeneration = + rustfs_protos::decode_delete_recovery_payload(&legacy_encoded).expect("default an omitted transition field"); + assert_eq!(legacy_decoded, generation); + } + } + } + #[test] fn boxed_apply_version_preserves_existing_wire_payload() { #[derive(Serialize)] @@ -2949,3 +3497,87 @@ mod delete_recovery_adoption_tests { assert!(!journal.matches_adoption_source(&journal.original_versions)); } } + +#[cfg(test)] +mod orphan_data_dir_protocol_tests { + use super::*; + + #[test] + fn delete_recovery_exists_roundtrips_without_cursor_ownership() { + let operation = DeleteRecoveryRpcOperation::Exists { + object: "object".to_owned(), + }; + assert_eq!( + operation.max_wire_payload_bytes(), + rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES + ); + let encoded = rmp_serde::to_vec_named(&operation).expect("encode exact-key existence probe"); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).expect("decode existence probe"), + DeleteRecoveryRpcOperation::Exists { object } if object == "object" + )); + for present in [false, true] { + let encoded = rmp_serde::to_vec_named(&DeleteRecoveryRpcReply::Evidence { present }).expect("encode evidence reply"); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).expect("decode evidence reply"), + DeleteRecoveryRpcReply::Evidence { present: decoded } if decoded == present + )); + } + #[derive(Deserialize)] + enum OlderOperation { + Discover { + _object: Option, + _cursor: Option, + _limit: u32, + }, + } + assert!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).is_err(), + "an older peer must reject existence probing instead of reporting absence" + ); + } + + #[test] + fn orphan_probe_roundtrips_without_mutation_options() { + let path = format!("object/{}", Uuid::new_v4()); + let operation = DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: path.clone() }; + assert_eq!( + operation.max_wire_payload_bytes(), + rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES + ); + let encoded = rmp_serde::to_vec_named(&operation).expect("encode orphan probe"); + let decoded: DeleteRecoveryRpcOperation = rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode probe"); + assert!(matches!(decoded, DeleteRecoveryRpcOperation::InspectOrphanDataDir { path: decoded } if decoded == path)); + let reply = rmp_serde::to_vec_named(&DeleteRecoveryRpcReply::OrphanDataDir { plain: true }).expect("encode probe reply"); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&reply).expect("decode probe reply"), + DeleteRecoveryRpcReply::OrphanDataDir { plain: true } + )); + #[derive(Deserialize)] + enum OlderOperation { + Inspect { _object: String, _transaction: Uuid }, + } + assert!( + rustfs_protos::decode_delete_recovery_payload::(&encoded).is_err(), + "older peers must reject the additive probe" + ); + } + + #[test] + fn orphan_delete_is_nonrecursive_when_older_peer_ignores_new_option() { + #[derive(Deserialize)] + struct OlderDeleteOptions { + recursive: bool, + immediate: bool, + } + let opts = DeleteOptions { + orphan_data_dir: true, + immediate: true, + ..Default::default() + }; + let encoded = serde_json::to_vec(&opts).expect("encode leaf deletion"); + let old: OlderDeleteOptions = serde_json::from_slice(&encoded).expect("older options ignore additive field"); + assert!(!old.recursive, "ignoring the new option must never broaden deletion"); + assert!(old.immediate); + } +} diff --git a/crates/ecstore/src/disk/os.rs b/crates/ecstore/src/disk/os.rs index 14a56b598..0f3e3f8a7 100644 --- a/crates/ecstore/src/disk/os.rs +++ b/crates/ecstore/src/disk/os.rs @@ -486,18 +486,25 @@ pub(crate) mod prepare_rename_test_hooks { /// Fsync a directory so recently created or renamed entries survive power loss. /// No-op on non-Unix platforms where directories cannot be opened for syncing. pub fn fsync_dir_std(dir: impl AsRef) -> io::Result<()> { + fsync_directory_handle_std(dir.as_ref(), None) +} + +pub(crate) fn fsync_directory_handle_std(dir: &Path, opened: Option<&std::fs::File>) -> io::Result<()> { #[cfg(test)] - fsync_dir_recorder::record(dir.as_ref()); + fsync_dir_recorder::record(dir); #[cfg(unix)] { #[cfg(test)] - if let Some(kind) = fsync_dir_recorder::take_failure(dir.as_ref()) { + if let Some(kind) = fsync_dir_recorder::take_failure(dir) { return Err(io::Error::from(kind)); } - std::fs::File::open(dir.as_ref())?.sync_all()?; + match opened { + Some(file) => file.sync_all()?, + None => std::fs::File::open(dir)?.sync_all()?, + } } #[cfg(not(unix))] - let _ = dir; + let _ = (dir, opened); Ok(()) } @@ -670,6 +677,8 @@ fn file_fdatasync_group_commit_wait() -> Duration { #[derive(Clone, Eq, Hash, PartialEq)] struct DstDirFsyncGroupKey { canonical_path: PathBuf, + // A batch must execute in the same resource class as its producers. + isolated: bool, #[cfg(unix)] dev: u64, #[cfg(unix)] @@ -686,13 +695,17 @@ impl DstDirFsyncGroupKey { use std::os::unix::fs::MetadataExt; Ok(Self { canonical_path, + isolated: FSYNC_ON_CURRENT_RUNTIME.get(), dev: metadata.dev(), ino: metadata.ino(), }) } #[cfg(not(unix))] { - Ok(Self { canonical_path }) + Ok(Self { + canonical_path, + isolated: FSYNC_ON_CURRENT_RUNTIME.get(), + }) } } } @@ -729,6 +742,17 @@ impl OpenedDstDirFsyncGroup { Ok(Self { key, dir }) } } + + #[cfg(target_os = "linux")] + fn from_file(dir: &Path, file: Arc) -> io::Result { + let key = DstDirFsyncGroupKey::from_metadata(dir.canonicalize()?, file.metadata()?)?; + Ok(Self { + #[cfg(test)] + dir: key.canonical_path.clone(), + key, + dir_file: file, + }) + } } struct DstDirFsyncWaiter { @@ -1002,19 +1026,45 @@ fn run_dst_dir_fsync_group_worker(group: Arc) -> impl std::fut } } +#[cfg(any(test, not(target_os = "linux")))] async fn fsync_dst_dir_group_commit_with_enabled( dir: impl AsRef, enabled: bool, namespace_owner: Option>, +) -> io::Result<()> { + fsync_dst_dir_group_commit_opened(dir.as_ref(), enabled, namespace_owner, None).await +} + +async fn fsync_dst_dir_group_commit_opened( + dir: &Path, + enabled: bool, + namespace_owner: Option>, + opened: Option>, ) -> io::Result<()> { if !enabled { - return fsync_dir_with_owner(dir.as_ref(), namespace_owner).await; + if let Some(file) = opened { + let dir = dir.to_path_buf(); + return fsync_spawn_blocking(move || { + let _owner = namespace_owner; + fsync_directory_handle_std(&dir, Some(&file)) + }) + .await?; + } + return fsync_dir_with_owner(dir, namespace_owner).await; } - let dir = dir.as_ref().to_path_buf(); - let opened = tokio::task::spawn_blocking(move || OpenedDstDirFsyncGroup::open(&dir)) - .await - .map_err(|err| io::Error::other(format!("blocking dst dir group open failed: {err}")))??; + let dir = dir.to_path_buf(); + let opened = tokio::task::spawn_blocking(move || { + #[cfg(target_os = "linux")] + if let Some(file) = opened { + return OpenedDstDirFsyncGroup::from_file(&dir, file); + } + #[cfg(not(target_os = "linux"))] + let _ = opened; + OpenedDstDirFsyncGroup::open(&dir) + }) + .await + .map_err(|err| io::Error::other(format!("blocking dst dir group open failed: {err}")))??; let (result_rx, worker) = DST_DIR_FSYNC_GROUP_COMMIT.enqueue_opened(opened, namespace_owner)?; if let Some(group) = worker { tokio::spawn(run_dst_dir_fsync_group_worker(group)); @@ -1027,6 +1077,49 @@ async fn fsync_dst_dir_group_commit_with_enabled( } } +/// Reuse the published directory without bypassing group commit or sync admission. +pub(crate) async fn fsync_commit_directory( + dir: &Path, + guard: &RenameCommitGuard, + lease: Arc, + admission: Option<&FileSyncAdmission>, +) -> io::Result<()> { + #[cfg(target_os = "linux")] + { + if dir != guard.directories.destination_path { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "sync directory does not match commit guard")); + } + let file = guard + .directories + .destination + .get() + .ok_or_else(|| io::Error::other("commit directory was not published"))? + .file + .clone(); + if !dst_dir_fsync_group_commit_enabled() + && let Some(admission) = admission + { + let dir = dir.to_path_buf(); + return run_blocking_namespace_file_sync_operation(lease, admission, move || { + #[cfg(test)] + fsync_dir_recorder::record_limited(&dir); + fsync_directory_handle_std(&dir, Some(&file)) + }) + .await; + } + fsync_dst_dir_group_commit_opened(dir, dst_dir_fsync_group_commit_enabled(), Some(lease), Some(file)).await + } + #[cfg(not(target_os = "linux"))] + { + let _ = guard; + match admission { + Some(admission) => fsync_dst_dir_group_commit_or_namespace_file_sync_limit(dir, lease, admission).await, + None => fsync_dst_dir_group_commit(dir, Some(lease)).await, + } + } +} + +#[cfg(not(target_os = "linux"))] pub(crate) async fn fsync_dst_dir_group_commit( dir: impl AsRef, namespace_owner: Option>, @@ -1034,6 +1127,7 @@ pub(crate) async fn fsync_dst_dir_group_commit( fsync_dst_dir_group_commit_with_enabled(dir, dst_dir_fsync_group_commit_enabled(), namespace_owner).await } +#[cfg(not(target_os = "linux"))] pub(crate) async fn fsync_dst_dir_group_commit_or_namespace_file_sync_limit( dir: impl AsRef, lease: Arc, @@ -1061,7 +1155,7 @@ fn clear_dst_dir_fsync_group_commit_for_test() { DST_DIR_FSYNC_GROUP_COMMIT.clear_for_test(); } -type FileFdatasyncGroupKey = usize; +type FileFdatasyncGroupKey = (usize, bool); struct FileFdatasyncWaiter { files: Vec, @@ -1133,7 +1227,7 @@ impl FileFdatasyncGroupCommit { )); } let (result_tx, result_rx) = oneshot::channel(); - let key = Arc::as_ptr(&disk_permits) as FileFdatasyncGroupKey; + let key = (Arc::as_ptr(&disk_permits) as usize, FSYNC_ON_CURRENT_RUNTIME.get()); let mut registry = self.inner.lock(); registry.groups.retain(|_, group| group.disk_permits.strong_count() > 0); if registry.total_waiters >= MAX_FILE_FDATASYNC_WAITERS { @@ -1371,9 +1465,26 @@ static FSYNC_RUNTIME: LazyLock> = LazyLock::new( } }); +thread_local! { + static FSYNC_ON_CURRENT_RUNTIME: std::cell::Cell = const { std::cell::Cell::new(false) }; +} + +/// Called only when starting threads belonging to an isolated I/O runtime. +/// Async and blocking threads use the same runtime's blocking budget for fsync. +pub(crate) fn use_current_runtime_for_fsync() { + FSYNC_ON_CURRENT_RUNTIME.set(true); +} + +pub(super) fn is_isolated_io_thread() -> bool { + FSYNC_ON_CURRENT_RUNTIME.get() +} + /// Spawn a blocking task on the fsync-dedicated runtime if configured, /// otherwise fall back to the main tokio blocking pool. fn fsync_spawn_blocking(f: impl FnOnce() -> T + Send + 'static) -> tokio::task::JoinHandle { + if FSYNC_ON_CURRENT_RUNTIME.get() { + return tokio::task::spawn_blocking(f); + } match FSYNC_RUNTIME.as_ref() { Some(rt) => rt.spawn_blocking(f), None => tokio::task::spawn_blocking(f), @@ -1485,6 +1596,7 @@ pub(crate) async fn acquire_metadata_mutation_lease( /// cancelled while a blocking filesystem call is still running. pub(crate) struct NamespaceMutationLease { _namespace_guard: OwnedMutexGuard<()>, + _staging_guard: Option>, _volume_guard: Option>, external_guard: Mutex>>, // Independent of the quota claim; both survive cancellation of the waiter. @@ -1507,6 +1619,7 @@ pub(in crate::disk) async fn acquire_namespace_mutation_lease_with_owner( ) -> Arc { Arc::new(NamespaceMutationLease { _namespace_guard: disk_namespace_mutation_lock(path).lock_owned().await, + _staging_guard: None, _volume_guard: None, external_guard: Mutex::new(None), _namespace_owner: namespace_owner, @@ -1520,19 +1633,34 @@ pub(crate) async fn acquire_rename_data_mutation_lease( volume: &str, destination_object: &Path, ) -> Arc { - acquire_rename_data_mutation_lease_with_owner(root, volume, destination_object, None).await + acquire_rename_data_mutation_lease_with_owner(root, volume, destination_object, None, None).await } pub(crate) async fn acquire_rename_data_mutation_lease_with_owner( root: &Path, volume: &str, destination_object: &Path, + staging_prefix: Option<&Path>, namespace_owner: Option>, ) -> Arc { - let namespace_guard = disk_namespace_mutation_lock(destination_object).lock_owned().await; + // Metadata ownership precedes namespace paths in lexical order, then volume + // ownership. Deduplicate identical keys so TMP-to-TMP publication cannot self-lock. + let destination_lock = disk_namespace_mutation_lock(destination_object); + let (namespace_guard, staging_guard) = match staging_prefix { + Some(source) if source < destination_object => { + let staging = disk_namespace_mutation_lock(source).lock_owned().await; + (destination_lock.lock_owned().await, Some(staging)) + } + Some(source) if source > destination_object => { + let destination = destination_lock.lock_owned().await; + (destination, Some(disk_namespace_mutation_lock(source).lock_owned().await)) + } + _ => (destination_lock.lock_owned().await, None), + }; let volume_guard = disk_volume_mutation_lock(root, volume).read_owned().await; Arc::new(NamespaceMutationLease { _namespace_guard: namespace_guard, + _staging_guard: staging_guard, _volume_guard: Some(volume_guard), external_guard: Mutex::new(None), _namespace_owner: namespace_owner, @@ -2071,9 +2199,19 @@ pub(crate) async fn remove_dir_with_owner( if namespace_owner.is_none() { return tokio::fs::remove_dir(path).await; } + let lease = acquire_namespace_mutation_lease_with_owner(path.as_ref(), namespace_owner).await; + remove_dir_with_lease(path, lease).await +} + +/// Remove a directory whose namespace is already leased by the caller. +pub(crate) async fn remove_dir_with_lease(path: impl AsRef, lease: Arc) -> io::Result<()> { let path = path.as_ref().to_path_buf(); - let lease = acquire_namespace_mutation_lease_with_owner(&path, namespace_owner).await; - run_blocking_namespace_operation(lease, move || std::fs::remove_dir(path)).await + run_blocking_namespace_operation(lease, move || { + #[cfg(all(any(test, feature = "test-util"), not(windows)))] + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Remove, &path); + std::fs::remove_dir(path) + }) + .await } /// Preserve raw rename semantics while retaining a counted owner in the syscall. @@ -2268,6 +2406,20 @@ pub(crate) fn create_prepared_rename_source_with_commit_guard( #[cfg(not(windows))] { let _ = (dst_file_path, commit_guard); + #[cfg(target_os = "linux")] + let source = { + use rustix::fs::{Mode, OFlags}; + let directories = &commit_guard.directories; + let name = directories.file_name(src_file_path, true)?; + directories.file_name(dst_file_path, false)?; + open_linux_relative( + &directories.source.file, + Path::new(name), + OFlags::CREATE | OFlags::TRUNC | OFlags::WRONLY, + Mode::from_raw_mode(0o666), + )? + }; + #[cfg(not(target_os = "linux"))] let source = std::fs::OpenOptions::new() .create(true) .write(true) @@ -2288,18 +2440,39 @@ pub(crate) fn create_prepared_rename_source_with_commit_guard( } } -#[cfg(windows)] +#[cfg(any(windows, target_os = "linux"))] pub(crate) fn read_destination_file_with_commit_guard( file_path: &Path, commit_guard: &RenameCommitGuard, ) -> io::Result>> { - if file_path.parent() != Some(commit_guard.destination_parent.as_path()) { - return Err(io::Error::new( - io::ErrorKind::InvalidInput, - "destination file parent does not match its commit guard", - )); + #[cfg(target_os = "linux")] + { + use rustix::fs::{Mode, OFlags}; + let directories = &commit_guard.directories; + let name = directories.file_name(file_path, false)?; + let Some(parent) = directories.destination.get() else { + return Ok(None); + }; + match open_linux_relative(&parent.file, Path::new(name), OFlags::RDONLY, Mode::empty()) { + Ok(mut file) => { + let mut data = Vec::new(); + std::io::Read::read_to_end(&mut file, &mut data)?; + Ok(Some(data)) + } + Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None), + Err(err) => Err(err), + } + } + #[cfg(windows)] + { + if file_path.parent() != Some(commit_guard.destination_parent.as_path()) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "destination file parent does not match its commit guard", + )); + } + read_windows_relative_file(file_path, &commit_guard.destination_parent_guard) } - read_windows_relative_file(file_path, &commit_guard.destination_parent_guard) } #[cfg(windows)] @@ -2337,23 +2510,40 @@ pub(crate) async fn rename_all_with_prepared_source( dst_file_path: impl AsRef, base_dir: impl AsRef, publication_root: &PublicationRoot, - _commit_guard: &RenameCommitGuard, + commit_guard: &RenameCommitGuard, lease: Arc, ) -> Result<()> { let src_file_path = src_file_path.as_ref().to_path_buf(); let dst_file_path = dst_file_path.as_ref().to_path_buf(); let base_dir = base_dir.as_ref().to_path_buf(); + #[cfg(not(target_os = "linux"))] let publication_root = publication_root.clone(); + #[cfg(target_os = "linux")] + let _ = publication_root; + let commit_guard = commit_guard.clone(); let operation = { let src_file_path = src_file_path.clone(); let dst_file_path = dst_file_path.clone(); + #[cfg(not(target_os = "linux"))] let base_dir = base_dir.clone(); move || { - validate_prepared_rename_source(&prepared_source, &src_file_path)?; - let preparation = prepare_rename_with_retry(&src_file_path, &dst_file_path, &base_dir, &publication_root)?; - #[cfg(any(test, feature = "test-util"))] - prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path); - rename_prepared(&src_file_path, &dst_file_path, &preparation) + #[cfg(target_os = "linux")] + { + #[cfg(any(test, feature = "test-util"))] + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path); + commit_guard + .directories + .rename(&src_file_path, &dst_file_path, Some(&prepared_source)) + } + #[cfg(not(target_os = "linux"))] + { + let _ = commit_guard; + validate_prepared_rename_source(&prepared_source, &src_file_path)?; + let preparation = prepare_rename_with_retry(&src_file_path, &dst_file_path, &base_dir, &publication_root)?; + #[cfg(any(test, feature = "test-util"))] + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::PreparedRename, &dst_file_path); + rename_prepared(&src_file_path, &dst_file_path, &preparation) + } } }; let result = run_blocking_namespace_operation(lease, operation).await; @@ -2370,10 +2560,35 @@ pub(crate) async fn rename_all_with_commit_guard( dst_file_path: impl AsRef, base_dir: impl AsRef, publication_root: &PublicationRoot, - _commit_guard: &RenameCommitGuard, + commit_guard: &RenameCommitGuard, lease: Arc, ) -> Result<()> { - rename_all_with_lease(src_file_path, dst_file_path, base_dir, publication_root, lease).await + #[cfg(target_os = "linux")] + { + let source = src_file_path.as_ref().to_path_buf(); + let destination = dst_file_path.as_ref().to_path_buf(); + let guard = commit_guard.clone(); + let _ = (base_dir, publication_root); + run_blocking_namespace_operation(lease, move || { + guard.directories.prepare_destination()?; + #[cfg(test)] + prepared_publication_test_hooks::run_rename_destination(&source, &destination); + #[cfg(any(test, feature = "test-util"))] + { + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, &source); + prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, &destination); + } + guard.directories.rename(&source, &destination, None) + }) + .await + .map_err(to_file_error) + .map_err(DiskError::from) + } + #[cfg(not(target_os = "linux"))] + { + let _ = commit_guard; + rename_all_with_lease(src_file_path, dst_file_path, base_dir, publication_root, lease).await + } } #[tracing::instrument(level = "debug", skip_all)] @@ -2501,7 +2716,7 @@ async fn reliable_rename_inner_with_lease( result } -#[cfg(not(windows))] +#[cfg(not(any(windows, target_os = "linux")))] fn validate_prepared_rename_source(prepared_source: &PreparedRenameSource, src_file_path: &Path) -> io::Result<()> { if prepared_source.path != src_file_path { return Err(io::Error::new( @@ -2701,6 +2916,8 @@ pub(crate) async fn fsync_dir_with_namespace_file_sync_limit( struct RenamePreparation { parent_guard: Option, + #[cfg(target_os = "linux")] + publication_root: PublicationRoot, #[cfg(windows)] _source_parent_guard: ExistingBaseDirectoryGuard, #[cfg(windows)] @@ -2709,15 +2926,32 @@ struct RenamePreparation { #[cfg(not(windows))] fn prepare_rename_with_retry( - src_file_path: &Path, + _src_file_path: &Path, dst_file_path: &Path, base_dir: &Path, publication_root: &PublicationRoot, ) -> io::Result { - let prune_budget = prepare_prune_budget(dst_file_path, base_dir); + let parent_guard = dst_file_path + .parent() + .map(|parent| prepare_destination_parent_with_retry(parent, base_dir, publication_root)) + .transpose()?; + Ok(RenamePreparation { + parent_guard, + #[cfg(target_os = "linux")] + publication_root: publication_root.clone(), + }) +} + +#[cfg(not(windows))] +fn prepare_destination_parent_with_retry( + parent: &Path, + base_dir: &Path, + publication_root: &PublicationRoot, +) -> io::Result { + let prune_budget = parent_prune_budget(parent, base_dir); let mut attempt = 0; loop { - match prepare_rename(src_file_path, dst_file_path, base_dir, publication_root) { + match mkdir_all_below_existing_base_std(parent, base_dir, publication_root) { Ok(preparation) => return Ok(preparation), Err(err) if should_retry_prepare(&err, attempt, prune_budget) => { attempt += 1; @@ -2807,20 +3041,6 @@ fn prepare_rename_with_retry( }) } -#[cfg(not(windows))] -fn prepare_rename( - _src_file_path: &Path, - dst_file_path: &Path, - base_dir: &Path, - publication_root: &PublicationRoot, -) -> io::Result { - let parent_guard = dst_file_path - .parent() - .map(|parent| mkdir_all_below_existing_base_std(parent, base_dir, publication_root)) - .transpose()?; - Ok(RenamePreparation { parent_guard }) -} - /// Publish a prepared rename. The retry budget starts fresh here: preparation /// keeps its own counter, so a chain rebuilt after a concurrent prune must not /// cost the rename its one retry. @@ -2836,9 +3056,23 @@ fn rename_prepared(_src_file_path: &Path, dst_file_path: &Path, preparation: &Re #[cfg(not(windows))] { + // A missing source parent is not a destination-prune race. Open it once + // outside both retry loops, retaining it through any rename retry. + #[cfg(target_os = "linux")] + let source_parent = preparation.publication_root.open_directory( + _src_file_path + .parent() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing rename source parent"))?, + )?; let mut attempt = 0; loop { - let rename_result = rename_into_existing_parent(_src_file_path, dst_file_path, preparation.parent_guard.as_ref()); + let rename_result = rename_into_existing_parent( + _src_file_path, + dst_file_path, + preparation.parent_guard.as_ref(), + #[cfg(target_os = "linux")] + &source_parent, + ); match rename_result { Ok(()) => return Ok(()), Err(err) if should_retry_rename(&err, attempt) => { @@ -2877,8 +3111,9 @@ fn rename_into_existing_parent( src_file_path: &Path, dst_file_path: &Path, parent_guard: Option<&ExistingBaseDirectoryGuard>, + #[cfg(target_os = "linux")] source_parent: &std::fs::File, ) -> io::Result<()> { - use rustix::fs::{Mode, OFlags, open, renameat}; + use rustix::fs::renameat; let Some(parent_guard) = parent_guard else { let rename_started = rustfs_io_metrics::put_stage_timer(); @@ -2889,6 +3124,7 @@ fn rename_into_existing_parent( ); return result; }; + #[cfg(not(target_os = "linux"))] let src_parent = src_file_path .parent() .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename source must have a parent directory"))?; @@ -2898,10 +3134,11 @@ fn rename_into_existing_parent( let dst_name = dst_file_path .file_name() .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must have a file name"))?; - let src_parent = open( + #[cfg(not(target_os = "linux"))] + let src_parent = rustix::fs::open( src_parent, - OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, - Mode::empty(), + rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::DIRECTORY | rustix::fs::OFlags::NOFOLLOW | rustix::fs::OFlags::CLOEXEC, + rustix::fs::Mode::empty(), ) .map_err(io::Error::from)?; let dst_parent = parent_guard @@ -2909,6 +3146,9 @@ fn rename_into_existing_parent( .ok_or_else(|| io::Error::other("rename destination parent guard is empty"))?; let rename_started = rustfs_io_metrics::put_stage_timer(); + #[cfg(target_os = "linux")] + let result = renameat(source_parent, src_name, dst_parent, dst_name).map_err(io::Error::from); + #[cfg(not(target_os = "linux"))] let result = renameat(&src_parent, src_name, dst_parent, dst_name).map_err(io::Error::from); rustfs_io_metrics::record_put_object_stage_duration_from( rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL, @@ -3091,16 +3331,18 @@ struct WindowsDirectoryHandle { /// Stable root for namespace-changing disk operations. /// -/// Windows opens the configured endpoint once and keeps that directory identity -/// pinned for the lifetime of the disk. Publication then resolves every source -/// and destination component relative to this handle instead of re-entering the -/// mutable pathname namespace. Other platforms retain the path so callers use a -/// uniform API while their existing `openat`/`renameat` guards remain unchanged. +/// Linux and Windows pin the configured directory for the lifetime of the disk. +/// Linux also exposes a procfs alias for operations not yet using relative IO; +/// both access forms share one root handle and therefore one mount identity. #[derive(Clone)] pub(crate) struct PublicationRoot { - path: PathBuf, + path: Arc, + #[cfg(target_os = "linux")] + directory: Arc, + #[cfg(target_os = "linux")] + io_path: Arc, #[cfg(windows)] - configured_path: PathBuf, + configured_path: Arc, #[cfg(windows)] directory: WindowsDirectoryHandle, } @@ -3114,15 +3356,34 @@ impl PublicationRoot { #[cfg(windows)] let (resolved_path, directory) = open_windows_publication_root(path)?; + #[cfg(target_os = "linux")] + let directory = { + use rustix::fs::{Mode, OFlags, open}; + Arc::new(std::fs::File::from(open( + path, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + Mode::empty(), + )?)) + }; + #[cfg(target_os = "linux")] + let io_path = { + use std::os::fd::AsRawFd; + Arc::new(PathBuf::from(format!("/proc/self/fd/{}/.", directory.as_raw_fd()))) + }; + Ok(Self { #[cfg(not(windows))] - path: path.to_path_buf(), + path: Arc::new(path.to_path_buf()), #[cfg(windows)] - path: resolved_path, + path: Arc::new(resolved_path), #[cfg(windows)] - configured_path: path.to_path_buf(), + configured_path: Arc::new(path.to_path_buf()), #[cfg(windows)] directory, + #[cfg(target_os = "linux")] + directory, + #[cfg(target_os = "linux")] + io_path, }) } @@ -3130,15 +3391,170 @@ impl PublicationRoot { &self.path } + #[cfg(target_os = "linux")] + pub(crate) fn directory(&self) -> &Arc { + &self.directory + } + + #[cfg(target_os = "linux")] + pub(crate) fn io_path(&self) -> &Path { + &self.io_path + } + + pub(crate) fn open_readonly(&self, path: &Path) -> io::Result { + #[cfg(target_os = "linux")] + return self.open_relative(path, rustix::fs::OFlags::RDONLY, rustix::fs::Mode::empty()); + #[cfg(not(target_os = "linux"))] + std::fs::File::open(path) + } + + pub(crate) fn create_truncate(&self, path: &Path) -> io::Result { + #[cfg(target_os = "linux")] + { + use rustix::fs::{Mode, OFlags}; + self.open_relative(path, OFlags::CREATE | OFlags::WRONLY | OFlags::TRUNC, Mode::from_raw_mode(0o666)) + } + #[cfg(not(target_os = "linux"))] + std::fs::OpenOptions::new().create(true).write(true).truncate(true).open(path) + } + + pub(crate) fn create_truncate_with_parent(&self, path: &Path) -> io::Result<(std::fs::File, Option)> { + #[cfg(target_os = "linux")] + { + use rustix::fs::{Mode, OFlags}; + let parent = self.open_directory( + path.parent() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing file parent"))?, + )?; + let name = path + .file_name() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing file name"))?; + let file = open_linux_relative( + &parent, + Path::new(name), + OFlags::CREATE | OFlags::WRONLY | OFlags::TRUNC, + Mode::from_raw_mode(0o666), + )?; + Ok((file, Some(parent))) + } + #[cfg(not(target_os = "linux"))] + Ok((self.create_truncate(path)?, None)) + } + + #[cfg(target_os = "linux")] + fn open_relative(&self, path: &Path, flags: rustix::fs::OFlags, mode: rustix::fs::Mode) -> io::Result { + let relative = path + .strip_prefix(self.io_path.as_path()) + .or_else(|_| path.strip_prefix(self.path.as_path())) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "path must remain below its publication root"))?; + open_linux_relative(&self.directory, relative, flags, mode) + } + + #[cfg(target_os = "linux")] + fn open_directory(&self, path: &Path) -> io::Result { + self.open_relative( + path, + rustix::fs::OFlags::RDONLY | rustix::fs::OFlags::DIRECTORY, + rustix::fs::Mode::empty(), + ) + } + #[cfg(windows)] fn relative_path<'a>(&self, path: &'a Path) -> io::Result<&'a Path> { // The configured path only derives a suffix; traversal stays rooted at the pinned directory handle. - path.strip_prefix(&self.path) - .or_else(|_| path.strip_prefix(&self.configured_path)) + path.strip_prefix(self.path.as_path()) + .or_else(|_| path.strip_prefix(self.configured_path.as_path())) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "path must remain below its publication root")) } } +/// Open the actual IO handle with the same containment policy as path validation. +/// Older kernels retain fd-relative traversal; only ENOSYS enables the fallback. +#[cfg(target_os = "linux")] +fn open_linux_relative( + directory: &std::fs::File, + relative: &Path, + flags: rustix::fs::OFlags, + mode: rustix::fs::Mode, +) -> io::Result { + use rustix::fs::{OFlags, ResolveFlags, openat2}; + if relative + .components() + .any(|component| !matches!(component, Component::Normal(_) | Component::CurDir)) + { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "invalid relative disk path")); + } + let relative = if relative.as_os_str().is_empty() { + Path::new(".") + } else { + relative + }; + let flags = flags | OFlags::CLOEXEC | OFlags::NOFOLLOW; + match openat2(directory, relative, flags, mode, ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS) { + Ok(fd) => Ok(fd.into()), + Err(rustix::io::Errno::NOSYS) => open_linux_relative_legacy(directory, relative, flags, mode), + Err(rustix::io::Errno::LOOP | rustix::io::Errno::XDEV) => Err(DiskError::InvalidPath.into()), + Err(err) => Err(err.into()), + } +} + +#[cfg(target_os = "linux")] +fn open_linux_relative_legacy( + directory: &std::fs::File, + relative: &Path, + flags: rustix::fs::OFlags, + mode: rustix::fs::Mode, +) -> io::Result { + use rustix::fs::OFlags; + + let mut parent = None; + let mut components = relative + .components() + .filter_map(|component| match component { + Component::Normal(name) => Some(name), + _ => None, + }) + .peekable(); + while let Some(name) = components.next() { + let current = parent.as_ref().unwrap_or(directory); + if components.peek().is_none() { + return open_linux_relative_component(current, name, flags, mode); + } + parent = Some(open_linux_relative_component( + current, + name, + OFlags::PATH | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC, + rustix::fs::Mode::empty(), + )?); + } + open_linux_relative_component(directory, std::ffi::OsStr::new("."), flags, mode) +} + +#[cfg(target_os = "linux")] +fn open_linux_relative_component( + directory: &std::fs::File, + name: &std::ffi::OsStr, + flags: rustix::fs::OFlags, + mode: rustix::fs::Mode, +) -> io::Result { + use rustix::fs::{AtFlags, FileType, openat, statat}; + use rustix::io::Errno; + match openat(directory, name, flags, mode) { + Ok(fd) => Ok(fd.into()), + Err(Errno::LOOP) => Err(DiskError::InvalidPath.into()), + // O_DIRECTORY | O_NOFOLLOW reports ENOTDIR for an intermediate link. + // Inspect only this failure so ordinary non-directory errors retain + // their classification without adding a stat to successful opens. + Err(Errno::NOTDIR) + if statat(directory, name, AtFlags::SYMLINK_NOFOLLOW) + .is_ok_and(|metadata| FileType::from_raw_mode(metadata.st_mode) == FileType::Symlink) => + { + Err(DiskError::InvalidPath.into()) + } + Err(err) => Err(err.into()), + } +} + #[cfg(windows)] #[derive(Clone)] pub(crate) struct ExistingBaseDirectoryGuard { @@ -3176,6 +3592,8 @@ pub(crate) type ExistingBaseDirectoryGuard = (); #[derive(Clone)] pub(crate) struct RenameCommitGuard { + #[cfg(target_os = "linux")] + directories: Arc, #[cfg(windows)] source_parent: PathBuf, #[cfg(windows)] @@ -3186,6 +3604,108 @@ pub(crate) struct RenameCommitGuard { destination_parent_guard: ExistingBaseDirectoryGuard, } +#[cfg(target_os = "linux")] +struct LinuxCommitDirectory { + file: Arc, + device: u64, + inode: u64, +} + +#[cfg(target_os = "linux")] +impl LinuxCommitDirectory { + fn new(file: std::fs::File) -> io::Result { + use std::os::unix::fs::MetadataExt; + let metadata = file.metadata()?; + Ok(Self { + file: Arc::new(file), + device: metadata.dev(), + inode: metadata.ino(), + }) + } +} + +#[cfg(target_os = "linux")] +struct LinuxCommitDirectories { + root: PublicationRoot, + source_path: PathBuf, + destination_path: PathBuf, + source: LinuxCommitDirectory, + destination: std::sync::OnceLock, + destination_base: PathBuf, +} + +#[cfg(target_os = "linux")] +impl LinuxCommitDirectories { + fn prepare_destination(&self) -> io::Result<&LinuxCommitDirectory> { + if let Some(directory) = self.destination.get() { + return Ok(directory); + } + let mut parents = prepare_destination_parent_with_retry(&self.destination_path, &self.destination_base, &self.root)?; + let directory = std::fs::File::from( + parents + .pop() + .ok_or_else(|| io::Error::other("missing destination directory guard"))?, + ); + let directory = LinuxCommitDirectory::new(directory)?; + Ok(self.destination.get_or_init(|| directory)) + } + + fn file_name<'a>(&self, path: &'a Path, source: bool) -> io::Result<&'a std::ffi::OsStr> { + let parent = if source { &self.source_path } else { &self.destination_path }; + if path.parent() != Some(parent) { + return Err(io::Error::new(io::ErrorKind::InvalidInput, "file parent does not match its commit guard")); + } + path.file_name() + .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "missing file name")) + } + + fn validate(&self) -> io::Result<()> { + use std::os::unix::fs::MetadataExt; + // A pinned directory may have been unlinked or replaced. Do not report a + // successful publication into a detached object tree. + for (path, directory) in [ + (&self.source_path, &self.source), + (&self.destination_path, self.prepare_destination()?), + ] { + let current = self.root.open_directory(path)?.metadata()?; + if (current.dev(), current.ino()) != (directory.device, directory.inode) { + return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed")); + } + } + Ok(()) + } + + fn rename(&self, source: &Path, destination: &Path, prepared: Option<&PreparedRenameSource>) -> io::Result<()> { + let source_name = self.file_name(source, true)?; + let destination_name = self.file_name(destination, false)?; + let destination_parent = self.prepare_destination()?; + self.validate()?; + let mut attempt = 0; + loop { + if let Some(prepared) = prepared { + let metadata = rustix::fs::statat(&*self.source.file, source_name, rustix::fs::AtFlags::SYMLINK_NOFOLLOW)?; + if prepared.path != source || metadata.st_dev != prepared.device || metadata.st_ino != prepared.inode { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "rename source identity changed while publication was prepared", + )); + } + } + let started = rustfs_io_metrics::put_stage_timer(); + let result = rustix::fs::renameat(&*self.source.file, source_name, &*destination_parent.file, destination_name) + .map_err(io::Error::from); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL, + started, + ); + match result { + Err(err) if should_retry_rename(&err, attempt) => attempt += 1, + result => return result, + } + } + } +} + pub(crate) struct RenameDestinationPathGuard { #[cfg(windows)] directory: PathBuf, @@ -3385,6 +3905,29 @@ pub(crate) fn prepare_rename_commit_guard( destination_base: &Path, publication_root: &PublicationRoot, ) -> io::Result { + #[cfg(target_os = "linux")] + { + existing_base_directory_suffix(destination_parent, destination_base)?; + let source = LinuxCommitDirectory::new(publication_root.open_directory(source_parent)?)?; + let destination = std::sync::OnceLock::new(); + match publication_root.open_directory(destination_parent) { + Ok(directory) => { + let _ = destination.set(LinuxCommitDirectory::new(directory)?); + } + Err(err) if err.kind() == io::ErrorKind::NotFound => {} + Err(err) => return Err(err), + } + Ok(RenameCommitGuard { + directories: Arc::new(LinuxCommitDirectories { + root: publication_root.clone(), + source_path: source_parent.to_path_buf(), + destination_path: destination_parent.to_path_buf(), + source, + destination, + destination_base: destination_base.to_path_buf(), + }), + }) + } #[cfg(windows)] { // A same-directory rename must use one shared-write parent handle: @@ -3413,7 +3956,7 @@ pub(crate) fn prepare_rename_commit_guard( }) } - #[cfg(not(windows))] + #[cfg(not(any(windows, target_os = "linux")))] { let _ = (source_parent, destination_parent, destination_base, publication_root); Ok(RenameCommitGuard {}) @@ -3532,7 +4075,10 @@ fn windows_final_path_with_flags(handle: &winapi_util::Handle, flags: u32) -> io .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "Windows final path length exceeds usize"))?, ); let path = PathBuf::from(OsString::from_wide(&buffer)); - Ok(rustfs_utils::simplified(&path).to_path_buf()) + // Keep the root in the non-verbatim form: callers extend it by joining + // `/`-separated relative paths, which a verbatim (`\\?\`) root would + // turn into ERROR_INVALID_NAME for UNC shares and roots past MAX_PATH. + Ok(rustfs_utils::non_verbatim(&path).into_owned()) } #[cfg(windows)] @@ -4209,11 +4755,7 @@ fn windows_rename_source_is_allowed(attributes: u32, reparse_tag: u32) -> bool { attributes & FILE_ATTRIBUTE_REPARSE_POINT == 0 || reparse_tag == IO_REPARSE_TAG_DEDUP } -pub(crate) fn mkdir_all_below_existing_base_std( - dir_path: &Path, - base_dir: &Path, - publication_root: &PublicationRoot, -) -> io::Result { +fn existing_base_directory_suffix<'a>(dir_path: &'a Path, base_dir: &Path) -> io::Result<&'a Path> { let relative = dir_path .strip_prefix(base_dir) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "rename destination must remain below its base directory"))?; @@ -4226,15 +4768,31 @@ pub(crate) fn mkdir_all_below_existing_base_std( } } + Ok(relative) +} + +pub(crate) fn mkdir_all_below_existing_base_std( + dir_path: &Path, + base_dir: &Path, + publication_root: &PublicationRoot, +) -> io::Result { + let relative = existing_base_directory_suffix(dir_path, base_dir)?; + #[cfg(unix)] { - let _ = publication_root; - use rustix::fs::{Mode, OFlags, mkdirat, open, openat}; + use rustix::fs::{Mode, OFlags, mkdirat, openat}; use rustix::io::Errno; let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC; let mode = Mode::RWXU | Mode::RWXG | Mode::RWXO; - let mut parents = vec![open(base_dir, flags, Mode::empty()).map_err(io::Error::from)?]; + #[cfg(target_os = "linux")] + let base = publication_root.open_directory(base_dir)?.into(); + #[cfg(not(target_os = "linux"))] + let base = { + let _ = publication_root; + rustix::fs::open(base_dir, flags, Mode::empty()).map_err(io::Error::from)? + }; + let mut parents = vec![base]; #[cfg(test)] let mut walked_path = base_dir.to_path_buf(); @@ -4354,10 +4912,18 @@ fn should_retry_rename(err: &io::Error, attempt: usize) -> bool { /// steal an attempt by making that same upward progress. A destination whose /// parent *is* the base gets a budget of zero, keeping `NotFound` immediately /// terminal for speculative cleanup renames. +#[cfg(any(windows, test))] fn prepare_prune_budget(dst_file_path: &Path, base_dir: &Path) -> usize { dst_file_path .parent() - .and_then(|parent| parent.strip_prefix(base_dir).ok()) + .map(|parent| parent_prune_budget(parent, base_dir)) + .unwrap_or(0) +} + +fn parent_prune_budget(parent: &Path, base_dir: &Path) -> usize { + parent + .strip_prefix(base_dir) + .ok() .map(|relative| relative.components().count()) .unwrap_or(0) } @@ -4484,6 +5050,107 @@ mod tests { use tempfile::tempdir; use tracing_subscriber::fmt::MakeWriter; + #[tokio::test] + async fn rename_staging_leases_are_ordered_and_deduplicated() { + let root = tempdir().expect("namespace fixture"); + let early = root.path().join("a"); + let late = root.path().join("b"); + let blocker = acquire_namespace_mutation_lease_with_owner(&late, None).await; + let first_root = root.path().to_path_buf(); + let first_early = early.clone(); + let first_late = late.clone(); + let first = tokio::spawn(async move { + drop( + acquire_rename_data_mutation_lease_with_owner(&first_root, "bucket", &first_late, Some(&first_early), None).await, + ); + }); + tokio::time::timeout(Duration::from_secs(10), async { + while disk_namespace_mutation_lock(&early).try_lock_owned().is_ok() { + tokio::task::yield_now().await; + } + }) + .await + .expect("source must acquire the earlier key before waiting for destination"); + let second_root = root.path().to_path_buf(); + let second_early = early.clone(); + let second_late = late.clone(); + let second = tokio::spawn(async move { + drop( + acquire_rename_data_mutation_lease_with_owner(&second_root, "bucket", &second_early, Some(&second_late), None) + .await, + ); + }); + drop(blocker); + tokio::time::timeout(Duration::from_secs(10), async { + first.await.expect("first publication"); + second.await.expect("reverse publication"); + }) + .await + .expect("reverse source/destination order must not deadlock"); + drop( + tokio::time::timeout( + Duration::from_secs(10), + acquire_rename_data_mutation_lease_with_owner(root.path(), "bucket", &early, Some(&early), None), + ) + .await + .expect("identical source and destination key must be acquired once"), + ); + } + + #[test] + fn fsync_groups_do_not_mix_cleanup_and_foreground_workers() { + let dir = tempdir().unwrap(); + let dst_groups = Arc::new(DstDirFsyncGroupCommit::default()); + let file_groups = Arc::new(FileFdatasyncGroupCommit::default()); + let disk_permits = Arc::new(Semaphore::new(1)); + let (_, dst_worker) = dst_groups.enqueue_for_test(dir.path()).unwrap(); + let (_, file_worker) = file_groups + .enqueue(disk_permits.clone(), vec![dir.path().join("one")]) + .unwrap(); + assert!(dst_worker.is_some()); + assert!(file_worker.is_some()); + let cleanup_dst_groups = dst_groups.clone(); + let cleanup_file_groups = file_groups.clone(); + std::thread::spawn(move || { + use_current_runtime_for_fsync(); + let (_, dst_worker) = cleanup_dst_groups.enqueue_for_test(dir.path()).unwrap(); + let (_, file_worker) = cleanup_file_groups + .enqueue(disk_permits, vec![dir.path().join("two")]) + .unwrap(); + assert!(dst_worker.is_some(), "cleanup must not join a foreground directory worker"); + assert!(file_worker.is_some(), "cleanup must not join a foreground file worker"); + }) + .join() + .unwrap(); + assert_eq!(dst_groups.inner.lock().groups.len(), 2); + assert_eq!(file_groups.inner.lock().groups.len(), 2); + } + + #[tokio::test] + async fn isolated_runtime_keeps_fsync_on_its_own_blocking_pool() { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(1) + .max_blocking_threads(1) + .thread_name("isolated-fsync-test") + .on_thread_start(use_current_runtime_for_fsync) + .enable_all() + .build() + .unwrap(); + let result = runtime + .spawn(async { + fsync_spawn_blocking(|| { + assert_eq!(std::thread::current().name(), Some("isolated-fsync-test")); + }) + .await + .unwrap(); + }) + .await; + // A runtime cannot be dropped from another runtime's async context. + runtime.shutdown_background(); + result.unwrap(); + assert!(!FSYNC_ON_CURRENT_RUNTIME.get(), "the calling runtime must retain normal fsync routing"); + } + fn file_sync_limiter() -> Arc { Arc::new(Semaphore::new(MAX_PARALLEL_FILE_SYNCS)) } @@ -4493,7 +5160,11 @@ mod tests { .first() .expect("test publication root requires at least one path") .to_path_buf(); - while !paths.iter().all(|path| path.starts_with(&common)) { + // The disk root must exist independently of a missing or symlinked + // rename base; those are inputs whose rejection the tests exercise. + while !paths.iter().all(|path| path.starts_with(&common)) + || !std::fs::symlink_metadata(&common).is_ok_and(|metadata| metadata.is_dir()) + { assert!(common.pop(), "test paths must share an absolute root"); } PublicationRoot::new(&common).expect("test publication root should open") @@ -7737,3 +8408,172 @@ mod tests { assert_eq!(err.kind(), io::ErrorKind::NotFound); } } + +#[cfg(all(test, target_os = "linux"))] +mod fd_relative_tests { + use super::*; + use std::io::{Read, Write}; + + #[tokio::test] + async fn guarded_first_publication_recovers_from_a_concurrent_parent_prune() { + let temp = tempfile::tempdir().expect("temporary disk"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let source = temp.path().join("tmp"); + let base = temp.path().join("bucket"); + let shared = base.join("prefix"); + let destination = shared.join("object"); + std::fs::create_dir(&source).expect("staging directory"); + std::fs::create_dir_all(&shared).expect("shared prefix"); + let guard = prepare_rename_commit_guard(&source, &destination, &base, &root).expect("commit directories"); + let src = source.join("xl.meta"); + let dst = destination.join("xl.meta"); + let mut staged = create_prepared_rename_source_with_commit_guard(&src, &dst, &guard).expect("staged metadata"); + staged.write_all(b"metadata", true).expect("write metadata"); + let pruned = shared.clone(); + prepare_rename_test_hooks::queue_after_component_opened(&shared, move || { + std::fs::remove_dir(pruned).expect("prune empty prefix"); + }); + let lease = acquire_namespace_mutation_lease(&dst).await; + rename_all_with_prepared_source(staged, &src, &dst, &base, &root, &guard, lease) + .await + .expect("publication must rebuild the pruned parent"); + assert_eq!(std::fs::read(dst).expect("published metadata"), b"metadata"); + assert!(!src.exists()); + } + + #[test] + fn legacy_relative_open_rejects_symlinks_and_opens_existing_components() { + use rustix::fs::{Mode, OFlags}; + let temp = tempfile::tempdir().expect("temporary disk"); + std::fs::create_dir(temp.path().join("parent")).expect("parent directory"); + std::fs::write(temp.path().join("parent/data"), b"data").expect("file data"); + std::os::unix::fs::symlink("parent", temp.path().join("link")).expect("directory link"); + std::os::unix::fs::symlink("data", temp.path().join("parent/link")).expect("file link"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let flags = OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC; + let mut bytes = Vec::new(); + open_linux_relative_legacy(root.directory(), Path::new("parent/data"), flags, Mode::empty()) + .expect("legacy open") + .read_to_end(&mut bytes) + .expect("legacy read"); + assert_eq!(bytes, b"data"); + for relative in ["link/data", "parent/link"] { + assert!(open_linux_relative_legacy(root.directory(), Path::new(relative), flags, Mode::empty()).is_err()); + } + } + + #[test] + fn root_relative_io_retains_mount_identity_after_path_replacement() { + let temp = tempfile::tempdir().expect("temporary disk"); + let disk = temp.path().join("disk"); + std::fs::create_dir(&disk).expect("disk root"); + std::fs::write(disk.join("data"), b"original").expect("original data"); + let root = PublicationRoot::new(&disk).expect("pinned root"); + let original = temp.path().join("original"); + std::fs::rename(&disk, &original).expect("move root"); + std::fs::create_dir(&disk).expect("replacement root"); + std::fs::write(disk.join("data"), b"replacement").expect("replacement data"); + for path in [disk.join("data"), root.io_path().join("data")] { + let mut actual = Vec::new(); + root.open_readonly(&path) + .expect("pinned read") + .read_to_end(&mut actual) + .expect("read bytes"); + assert_eq!(actual, b"original"); + } + let (mut file, parent) = root.create_truncate_with_parent(&disk.join("new")).expect("pinned creation"); + file.write_all(b"new").expect("write bytes"); + file.sync_data().expect("file sync"); + fsync_directory_handle_std(&disk, parent.as_ref()).expect("pinned parent sync"); + assert_eq!(std::fs::read(original.join("new")).expect("original disk file"), b"new"); + assert!(!disk.join("new").exists()); + } + + #[test] + fn root_relative_io_rejects_links_and_parent_escape_without_truncation() { + let temp = tempfile::tempdir().expect("temporary disk"); + let disk = temp.path().join("disk"); + std::fs::create_dir(&disk).expect("disk root"); + let outside = temp.path().join("outside"); + std::fs::write(&outside, b"preserved").expect("outside data"); + std::os::unix::fs::symlink(&outside, disk.join("link")).expect("file link"); + std::os::unix::fs::symlink(temp.path(), disk.join("parent")).expect("directory link"); + let root = PublicationRoot::new(&disk).expect("pinned root"); + for path in [disk.join("link"), disk.join("parent/outside"), disk.join("../outside")] { + assert!(root.open_readonly(&path).is_err(), "unsafe read: {path:?}"); + assert!(root.create_truncate(&path).is_err(), "unsafe write: {path:?}"); + } + assert_eq!(std::fs::read(&outside).expect("outside remains readable"), b"preserved"); + } + + #[tokio::test] + async fn guarded_publication_reuses_parents_and_syncs_without_changing_bytes() { + for grouped in [false, true] { + let _grouping = set_dst_dir_fsync_group_commit_for_test(grouped); + let temp = tempfile::tempdir().expect("temporary disk"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let source = temp.path().join("tmp"); + let destination = temp.path().join("bucket/object"); + let base = temp.path().join("bucket"); + std::fs::create_dir(&source).expect("staging directory"); + std::fs::create_dir(&base).expect("bucket directory"); + let guard = prepare_rename_commit_guard(&source, &destination, &base, &root).expect("commit directories"); + assert!(!destination.exists(), "preparation must not create the object directory"); + let src = source.join("xl.meta"); + let dst = destination.join("xl.meta"); + assert!( + read_destination_file_with_commit_guard(&dst, &guard) + .expect("absent read") + .is_none() + ); + let mut staged = create_prepared_rename_source_with_commit_guard(&src, &dst, &guard).expect("staged metadata"); + staged.write_all(b"metadata", true).expect("write metadata"); + let lease = acquire_namespace_mutation_lease(&dst).await; + rename_all_with_prepared_source(staged, &src, &dst, &base, &root, &guard, lease.clone()) + .await + .expect("publish"); + fsync_commit_directory(&destination, &guard, lease, None) + .await + .expect("directory sync"); + assert_eq!( + read_destination_file_with_commit_guard(&dst, &guard).expect("committed read"), + Some(b"metadata".to_vec()) + ); + assert!(!src.exists()); + assert_eq!(std::fs::read_dir(&destination).expect("object directory").count(), 1); + } + } + + #[tokio::test] + async fn guarded_publication_rejects_replaced_directory_and_source() { + for replace_directory in [false, true] { + let temp = tempfile::tempdir().expect("temporary disk"); + let root = PublicationRoot::new(temp.path()).expect("pinned root"); + let source = temp.path().join("tmp"); + let destination = temp.path().join("object"); + std::fs::create_dir(&source).expect("staging directory"); + std::fs::create_dir(&destination).expect("existing object directory"); + let guard = prepare_rename_commit_guard(&source, &destination, temp.path(), &root).expect("commit directories"); + let src = source.join("xl.meta"); + let dst = destination.join("xl.meta"); + std::fs::write(&dst, b"old").expect("old metadata"); + let mut staged = create_prepared_rename_source_with_commit_guard(&src, &dst, &guard).expect("staged metadata"); + staged.write_all(b"new", false).expect("new metadata"); + if replace_directory { + std::fs::rename(&destination, temp.path().join("detached")).expect("replace destination"); + std::fs::create_dir(&destination).expect("replacement directory"); + std::fs::write(&dst, b"old").expect("replacement metadata"); + } else { + std::fs::rename(&src, source.join("detached")).expect("replace staged file"); + std::fs::write(&src, b"unrelated").expect("replacement file"); + } + let lease = acquire_namespace_mutation_lease(&dst).await; + assert!( + rename_all_with_prepared_source(staged, &src, &dst, temp.path(), &root, &guard, lease) + .await + .is_err() + ); + assert_eq!(std::fs::read(&dst).expect("old metadata survives"), b"old"); + } + } +} diff --git a/crates/ecstore/src/ecstore_validation_blackbox.rs b/crates/ecstore/src/ecstore_validation_blackbox.rs index 6fccc53b5..1ce563974 100644 --- a/crates/ecstore/src/ecstore_validation_blackbox.rs +++ b/crates/ecstore/src/ecstore_validation_blackbox.rs @@ -560,7 +560,10 @@ async fn blackbox_heal_requests_preserve_repair_scope() { assert_eq!(request.pool_index, Some(0)); assert_eq!(request.set_index, Some(0)); assert_eq!(request.priority, HealChannelPriority::Low); - assert_eq!(request.recreate_missing, Some(true)); + assert_eq!( + request.recreate_missing, None, + "a degraded read does not instruct recreation; storage decides absence" + ); let mpu_bucket = "bb-mpu-convergence-heal"; let partial_object = "partial.bin"; diff --git a/crates/ecstore/src/error/conversion_roundtrip_tests.rs b/crates/ecstore/src/error/conversion_roundtrip_tests.rs index 9a01d16a2..9a1db49cb 100644 --- a/crates/ecstore/src/error/conversion_roundtrip_tests.rs +++ b/crates/ecstore/src/error/conversion_roundtrip_tests.rs @@ -34,6 +34,19 @@ use crate::disk::error_reduce::{OBJECT_OP_IGNORED_ERRS, reduce_errs}; use crate::error::StorageError; use rustfs_protos::proto_gen::node_service::Error as WireError; +#[test] +fn conditional_delete_errors_preserve_disk_storage_and_wire_identity() { + for disk_error in [DiskError::ConditionalDeleteConflict, DiskError::GuardedDeleteUnsupported] { + let storage: StorageError = disk_error.clone().into(); + let raw_code = storage.to_u32(); + assert_eq!(StorageError::from_u32(raw_code), Some(storage.clone())); + assert_eq!(storage.narrow_to_disk().expect("guard errors must narrow"), disk_error); + let wire: WireError = disk_error.clone().into(); + let decoded: DiskError = wire.into(); + assert_eq!(decoded, disk_error, "guard errors must survive peer RPC"); + } +} + /// The variants heal / replication / quorum repair match on, and that must /// survive the DiskError → StorageError → DiskError loop unchanged. fn heal_matched_disk_variants() -> Vec { diff --git a/crates/ecstore/src/error/mod.rs b/crates/ecstore/src/error/mod.rs index aba4fc411..e3824a8cb 100644 --- a/crates/ecstore/src/error/mod.rs +++ b/crates/ecstore/src/error/mod.rs @@ -264,6 +264,11 @@ pub enum StorageError { StorageFull, #[error("Please reduce your request rate")] SlowDown, + /// Delete recovery found a legacy residue it cannot classify. This is a + /// deterministic ambiguity that retrying cannot resolve; the object is + /// retained for operator inspection. + #[error("delete recovery found an ambiguous legacy residue; operator inspection is required")] + DeleteRecoveryLegacyAmbiguous, #[error("Decommission not started")] DecommissionNotStarted, #[error("Decommission already running")] @@ -290,6 +295,10 @@ pub enum StorageError { ConfigNotFound, #[error("Precondition failed")] PreconditionFailed, + #[error("Object generation changed during conditional deletion")] + ConditionalDeleteConflict, + #[error("The storage peer does not support guarded deletion")] + GuardedDeleteUnsupported, #[error("Not modified")] NotModified, #[error("Invalid range specified: {0}")] @@ -469,6 +478,8 @@ impl From for StorageError { DiskError::SourceStalled => StorageError::SourceStalled, DiskError::Timeout => StorageError::Timeout, DiskError::InvalidPath => StorageError::InvalidPath, + DiskError::ConditionalDeleteConflict => StorageError::ConditionalDeleteConflict, + DiskError::GuardedDeleteUnsupported => StorageError::GuardedDeleteUnsupported, DiskError::RemoteClientUnavailable(detail) => StorageError::RemoteClientUnavailable(detail), } } @@ -512,6 +523,8 @@ impl StorageError { StorageError::SourceStalled => DiskError::SourceStalled, StorageError::Timeout => DiskError::Timeout, StorageError::InvalidPath => DiskError::InvalidPath, + StorageError::ConditionalDeleteConflict => DiskError::ConditionalDeleteConflict, + StorageError::GuardedDeleteUnsupported => DiskError::GuardedDeleteUnsupported, StorageError::ErasureReadQuorum => DiskError::ErasureReadQuorum, StorageError::ErasureWriteQuorum => DiskError::ErasureWriteQuorum, StorageError::TooManyOpenFiles => DiskError::TooManyOpenFiles, @@ -696,6 +709,7 @@ impl Clone for StorageError { StorageError::BucketExists(a) => StorageError::BucketExists(a.clone()), StorageError::StorageFull => StorageError::StorageFull, StorageError::SlowDown => StorageError::SlowDown, + StorageError::DeleteRecoveryLegacyAmbiguous => StorageError::DeleteRecoveryLegacyAmbiguous, StorageError::PrefixAccessDenied(a, b) => StorageError::PrefixAccessDenied(a.clone(), b.clone()), StorageError::InvalidUploadIDKeyCombination(a, b) => { StorageError::InvalidUploadIDKeyCombination(a.clone(), b.clone()) @@ -758,6 +772,8 @@ impl Clone for StorageError { StorageError::InsufficientReadQuorum(a, b) => StorageError::InsufficientReadQuorum(a.clone(), b.clone()), StorageError::InsufficientWriteQuorum(a, b) => StorageError::InsufficientWriteQuorum(a.clone(), b.clone()), StorageError::PreconditionFailed => StorageError::PreconditionFailed, + StorageError::ConditionalDeleteConflict => StorageError::ConditionalDeleteConflict, + StorageError::GuardedDeleteUnsupported => StorageError::GuardedDeleteUnsupported, StorageError::NotModified => StorageError::NotModified, StorageError::InvalidPartNumber(a) => StorageError::InvalidPartNumber(*a), StorageError::InvalidRangeSpec(a) => StorageError::InvalidRangeSpec(a.clone()), @@ -835,6 +851,7 @@ impl StorageError { StorageError::BucketExists(_) => StorageErrorCode::BucketExists, StorageError::StorageFull => StorageErrorCode::StorageFull, StorageError::SlowDown => StorageErrorCode::SlowDown, + StorageError::DeleteRecoveryLegacyAmbiguous => StorageErrorCode::DeleteRecoveryLegacyAmbiguous, StorageError::PrefixAccessDenied(_, _) => StorageErrorCode::PrefixAccessDenied, StorageError::InvalidUploadIDKeyCombination(_, _) => StorageErrorCode::InvalidUploadIDKeyCombination, StorageError::MalformedUploadID(_) => StorageErrorCode::MalformedUploadID, @@ -871,6 +888,8 @@ impl StorageError { StorageError::InsufficientReadQuorum(_, _) => StorageErrorCode::InsufficientReadQuorum, StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum, StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed, + StorageError::ConditionalDeleteConflict => StorageErrorCode::ConditionalDeleteConflict, + StorageError::GuardedDeleteUnsupported => StorageErrorCode::GuardedDeleteUnsupported, StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall, StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge, StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec, @@ -938,6 +957,7 @@ impl StorageError { StorageErrorCode::BucketExists => Some(StorageError::BucketExists(Default::default())), StorageErrorCode::StorageFull => Some(StorageError::StorageFull), StorageErrorCode::SlowDown => Some(StorageError::SlowDown), + StorageErrorCode::DeleteRecoveryLegacyAmbiguous => Some(StorageError::DeleteRecoveryLegacyAmbiguous), StorageErrorCode::PrefixAccessDenied => { Some(StorageError::PrefixAccessDenied(Default::default(), Default::default())) } @@ -992,6 +1012,8 @@ impl StorageError { Some(StorageError::InsufficientWriteQuorum(Default::default(), Default::default())) } StorageErrorCode::PreconditionFailed => Some(StorageError::PreconditionFailed), + StorageErrorCode::ConditionalDeleteConflict => Some(StorageError::ConditionalDeleteConflict), + StorageErrorCode::GuardedDeleteUnsupported => Some(StorageError::GuardedDeleteUnsupported), StorageErrorCode::EntityTooSmall => { Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default())) } diff --git a/crates/ecstore/src/layout/endpoint.rs b/crates/ecstore/src/layout/endpoint.rs index 5e9ca0c58..f93e68808 100644 --- a/crates/ecstore/src/layout/endpoint.rs +++ b/crates/ecstore/src/layout/endpoint.rs @@ -101,23 +101,30 @@ impl TryFrom<&str> for Endpoint { let path = Path::new(&path).absolutize()?; #[cfg(windows)] - let path = if has_leading_slash_windows_drive(&path) { + let path = match strip_windows_drive_url_path(&path) { // Url::path() exposes file-like Windows paths as `/C:/...`. // Strip only that synthetic leading slash; plain URL paths // such as `/export1` must stay URL paths, not become // relative paths under the current drive. - Path::new(&path[1..]).absolutize()?.to_string_lossy().into_owned() - } else { - path + Some(drive_path) => Path::new(drive_path).absolutize()?.to_string_lossy().into_owned(), + None => path, }; #[cfg(windows)] let path = Path::new(&path); debug!("endpoint try_from: path={}", path.display()); - if path.parent().is_none() || path.as_os_str().is_empty() { + // A Windows drive root such as `C:\` has no parent, but it is + // accepted for parity with the path-style `C:\` endpoint. + #[cfg(windows)] + let is_root = path.parent().is_none() && !is_windows_drive_root(path); + #[cfg(not(windows))] + let is_root = path.parent().is_none(); + if is_root || path.as_os_str().is_empty() { return Err(Error::other("empty or root path is not supported in URL endpoint")); } + #[cfg(windows)] + reject_windows_unc_path(path)?; match path.to_str() { Some(v) => url.set_path(v), @@ -230,20 +237,63 @@ impl Endpoint { std::borrow::Cow::Borrowed(path) } }; + // Url::path() always starts with `/`, so every scheme stores a Windows + // drive path as `/C:/...`. Restore the drive path for `file` and + // `http(s)` endpoints alike; Windows would otherwise resolve `/C:/data` + // as an invalid `C:` component under the current drive. #[cfg(windows)] - if self.url.scheme() == "file" { - let stripped: &str = decoded.strip_prefix('/').unwrap_or(&decoded); - debug!("get_file_path windows: path={}", stripped); - return stripped.to_string(); + if let Some(drive_path) = strip_windows_drive_url_path(&decoded) { + debug!("get_file_path windows: path={}", drive_path); + return drive_path.to_string(); } decoded.into_owned() } } -#[cfg(windows)] -fn has_leading_slash_windows_drive(path: &str) -> bool { +/// Returns `C:/...` for a URL path shaped like `/C:/...`, otherwise `None`. +/// +/// The separator after the colon is required: `/C:foo` is a plain URL path, +/// and `/export1` must never be reinterpreted as a drive path. +#[cfg_attr(not(windows), allow(dead_code))] +fn strip_windows_drive_url_path(path: &str) -> Option<&str> { let bytes = path.as_bytes(); - bytes.len() >= 4 && bytes[0] == b'/' && bytes[1].is_ascii_alphabetic() && bytes[2] == b':' && bytes[3] == b'/' + if bytes.len() >= 4 && bytes[0] == b'/' && bytes[1].is_ascii_alphabetic() && bytes[2] == b':' && bytes[3] == b'/' { + Some(&path[1..]) + } else { + None + } +} + +#[cfg(windows)] +fn is_windows_drive_root(path: &Path) -> bool { + use std::path::{Component, Prefix}; + + let mut components = path.components(); + matches!( + components.next(), + Some(Component::Prefix(prefix)) if matches!(prefix.kind(), Prefix::Disk(_) | Prefix::VerbatimDisk(_)) + ) && matches!(components.next(), Some(Component::RootDir)) + && components.next().is_none() +} + +/// UNC shares are rejected instead of supported: `Url::from_file_path` turns +/// `\\nas\share\d1` into `file://nas/share/d1`, so the share server would be +/// read as the endpoint host and the share name dropped from the drive path, +/// and a network share does not provide the local-volume semantics (atomic +/// rename, fsync, volume identity) that erasure-coded drives rely on. +#[cfg(windows)] +fn reject_windows_unc_path(path: &Path) -> Result<()> { + use std::path::{Component, Prefix}; + + if let Some(Component::Prefix(prefix)) = path.components().next() + && matches!(prefix.kind(), Prefix::UNC(..) | Prefix::VerbatimUNC(..)) + { + return Err(Error::other( + "UNC network share paths are not supported as drive endpoints; use a local volume path such as \ +D:\\rustfs\\data1, or http://host:9000/D:/rustfs/data1 for URL endpoints", + )); + } + Ok(()) } /// parse a file path into a URL. @@ -261,6 +311,9 @@ fn url_parse_from_file_path(value: &str) -> Result { Err(err) => return Err(Error::other(format!("absolute path failed: {err}"))), }; + #[cfg(windows)] + reject_windows_unc_path(&file_path)?; + match Url::from_file_path(file_path) { Ok(url) => Ok(url), Err(_) => Err(Error::other("Convert a file path into an URL failed")), @@ -538,17 +591,110 @@ mod test { assert_eq!(url_endpoint.get_file_path(), "/path/to/data"); } + #[test] + fn strip_windows_drive_url_path_requires_drive_separator() { + assert_eq!(strip_windows_drive_url_path("/C:/data1"), Some("C:/data1")); + assert_eq!(strip_windows_drive_url_path("/d:/rustfs/data1"), Some("d:/rustfs/data1")); + assert_eq!(strip_windows_drive_url_path("/C:/"), Some("C:/")); + assert_eq!(strip_windows_drive_url_path("/C:foo"), None); + assert_eq!(strip_windows_drive_url_path("/C:"), None); + assert_eq!(strip_windows_drive_url_path("/export1"), None); + assert_eq!(strip_windows_drive_url_path("/1:/data"), None); + assert_eq!(strip_windows_drive_url_path("C:/data1"), None); + } + + #[cfg(not(windows))] + #[test] + fn url_endpoint_drive_shaped_path_stays_a_url_path_off_windows() { + let endpoint = Endpoint::try_from("http://host:9000/C:/data1").unwrap(); + assert_eq!(endpoint.get_file_path(), "/C:/data1"); + } + #[cfg(windows)] #[test] - fn test_windows_url_drive_path_requires_separator_after_colon() { - let drive_path_endpoint = Endpoint::try_from("http://host/C:/data").unwrap(); - assert_eq!(drive_path_endpoint.get_type(), EndpointType::Url); - assert!(has_leading_slash_windows_drive(Url::parse("http://host/C:/data").unwrap().path())); + fn windows_url_endpoint_file_path_restores_drive_path() { + for (arg, expected) in [ + ("http://host:9000/C:/data1", "C:/data1"), + ("https://host:9000/d:/rustfs/data4", "d:/rustfs/data4"), + ("http://host:9000/C:\\data1", "C:/data1"), + ("http://host:9000/C:/rustfs/../data1", "C:/data1"), + ("http://host:9000/C:/Program%20Files/rustfs", "C:/Program Files/rustfs"), + ] { + let endpoint = Endpoint::try_from(arg).unwrap_or_else(|err| panic!("{arg}: {err}")); + assert_eq!(endpoint.get_type(), EndpointType::Url, "{arg}"); + assert_eq!(endpoint.get_file_path(), expected, "{arg}"); + assert!(Path::new(&endpoint.get_file_path()).is_absolute(), "{arg}"); + } + } - let url_path_endpoint = Endpoint::try_from("http://host/C:foo").unwrap(); - assert_eq!(url_path_endpoint.get_type(), EndpointType::Url); - assert!(!has_leading_slash_windows_drive(Url::parse("http://host/C:foo").unwrap().path())); - assert_eq!(url_path_endpoint.get_file_path(), "/C:foo"); + #[cfg(windows)] + #[test] + fn windows_url_endpoint_keeps_url_identity_after_drive_path_parse() { + let endpoint = Endpoint::try_from("http://host:9000/C:/data1").unwrap(); + assert_eq!(endpoint.url.as_str(), "http://host:9000/C:/data1"); + assert_eq!(endpoint.to_string(), "http://host:9000/C:/data1"); + assert_eq!(endpoint.host_port(), "host:9000"); + } + + #[cfg(windows)] + #[test] + fn windows_url_endpoint_keeps_plain_url_paths() { + for (arg, expected) in [("http://host:9000/export1", "/export1"), ("http://host:9000/C:foo", "/C:foo")] { + let endpoint = Endpoint::try_from(arg).unwrap(); + assert_eq!(endpoint.get_type(), EndpointType::Url, "{arg}"); + assert_eq!(endpoint.get_file_path(), expected, "{arg}"); + } + } + + #[cfg(windows)] + #[test] + fn windows_url_and_path_endpoints_share_drive_file_path() { + let path_endpoint = Endpoint::try_from("C:\\rustfs\\data1").unwrap(); + let url_endpoint = Endpoint::try_from("http://host:9000/C:/rustfs/data1").unwrap(); + let verbatim_endpoint = Endpoint::try_from("\\\\?\\C:\\rustfs\\data1").unwrap(); + + assert_eq!(path_endpoint.get_type(), EndpointType::Path); + assert_eq!(path_endpoint.get_file_path(), "C:/rustfs/data1"); + assert_eq!(url_endpoint.get_file_path(), path_endpoint.get_file_path()); + assert_eq!(verbatim_endpoint.get_file_path(), path_endpoint.get_file_path()); + } + + #[cfg(windows)] + #[test] + fn windows_url_endpoint_resolves_existing_drive_directory() { + let dir = tempfile::tempdir().unwrap(); + let drive_path = dir.path().to_string_lossy().replace('\\', "/"); + let endpoint = Endpoint::try_from(format!("http://127.0.0.1:9000/{drive_path}").as_str()).unwrap(); + + let file_path = endpoint.get_file_path(); + assert_eq!(Path::new(&file_path), dir.path()); + rustfs_utils::canonicalize(&file_path) + .unwrap_or_else(|err| panic!("URL endpoint path '{file_path}' must resolve to the temp directory: {err}")); + } + + #[cfg(windows)] + #[test] + fn windows_drive_root_endpoints_are_accepted_in_both_forms() { + let path_endpoint = Endpoint::try_from("D:\\").unwrap(); + assert_eq!(path_endpoint.get_file_path(), "D:/"); + + let url_endpoint = Endpoint::try_from("http://host:9000/D:/").unwrap(); + assert_eq!(url_endpoint.get_type(), EndpointType::Url); + assert_eq!(url_endpoint.get_file_path(), "D:/"); + } + + #[cfg(windows)] + #[test] + fn windows_unc_endpoints_are_rejected() { + for arg in [ + "\\\\nas\\share\\d1", + "//nas/share/d1", + "\\\\?\\UNC\\nas\\share\\d1", + "http://host:9000//nas/share/d1", + ] { + let err = Endpoint::try_from(arg).expect_err(arg); + assert!(err.to_string().contains("UNC network share paths are not supported"), "{arg}: {err}"); + } } #[test] diff --git a/crates/ecstore/src/layout/endpoints.rs b/crates/ecstore/src/layout/endpoints.rs index d6f17d9b3..95abdc22a 100644 --- a/crates/ecstore/src/layout/endpoints.rs +++ b/crates/ecstore/src/layout/endpoints.rs @@ -1729,11 +1729,21 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()> Err(Error::other(format!( "local erasure endpoints must use distinct physical disks; detected shared devices [{details}]. \ -validation diagnostics: [{diagnostics_summary}]. \ +validation diagnostics: [{diagnostics_summary}]. {SHARED_DEVICE_PLATFORM_HINT}\ Set {ENV_UNSAFE_BYPASS_DISK_CHECK}=true only for local testing or CI to bypass this safety check" ))) } +/// Windows reports the volume mount point (for example `C:\`) as the device +/// identity, so every directory on one drive letter or mounted volume counts +/// as the same physical disk. +#[cfg(windows)] +const SHARED_DEVICE_PLATFORM_HINT: &str = "On Windows the device identity is the volume mount point, so directories on the same \ +drive letter or mounted volume (for example C:\\data1 and C:\\data2) count as one disk; place each endpoint on its own \ +volume. "; +#[cfg(not(windows))] +const SHARED_DEVICE_PLATFORM_HINT: &str = ""; + fn validate_local_cross_device_mounts(local_paths: &[String]) -> Result<()> { rustfs_utils::os::check_cross_device_mounts(local_paths) .map_err(|err| Error::other(format!("local endpoint cross-device mount validation failed: {err}"))) @@ -3581,6 +3591,38 @@ mod test { } } + #[cfg(windows)] + #[serial] + #[tokio::test] + async fn windows_url_endpoints_on_one_volume_resolve_and_report_shared_volume() { + async_with_vars([(ENV_UNSAFE_BYPASS_DISK_CHECK, None::<&str>), (ENV_MINIO_CI, None::<&str>)], async { + let dir = tempfile::tempdir().unwrap(); + let mut endpoints = Vec::new(); + for name in ["disk1", "disk2"] { + let disk = dir.path().join(name); + std::fs::create_dir_all(&disk).unwrap(); + let drive_path = disk.to_string_lossy().replace('\\', "/"); + let mut endpoint = Endpoint::try_from(format!("http://127.0.0.1:9000/{drive_path}").as_str()).unwrap(); + endpoint.is_local = true; + endpoints.push(endpoint); + } + + let err = validate_local_physical_disk_independence(&[Endpoints::from(endpoints)]).unwrap_err(); + + // Before the drive-path fix, `/C:/...` failed canonicalization here + // with "failed to resolve local endpoint path". + let err_text = err.to_string(); + assert!( + !err_text.contains("failed to resolve local endpoint path"), + "unexpected error: {err_text}" + ); + assert!(err_text.contains("distinct physical disks"), "unexpected error: {err_text}"); + assert!(err_text.contains("same drive letter or mounted volume"), "unexpected error: {err_text}"); + assert!(err_text.contains(ENV_UNSAFE_BYPASS_DISK_CHECK), "unexpected error: {err_text}"); + }) + .await; + } + #[cfg(target_os = "linux")] #[serial] #[tokio::test] diff --git a/crates/ecstore/src/object_api/types.rs b/crates/ecstore/src/object_api/types.rs index 1b9048bba..179ac5b9c 100644 --- a/crates/ecstore/src/object_api/types.rs +++ b/crates/ecstore/src/object_api/types.rs @@ -898,6 +898,10 @@ pub enum WriteCompletion { /// Drain the rename fan-out before returning. Minority failures still heal /// after a successful quorum commit; this does not require every disk to succeed. TailDrained, + /// Drain the rename fan-out and return system-metadata repair to the + /// caller's existing deployment/transaction owner. This does not grant + /// write or heal authority and does not change the quorum requirement. + TailDrainedWithOwnerRepair, } /// Storage-owned write mode inherited by physical rewrites. This selects the @@ -922,6 +926,10 @@ pub struct ObjectOptions { /// Lifecycle-only staged purge request checked under the object write lock. #[doc(hidden)] pub lifecycle_delete_all: Option, + /// Policy observed by lifecycle evaluation, checked under the metadata + /// transaction lock before logical deletion. Not persisted or sent over RPC. + #[doc(hidden)] + pub lifecycle_expiry_policy: Option, #[doc(hidden)] pub lifecycle_delete_all_journal: Option>>, /// Whole-operation authorization created only by consuming a validated @@ -942,6 +950,11 @@ pub struct ObjectOptions { /// serialized or copied into object metadata. #[doc(hidden)] pub delete_mutation_owner: Option>, + /// Pool-local generations for admitted current-object conditional actions. + /// Ordinary actions execute in a separate phase without this transient map. + /// This must never be populated from client metadata. + #[doc(hidden)] + pub conditional_delete_generations: Option>>, /// Internal read-only inspection must not enqueue metadata or payload repairs. pub suppress_read_repair: bool, /// Control-plane writers that immediately read or CAS the same namespace @@ -1006,6 +1019,9 @@ pub struct ObjectOptions { /// Proves an upper layer holds the bucket lifecycle sentinel. A separate /// fence avoids recursively acquiring the read lock behind a queued writer. pub bucket_lifecycle_lock_fence: Option, + /// Retains the actual lifecycle guard for an owned multipart commit. A + /// lock-loss fence alone does not keep a cancelled caller's lock alive. + pub bucket_lifecycle_lock_owner: Option>, pub replication_request: bool, /// True when the inbound request carried the /// `{x-rustfs-,x-minio-}source-proxy-request` header family with the @@ -1163,8 +1179,10 @@ impl std::fmt::Debug for ObjectOptions { .field("delete_replication", &self.delete_replication.is_some()) .field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot) .field("delete_mutation_owner", &self.delete_mutation_owner.is_some()) + .field("conditional_delete_generations", &self.conditional_delete_generations.is_some()) .field("namespace_lock_fence", &self.namespace_lock_fence.is_some()) .field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some()) + .field("bucket_lifecycle_lock_owner", &self.bucket_lifecycle_lock_owner.is_some()) .field("replication_request", &self.replication_request) .field("proxy_request", &self.proxy_request) .field("proxy_header_set", &self.proxy_header_set) diff --git a/crates/ecstore/src/object_api/write_commit_context.rs b/crates/ecstore/src/object_api/write_commit_context.rs index 769255c1d..84a7cb389 100644 --- a/crates/ecstore/src/object_api/write_commit_context.rs +++ b/crates/ecstore/src/object_api/write_commit_context.rs @@ -14,6 +14,7 @@ use super::{ObjectOptions, WriteCompletion}; use crate::error::{Error, Result}; +use rustfs_utils::path::object_name_disk_identity; use std::sync::Arc; use std::time::Duration; @@ -53,7 +54,7 @@ impl WriteCommitGuard { pub(crate) fn from_guard(guard: Arc, bucket: &str, object: &str) -> Self { Self { bucket: bucket.into(), - object: object.into(), + object: object_name_disk_identity(object).as_ref().into(), guards: Arc::new(vec![guard]), } } @@ -68,8 +69,10 @@ impl WriteCommitGuard { self.guards.iter().any(|guard| guard.is_lock_lost() || guard.is_released()) } + /// Namespace locks key the disk identity of the name (see `new_ns_lock`), + /// so a guard proves every spelling that resolves to the same file. fn matches(&self, bucket: &str, object: &str) -> bool { - self.bucket.as_ref() == bucket && self.object.as_ref() == object + self.bucket.as_ref() == bucket && self.object.as_ref() == object_name_disk_identity(object).as_ref() } } diff --git a/crates/ecstore/src/runtime/instance.rs b/crates/ecstore/src/runtime/instance.rs index d5c575ef0..4438665d6 100644 --- a/crates/ecstore/src/runtime/instance.rs +++ b/crates/ecstore/src/runtime/instance.rs @@ -49,6 +49,7 @@ use crate::layout::endpoints::{EndpointServerPools, SetupType}; use crate::object_api::ObjectEncryptionResolver; use crate::services::event_notification::EventNotifier; use crate::services::tier::tier::TierConfigMgr; +use crate::store::delete_recovery_runtime::DeleteRecoveryHints; use rustfs_lock::{GlobalLockManager, get_global_lock_manager}; use s3s::region::Region; use std::collections::{HashMap, HashSet}; @@ -240,6 +241,7 @@ pub struct InstanceContext { namespace_commit_generation: AtomicU64, /// Resolves object-encryption material at the application boundary. object_encryption_resolver: OnceLock>, + delete_recovery_hints: DeleteRecoveryHints, tier_delete_journal_recovery_stores: std::sync::Mutex>, #[cfg(test)] suppress_tier_delete_journal_recovery: bool, @@ -294,6 +296,7 @@ impl InstanceContext { namespace_commits: AtomicU64::new(0), namespace_commit_generation: AtomicU64::new(0), object_encryption_resolver: OnceLock::new(), + delete_recovery_hints: DeleteRecoveryHints::default(), tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()), #[cfg(test)] suppress_tier_delete_journal_recovery: false, @@ -465,6 +468,10 @@ impl InstanceContext { self.namespace_commit_generation.load(Ordering::Acquire) } + /// Whether a namespace commit is still pending. The scanner publication + /// barrier no longer consults it (an ordinary PUT shades its bucket + /// instead); fixtures use it to drain rename tails deterministically. + #[cfg(any(test, feature = "test-util"))] pub(crate) fn namespace_commits_pending(&self) -> bool { self.namespace_commits.load(Ordering::Acquire) != 0 || self.namespace_commit_generation() == u64::MAX } @@ -728,6 +735,10 @@ impl InstanceContext { self.background_cancel_token.get_or_init(|| token).clone() } + pub(crate) fn delete_recovery_hints(&self) -> &DeleteRecoveryHints { + &self.delete_recovery_hints + } + pub(crate) fn mark_tier_delete_journal_recovery_started(&self, store_id: Uuid) -> bool { #[cfg(test)] if self.suppress_tier_delete_journal_recovery { diff --git a/crates/ecstore/src/services/metrics_realtime.rs b/crates/ecstore/src/services/metrics_realtime.rs index 41d2843c3..551bae35c 100644 --- a/crates/ecstore/src/services/metrics_realtime.rs +++ b/crates/ecstore/src/services/metrics_realtime.rs @@ -144,6 +144,8 @@ fn to_madmin_scanner_metrics(metrics: rustfs_scanner_metrics::metrics::ScannerMe current_cycle_yield_duration_seconds: metrics.current_cycle_yield_duration_seconds, current_cycle_throttle_sleep_events: metrics.current_cycle_throttle_sleep_events, current_cycle_throttle_sleep_duration_seconds: metrics.current_cycle_throttle_sleep_duration_seconds, + current_cycle_disk_io_budget_wait_events: metrics.current_cycle_disk_io_budget_wait_events, + current_cycle_disk_io_budget_wait_duration_seconds: metrics.current_cycle_disk_io_budget_wait_duration_seconds, current_cycle_ilm_actions: metrics.current_cycle_ilm_actions, current_cycle_lifecycle_expiry_actions: metrics.current_cycle_lifecycle_expiry_actions, current_cycle_lifecycle_transition_actions: metrics.current_cycle_lifecycle_transition_actions, @@ -167,6 +169,8 @@ fn to_madmin_scanner_metrics(metrics: rustfs_scanner_metrics::metrics::ScannerMe last_cycle_yield_duration_seconds: metrics.last_cycle_yield_duration_seconds, last_cycle_throttle_sleep_events: metrics.last_cycle_throttle_sleep_events, last_cycle_throttle_sleep_duration_seconds: metrics.last_cycle_throttle_sleep_duration_seconds, + last_cycle_disk_io_budget_wait_events: metrics.last_cycle_disk_io_budget_wait_events, + last_cycle_disk_io_budget_wait_duration_seconds: metrics.last_cycle_disk_io_budget_wait_duration_seconds, last_cycle_ilm_actions: metrics.last_cycle_ilm_actions, last_cycle_heal_objects: metrics.last_cycle_heal_objects, last_cycle_replication_checks: metrics.last_cycle_replication_checks, @@ -184,6 +188,7 @@ fn to_madmin_scanner_metrics(metrics: rustfs_scanner_metrics::metrics::ScannerMe last_cycle_budget_limited: metrics.pacing_pressure.last_cycle_budget_limited, last_cycle_pause_observed: metrics.pacing_pressure.last_cycle_pause_observed, last_cycle_throttle_sleep_ratio: metrics.pacing_pressure.last_cycle_throttle_sleep_ratio, + last_cycle_disk_io_budget_wait_ratio: metrics.pacing_pressure.last_cycle_disk_io_budget_wait_ratio, last_cycle_yield_ratio: metrics.pacing_pressure.last_cycle_yield_ratio, last_cycle_total_pause_ratio: metrics.pacing_pressure.last_cycle_total_pause_ratio, }, @@ -231,6 +236,12 @@ fn to_madmin_scanner_metrics(metrics: rustfs_scanner_metrics::metrics::ScannerMe deferred_total: metrics.usage_freshness.deferred_total, last_deferred_unix_secs: metrics.usage_freshness.last_deferred_unix_secs, last_deferred_reason: metrics.usage_freshness.last_deferred_reason, + guardrail_veto_total: metrics.usage_freshness.guardrail_veto_total, + last_guardrail_veto_unix_secs: metrics.usage_freshness.last_guardrail_veto_unix_secs, + last_guardrail_veto_reason: metrics.usage_freshness.last_guardrail_veto_reason, + disk_io_budget_wait_events: metrics.usage_freshness.disk_io_budget_wait_events, + disk_io_budget_wait_millis: metrics.usage_freshness.disk_io_budget_wait_millis, + last_disk_io_budget_wait_unix_secs: metrics.usage_freshness.last_disk_io_budget_wait_unix_secs, }, maintenance_control: MadminScannerMaintenanceControlSnapshot { primary_control: metrics.maintenance_control.primary_control, @@ -747,6 +758,7 @@ mod test { last_cycle_budget_limited: true, last_cycle_pause_observed: true, last_cycle_throttle_sleep_ratio: 0.25, + last_cycle_disk_io_budget_wait_ratio: 0.15, last_cycle_yield_ratio: 0.05, last_cycle_total_pause_ratio: 0.3, }, @@ -886,10 +898,19 @@ mod test { deferred_total: 15, last_deferred_unix_secs: 16, last_deferred_reason: "data_movement".to_string(), + guardrail_veto_total: 4, + last_guardrail_veto_unix_secs: 17, + last_guardrail_veto_reason: "publication_blocked@".to_string(), + disk_io_budget_wait_events: 5, + disk_io_budget_wait_millis: 2_500, + last_disk_io_budget_wait_unix_secs: 18, }, ..Default::default() }); + assert_eq!(scanner.usage_freshness.guardrail_veto_total, 4); + assert_eq!(scanner.usage_freshness.last_guardrail_veto_unix_secs, 17); + assert_eq!(scanner.usage_freshness.last_guardrail_veto_reason, "publication_blocked@"); assert_eq!(scanner.usage_freshness.dirty_pending_buckets, 3); assert_eq!(scanner.usage_freshness.last_dirty_mark_unix_secs, 10); assert_eq!(scanner.usage_freshness.last_dirty_clear_unix_secs, 11); diff --git a/crates/ecstore/src/services/notification_sys.rs b/crates/ecstore/src/services/notification_sys.rs index 655caa47e..7db0ccba6 100644 --- a/crates/ecstore/src/services/notification_sys.rs +++ b/crates/ecstore/src/services/notification_sys.rs @@ -157,6 +157,18 @@ struct PeerAdminCache { last_server_success: Option, } +fn scanner_publication_lease_acquisition_error(host: &str, observational_only: bool, error: Error) -> Error { + let purpose = if observational_only { + "observational" + } else { + "authoritative" + }; + // Retry classification depends on the original error suffix. + Error::other(format!( + "scanner publication lease acquisition failed: peer={host}, purpose={purpose}: {error}" + )) +} + #[derive(Default)] struct TierConfigReloadWorkers { peers: HashMap, @@ -546,6 +558,17 @@ pub fn acquire_cross_pool_fence_fleet_proof() -> Option Option<&'static str> { + let state = cross_pool_fence_fleet_proof_slot() + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if state.topology_conflict { + return None; + } + REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get().map(String::as_str) +} + pub fn cross_pool_fence_fleet_proof_matches(proof: &CrossPoolFenceFleetProofToken) -> bool { fleet_capability_proof_matches(cross_pool_fence_fleet_proof_slot(), &proof.0) } @@ -970,8 +993,8 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() { } } -#[cfg(test)] -pub(crate) struct CrossPoolFenceFleetProofGuard { +#[cfg(any(test, feature = "test-util"))] +pub struct CrossPoolFenceFleetProofGuard { previous_proof: Option, previous_topology_conflict: bool, previous_journal_proof: Option, @@ -980,7 +1003,7 @@ pub(crate) struct CrossPoolFenceFleetProofGuard { previous_decommission_topology_conflict: bool, } -#[cfg(test)] +#[cfg(any(test, feature = "test-util"))] impl Drop for CrossPoolFenceFleetProofGuard { fn drop(&mut self) { let mut state = cross_pool_fence_fleet_proof_slot() @@ -1020,8 +1043,19 @@ impl Drop for CrossPoolFenceFleetProofGuard { /// Temporarily revoke the test proof so activation paths can exercise their /// fail-closed behavior without changing the process-wide topology binding. +#[cfg(any(test, feature = "test-util"))] +pub fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard { + revoke_cross_pool_fence_fleet_proof_for_test(true) +} + +/// Simulate peer unavailability while retaining the configured topology identity. #[cfg(test)] -pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard { +pub(crate) fn without_live_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard { + revoke_cross_pool_fence_fleet_proof_for_test(false) +} + +#[cfg(any(test, feature = "test-util"))] +fn revoke_cross_pool_fence_fleet_proof_for_test(topology_conflict: bool) -> CrossPoolFenceFleetProofGuard { let mut state = cross_pool_fence_fleet_proof_slot() .write() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -1045,21 +1079,21 @@ pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceF state.draining_generation = Some(proof.generation); } } - state.topology_conflict = true; + state.topology_conflict |= topology_conflict; if let Some(proof) = journal_state.proof.take() { proof.generation.revoke(); if !proof.generation.is_drained() { journal_state.draining_generation = Some(proof.generation); } } - journal_state.topology_conflict = true; + journal_state.topology_conflict |= topology_conflict; if let Some(proof) = decommission_state.proof.take() { proof.generation.revoke(); if !proof.generation.is_drained() { decommission_state.draining_generation = Some(proof.generation); } } - decommission_state.topology_conflict = true; + decommission_state.topology_conflict |= topology_conflict; guard } @@ -2932,7 +2966,7 @@ impl NotificationSys { Ok(lease) => grants.push(ScannerPublicationLeaseGrant { host, lease }), Err(err) => { let _ = self.release_scanner_publication_leases(grants).await; - return Err(Error::other(format!("scanner publication lease acquisition failed: {err}"))); + return Err(scanner_publication_lease_acquisition_error(&host, observational_only, err)); } } } @@ -3931,6 +3965,22 @@ mod tests { use crate::bucket::lifecycle::tier_last_day_stats::LastDayTierStats; use rustfs_data_usage::TierStats; + #[test] + fn scanner_lease_acquisition_error_identifies_peer_and_preserves_retry_suffix() { + for (observational_only, purpose) in [(false, "authoritative"), (true, "observational")] { + let reason = "scanner publication lease is blocked by pending namespace commit"; + let error = scanner_publication_lease_acquisition_error( + "peer-a.example.invalid:9000", + observational_only, + Error::other(reason), + ) + .to_string(); + assert!(error.contains("peer-a.example.invalid:9000"), "missing peer: {error}"); + assert!(error.contains(purpose), "missing purpose: {error}"); + assert!(error.ends_with(reason), "retry classification must retain the original suffix"); + } + } + fn ring(total_size: u64) -> LastDayTierStats { let mut stats = LastDayTierStats::default(); stats.add_stats(TierStats { diff --git a/crates/ecstore/src/services/rebalance/control.rs b/crates/ecstore/src/services/rebalance/control.rs index a8fef5fd8..0ee302157 100644 --- a/crates/ecstore/src/services/rebalance/control.rs +++ b/crates/ecstore/src/services/rebalance/control.rs @@ -18,8 +18,8 @@ use super::{ encode_rebalance_stop_propagation_record, }; use crate::core::pools::{ - PoolMeta, PoolRebalanceActivationFence, acquire_pool_activation_fleet_proof, acquire_pool_rebalance_activation_locks, - pool_meta_has_active_decommission, + PoolMeta, PoolRebalanceActivationFence, PoolRebalanceWorkerFence, acquire_pool_activation_fleet_proof, + acquire_pool_rebalance_activation_locks, acquire_pool_rebalance_worker_locks, pool_meta_has_active_decommission, }; use crate::error::{Error, Result}; use crate::object_api::ObjectOptions; @@ -77,10 +77,33 @@ fn ensure_rebalance_activation_pool_meta_allowed(meta: &PoolMeta) -> Result<()> } pub(super) enum RebalanceWorkerActivationFence { - Ready(Box), + Ready(Box), NotStartedTerminal, } +async fn persisted_rebalance_requires_worker_activation(pool: Arc, expected_id: &str) -> Result +where + S: EcstoreObjectIO, +{ + let mut persisted = RebalanceMeta::new(); + persisted + .load_with_opts( + pool, + ObjectOptions { + no_lock: true, + ..Default::default() + }, + ) + .await?; + if persisted.id != expected_id { + return Err(Error::other(format!( + "rebalance metadata changed before worker activation: expected {expected_id}, found {}", + persisted.id + ))); + } + Ok(crate::services::rebalance::rebalance_requires_worker_activation(&persisted)) +} + pub(super) struct RebalanceRunGuard { _local_guard: tokio::sync::OwnedRwLockReadGuard<()>, persisted_guard: rustfs_lock::NamespaceLockGuard, @@ -577,48 +600,58 @@ impl ECStore { where S: EcstoreObjectIO + StorageNamespaceLocking, { - // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. + // Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These + // shared fences admit a peer without waiting for the run's own entries. let mut pool_meta_guard = self.pool_meta_save_gate.lock().await; pool_meta_guard.ensure_write_safe("rebalance worker activation")?; - // Classify the durable rebalance record while holding both namespace - // fences. A terminal record is a no-op and must not depend on the - // notification subsystem having published a fleet proof yet. - let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?; + let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?; let pool_meta = self - .load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation") + .load_runtime_pool_meta_under_worker_fence(&mut pool_meta_guard, &worker_fence, "rebalance worker activation") .await?; ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?; - - let mut persisted = RebalanceMeta::new(); - persisted - .load_with_opts( - pool, - ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await?; - if persisted.id != expected_id { - return Err(Error::other(format!( - "rebalance metadata changed before worker activation: expected {expected_id}, found {}", - persisted.id - ))); - } - - activation_fence.ensure_held()?; - if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) { + let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?; + worker_fence.ensure_held()?; + if !needs_activation { return Ok(RebalanceWorkerActivationFence::NotStartedTerminal); } - // Active worker admission still requires the fail-closed fleet proof. - // Attach it immediately before the final fence validation so expiry or - // topology changes are checked again at every later commit boundary. + let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?; + worker_fence.set_fleet_proof(fleet_proof); + worker_fence.ensure_held()?; + Ok(RebalanceWorkerActivationFence::Ready(Box::new(worker_fence))) + } + + pub(super) async fn fence_rebalance_worker_metadata_update( + &self, + pool: Arc, + expected_id: &str, + ) -> Result>> + where + S: EcstoreObjectIO + StorageNamespaceLocking, + { + // The admission read fences must already be dropped. Reacquire in the + // original order and revalidate every durable boundary across that gap. + let mut pool_meta_guard = self.pool_meta_save_gate.lock().await; + pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?; + let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?; + let pool_meta = self + .load_runtime_pool_meta_under_activation_fence( + &mut pool_meta_guard, + &activation_fence, + "rebalance worker metadata update", + ) + .await?; + ensure_rebalance_activation_pool_meta_allowed(&pool_meta)?; + let needs_activation = persisted_rebalance_requires_worker_activation(pool, expected_id).await?; + activation_fence.ensure_held()?; + if !needs_activation { + return Ok(None); + } + let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?; activation_fence.set_fleet_proof(fleet_proof); activation_fence.ensure_held()?; - - Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence))) + Ok(Some(Box::new(activation_fence))) } #[tracing::instrument(skip_all)] @@ -1816,6 +1849,107 @@ mod tests { assert_activation_locks_released(&store).await; } + #[tokio::test] + #[serial_test::serial] + async fn rebalance_init_missing_fleet_proof_preserves_metadata_and_rechecks_competing_run() { + let previous_id = "completed-before-capability-refresh"; + let terminal = RebalanceMeta { + id: previous_id.to_string(), + stopped_at: Some(OffsetDateTime::now_utc()), + pool_stats: vec![RebalanceStats::default(); 2], + ..Default::default() + }; + let (_temp_dirs, store, competing_store) = crate::services::rebalance::test_two_pool_stores(Some(terminal.clone())).await; + let disk_stats = vec![ + DiskStat { + total_space: 100, + available_space: 0, + }, + DiskStat { + total_space: 100, + available_space: 100, + }, + ]; + let initial_epoch = store.ctx.data_movement_operation_epoch(); + let proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test(); + set_rebalance_disk_stats_override_for_test(store.id, disk_stats.clone()); + let err = store + .init_rebalance_start(vec!["waiting-request".to_string()]) + .await + .expect_err("initialization must fail closed while the live proof is missing"); + assert!( + err.to_string() + .contains("pool activation requires a live fleet capability proof") + ); + assert!( + crate::core::pools::is_pool_activation_fleet_proof_error(&err), + "the admin handler retries init only on this error shape: {err}" + ); + assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch); + { + let local = store.rebalance_meta.read().await; + let local = local.as_ref().expect("failed initialization must preserve the previous run"); + assert_eq!(local.id, previous_id); + assert_eq!(local.stopped_at, terminal.stopped_at); + assert!(local.cancel.is_none(), "failed initialization must not start a worker"); + } + let mut persisted = RebalanceMeta::new(); + persisted + .load(store.pools[0].clone()) + .await + .expect("failed initialization must leave durable metadata readable"); + assert_eq!(persisted.id, previous_id); + assert_eq!(persisted.stopped_at, terminal.stopped_at); + + drop(proof_guard); + set_rebalance_disk_stats_override_for_test(competing_store.id, disk_stats.clone()); + let winning_id = competing_store + .init_rebalance_start(vec!["winning-request".to_string()]) + .await + .expect("a ready request must be able to initialize one run after the failed attempt"); + assert_ne!(winning_id, previous_id); + + // The waiting request still holds its old local snapshot. Its retry must + // recheck durable state under the activation locks before publishing. + set_rebalance_disk_stats_override_for_test(store.id, disk_stats); + let remote_conflict = store + .init_rebalance_start(vec!["waiting-request".to_string()]) + .await + .expect_err("the waiting request must not replace a durable competing run"); + assert!( + matches!( + crate::data_movement::data_movement_stage_source(&remote_conflict), + Some(Error::RebalanceAlreadyRunning) + ), + "the save-stage error must retain the typed competing-run conflict: {remote_conflict:?}" + ); + let local_conflict = competing_store + .init_rebalance_start(vec!["another-request".to_string()]) + .await + .expect_err("a local retry must not replace its active run"); + assert!( + matches!(local_conflict, Error::RebalanceAlreadyRunning), + "the local admission check must report the competing-run conflict: {local_conflict:?}" + ); + persisted + .load(store.pools[0].clone()) + .await + .expect("the competing run must remain durable after both rejected retries"); + assert_eq!(persisted.id, winning_id); + assert_eq!(persisted.pool_stats[0].buckets, vec!["winning-request".to_string()]); + assert_eq!( + store + .rebalance_meta + .read() + .await + .as_ref() + .expect("old local run must survive") + .id, + previous_id + ); + assert_eq!(store.ctx.data_movement_operation_epoch(), initial_epoch + 1); + } + #[tokio::test] #[serial_test::serial] async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() { @@ -1874,6 +2008,128 @@ mod tests { ); } + async fn assert_worker_write_upgrade_revalidates_durable_state(change: &str) { + let rebalance_id = "rebalance-worker-write-upgrade"; + let active = RebalanceMeta { + id: rebalance_id.to_string(), + percent_free_goal: 0.5, + pool_stats: vec![ + RebalanceStats { + participating: true, + init_capacity: 100, + info: RebalanceInfo { + status: RebalStatus::Started, + ..Default::default() + }, + ..Default::default() + }, + RebalanceStats::default(), + ], + ..Default::default() + }; + let (_temp_dirs, store, peer) = + crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some(active.clone())).await; + let arrived = Arc::new(tokio::sync::Notify::new()); + let release = Arc::new(tokio::sync::Notify::new()); + let worker = Arc::clone(&store); + let mut tasks = tokio::task::JoinSet::new(); + tasks.spawn( + super::super::runtime::REBALANCE_WORKER_WRITE_UPGRADE_BARRIER + .scope((Arc::clone(&arrived), Arc::clone(&release)), async move { + worker.start_rebalance_for_id(rebalance_id).await + }), + ); + tokio::time::timeout(std::time::Duration::from_secs(30), arrived.notified()) + .await + .expect("empty-queue activation must release its read fences before requesting write authority"); + + let mut changed = active.clone(); + match change { + "stopped" => { + changed.stopped_at = Some(OffsetDateTime::now_utc()); + changed.pool_stats[0].info.status = RebalStatus::Stopped; + } + "replaced" => changed.id = "replacement-rebalance-run".to_string(), + "decommission" => { + let mut pool_meta = peer.pool_meta.read().await.clone(); + pool_meta.pools[0].decommission = Some(crate::core::pools::PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + pool_meta + .save(peer.pools.clone()) + .await + .expect("a competing pool metadata writer must acquire the released pool fence"); + } + _ => panic!("unknown durable state transition"), + } + changed + .save_with_opts( + peer.pools[0].clone(), + ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("a competing run writer must acquire the released rebalance fence"); + release.notify_one(); + let result = tokio::time::timeout(std::time::Duration::from_secs(30), tasks.join_next()) + .await + .expect("activation must finish after the durable transition") + .expect("activation task must be tracked") + .expect("activation must not panic"); + match change { + "stopped" => result.expect("a durably stopped run must become an activation no-op"), + "replaced" => assert!( + result + .expect_err("a replaced run must reject stale activation") + .to_string() + .contains("expected"), + ), + "decommission" => assert!(matches!(result, Err(Error::DecommissionAlreadyRunning))), + _ => unreachable!(), + } + let mut persisted = RebalanceMeta::new(); + persisted + .load(peer.pools[0].clone()) + .await + .expect("the competing durable state must remain readable"); + assert_eq!(persisted.id, changed.id, "activation must not overwrite the run identity"); + assert_eq!(persisted.stopped_at, changed.stopped_at, "activation must not overwrite the stop marker"); + assert_eq!(persisted.pool_stats[0].info.status, changed.pool_stats[0].info.status); + assert!( + store + .rebalance_meta + .read() + .await + .as_ref() + .expect("local metadata must remain present") + .cancel + .is_none(), + "rejected activation must not install a worker token", + ); + assert_activation_locks_released(&store).await; + } + + #[tokio::test] + #[serial_test::serial] + async fn rebalance_worker_write_upgrade_rechecks_stop() { + assert_worker_write_upgrade_revalidates_durable_state("stopped").await; + } + + #[tokio::test] + #[serial_test::serial] + async fn rebalance_worker_write_upgrade_rechecks_run_identity() { + assert_worker_write_upgrade_revalidates_durable_state("replaced").await; + } + + #[tokio::test] + #[serial_test::serial] + async fn rebalance_worker_write_upgrade_rechecks_decommission() { + assert_worker_write_upgrade_revalidates_durable_state("decommission").await; + } + #[tokio::test] #[serial_test::serial] async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() { diff --git a/crates/ecstore/src/services/rebalance/entry.rs b/crates/ecstore/src/services/rebalance/entry.rs index 6a9987707..09cbdc4f4 100644 --- a/crates/ecstore/src/services/rebalance/entry.rs +++ b/crates/ecstore/src/services/rebalance/entry.rs @@ -50,7 +50,7 @@ fn ensure_rebalance_entry_active(cancel: &CancellationToken) -> Result<()> { Ok(()) } -#[cfg(test)] +#[cfg(any(test, feature = "test-util"))] tokio::task_local! { static REBALANCE_ENTRY_RUN_FENCE_BARRIER: (Arc, Arc); } @@ -268,7 +268,7 @@ impl ECStore { self.rebalance_run_guard(rebalance_id.as_ref(), "rebalance entry") }) .await?; - #[cfg(test)] + #[cfg(any(test, feature = "test-util"))] if let Ok((arrived, release)) = REBALANCE_ENTRY_RUN_FENCE_BARRIER.try_with(Clone::clone) { arrived.notify_one(); release.notified().await; @@ -836,7 +836,8 @@ pub mod test_util { _temp_dirs: Vec, store: Arc, cancel: CancellationToken, - barrier: crate::data_movement::SourceCleanupDeleteBarrier, + entry_arrived: Arc, + entry_release: Arc, stop_probe: super::super::control::RebalanceStopWaitProbe, entry_task: Option>>, } @@ -897,31 +898,36 @@ pub mod test_util { cached: Some(file_meta), reusable: false, }; - let barrier = crate::data_movement::SourceCleanupDeleteBarrier::install(bucket, object); + let entry_arrived = Arc::new(tokio::sync::Notify::new()); + let entry_release = Arc::new(tokio::sync::Notify::new()); let entry_store = Arc::clone(&store); let entry_set = Arc::clone(&set); let entry_cancel = cancel.clone(); - let entry_task = tokio::spawn(async move { - entry_store - .rebalance_entry( - RebalanceEntryTarget { - bucket: bucket.to_string(), - pool_index: 0, - }, - entry, - entry_set, - Arc::new(RebalanceBucketConfigs::default()), - Arc::from(rebalance_id), - entry_cancel, - ) - .await - }); + let entry_task = tokio::spawn(REBALANCE_ENTRY_RUN_FENCE_BARRIER.scope( + (Arc::clone(&entry_arrived), Arc::clone(&entry_release)), + async move { + entry_store + .rebalance_entry( + RebalanceEntryTarget { + bucket: bucket.to_string(), + pool_index: 0, + }, + entry, + entry_set, + Arc::new(RebalanceBucketConfigs::default()), + Arc::from(rebalance_id), + entry_cancel, + ) + .await + }, + )); Self { _temp_dirs, store, cancel, - barrier, + entry_arrived, + entry_release, stop_probe: super::super::control::RebalanceStopWaitProbe::install(rebalance_id), entry_task: Some(entry_task), } @@ -932,7 +938,9 @@ pub mod test_util { } pub async fn wait_until_entry_paused(&self) { - self.barrier.wait_until_paused().await; + tokio::time::timeout(std::time::Duration::from_secs(30), self.entry_arrived.notified()) + .await + .expect("the entry should acquire its run fence before pausing"); } pub async fn wait_until_admission_cancelled(&self) { @@ -946,7 +954,7 @@ pub mod test_util { } pub fn release_entry(&self) { - self.barrier.release(); + self.entry_release.notify_one(); } pub async fn assert_entry_cancelled(&mut self) { @@ -964,7 +972,7 @@ pub mod test_util { impl Drop for PausedRebalanceEntryTestFixture { fn drop(&mut self) { - self.barrier.release(); + self.entry_release.notify_one(); if let Some(task) = self.entry_task.take() { task.abort(); } @@ -1432,7 +1440,7 @@ mod tests { #[tokio::test] #[serial_test::serial] - async fn real_rebalance_entry_progresses_while_peer_activation_waits_for_run_fence() { + async fn real_rebalance_peer_activates_while_entry_retains_run_fence() { const REBALANCE_ID: &str = "rebalance-peer-activation-lock-order"; let (_temp_dirs, store, peer) = crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(Some( active_rebalance_meta(REBALANCE_ID), @@ -1448,7 +1456,7 @@ mod tests { let bucket = crate::disk::RUSTFS_META_BUCKET; let object = "rebalance-peer-activation-object"; let version_id = uuid::Uuid::new_v4(); - let payload = b"entry must drain before peer activation takes the pool fence".repeat(1024); + let payload = b"peer activation must not wait for this entry to drain".repeat(1024); let source_set = store.pools[0].get_disks_by_key(object); let target_set = store.pools[1].get_disks_by_key(object); let opts = ObjectOptions { @@ -1508,7 +1516,12 @@ mod tests { ); tokio::time::timeout(StdDuration::from_secs(30), attempted.notified()) .await - .expect("peer activation must attempt the persisted rebalance write fence"); + .expect("peer activation must attempt the persisted rebalance fence"); + tokio::time::timeout(StdDuration::from_secs(30), activation_result) + .await + .expect("peer activation must finish while the entry remains paused") + .expect("peer activation result should be sent") + .expect("joining the same run must not require its active entry to release the run fence"); release.notify_one(); tokio::time::timeout(StdDuration::from_secs(30), async { @@ -1523,10 +1536,6 @@ mod tests { }) .await .expect("entry and peer activation must both make progress"); - activation_result - .await - .expect("peer activation result should be sent") - .expect("peer activation must not time out behind the entry it blocks"); let mut reader = target_set .get_object_reader(bucket, object, None, HeaderMap::new(), &opts) diff --git a/crates/ecstore/src/services/rebalance/meta.rs b/crates/ecstore/src/services/rebalance/meta.rs index 8983f40cd..54f5efe8e 100644 --- a/crates/ecstore/src/services/rebalance/meta.rs +++ b/crates/ecstore/src/services/rebalance/meta.rs @@ -71,6 +71,31 @@ impl From for RebalStatus { } impl RebalanceMeta { + /// A historical run has no authority over the current deployment's pool + /// indices. Stop timestamps alone do not prove that its workers stopped. + pub(crate) fn has_terminal_topology(&self) -> bool { + !self.id.is_empty() + && !self.pool_stats.is_empty() + && self.pool_stats.iter().all(|pool| { + if pool.info.stopping + || !matches!(pool.info.status, RebalStatus::Completed | RebalStatus::Stopped | RebalStatus::Failed) + && (pool.participating || pool.info.status != RebalStatus::None) + { + return false; + } + let Some(message) = pool.info.last_error.as_deref() else { + return true; + }; + if !is_rebalance_stop_propagation_error(Some(message)) { + return true; + } + decode_rebalance_stop_propagation_record(message).is_some_and(|record| { + !record.has_failures() + && (record.stop_attempt_at.or(self.stopped_at).is_none() || record.terminal_reload_attempt_at.is_some()) + }) + }) + } + pub fn new() -> Self { Self::default() } diff --git a/crates/ecstore/src/services/rebalance/runtime.rs b/crates/ecstore/src/services/rebalance/runtime.rs index 39302d592..abead2485 100644 --- a/crates/ecstore/src/services/rebalance/runtime.rs +++ b/crates/ecstore/src/services/rebalance/runtime.rs @@ -27,6 +27,11 @@ use tokio::time::{Duration, Instant}; use tokio_util::sync::CancellationToken; use tracing::{debug, error, info, warn}; +#[cfg(test)] +tokio::task_local! { + pub(super) static REBALANCE_WORKER_WRITE_UPGRADE_BARRIER: (Arc, Arc); +} + pub(super) fn should_fail_repeated_rebalance_bucket_defer( deferred_buckets: &mut HashSet, bucket: &str, @@ -213,6 +218,22 @@ impl ECStore { } if must_persist { + // Shared admission grants no write authority. Release it before + // taking the exclusive metadata fences, then recheck stop/run/pool + // state before publishing the locally staged terminal candidate. + drop(activation_fence); + #[cfg(test)] + if let Ok((arrived, release)) = REBALANCE_WORKER_WRITE_UPGRADE_BARRIER.try_with(Clone::clone) { + arrived.notify_one(); + release.notified().await; + } + let Some(activation_fence) = self + .fence_rebalance_worker_metadata_update(pool.clone(), expected_id.as_ref()) + .await? + else { + cancel_tx.cancel(); + return Ok(false); + }; let save_result = resolve_rebalance_meta_save_result( self.save_rebalance_meta_under_activation_fence( pool, @@ -242,13 +263,14 @@ impl ECStore { cancel_tx.cancel(); return Err(err); } + } else { + if let Err(err) = activation_fence.ensure_held() { + let mut rebalance_meta = self.rebalance_meta.write().await; + rollback_local_rebalance_worker_activation(rebalance_meta.as_mut(), expected_id.as_ref(), &rx); + return Err(err); + } + drop(activation_fence); } - if !must_persist && let Err(err) = activation_fence.ensure_held() { - let mut rebalance_meta = self.rebalance_meta.write().await; - rollback_local_rebalance_worker_activation(rebalance_meta.as_mut(), expected_id.as_ref(), &rx); - return Err(err); - } - drop(activation_fence); if activation_outcome != RebalanceLocalActivationOutcome::Started { return Ok(must_persist); diff --git a/crates/ecstore/src/services/tier/tier.rs b/crates/ecstore/src/services/tier/tier.rs index 8d80f37e9..9c0f84907 100644 --- a/crates/ecstore/src/services/tier/tier.rs +++ b/crates/ecstore/src/services/tier/tier.rs @@ -2270,7 +2270,7 @@ pub(crate) fn tier_destination_id_from_metadata(metadata: &HashMap(handle: &Arc>, api: Arc) -> io::Result<()> + where + S: EcstoreObjectIO + + EcstoreObjectOperations + + NamespaceLocking + + ListOperations>, + { + // Periodic and committed-mutation refreshes call this shared entry directly. + let reload = Self::reload_handle_inner(handle, api); + #[cfg(feature = "diagnostic-read-version-caller")] + let reload = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::TierConfigReload, + reload, + ); + reload.await + } + + async fn reload_handle_inner(handle: &Arc>, api: Arc) -> io::Result<()> where S: EcstoreObjectIO + EcstoreObjectOperations @@ -12533,6 +12550,51 @@ mod tests { assert!(reloaded.iter().any(|intent| intent.mutation_id == prepared_id)); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn diagnostic_reload_shared_entry_preserves_source_and_api() { + use crate::api::diagnostic::{ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry}; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_isolated_s3_http_operation_context, + }; + + let store = Arc::new(CasConfigStore::default()); + empty_mgr() + .save_tiering_config_if_current(store.clone(), None) + .await + .expect("empty tier config fixture should persist"); + let mut request = S3HttpRequestGuard::new("GET"); + let get = request.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context().expect("GetObject context should be captured") + }); + + for (earlier, api) in [(None, None), (Some(ReadVersionEntrySource::ConfigRead), Some(get))] { + store.reload_contexts.lock().expect("context recorder").clear(); + let handle = TierConfigMgr::new(); + with_isolated_s3_http_operation_context(api, async { + let reload = TierConfigMgr::reload_handle_with(&handle, store.clone()); + if let Some(source) = earlier { + with_read_version_entry(source, reload).await + } else { + reload.await + } + }) + .await + .expect("shared reload entry should load config and scan intents"); + + let observed = store.reload_contexts.lock().expect("context recorder"); + assert!(observed.iter().any(|(boundary, _, _)| *boundary == "reader")); + assert!(observed.iter().any(|(boundary, _, _)| *boundary == "list")); + let expected = earlier.unwrap_or(ReadVersionEntrySource::TierConfigReload); + assert!(observed.iter().all(|(_, source, operation)| { + *source == Some(expected) && *operation == api.map(|context| context.operation_name()) + })); + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + } + #[tokio::test] async fn reload_handle_keeps_committed_block_until_publish_swaps_generation() { let store = Arc::new(CasConfigStore::default()); @@ -14725,8 +14787,17 @@ mod tests { release: tokio::sync::Notify, } + #[cfg(feature = "diagnostic-read-version-caller")] + type ReloadDiagnosticObservation = ( + &'static str, + Option, + Option<&'static str>, + ); + #[derive(Debug)] struct CasConfigStore { + #[cfg(feature = "diagnostic-read-version-caller")] + reload_contexts: Mutex>, objects: tokio::sync::Mutex, String)>>, legacy_state: tokio::sync::Mutex>>, if_match_race_rewrite: tokio::sync::Mutex)>>, @@ -14750,6 +14821,8 @@ mod tests { impl Default for CasConfigStore { fn default() -> Self { Self { + #[cfg(feature = "diagnostic-read-version-caller")] + reload_contexts: Mutex::new(Vec::new()), objects: tokio::sync::Mutex::new(HashMap::new()), legacy_state: tokio::sync::Mutex::new(None), if_match_race_rewrite: tokio::sync::Mutex::new(std::collections::VecDeque::new()), @@ -14904,6 +14977,12 @@ mod tests { _headers: Self::HeaderMap, _opts: &Self::ObjectOptions, ) -> Result { + #[cfg(feature = "diagnostic-read-version-caller")] + self.reload_contexts.lock().expect("context recorder").push(( + "reader", + crate::api::diagnostic::current_read_version_entry_source(), + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + )); let legacy_config_path = tier_config_path(TIER_CONFIG_LEGACY_FILE); let (data, etag) = if object == legacy_config_path { let state = self.legacy_state.lock().await; @@ -15259,6 +15338,12 @@ mod tests { _start_after: Option, _incl_deleted: bool, ) -> Result { + #[cfg(feature = "diagnostic-read-version-caller")] + self.reload_contexts.lock().expect("context recorder").push(( + "list", + crate::api::diagnostic::current_read_version_entry_source(), + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + )); if matches!( prefix, crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX diff --git a/crates/ecstore/src/set_disk/core/io_primitives.rs b/crates/ecstore/src/set_disk/core/io_primitives.rs index 8dfb6441a..88bee85d4 100644 --- a/crates/ecstore/src/set_disk/core/io_primitives.rs +++ b/crates/ecstore/src/set_disk/core/io_primitives.rs @@ -43,10 +43,10 @@ use super::super::{ EVENT_SET_DISK_RENAME_TAIL_DRAIN_FAILED, EVENT_SET_DISK_WRITE, Error, FileInfo, FileMeta, FileMetaShallowVersion, GetCodecStreamingFallbackReason, GetObjectMetadataCacheEntry, HTTPPreconditions, HashAlgorithm, HealAdmissionResult, HealChannelPriority, HealRequestSource, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, MultipartWriteQuorumContext, - OBJECT_OP_IGNORED_ERRS, ObjectOptions, ObjectPartInfo, OffsetDateTime, RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, - RawFileInfo, ReadMultipleReq, ReadMultipleResp, Result, SLASH_SEPARATOR, STORAGE_FORMAT_FILE, SetDisks, SnapshotLeaseToken, - StorageError, UpdateMetadataOpts, Uuid, capacity_scope_from_disks, current_dirty_generation, debug, disk, - file_info_is_valid_for_metadata, info, is_object_dangling, issue3031_diag_enabled, join_all, join_errs, + OBJECT_OP_IGNORED_ERRS, ObjectInfo, ObjectOptions, ObjectPartInfo, OffsetDateTime, RUSTFS_META_BUCKET, + RUSTFS_META_MULTIPART_BUCKET, RawFileInfo, ReadMultipleReq, ReadMultipleResp, Result, SLASH_SEPARATOR, STORAGE_FORMAT_FILE, + SetDisks, SnapshotLeaseToken, StorageError, UpdateMetadataOpts, Uuid, capacity_scope_from_disks, current_dirty_generation, + debug, disk, file_info_is_valid_for_metadata, info, is_object_dangling, issue3031_diag_enabled, join_all, join_errs, log_multipart_write_quorum_failure, merge_file_meta_versions, path_join_buf, record_global_dirty_scope, reduce_read_quorum_errs, reduce_write_quorum_errs, send_heal_request_with_admission, should_prevent_write, to_object_err, warn, @@ -145,7 +145,7 @@ use std::{ time::{Duration, Instant}, }; use tokio::io::{AsyncRead, ReadBuf}; -use tokio::sync::RwLock; +use tokio::sync::{RwLock, oneshot}; use tokio::task::JoinSet; pub(in crate::set_disk) const EVENT_SET_DISK_READ: &str = "set_disk_read"; @@ -601,7 +601,8 @@ pub(in crate::set_disk) async fn submit_read_repair_heal_with_submitter( ); request.source = HealRequestSource::ReadRepair; request.object_version_id = version_id.filter(|value| !value.is_empty()).map(str::to_string); - request.recreate_missing = Some(true); + // A degraded read's surviving minority is not a recreate instruction; the + // locked storage read decides whether the target is absent (backlog#2900). // Reader failures include checksum damage that metadata-only healing cannot detect. if matches!(reason, "missing_shards" | "decode_error") { request.scan_mode = Some(rustfs_heal_contracts::heal_channel::HealScanMode::Deep); @@ -2455,6 +2456,17 @@ impl RenameConvergence { } } +struct RenameDataObservations<'a> { + disks: &'a [Option], + file_infos: &'a [FileInfo], + disk_versions: &'a [Option>], + errs: &'a [Option], + cleanup_data_dirs: &'a [Option], + old_current_sizes: &'a [Option], + old_current_source_checked: &'a [bool], + old_current_sources: &'a [Option], +} + pub(in crate::set_disk) struct RenameDataCommit { pub(in crate::set_disk) online_disks: Vec>, /// Disks whose capacity must be marked for this commit. Under early ACK, @@ -2464,6 +2476,9 @@ pub(in crate::set_disk) struct RenameDataCommit { pub(in crate::set_disk) data_dir: Option, pub(in crate::set_disk) cleanup_disks: Vec>, pub(in crate::set_disk) old_current_size: Option, + pub(in crate::set_disk) old_current_source_checked: bool, + pub(in crate::set_disk) old_current_source_all_successful_checked: bool, + pub(in crate::set_disk) old_current_source: Option, pub(in crate::set_disk) committed_file_info: FileInfo, pub(in crate::set_disk) tail_drain: Option>>, } @@ -2479,6 +2494,500 @@ pub(in crate::set_disk) struct RenameTailOutcome { pub(in crate::set_disk) cleanup: Vec, } +const RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP"; +const RENAME_TAIL_CLEANUP_DEFER_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ENABLE"; +const RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY"; +const RENAME_TAIL_CLEANUP_DEFER_WORKERS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS"; +const RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY_MS"; +const RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW_MS"; +const RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_ENABLE_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_ENABLE"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_MIN_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_MIN"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_LOW_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_LOW"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_HIGH_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_HIGH"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS"; +const RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS"; +const RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER"; +const RENAME_FOREGROUND_ACTIVE_LANES_ENV: &str = "RUSTFS_PUT_RENAME_FOREGROUND_ACTIVE_LANES"; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY: usize = 1024; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_WORKERS: usize = 2; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX: usize = 1; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW: usize = 128; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH: usize = 512; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS: u64 = 500; +const DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS: u64 = 2_000; +const DEFAULT_RENAME_FOREGROUND_ACTIVE_LANES: usize = 1024; + +fn rename_tail_cleanup_counterfactual_skip_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_defer_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_DEFER_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_defer_queue_capacity() -> usize { + static CAPACITY: OnceLock = OnceLock::new(); + *CAPACITY.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY, + ) + .max(1) + }) +} + +fn rename_tail_cleanup_defer_workers() -> usize { + static WORKERS: OnceLock = OnceLock::new(); + *WORKERS.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_WORKERS_ENV, DEFAULT_RENAME_TAIL_CLEANUP_DEFER_WORKERS).max(1) + }) +} + +fn rename_tail_cleanup_defer_worker_delay() -> Duration { + static DELAY: OnceLock = OnceLock::new(); + *DELAY.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY_MS_ENV, 0))) +} + +fn rename_tail_cleanup_defer_batch_window() -> Duration { + static WINDOW: OnceLock = OnceLock::new(); + *WINDOW.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW_MS_ENV, 0))) +} + +fn rename_tail_cleanup_defer_batch_max() -> usize { + static BATCH_MAX: OnceLock = OnceLock::new(); + *BATCH_MAX.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX_ENV, DEFAULT_RENAME_TAIL_CLEANUP_DEFER_BATCH_MAX).max(1) + }) +} + +fn rename_tail_cleanup_defer_adaptive_batch_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_ENABLE_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_defer_adaptive_batch_min() -> usize { + static BATCH_MIN: OnceLock = OnceLock::new(); + *BATCH_MIN.get_or_init(|| rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_MIN_ENV, 1).max(1)) +} + +fn rename_tail_cleanup_defer_adaptive_batch_low() -> usize { + static BATCH_LOW: OnceLock = OnceLock::new(); + *BATCH_LOW.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_LOW_ENV, rename_tail_cleanup_defer_batch_max()) + .max(1) + }) +} + +fn rename_tail_cleanup_defer_adaptive_batch_high() -> usize { + static BATCH_HIGH: OnceLock = OnceLock::new(); + *BATCH_HIGH.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_BATCH_HIGH_ENV, + rename_tail_cleanup_defer_adaptive_batch_low().max(rename_tail_cleanup_defer_batch_max()), + ) + .max(1) + }) +} + +fn rename_tail_cleanup_defer_adaptive_queue_low() -> usize { + static QUEUE_LOW: OnceLock = OnceLock::new(); + *QUEUE_LOW.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_LOW, + ) + }) +} + +fn rename_tail_cleanup_defer_adaptive_queue_high() -> usize { + static QUEUE_HIGH: OnceLock = OnceLock::new(); + *QUEUE_HIGH.get_or_init(|| { + rustfs_utils::get_env_usize( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_QUEUE_HIGH, + ) + }) +} + +fn rename_tail_cleanup_defer_adaptive_age_low() -> Duration { + static AGE_LOW: OnceLock = OnceLock::new(); + *AGE_LOW.get_or_init(|| { + Duration::from_millis(rustfs_utils::get_env_u64( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_LOW_MS, + )) + }) +} + +fn rename_tail_cleanup_defer_adaptive_age_high() -> Duration { + static AGE_HIGH: OnceLock = OnceLock::new(); + *AGE_HIGH.get_or_init(|| { + Duration::from_millis(rustfs_utils::get_env_u64( + RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS_ENV, + DEFAULT_RENAME_TAIL_CLEANUP_DEFER_ADAPTIVE_AGE_HIGH_MS, + )) + }) +} + +fn rename_tail_cleanup_defer_batch_target(queued_depth: usize, oldest_age: Option) -> usize { + let fixed = rename_tail_cleanup_defer_batch_max(); + if !rename_tail_cleanup_defer_adaptive_batch_enabled() { + return fixed; + } + + let min = rename_tail_cleanup_defer_adaptive_batch_min(); + let low = rename_tail_cleanup_defer_adaptive_batch_low().max(min); + let high = rename_tail_cleanup_defer_adaptive_batch_high().max(low); + let queue_low = rename_tail_cleanup_defer_adaptive_queue_low(); + let queue_high = rename_tail_cleanup_defer_adaptive_queue_high().max(queue_low); + let age_low = rename_tail_cleanup_defer_adaptive_age_low(); + let age_high = rename_tail_cleanup_defer_adaptive_age_high().max(age_low); + + let age = oldest_age.unwrap_or_default(); + if queued_depth >= queue_high || age >= age_high { + high + } else if queued_depth >= queue_low || age >= age_low { + low + } else { + min + } +} + +fn rename_tail_cleanup_defer_hold_worker_enabled() -> bool { + static ENABLED: OnceLock = OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_foreground_active_lanes() -> &'static Vec { + static LANES: OnceLock> = OnceLock::new(); + LANES.get_or_init(|| { + let lanes = + rustfs_utils::get_env_usize(RENAME_FOREGROUND_ACTIVE_LANES_ENV, DEFAULT_RENAME_FOREGROUND_ACTIVE_LANES).max(1); + (0..lanes).map(|_| AtomicUsize::new(0)).collect() + }) +} + +pub(in crate::set_disk) fn rename_foreground_active_count(disk_index: usize) -> usize { + let lanes = rename_foreground_active_lanes(); + lanes[disk_index % lanes.len()].load(Ordering::Relaxed) +} + +struct RenameForegroundActiveGuard { + disk_index: usize, +} + +impl Drop for RenameForegroundActiveGuard { + fn drop(&mut self) { + let lanes = rename_foreground_active_lanes(); + lanes[self.disk_index % lanes.len()].fetch_sub(1, Ordering::Relaxed); + } +} + +fn rename_foreground_active_guard(disk_index: usize) -> RenameForegroundActiveGuard { + let lanes = rename_foreground_active_lanes(); + let active = lanes[disk_index % lanes.len()].fetch_add(1, Ordering::Relaxed) + 1; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_FOREGROUND_ACTIVE, + active as f64, + ); + RenameForegroundActiveGuard { disk_index } +} + +type RenameTailCleanupFuture = Pin + Send + 'static>>; + +struct RenameTailCleanupJob { + enqueued_at: Instant, + targets: usize, + future: RenameTailCleanupFuture, +} + +struct RenameTailCleanupQueue { + sender: tokio::sync::mpsc::Sender, + queued: Arc, + oldest_enqueued_at: Arc>>, +} + +async fn run_rename_tail_cleanup_job(job: RenameTailCleanupJob) { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_START_DELAY, + job.enqueued_at.elapsed().as_secs_f64() * 1000.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_TARGETS, + job.targets as f64, + ); + let worker_delay = rename_tail_cleanup_defer_worker_delay(); + if !worker_delay.is_zero() { + let delay_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(worker_delay).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY, + delay_started, + ); + } + let cleanup_started = rustfs_io_metrics::put_stage_timer(); + job.future.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DURATION, + cleanup_started, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED_LAG, + job.enqueued_at.elapsed().as_secs_f64() * 1000.0, + ); +} + +impl RenameTailCleanupQueue { + fn new() -> Self { + let (sender, receiver) = tokio::sync::mpsc::channel::(rename_tail_cleanup_defer_queue_capacity()); + let queued = Arc::new(AtomicUsize::new(0)); + let active_workers = Arc::new(AtomicUsize::new(0)); + let oldest_enqueued_at = Arc::new(tokio::sync::Mutex::new(None::)); + let receiver = Arc::new(tokio::sync::Mutex::new(receiver)); + if !rename_tail_cleanup_defer_hold_worker_enabled() { + for _ in 0..rename_tail_cleanup_defer_workers() { + let receiver = Arc::clone(&receiver); + let queued = Arc::clone(&queued); + let active_workers = Arc::clone(&active_workers); + let oldest_enqueued_at = Arc::clone(&oldest_enqueued_at); + tokio::spawn(async move { + loop { + let queued_before_poll = queued.load(Ordering::Relaxed); + let worker_poll_started = rustfs_io_metrics::put_stage_timer(); + let Some(job) = receiver.lock().await.recv().await else { + return; + }; + if queued_before_poll == 0 { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_IDLE_WAIT, + worker_poll_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_POLL_WAIT, + worker_poll_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_JOB_WAIT, + worker_poll_started, + ); + } + let mut batch = vec![job]; + let batch_window = rename_tail_cleanup_defer_batch_window(); + let queued_snapshot = queued.load(Ordering::Relaxed); + let oldest_snapshot = *oldest_enqueued_at.lock().await; + let batch_max = rename_tail_cleanup_defer_batch_target( + queued_snapshot, + oldest_snapshot.map(|oldest| oldest.elapsed()), + ); + if batch_max > 1 && !batch_window.is_zero() { + let window_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(batch_window).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW, + window_started, + ); + } + while batch.len() < batch_max { + let next = { + let mut locked = receiver.lock().await; + locked.try_recv().ok() + }; + let Some(next) = next else { + break; + }; + batch.push(next); + } + let queued_after_dequeue = queued.fetch_sub(batch.len(), Ordering::Relaxed).saturating_sub(batch.len()); + if queued_after_dequeue == 0 { + *oldest_enqueued_at.lock().await = None; + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + queued_after_dequeue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_dequeue as f64, + ); + if let Some(oldest) = *oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_SIZE, + batch.len() as f64, + ); + if batch.len() > 1 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BURST_BATCH_SIZE, + batch.len() as f64, + ); + } + let active = active_workers.fetch_add(1, Ordering::Relaxed) + 1; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_ACTIVE, + active as f64, + ); + for job in batch { + run_rename_tail_cleanup_job(job).await; + } + active_workers.fetch_sub(1, Ordering::Relaxed); + } + }); + } + } + Self { + sender, + queued, + oldest_enqueued_at, + } + } + + async fn try_enqueue(&self, job: RenameTailCleanupJob) -> std::result::Result<(), RenameTailCleanupJob> { + let queue_push_started = rustfs_io_metrics::put_stage_timer(); + { + let mut oldest = self.oldest_enqueued_at.lock().await; + if oldest.is_none() { + *oldest = Some(job.enqueued_at); + } + } + let queued_after_enqueue = self.queued.fetch_add(1, Ordering::Relaxed) + 1; + if rename_tail_cleanup_defer_hold_worker_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + queued_after_enqueue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_enqueue as f64, + ); + if let Some(oldest) = *self.oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + queue_push_started, + ); + return Ok(()); + } + match self.sender.try_send(job) { + Ok(()) => { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + queued_after_enqueue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_enqueue as f64, + ); + if let Some(oldest) = *self.oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + queue_push_started, + ); + Ok(()) + } + Err(err) => { + self.queued.fetch_sub(1, Ordering::Relaxed); + if self.queued.load(Ordering::Relaxed) == 0 { + *self.oldest_enqueued_at.lock().await = None; + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_FALLBACK_SYNC, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + self.queued.load(Ordering::Relaxed) as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + self.queued.load(Ordering::Relaxed) as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + queue_push_started, + ); + Err(err.into_inner()) + } + } + } +} + +fn rename_tail_cleanup_queue() -> &'static RenameTailCleanupQueue { + static QUEUE: OnceLock = OnceLock::new(); + QUEUE.get_or_init(RenameTailCleanupQueue::new) +} + const EVENT_SET_DISK_RENAME_ROLLBACK: &str = "set_disk_rename_rollback"; #[derive(Clone, Copy)] @@ -2573,10 +3082,9 @@ async fn inspect_incomplete_rename_rollback( request.recreate_missing = Some(false); request.update_parity = Some(false); request.recursive = Some(false); - match tokio::time::timeout(Duration::from_secs(1), submitter(request)).await { - Ok(result) => result, - Err(_) => ReadRepairAdmissionOutcome::Failed("rollback inspection admission timed out".to_string()), - } + tokio::time::timeout(Duration::from_secs(1), submitter(request)) + .await + .unwrap_or_else(|_| ReadRepairAdmissionOutcome::Failed("rollback inspection admission timed out".to_string())) } fn rename_rollback_task_outcome( @@ -2863,21 +3371,24 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal< SubmitFuture, >( tail_drain: tokio::task::JoinHandle>, - guard_release: tokio::sync::oneshot::Receiver, + guard_release: oneshot::Receiver, guards: Guards, request: rustfs_heal_contracts::heal_channel::HealChannelRequest, finalize: Finalize, cleanup: Cleanup, submit: Submit, ) where - Guards: Send, + Guards: Send + 'static, Finalize: FnOnce() -> FinalizeFuture + Send, FinalizeFuture: Future + Send, Cleanup: FnOnce(Guards, Vec) -> CleanupFuture + Send, CleanupFuture: Future + Send, + Cleanup: 'static, + CleanupFuture: 'static, Submit: FnOnce(rustfs_heal_contracts::heal_channel::HealChannelRequest) -> SubmitFuture + Send, SubmitFuture: Future + Send, { + let tail_async_started = rustfs_io_metrics::put_stage_timer(); let (needs_heal, tail_cleanup, tail_complete) = match tail_drain.await { Ok(Some(outcome)) => (outcome.convergence.needs_heal(), outcome.cleanup, true), Ok(None) => { @@ -2906,16 +3417,80 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal< (true, Vec::new(), false) } }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT, + tail_async_started, + ); // A dropped sender means the request continuation was cancelled. Cleanup // must still run; only the explicit `false` used by the hard-crash harness // suppresses all in-process continuation work. + let guard_release_wait_started = rustfs_io_metrics::put_stage_timer(); if matches!(guard_release.await, Ok(false)) { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT, + guard_release_wait_started, + ); drop(guards); return; } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT, + guard_release_wait_started, + ); + let finalize_started = rustfs_io_metrics::put_stage_timer(); finalize().await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_FINALIZE, + finalize_started, + ); if tail_complete { - cleanup(guards, tail_cleanup).await; + let cleanup_submit_started = rustfs_io_metrics::put_stage_timer(); + if rename_tail_cleanup_counterfactual_skip_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP, + 1.0, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + drop(guards); + } else if rename_tail_cleanup_defer_enabled() { + let targets = tail_cleanup.len(); + let job = RenameTailCleanupJob { + enqueued_at: Instant::now(), + targets, + future: Box::pin(cleanup(guards, tail_cleanup)), + }; + if let Err(job) = rename_tail_cleanup_queue().try_enqueue(job).await { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + let cleanup_started = rustfs_io_metrics::put_stage_timer(); + job.future.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP, + cleanup_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + } + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + cleanup_submit_started, + ); + let cleanup_started = rustfs_io_metrics::put_stage_timer(); + cleanup(guards, tail_cleanup).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP, + cleanup_started, + ); + } } else { // The outer coordinator may fail while a disk rename still owns the // staging source. Preserve it for later healing/GC instead of racing @@ -2923,7 +3498,12 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal< drop(guards); } if needs_heal { + let heal_submit_started = rustfs_io_metrics::put_stage_timer(); submit(request).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_HEAL_SUBMIT, + heal_submit_started, + ); } } @@ -2977,8 +3557,8 @@ impl SetDisks { bucket: &str, object: &str, write_quorum: usize, - ) -> crate::error::Result<(Vec>, Vec>)> { - let fence_path = crate::disk::quota_mutation_fence_path(bucket, object); + ) -> Result<(Vec>, Vec>)> { + let fence_path = disk::quota_mutation_fence_path(bucket, object); let results = join_all(disks.iter().map(|disk| { let disk = disk.clone(); let fence_path = fence_path.clone(); @@ -3020,8 +3600,8 @@ impl SetDisks { bucket: &str, object: &str, write_quorum: usize, - ) -> crate::error::Result<()> { - let fence_path = crate::disk::quota_mutation_fence_path(bucket, object); + ) -> Result<()> { + let fence_path = disk::quota_mutation_fence_path(bucket, object); let results = join_all(disks.iter().zip(tokens).filter_map(|(disk, token)| { let disk = disk.as_ref()?.clone(); let token = (*token)?; @@ -3043,18 +3623,36 @@ impl SetDisks { } } + #[allow(clippy::too_many_arguments)] fn rename_data_commit_from_observations( - disks: &[Option], - file_infos: &[FileInfo], - disk_versions: &[Option>], - errs: &[Option], - cleanup_data_dirs: &[Option], - old_current_sizes: &[Option], + observations: RenameDataObservations<'_>, write_quorum: usize, ) -> disk::error::Result { + let RenameDataObservations { + disks, + file_infos, + disk_versions, + errs, + cleanup_data_dirs, + old_current_sizes, + old_current_source_checked, + old_current_sources, + } = observations; let data_dir = Self::reduce_common_data_dir(cleanup_data_dirs, write_quorum); let convergence = Self::classify_rename_convergence(disk_versions, errs); let old_current_size = Self::reduce_common_old_current_size(old_current_sizes, write_quorum); + let old_current_source_quorum_checked = + old_current_source_checked.iter().filter(|checked| **checked).count() >= write_quorum; + let old_current_source_all_successful_checked = errs + .iter() + .enumerate() + .filter(|(_, err)| err.is_none()) + .all(|(idx, _)| old_current_source_checked.get(idx).copied().unwrap_or(false)); + let old_current_source = if old_current_source_quorum_checked { + Self::reduce_common_old_current_source(old_current_sources, write_quorum) + } else { + None + }; let online_disks = Self::eval_disks(disks, errs); let committed_slot = online_disks.iter().position(Option::is_some).ok_or(DiskError::Unexpected)?; let committed_file_info = file_infos.get(committed_slot).cloned().ok_or(DiskError::Unexpected)?; @@ -3082,6 +3680,9 @@ impl SetDisks { data_dir, cleanup_disks, old_current_size, + old_current_source_checked: old_current_source_quorum_checked, + old_current_source_all_successful_checked, + old_current_source, committed_file_info, tail_drain: None, }) @@ -3199,9 +3800,10 @@ impl SetDisks { let src_object = Arc::new(src_object.to_string()); let dst_bucket = Arc::new(dst_bucket.to_string()); let dst_object = Arc::new(dst_object.to_string()); - let (commit_tx, commit_rx) = tokio::sync::oneshot::channel(); + let (commit_tx, commit_rx) = oneshot::channel(); let coordinator_failure_receipt = rollback_receipt.clone(); + let early_ack_spawn_to_quorum_wait_started = rustfs_io_metrics::put_stage_timer(); let tail_drain = tokio::spawn({ let fanout_src_bucket = src_bucket.clone(); let fanout_src_object = src_object.clone(); @@ -3234,6 +3836,11 @@ impl SetDisks { let Some(disk) = disk else { return Err(DiskError::DiskNotFound); }; + let disk_wait_location_stage = if disk.is_local() { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL + } else { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE + }; let is_delete_marker = file_info.is_canonical_delete_marker(); // Clone FileInfo and set erasure.index for this disk @@ -3253,6 +3860,7 @@ impl SetDisks { let disk_wait_started = rustfs_io_metrics::put_stage_timer(); dispatch_state = RenameDispatchState::MayHavePublished; + let _foreground_active_guard = rename_foreground_active_guard(i); let observed = disk .rename_data_borrowed_with_fence_observed( &src_bucket, @@ -3260,7 +3868,7 @@ impl SetDisks { &file_info, &dst_bucket, &dst_object, - crate::disk::RenameDataGuards { + disk::RenameDataGuards { scanner_publication_lease_token, namespace_owner: namespace_commit_guard .clone() @@ -3281,6 +3889,7 @@ impl SetDisks { rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT, duration_ms, ); + rustfs_io_metrics::record_put_object_stage_duration(disk_wait_location_stage, duration_ms); let position = if result.is_ok() { let rank = successful_rename_completion_rank .as_ref() @@ -3319,9 +3928,12 @@ impl SetDisks { let mut data_dirs = vec![None; disk_count]; let mut cleanup_data_dirs = vec![None; disk_count]; let mut old_current_sizes = vec![None; disk_count]; + let mut old_current_source_checked = vec![false; disk_count]; + let mut old_current_sources = vec![None; disk_count]; let mut capacity_scope_generation = None; let mut cleanup_at_snapshot = vec![false; disk_count]; + let mut tail_after_quorum_started = None; while let Some(joined) = tasks.join_next().await { results_seen += 1; match joined { @@ -3331,6 +3943,8 @@ impl SetDisks { cleanup_data_dirs[idx] = res.cleanup_data_dir; disk_versions[idx] = res.sign; old_current_sizes[idx] = res.old_current_size; + old_current_source_checked[idx] = res.old_current_source_checked; + old_current_sources[idx] = res.old_current_source; errs[idx] = None; success_count += 1; } @@ -3350,12 +3964,16 @@ impl SetDisks { if !sent_commit && success_count >= write_quorum { let snapshot_commit = Self::rename_data_commit_from_observations( - &coordinator_disks, - &file_infos, - &disk_versions, - &errs, - &cleanup_data_dirs, - &old_current_sizes, + RenameDataObservations { + disks: &coordinator_disks, + file_infos: &file_infos, + disk_versions: &disk_versions, + errs: &errs, + cleanup_data_dirs: &cleanup_data_dirs, + old_current_sizes: &old_current_sizes, + old_current_source_checked: &old_current_source_checked, + old_current_sources: &old_current_sources, + }, write_quorum, ); if let Ok(commit) = snapshot_commit.as_ref() { @@ -3366,8 +3984,13 @@ impl SetDisks { let _ = commit_tx.send(snapshot_commit); } sent_commit = true; + tail_after_quorum_started = rustfs_io_metrics::put_stage_timer(); } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_AFTER_QUORUM_WAIT, + tail_after_quorum_started, + ); #[cfg(test)] rollback_fault_injection::after_fanout(&fanout_dst_object); @@ -3487,6 +4110,10 @@ impl SetDisks { } }); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT, + early_ack_spawn_to_quorum_wait_started, + ); let quorum_wait_started = rustfs_io_metrics::put_stage_timer(); let commit = match commit_rx.await { Ok(commit) => commit, @@ -3499,8 +4126,13 @@ impl SetDisks { rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT, quorum_wait_started, ); + let attach_tail_owner_started = rustfs_io_metrics::put_stage_timer(); commit.map(|mut commit| { commit.tail_drain = Some(tail_drain); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_ATTACH_TAIL_OWNER, + attach_tail_owner_started, + ); commit }) } @@ -3623,6 +4255,11 @@ impl SetDisks { let Some(disk) = disk else { return Err(DiskError::DiskNotFound); }; + let disk_wait_location_stage = if disk.is_local() { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL + } else { + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE + }; let is_delete_marker = file_info.is_canonical_delete_marker(); let mut local_file_info; @@ -3656,6 +4293,7 @@ impl SetDisks { let disk_wait_started = rustfs_io_metrics::put_stage_timer(); dispatch_state = RenameDispatchState::MayHavePublished; + let _foreground_active_guard = rename_foreground_active_guard(i); let observed = disk .rename_data_borrowed_with_fence_observed( &src_bucket, @@ -3663,7 +4301,7 @@ impl SetDisks { file_info, &dst_bucket, &dst_object, - crate::disk::RenameDataGuards { + disk::RenameDataGuards { scanner_publication_lease_token, namespace_owner: namespace_commit_guard .clone() @@ -3684,6 +4322,7 @@ impl SetDisks { rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT, duration_ms, ); + rustfs_io_metrics::record_put_object_stage_duration(disk_wait_location_stage, duration_ms); let position = if result.is_ok() { let rank = successful_rename_completion_rank .as_ref() @@ -3719,9 +4358,16 @@ impl SetDisks { let mut data_dirs = vec![None; disk_count]; let mut cleanup_data_dirs = vec![None; disk_count]; let mut old_current_sizes = vec![None; disk_count]; + let mut old_current_source_checked = vec![false; disk_count]; + let mut old_current_sources = vec![None; disk_count]; let quorum_wait_started = rustfs_io_metrics::put_stage_timer(); + let sync_full_fanout_started = rustfs_io_metrics::put_stage_timer(); let fanout_result = fanout.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_SYNC_FULL_FANOUT_WAIT, + sync_full_fanout_started, + ); rustfs_io_metrics::record_put_object_stage_duration_from( rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT, quorum_wait_started, @@ -3762,6 +4408,8 @@ impl SetDisks { cleanup_data_dirs[idx] = res.cleanup_data_dir; disk_versions[idx].clone_from(&res.sign); old_current_sizes[idx] = res.old_current_size; + old_current_source_checked[idx] = res.old_current_source_checked; + old_current_sources[idx] = res.old_current_source.clone(); errs.push(None); } Ok(Err(e)) => { @@ -3898,6 +4546,18 @@ impl SetDisks { let data_dir = Self::reduce_common_data_dir(&cleanup_data_dirs, write_quorum); let convergence = Self::classify_rename_convergence(&disk_versions, &errs); let old_current_size = Self::reduce_common_old_current_size(&old_current_sizes, write_quorum); + let old_current_source_quorum_checked = + old_current_source_checked.iter().filter(|checked| **checked).count() >= write_quorum; + let old_current_source_all_successful_checked = errs + .iter() + .enumerate() + .filter(|(_, err)| err.is_none()) + .all(|(idx, _)| old_current_source_checked.get(idx).copied().unwrap_or(false)); + let old_current_source = if old_current_source_quorum_checked { + Self::reduce_common_old_current_source(&old_current_sources, write_quorum) + } else { + None + }; let online_disks = Self::eval_disks(disks, &errs); let committed_slot = online_disks.iter().position(Option::is_some).ok_or(DiskError::Unexpected)?; let committed_file_info = std::mem::take(&mut file_infos[committed_slot]); @@ -3925,11 +4585,34 @@ impl SetDisks { data_dir, cleanup_disks, old_current_size, + old_current_source_checked: old_current_source_quorum_checked, + old_current_source_all_successful_checked, + old_current_source, committed_file_info, tail_drain: None, }) } + pub(in crate::set_disk) fn reduce_common_old_current_source( + old_current_sources: &[Option], + write_quorum: usize, + ) -> Option { + let mut counts: HashMap<[u8; 32], (usize, FileInfo)> = HashMap::new(); + + for source in old_current_sources.iter().flatten() { + let entry = counts + .entry(Self::file_info_quorum_hash(source)) + .or_insert_with(|| (0, source.clone())); + entry.0 += 1; + } + + counts + .into_values() + .filter(|(count, _)| *count >= write_quorum) + .max_by_key(|(count, _)| *count) + .map(|(_, source)| source) + } + /// rustfs/backlog#1009: reduce the per-disk observations of the /// destination's previous current version to one set-level value, mirroring /// `reduce_common_data_dir`: the observation reported by at least @@ -4995,6 +5678,7 @@ impl SetDisks { fi.set_tier_free_version_id(&Uuid::new_v4().to_string()); let disks = self.get_disks_internal().await; + self.invalidate_get_object_metadata_cache(bucket, object).await; let mut futures = Vec::with_capacity(disks.len()); for (disk_index, disk_op) in disks.iter().enumerate() { @@ -5075,6 +5759,9 @@ impl SetDisks { } else { false }; + if absent { + self.invalidate_get_object_metadata_cache(bucket, object).await; + } Ok((m, absent)) } @@ -5523,12 +6210,10 @@ impl SetDisks { /// * The set of referenced data dirs is the UNION of `get_data_dirs()` across /// every online disk's `xl.meta`, so a dir named by *any* replica is kept. /// * If a disk holds the object directory but its `xl.meta` is missing or - /// unparsable, the object is treated as degraded and unmarked data dirs are - /// never removed. A data dir carrying a committed delete-transaction marker - /// remains reclaimable after a downgrade/re-upgrade cleanup interruption. - /// * Only subdirectories whose names parse as a UUID are ever considered; - /// removal is non-recursive-safe via a recursive delete of the full stray - /// data-dir path only. + /// unparsable, leave its data to quorum-aware recovery and healing. + /// * Any outstanding delete transaction defers all candidates to recovery. + /// * UUID-shaped S3 prefixes are protected: only regular part files and native + /// sidecars qualify. The final mutation repeats this check and never recurses. /// /// Returns the number of stray data directories removed across the set. This /// is best-effort maintenance: individual delete failures are logged and @@ -5543,13 +6228,16 @@ impl SetDisks { async fn reclaim_orphan_data_dirs_inner(&self, bucket: &str, object: &str, dry_run: bool) -> disk::error::Result { let disks = self.get_disks_internal().await; + if disks.len() != self.set_drive_count || disks.iter().any(Option::is_none) { + // An unavailable participant can still reference an apparent orphan. + return Ok(0); + } // Phase 1 (read-only): build the referenced-data-dir union and record the // physical UUID subdirectories present on each disk. Abort on any degraded // copy so a healable object is never stripped of a referenced data dir. let mut referenced: HashSet = HashSet::new(); - let mut per_disk_dirs: Vec<(usize, Vec<(Uuid, bool)>)> = Vec::new(); - let mut healthy_metas = 0usize; + let mut per_disk_dirs: Vec<(usize, Vec)> = Vec::new(); for (i, disk) in disks.iter().enumerate() { let Some(disk) = disk else { continue }; @@ -5584,22 +6272,6 @@ impl SetDisks { // to the orphan-dir / dangling-object heal paths. continue; } - let mut committed = Vec::with_capacity(physical.len()); - for dir in physical { - let data_dir = format!("{object}/{dir}"); - let committed_delete = disk.list_dir("", bucket, &data_dir, 0).await.is_ok_and(|entries| { - entries.iter().any(|entry| { - entry - .strip_prefix(DELETE_DATA_DIR_MARKER_PREFIX) - .is_some_and(|transaction| Uuid::parse_str(transaction).is_ok()) - }) - }); - committed.push((dir, committed_delete)); - } - if committed.iter().all(|(_, committed_delete)| *committed_delete) { - per_disk_dirs.push((i, committed)); - continue; - } warn!( target: "rustfs_ecstore::set_disk", bucket, object, @@ -5634,21 +6306,39 @@ impl SetDisks { } } - healthy_metas += 1; if !physical.is_empty() { - per_disk_dirs.push((i, physical.into_iter().map(|dir| (dir, false)).collect())); + per_disk_dirs.push((i, physical)); } } - // Phase 2: delete every physical data dir not referenced by the union. + if !per_disk_dirs + .iter() + .any(|(_, dirs)| dirs.iter().any(|dir| !referenced.contains(dir))) + { + return Ok(0); + } + // A neighboring journal can still need an unreferenced payload for + // rollback. Objects whose physical dirs are all referenced pay no + // recovery-discovery overhead. + for disk in disks.iter().flatten() { + match disk.has_delete_transaction_evidence(bucket, object).await { + Ok(false) => {} + _ => return Ok(0), + } + } + + // Phase 2: classify candidates consistently for dry-run and reclamation. let mut removed = 0usize; for (i, physical) in per_disk_dirs { let Some(disk) = disks[i].as_ref() else { continue }; - for (dir, committed_delete) in physical { - if referenced.contains(&dir) || (healthy_metas == 0 && !committed_delete) { + for dir in physical { + if referenced.contains(&dir) { continue; } let stray = format!("{object}/{dir}"); + if !disk.is_plain_orphan_data_dir(bucket, &stray).await.unwrap_or(false) { + continue; + } if dry_run { removed += 1; debug!( @@ -5664,18 +6354,18 @@ impl SetDisks { continue; } match disk - .delete( + .delete_data_dir( bucket, &stray, DeleteOptions { - recursive: true, + orphan_data_dir: true, immediate: true, ..Default::default() }, ) .await { - Ok(()) => { + Ok(DataDirDeleteStatus::Deleted) => { removed += 1; info!( target: "rustfs_ecstore::set_disk", @@ -5683,7 +6373,8 @@ impl SetDisks { "reclaim_orphan_data_dirs: removed orphaned data directory" ); } - Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Ok(DataDirDeleteStatus::Deferred) + | Err(DiskError::FileNotFound | DiskError::VolumeNotFound | DiskError::DiskOngoingReq) => {} Err(err) => { warn!( target: "rustfs_ecstore::set_disk", @@ -5716,6 +6407,45 @@ impl SetDisks { } } + pub(crate) fn evaluate_write_precondition( + oi: Result, + http_preconditions: &HTTPPreconditions, + bucket: &str, + object: &str, + opts: &ObjectOptions, + ) -> Option { + match oi { + Ok(oi) => { + // Ordinary writes may proceed past a top-level delete marker; + // data movement and guarded internal writes must preserve it. + if oi.delete_marker { + if opts.data_movement || opts.preserve_delete_marker { + return Some(StorageError::PreconditionFailed); + } + return Self::write_precondition_lookup_error( + StorageError::ObjectNotFound(bucket.to_string(), object.to_string()), + http_preconditions, + bucket, + object, + ); + } + let if_none_match = http_preconditions.if_none_match_value().map(str::to_owned); + let if_match = http_preconditions.if_match_value().map(str::to_owned); + if should_prevent_write(&oi, if_none_match, if_match) + && !crate::data_movement::can_replace_stale_data_movement_target(&oi, opts) + { + return Some(StorageError::PreconditionFailed); + } + } + + Err(error) => { + return Self::write_precondition_lookup_error(error, http_preconditions, bucket, object); + } + } + + None + } + pub(in crate::set_disk) async fn check_write_precondition( &self, bucket: &str, @@ -5731,30 +6461,19 @@ impl SetDisks { // - If no_lock is false, we must have obtained the lock out side of this function // - If no_lock is true, we should not obtain locks lookup_opts.no_lock = true; - let oi = self.get_object_info(bucket, object, &lookup_opts).await; - - match oi { - Ok(oi) => { - // Ordinary writes may proceed past a top-level delete marker; - // data movement and guarded internal writes must preserve it. - if oi.delete_marker { - return (opts.data_movement || opts.preserve_delete_marker).then_some(StorageError::PreconditionFailed); - } - let if_none_match = http_preconditions.if_none_match_value().map(str::to_owned); - let if_match = http_preconditions.if_match_value().map(str::to_owned); - if should_prevent_write(&oi, if_none_match, if_match) - && !crate::data_movement::can_replace_stale_data_movement_target(&oi, opts) - { - return Some(StorageError::PreconditionFailed); - } - } - - Err(error) => { - return Self::write_precondition_lookup_error(error, &http_preconditions, bucket, object); - } - } - - None + let lookup = self.get_object_info(bucket, object, &lookup_opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let lookup = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::WritePreconditionRead, + async { + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::WritePreconditionLookup, + ); + lookup.await + }, + ); + let oi = lookup.await; + Self::evaluate_write_precondition(oi, &http_preconditions, bucket, object, opts) } } @@ -6094,6 +6813,12 @@ pub(crate) mod disk_call_counters { /// Registry key: (object, kind, disk_index). type CountKey = (String, String, usize); + #[cfg(feature = "diagnostic-read-version-caller")] + type ReadVersionContext = ( + Option<&'static str>, + Option, + ); + #[derive(Default)] struct Registry { /// Object names with an active observing scope. Only these accumulate, @@ -6101,6 +6826,8 @@ pub(crate) mod disk_call_counters { observed: HashSet, /// (object, kind, disk_index) -> call count. counts: HashMap, + #[cfg(feature = "diagnostic-read-version-caller")] + read_version_contexts: HashMap<(String, usize), Vec>, } fn registry() -> &'static Mutex { @@ -6118,6 +6845,17 @@ pub(crate) mod disk_call_counters { *reg.counts .entry((object.to_string(), kind.to_string(), disk_index)) .or_insert(0) += 1; + #[cfg(feature = "diagnostic-read-version-caller")] + if kind == KIND_READ_VERSION { + let context = ( + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + crate::cluster::rpc::runtime_sources::current_read_version_request_caller(), + ); + reg.read_version_contexts + .entry((object.to_string(), disk_index)) + .or_default() + .push(context); + } } /// Matches shard paths to the observed object's namespace path. This also @@ -6160,6 +6898,15 @@ pub(crate) mod disk_call_counters { } impl CallCounterScope { + #[cfg(feature = "diagnostic-read-version-caller")] + pub fn read_version_contexts_for_disk(&self, disk_index: usize) -> Vec { + let reg = registry().lock().expect("disk call-counter registry poisoned"); + reg.read_version_contexts + .get(&(self.object.clone(), disk_index)) + .cloned() + .unwrap_or_default() + } + /// Total number of `kind` calls across all disks for the observed object. pub fn total(&self, kind: &str) -> u64 { let reg = registry().lock().expect("disk call-counter registry poisoned"); @@ -6185,6 +6932,8 @@ pub(crate) mod disk_call_counters { let mut reg = registry().lock().expect("disk call-counter registry poisoned"); reg.observed.remove(&self.object); reg.counts.retain(|(obj, _, _), _| obj != &self.object); + #[cfg(feature = "diagnostic-read-version-caller")] + reg.read_version_contexts.retain(|(obj, _), _| obj != &self.object); } } } @@ -6570,9 +7319,9 @@ mod tests { ); scope.try_begin().expect("delete scope should enter flight"); let scope_guard = crate::object_api::ScannerPublicationCommitScopeGuard::new(scope.clone()); - let (started_tx, started_rx) = tokio::sync::oneshot::channel(); - let (release_tx, release_rx) = tokio::sync::oneshot::channel(); - let (finished_tx, finished_rx) = tokio::sync::oneshot::channel(); + let (started_tx, started_rx) = oneshot::channel(); + let (release_tx, release_rx) = oneshot::channel(); + let (finished_tx, finished_rx) = oneshot::channel(); let waiter = tokio::spawn(run_scanner_publication_delete_owner(Some(scope.clone()), move || async move { started_tx.send(()).expect("delete owner should start"); @@ -7326,6 +8075,290 @@ mod tests { drop(dirs); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_coalescer_restores_each_item_after_delayed_and_threshold_flush() { + use crate::cluster::rpc::runtime_sources::{ + ReadVersionDiagnosticContext, with_optional_read_version_request_caller, with_read_version_diagnostic_context, + with_read_version_entry, + }; + use crate::set_disk::core::metadata_read::enqueue_coalesced_read_version; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::{ + ReadVersionContextBoundary, ReadVersionEntrySource, ReadVersionRequestCaller, + }; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_isolated_s3_http_operation_context, + }; + let operation = |op| { + let mut guard = S3HttpRequestGuard::new("GET"); + guard.in_scope(|| { + rustfs_io_metrics::record_s3_op(op); + current_s3_http_operation_context().expect("operation should be captured") + }) + }; + let get = operation(rustfs_s3_ops::S3Operation::GetObject); + let list = operation(rustfs_s3_ops::S3Operation::ListObjectsV2); + let foreign = operation(rustfs_s3_ops::S3Operation::DeleteObject); + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + for (items, delay) in [(4, "2000"), (crate::disk::BATCH_READ_VERSION_MAX_ITEMS, "10000")] { + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + let bucket = "diagnostic-coalescer-context"; + let (_dirs, disks) = call_counter_local_disks(bucket, 1).await; + let disk = disks[0].clone().expect("test disk should exist"); + for case in 0..4 { + install_metadata_fanout_fileinfo(&disks, bucket, &format!("context-{case}"), None).await; + } + let mut contexts = Vec::new(); + for (api, caller, source) in [ + (Some(get), Some(ReadVersionRequestCaller::Object), ReadVersionEntrySource::ObjectRead), + (None, None, ReadVersionEntrySource::ConfigRead), + ( + Some(list), + Some(ReadVersionRequestCaller::ListObjects), + ReadVersionEntrySource::ListObjects, + ), + (None, Some(ReadVersionRequestCaller::Heal), ReadVersionEntrySource::LifecycleRecovery), + ] { + contexts.push( + with_isolated_s3_http_operation_context( + api, + with_optional_read_version_request_caller( + caller, + with_read_version_entry(source, async { ReadVersionDiagnosticContext::capture() }), + ), + ) + .await, + ); + } + let opts = ReadOptions::default(); + let foreign_context = with_isolated_s3_http_operation_context( + Some(foreign), + with_optional_read_version_request_caller( + Some(ReadVersionRequestCaller::Multipart), + with_read_version_entry(ReadVersionEntrySource::ObjectRead, async { + ReadVersionDiagnosticContext::capture() + }), + ), + ) + .await; + temp_env::async_with_vars([("RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS", Some(delay))], async { + with_read_version_diagnostic_context(foreign_context, ReadVersionContextBoundary::Entry, async { + let reads = (0..items).map(|index| { + let case = index % contexts.len(); + let item = BatchReadVersionItem { + org_volume: String::new(), + volume: bucket.to_owned(), + path: format!("context-{case}"), + version_id: String::new(), + }; + with_read_version_diagnostic_context( + contexts[case], + ReadVersionContextBoundary::EarlyStop, + enqueue_coalesced_read_version(disk.clone(), item, &opts), + ) + }); + let results = futures::future::join_all(reads).await; + for (index, result) in results.into_iter().enumerate() { + assert_eq!(result.expect("fallback read should succeed").name, format!("context-{}", index % 4)); + } + assert_eq!( + current_s3_http_operation_context(), + Some(foreign), + "flush must restore its caller's ambient scope" + ); + }) + .await; + }) + .await; + assert_eq!(current_s3_http_operation_context(), None); + }) + }); + let rows = snapshots.snapshot().into_vec(); + let item_label = items.to_string(); + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_get_metadata_read_version_coalescer_total" + && labels.get("event") == Some(&"attempted_batch") + && labels.get("item_count") == Some(&item_label.as_str()) + && matches!(value, DebugValue::Counter(1)) + }), + "all requests should flush in exactly one batch" + ); + for (source, api, caller) in [ + ("object_read", "s3:GetObject", "object"), + ("config_read", "unattributed", "unattributed"), + ("list_objects", "s3:ListObjectsV2", "list_objects"), + ("lifecycle_recovery", "unattributed", "heal"), + ] { + assert!(rows.iter().any(|(key, _, _, value)| { + let labels = key.key().labels().map(|label| (label.key(), label.value())).collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("boundary") == Some(&"unary_fallback") + && labels.get("entry_source") == Some(&source) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("parent_caller") == Some(&caller) + && labels.get("child_caller") == Some(&caller) + && matches!(value, DebugValue::Counter(count) if *count == u64::try_from(items / 4).expect("small item count")) + }), "each item must retain its own API, caller, and entry source"); + } + } + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn metadata_full_wait_preserves_external_request_diagnostic_context() { + use crate::cluster::rpc::runtime_sources::with_read_version_request_caller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context, + }; + + const DISKS: usize = 4; + let bucket = "full-wait-external-context-bucket"; + let object = "full-wait-external-context-object"; + let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + install_metadata_fanout_fileinfo(&disks, bucket, object, None).await; + let calls = disk_call_counters::observe(object); + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context() + }) + .expect("external request operation should be captured"); + let (metadata, errors, _) = with_s3_http_operation_context( + Some(operation), + with_read_version_request_caller( + ReadVersionRequestCaller::Object, + SetDisks::read_all_fileinfo_observed(&disks, "", bucket, object, "", false, false, false, false, 2), + ), + ) + .await + .expect("full-wait metadata read should complete"); + + assert_eq!(metadata.iter().filter(|info| info.name == object).count(), DISKS); + assert!(errors.iter().all(Option::is_none)); + for disk_index in 0..DISKS { + assert_eq!( + calls.read_version_contexts_for_disk(disk_index), + vec![(Some("s3:GetObject"), Some(ReadVersionRequestCaller::Object))], + "every full-wait child must retain the external request context" + ); + } + assert_eq!(current_s3_http_operation_context(), None); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn metadata_full_wait_keeps_internal_reads_without_s3_context() { + use crate::cluster::rpc::runtime_sources::with_optional_read_version_request_caller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + + const DISKS: usize = 4; + let bucket = "full-wait-internal-context-bucket"; + let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + for (case, caller) in [None, Some(ReadVersionRequestCaller::Heal)].into_iter().enumerate() { + let object = format!("full-wait-internal-context-object-{case}"); + install_metadata_fanout_fileinfo(&disks, bucket, &object, None).await; + let calls = disk_call_counters::observe(&object); + let (metadata, errors, _) = with_optional_read_version_request_caller( + caller, + SetDisks::read_all_fileinfo_observed(&disks, "", bucket, &object, "", false, false, false, false, 2), + ) + .await + .expect("internal full-wait metadata read should complete"); + + assert_eq!(metadata.iter().filter(|info| info.name == object).count(), DISKS); + assert!(errors.iter().all(Option::is_none)); + for disk_index in 0..DISKS { + assert_eq!( + calls.read_version_contexts_for_disk(disk_index), + vec![(None, caller)], + "internal reads must retain their caller without acquiring an external request context" + ); + } + } + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn metadata_early_stop_preserves_external_and_internal_diagnostic_context() { + use crate::cluster::rpc::runtime_sources::with_optional_read_version_request_caller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + use rustfs_io_metrics::s3_http_metrics::{ + S3HttpRequestGuard, current_s3_http_operation_context, with_s3_http_operation_context, + }; + + const DISKS: usize = 4; + let bucket = "early-stop-diagnostic-context-bucket"; + let (_dirs, disks) = call_counter_local_disks(bucket, DISKS).await; + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::GetObject); + current_s3_http_operation_context() + }) + .expect("external request operation should be captured"); + + temp_env::async_with_vars( + [ + ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")), + ], + async { + for (case, api, caller) in [ + (0, Some(operation), Some(ReadVersionRequestCaller::Object)), + (1, None, None), + (2, None, Some(ReadVersionRequestCaller::Heal)), + ] { + let object = object_with_initial_data_shards(bucket, &format!("early-stop-context-object-{case}"), 2, 3); + install_inline_metadata_fanout_fileinfo(&disks, bucket, &object, b"context probe", |_| {}).await; + let calls = disk_call_counters::observe(&object); + let (metadata, errors, _) = with_s3_http_operation_context( + api, + with_optional_read_version_request_caller( + caller, + SetDisks::read_all_fileinfo_observed(&disks, "", bucket, &object, "", true, false, false, true, 2), + ), + ) + .await + .expect("early-stop metadata read should complete"); + + assert!(metadata.iter().filter(|info| info.name == object).count() >= 2); + assert!(errors.iter().all(Option::is_none)); + assert!(calls.total(disk_call_counters::KIND_READ_VERSION) >= 2); + let expected_api = api.map(|context| context.operation_name()); + for disk_index in 0..DISKS { + for context in calls.read_version_contexts_for_disk(disk_index) { + assert_eq!( + context, + (expected_api, caller), + "every polled early-stop child must retain its parent's diagnostic context" + ); + } + } + } + }, + ) + .await; + assert_eq!(current_s3_http_operation_context(), None); + } + #[test] fn batch_read_version_response_mapping_preserves_index_and_errors() { let expected_items = vec![ @@ -9030,7 +10063,7 @@ mod tests { #[allow(unreachable_code)] None }); - let (release, released) = tokio::sync::oneshot::channel::(); + let (release, released) = oneshot::channel::(); drop(release); finish_rename_tail_heal( @@ -10318,7 +11351,7 @@ mod tests { info.size = 11; info.parts.clear(); info.add_object_part(1, "new-etag".to_string(), 11, None, 11, None, None); - let crate::disk::Disk::Local(local) = disk.as_ref() else { + let disk::Disk::Local(local) = disk.as_ref() else { panic!("physical publication fixture requires local disks"); }; // Linux IO paths use a mount FD, which is also the namespace lock key. @@ -10326,7 +11359,7 @@ mod tests { .get_disk() .get_object_path_for_io(bucket, object) .expect("the publication path must resolve through the disk's mount lease"); - let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (entered_tx, entered_rx) = oneshot::channel(); let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); hooks.push(os::prepared_publication_test_hooks::install( &destination.join(STORAGE_FORMAT_FILE), @@ -10512,11 +11545,11 @@ mod tests { info.add_object_part(1, "new-etag".to_string(), 11, None, 11, None, None); } let disk = disks[3].as_ref().expect("tail disk"); - let crate::disk::Disk::Local(local) = disk.as_ref() else { + let disk::Disk::Local(local) = disk.as_ref() else { panic!("local fixture"); }; let destination = local.get_disk().get_object_path_for_io(bucket, object).expect("tail IO path"); - let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); + let (entered_tx, entered_rx) = oneshot::channel(); let (release_tx, release_rx) = std::sync::mpsc::channel::<()>(); let _hook = os::prepared_publication_test_hooks::install(&destination.join(STORAGE_FORMAT_FILE), move || { let _ = entered_tx.send(()); @@ -10528,7 +11561,7 @@ mod tests { let receipt = RenameRollbackReceipt::default(); let caller_receipt = receipt.clone(); let caller_disks = disks.clone(); - let (ack_tx, ack_rx) = tokio::sync::oneshot::channel(); + let (ack_tx, ack_rx) = oneshot::channel(); let caller = tokio::spawn(async move { let commit = SetDisks::rename_data_owned_with_fence( &caller_disks, @@ -12005,6 +13038,9 @@ mod tests { let bucket = "dangling-invalid-meta-bucket"; let object = "object"; let set = io_primitives_test_set(vec![None, None, None, None], 1).await; + let cached_generation = set + .get_object_metadata_cache_generation(bucket, object) + .expect("local cache generation should be enabled"); let mut data_errs_by_part = HashMap::new(); data_errs_by_part.insert( 1, @@ -12039,6 +13075,10 @@ mod tests { .expect("invalid metadata should be cleanable when part results prove the object is dangling"); assert!(!deleted.is_valid()); + assert!( + !set.is_get_object_metadata_cache_generation_current(cached_generation), + "dangling cleanup must retire locally cached metadata before deletion" + ); } #[tokio::test] @@ -12373,7 +13413,7 @@ mod tests { assert_eq!(request.object_prefix.as_deref(), Some(object.as_str())); assert_eq!(request.object_version_id.as_deref(), Some(version.as_str())); assert_eq!((request.pool_index, request.set_index), (Some(1), Some(2))); - assert_eq!(request.recreate_missing, Some(true)); + assert_eq!(request.recreate_missing, None, "storage evidence decides absence"); let key = ReadRepairHealCacheKey::new("reader-failure", &object, Some(&version), 1, 2); release_read_repair_heal_reservation(&key).await; } diff --git a/crates/ecstore/src/set_disk/core/metadata_read.rs b/crates/ecstore/src/set_disk/core/metadata_read.rs index 37e727395..e3d318589 100644 --- a/crates/ecstore/src/set_disk/core/metadata_read.rs +++ b/crates/ecstore/src/set_disk/core/metadata_read.rs @@ -171,6 +171,23 @@ fn read_version_coalescing_delay() -> Duration { struct CoalescedReadVersionRequest { item: BatchReadVersionItem, tx: oneshot::Sender>, + #[cfg(feature = "diagnostic-read-version-caller")] + context: crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext, +} + +impl CoalescedReadVersionRequest { + fn new(item: BatchReadVersionItem, tx: oneshot::Sender>) -> Self { + #[cfg(feature = "diagnostic-read-version-caller")] + let context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture(); + #[cfg(feature = "diagnostic-read-version-caller")] + context.observe(rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::CoalescerEnqueue); + Self { + item, + tx, + #[cfg(feature = "diagnostic-read-version-caller")] + context, + } + } } #[derive(Clone, Debug, Eq, PartialEq)] @@ -246,26 +263,49 @@ async fn read_version_via_coalescer( allow_coalescing: bool, ) -> disk::error::Result { if !allow_coalescing || !read_version_coalescing_enabled() { + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::MetadataRead, + disk.read_version(org_bucket, bucket, object, version_id, opts), + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] return disk.read_version(org_bucket, bucket, object, version_id, opts).await; } if !matches!(disk.as_ref(), Disk::Remote(_)) { record_read_version_coalescer_event("bypass_non_remote", 1); + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::MetadataRead, + disk.read_version(org_bucket, bucket, object, version_id, opts), + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] return disk.read_version(org_bucket, bucket, object, version_id, opts).await; } - let (tx, rx) = oneshot::channel(); let item = BatchReadVersionItem { org_volume: org_bucket.to_string(), volume: bucket.to_string(), path: object.to_string(), version_id: version_id.to_string(), }; + enqueue_coalesced_read_version(disk, item, opts).await +} + +pub(super) async fn enqueue_coalesced_read_version( + disk: DiskStore, + item: BatchReadVersionItem, + opts: &ReadOptions, +) -> disk::error::Result { + let (tx, rx) = oneshot::channel(); let lane_key = ReadVersionCoalescerKey::new(&disk, opts); + let request = CoalescedReadVersionRequest::new(item, tx); let pending = { let mut coalescer = read_version_coalescer().lock().await; let lane = coalescer.lanes.entry(lane_key).or_default(); let schedule_delayed_flush = lane.is_empty(); - lane.push(CoalescedReadVersionRequest { item, tx }); + lane.push(request); if lane.len() >= BATCH_READ_VERSION_MAX_ITEMS { coalescer.lanes.remove(&lane_key) } else if schedule_delayed_flush { @@ -326,20 +366,24 @@ async fn flush_read_version_coalescer_pending( let mut senders = Vec::with_capacity(pending.len()); let mut items = Vec::with_capacity(pending.len()); + #[cfg(feature = "diagnostic-read-version-caller")] + let mut contexts = Vec::with_capacity(pending.len()); for request in pending { senders.push(request.tx); items.push(request.item); + #[cfg(feature = "diagnostic-read-version-caller")] + contexts.push(request.context); } let expected_items = items.iter().map(ExpectedBatchReadVersionItem::from).collect::>(); record_read_version_coalescer_event("attempted_batch", items.len()); - let result = - match tokio::time::timeout(get_drive_metadata_timeout(), disk.batch_read_version(BatchReadVersionReq { items, opts })) - .await - { - Ok(result) => result, - Err(_) => Err(DiskError::Timeout), - }; + let batch = disk.batch_read_version(BatchReadVersionReq { items, opts }); + #[cfg(feature = "diagnostic-read-version-caller")] + let batch = crate::cluster::rpc::runtime_sources::with_read_version_batch_contexts(contexts, batch); + let result = match tokio::time::timeout(get_drive_metadata_timeout(), batch).await { + Ok(result) => result, + Err(_) => Err(DiskError::Timeout), + }; match result { Ok(responses) => { let map_started = batch_read_version_stage_timer(); @@ -576,6 +620,7 @@ impl MetadataFanoutDiagnostics { pub(in crate::set_disk) fn record_quorum_candidate_latency(&self, path: &'static str, read_quorum: usize) { if let Some(latency) = self.quorum_candidate_latency(read_quorum) { + rustfs_io_metrics::record_get_object_metadata_quorum_result(path, "reached"); rustfs_io_metrics::record_get_object_quorum_reached_latency(path, latency.as_secs_f64()); } } @@ -1015,6 +1060,10 @@ impl SetDisks { let object: Arc = Arc::from(object); let version_id: Arc = Arc::from(version_id); let slowtail_fault = get_metadata_slowtail_fault_request(bucket.as_ref(), object.as_ref(), read_data); + #[cfg(feature = "diagnostic-read-version-caller")] + let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::MetadataRead, + ); let futures = disks.iter().enumerate().map(|(disk_index, disk)| { let disk = disk.clone(); let task_opts = opts; @@ -1023,21 +1072,35 @@ impl SetDisks { let object = object.clone(); let version_id = version_id.clone(); let slowtail_fault = slowtail_fault.clone(); + #[cfg(feature = "diagnostic-read-version-caller")] + #[allow(clippy::redundant_locals)] + let diagnostic_context = diagnostic_context; AbortOnDropJoinHandle(tokio::spawn(async move { - let response_start = observe.then(Instant::now); - let result = if let Some(disk) = disk { - Self::record_read_version_call(&object, disk_index); - if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) { - Self::record_metadata_slowtail_fault(&object, disk_index); - tokio::time::sleep(delay).await; - } - read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) - .await - } else { - Err(DiskError::DiskNotFound) + let read = async move { + let response_start = observe.then(Instant::now); + let result = if let Some(disk) = disk { + Self::record_read_version_call(&object, disk_index); + if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(disk_index)) { + Self::record_metadata_slowtail_fault(&object, disk_index); + tokio::time::sleep(delay).await; + } + read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) + .await + } else { + Err(DiskError::DiskNotFound) + }; + let elapsed = response_start.map(|start| start.elapsed()); + (result, elapsed) }; - let elapsed = response_start.map(|start| start.elapsed()); - (result, elapsed) + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::FullWait, + read, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + read.await })) }); @@ -1133,6 +1196,10 @@ impl SetDisks { let mut non_inline_candidate_eligible = None; let mut single_pending_hedge_deadline = None; let slowtail_fault = get_metadata_slowtail_fault_request(bucket.as_ref(), object.as_ref(), read_data); + #[cfg(feature = "diagnostic-read-version-caller")] + let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::MetadataRead, + ); let spawn_read_version = |join_set: &mut JoinSet<(usize, disk::error::Result, Duration)>, task_ids: &mut [Option], index: usize, @@ -1143,27 +1210,45 @@ impl SetDisks { let object = object.clone(); let version_id = version_id.clone(); let slowtail_fault = slowtail_fault.clone(); + #[cfg(feature = "diagnostic-read-version-caller")] + #[allow(clippy::redundant_locals)] + let diagnostic_context = diagnostic_context; let task = join_set.spawn(async move { - let response_start = Instant::now(); - let result = if let Some(disk) = disk { - #[cfg(test)] - let _fanout_task_guard = rename_fanout_barrier::task_guard(&object); - Self::record_read_version_call(&object, index); - #[cfg(test)] - Self::read_version_fanout_barrier(&object, index).await; - if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) { - Self::record_metadata_slowtail_fault(&object, index); + let read = async move { + let response_start = Instant::now(); + let result = if let Some(disk) = disk { #[cfg(test)] - rename_fanout_barrier::checkpoint(&object, index, rename_fanout_barrier::PHASE_METADATA_SLOWTAIL_FAULT) + let _fanout_task_guard = rename_fanout_barrier::task_guard(&object); + Self::record_read_version_call(&object, index); + #[cfg(test)] + Self::read_version_fanout_barrier(&object, index).await; + if let Some(delay) = slowtail_fault.as_ref().and_then(|fault| fault.delay_for_disk(index)) { + Self::record_metadata_slowtail_fault(&object, index); + #[cfg(test)] + rename_fanout_barrier::checkpoint( + &object, + index, + rename_fanout_barrier::PHASE_METADATA_SLOWTAIL_FAULT, + ) .await; - tokio::time::sleep(delay).await; - } - read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) - .await - } else { - Err(DiskError::DiskNotFound) + tokio::time::sleep(delay).await; + } + read_version_via_coalescer(disk, &org_bucket, &bucket, &object, &version_id, &task_opts, allow_coalescing) + .await + } else { + Err(DiskError::DiskNotFound) + }; + (index, result, response_start.elapsed()) }; - (index, result, response_start.elapsed()) + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::EarlyStop, + read, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + read.await }); task_ids[index] = Some(task.id()); }; diff --git a/crates/ecstore/src/set_disk/metadata.rs b/crates/ecstore/src/set_disk/metadata.rs index 36e3bf6d6..6be934391 100644 --- a/crates/ecstore/src/set_disk/metadata.rs +++ b/crates/ecstore/src/set_disk/metadata.rs @@ -35,6 +35,58 @@ struct FileInfoIdentityGroup { } impl SetDisks { + /// Read one key's complete version stack without enumerating its parent. + pub(crate) async fn read_object_versions_for_lifecycle( + &self, + bucket: &str, + object: &str, + ) -> disk::error::Result> { + let disks = self.get_disks_internal().await; + if disks.len() != self.set_drive_count || disks.is_empty() { + return Err(DiskError::ErasureReadQuorum); + } + let disk_object = rustfs_utils::path::encode_dir_object(object); + let (mut raw, mut errors) = Self::read_all_raw_file_info(&disks, bucket, &disk_object, false).await; + for (metadata, error) in raw.iter_mut().zip(&mut errors) { + if metadata.as_ref().is_some_and(|metadata| metadata.buf.is_empty()) { + *metadata = None; + *error = Some(DiskError::FileCorrupt); + } + } + // An unavailable disk is not evidence that a version group is absent. + // Unlike a walk, an exact read must also decide the all-missing case. + if raw.iter().all(Option::is_none) { + return if errors.iter().all(|error| { + matches!( + error, + Some(DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::VolumeNotFound) + ) + }) { + Ok(None) + } else { + Err(DiskError::ErasureReadQuorum) + }; + } + let entries = rustfs_filemeta::MetaCacheEntries( + raw.into_iter() + .map(|raw| { + raw.map(|raw| rustfs_filemeta::MetaCacheEntry { + name: object.to_owned(), + metadata: raw.buf, + cached: None, + reusable: false, + }) + }) + .collect(), + ); + // Reuse the listing's per-version authority, including the object's + // actual erasure layout. Never turn unreadable/stale metadata into a + // singleton made from the preceding write's ObjectInfo. + Self::resolve_listed_versions(bucket, entries, &errors, self.set_drive_count, self.default_parity_count)? + .map(Some) + .ok_or(DiskError::ErasureReadQuorum) + } + /// Resolve each listed version using the same metadata authority as an exact read. pub(crate) fn resolve_listed_versions( bucket: &str, diff --git a/crates/ecstore/src/set_disk/mod.rs b/crates/ecstore/src/set_disk/mod.rs index 560d6671a..87eb1acca 100644 --- a/crates/ecstore/src/set_disk/mod.rs +++ b/crates/ecstore/src/set_disk/mod.rs @@ -160,13 +160,14 @@ use rustfs_utils::http::{ use rustfs_utils::{ HashAlgorithm, crypto::hex, - path::{SLASH_SEPARATOR, encode_dir_object, encode_dir_object_ref, has_suffix, path_join_buf}, + path::{SLASH_SEPARATOR, encode_dir_object, encode_dir_object_ref, has_suffix, object_name_disk_identity, path_join_buf}, }; use sha2::Sha256; +use std::borrow::Cow; use std::hash::{BuildHasher, Hash, Hasher}; use std::mem::{self}; use std::pin::Pin; -use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, OnceLock}; use std::task::{Context, Poll}; use std::time::{Instant, SystemTime, UNIX_EPOCH}; @@ -410,6 +411,12 @@ impl Drop for ObjectLockDiagGuard { } let hold = self.acquired_at.elapsed(); + if self.op == "put_object_commit" && self.mode == "write" && rustfs_io_metrics::put_stage_metrics_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD, + hold.as_secs_f64() * 1000.0, + ); + } record_object_lock_diag_hold_duration(self.op, self.mode, hold); let threshold = get_object_lock_diag_slow_hold_threshold(); if hold >= threshold { @@ -927,16 +934,21 @@ pub(in crate::set_disk) struct GetObjectFileInfo { owned: Option, shared: Option>, condition_stopped: bool, + late_metadata_fanout: Option, } struct OwnedGetObjectFileInfo { fi: FileInfo, parts_metadata: Vec, online_disks: Vec>, - late_metadata_fanout_disks: Option>>, } -type OwnedGetObjectFileInfoParts = (FileInfo, Vec, Vec>, Option>>); +enum LateMetadataFanout { + CapturedDisks(Vec>), + RefreshCurrentSet, +} + +type OwnedGetObjectFileInfoParts = (FileInfo, Vec, Vec>, Option); impl GetObjectFileInfo { fn owned(fi: FileInfo, parts_metadata: Vec, online_disks: Vec>) -> Self { @@ -945,10 +957,10 @@ impl GetObjectFileInfo { fi, parts_metadata, online_disks, - late_metadata_fanout_disks: None, }), shared: None, condition_stopped: false, + late_metadata_fanout: None, } } @@ -958,16 +970,9 @@ impl GetObjectFileInfo { online_disks: Vec>, late_metadata_fanout_disks: Vec>, ) -> Self { - Self { - owned: Some(OwnedGetObjectFileInfo { - fi, - parts_metadata, - online_disks, - late_metadata_fanout_disks: Some(late_metadata_fanout_disks), - }), - shared: None, - condition_stopped: false, - } + let mut snapshot = Self::owned(fi, parts_metadata, online_disks); + snapshot.late_metadata_fanout = Some(LateMetadataFanout::CapturedDisks(late_metadata_fanout_disks)); + snapshot } fn shared(entry: Arc) -> Self { @@ -976,9 +981,16 @@ impl GetObjectFileInfo { owned: None, shared: Some(entry), condition_stopped, + late_metadata_fanout: None, } } + fn shared_with_late_metadata_refresh(entry: Arc) -> Self { + let mut snapshot = Self::shared(entry); + snapshot.late_metadata_fanout = Some(LateMetadataFanout::RefreshCurrentSet); + snapshot + } + fn fi(&self) -> &FileInfo { match (&self.owned, &self.shared) { (Some(snapshot), None) => &snapshot.fi, @@ -1004,9 +1016,7 @@ impl GetObjectFileInfo { } fn has_late_metadata_fanout(&self) -> bool { - self.owned - .as_ref() - .is_some_and(|snapshot| snapshot.late_metadata_fanout_disks.is_some()) + self.late_metadata_fanout.is_some() } fn into_owned(self) -> (FileInfo, Vec, Vec>) { @@ -1015,16 +1025,22 @@ impl GetObjectFileInfo { } fn into_owned_with_late_metadata_fanout(self) -> OwnedGetObjectFileInfoParts { - match (self.owned, self.shared) { - (Some(snapshot), None) => ( - snapshot.fi, - snapshot.parts_metadata, - snapshot.online_disks, - snapshot.late_metadata_fanout_disks, - ), + let Self { + owned, + shared, + late_metadata_fanout, + .. + } = self; + match (owned, shared) { + (Some(snapshot), None) => (snapshot.fi, snapshot.parts_metadata, snapshot.online_disks, late_metadata_fanout), (None, Some(entry)) => match Arc::try_unwrap(entry) { - Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, None), - Err(entry) => (entry.fi.clone(), entry.parts_metadata.clone(), entry.online_disks.clone(), None), + Ok(entry) => (entry.fi, entry.parts_metadata, entry.online_disks, late_metadata_fanout), + Err(entry) => ( + entry.fi.clone(), + entry.parts_metadata.clone(), + entry.online_disks.clone(), + late_metadata_fanout, + ), }, _ => unreachable!("GET metadata snapshot representation must be exclusive"), } @@ -1415,11 +1431,11 @@ mod prepared_get_object_metadata_tests { let snapshot = GetObjectFileInfo::shared(Arc::clone(&cached)); assert!(snapshot.has_valid_representation()); - assert!(std::mem::size_of::() >= std::mem::size_of::()); - assert!( - std::mem::size_of::() - <= std::mem::size_of::() + 2 * std::mem::size_of::() - ); + // Keep the original payload budget after moving the captured disks out + // of OwnedGetObjectFileInfo. + let owned_payload_size = std::mem::size_of::<(OwnedGetObjectFileInfo, Option>>)>(); + assert!(std::mem::size_of::() >= owned_payload_size); + assert!(std::mem::size_of::() <= owned_payload_size + 2 * std::mem::size_of::()); assert_eq!(Arc::strong_count(&cached), 2, "a cache hit must add one snapshot reference"); assert_eq!(snapshot.fi().name, "object"); assert_eq!(snapshot.parts_metadata().len(), 1); @@ -4339,6 +4355,9 @@ pub struct SetDisks { /// writes skip the global registry mutex (backlog#1315). `Arc` so clones of /// a set share one generation marker. capacity_dirty_generation: Arc, + /// Per-set proof that successful rename_data peers return the old-current + /// source capability needed before PUT may skip its pre-rename lookup. + tier_free_version_rename_data_source_capable: Arc, /// Orphan prefixes whose last purge scan met data that can never be /// purged by listing (residue without a committed marker, an in-flight /// write), keyed by `bucket/prefix` with the time of that scan. Empty @@ -4594,6 +4613,14 @@ struct GetObjectMetadataCacheKey { hash: u64, } +#[derive(Clone, Debug)] +struct GetObjectMetadataCacheLookupKey<'a> { + bucket: &'a str, + object: Cow<'a, str>, + generation: u64, + hash: u64, +} + #[derive(Clone, Copy, Debug, Eq, PartialEq)] struct GetObjectMetadataCacheGeneration { index: usize, @@ -4675,13 +4702,40 @@ impl GetObjectMetadataCacheKey { fn new(bucket: &str, object: &str, generation: GetObjectMetadataCacheGeneration) -> Self { Self { bucket: Arc::from(bucket), - object: Arc::from(object), + object: Arc::from(object_name_disk_identity(object).as_ref()), generation: generation.value, hash: generation.hash, } } } +impl<'a> GetObjectMetadataCacheLookupKey<'a> { + fn new(bucket: &'a str, object: &'a str, generation: GetObjectMetadataCacheGeneration) -> Self { + Self { + bucket, + object: object_name_disk_identity(object), + generation: generation.value, + hash: generation.hash, + } + } +} + +impl Hash for GetObjectMetadataCacheLookupKey<'_> { + fn hash(&self, state: &mut H) { + self.hash.hash(state); + self.generation.hash(state); + } +} + +impl moka::Equivalent for GetObjectMetadataCacheLookupKey<'_> { + fn equivalent(&self, key: &GetObjectMetadataCacheKey) -> bool { + self.bucket == key.bucket.as_ref() + && self.object.as_ref() == key.object.as_ref() + && self.generation == key.generation + && self.hash == key.hash + } +} + impl Hash for GetObjectMetadataCacheKey { fn hash(&self, state: &mut H) { self.hash.hash(state); @@ -4692,7 +4746,9 @@ impl Hash for GetObjectMetadataCacheKey { #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum GetObjectMetadataCacheCapability { Full, - // Quorum-validated metadata without a complete body or write-target layout. + // A quorum-selected layout can serve body reads, but is not a mutation layout. + QuorumSelected, + // Quorum-validated metadata used only to evaluate a terminal read condition. ConditionOnly, } @@ -4725,6 +4781,10 @@ impl DiskHealthEntry { } } +#[derive(Debug, thiserror::Error)] +#[error("committed system metadata write requires repair by its owning writer")] +pub(crate) struct SystemMetadataRepairRequired; + impl SetDisks { pub(in crate::set_disk) async fn persist_delete_marker_purge( &self, @@ -4835,7 +4895,7 @@ impl SetDisks { state = "mrf_durable_admission_failed", bucket, object, version_id, pool_index = self.pool_index, set_index = self.set_index, error = %error, - "Partial write repair falling back to the heal channel" + "Partial write repair responsibility could not be persisted" ); false } @@ -4845,7 +4905,17 @@ impl SetDisks { pub(in crate::set_disk) async fn submit_rename_tail_heal( &self, mut request: rustfs_heal_contracts::heal_channel::HealChannelRequest, - ) { + completion: crate::object_api::WriteCompletion, + ) -> Result<()> { + // System metadata repair remains with the writer's deployment and + // transaction fence; a user-bucket MRF authority cannot authorize it. + if crate::bucket::utils::is_meta_bucketname(&request.bucket) { + return if completion == crate::object_api::WriteCompletion::TailDrainedWithOwnerRepair { + Err(Error::other(SystemMetadataRepairRequired)) + } else { + Ok(()) + }; + } if let Some(object) = request.object_prefix.as_deref() && self .persist_partial_write( @@ -4856,10 +4926,10 @@ impl SetDisks { ) .await { - return; + return Ok(()); } if request.expected_bucket_incarnation_id.is_none() { - return; + return Ok(()); } request.source = rustfs_heal_contracts::heal_channel::HealRequestSource::Mrf; #[cfg(test)] @@ -4871,11 +4941,12 @@ impl SetDisks { .clone(); if let Some(capture) = capture { let _ = capture.send(request); - return; + return Ok(()); } } let _ = rustfs_heal_contracts::heal_channel::send_heal_request(request).await; + Ok(()) } #[cfg(test)] @@ -4917,7 +4988,9 @@ impl SetDisks { fn get_object_metadata_cache_hash(&self, bucket: &str, object: &str) -> u64 { let mut hasher = self.get_object_metadata_cache_hash_builder.build_hasher(); bucket.hash(&mut hasher); - object.hash(&mut hasher); + // Every spelling of one xl.meta (Windows `a\b` vs `a/b`) shares one + // cache identity, so a mutation through any of them retires the rest. + object_name_disk_identity(object).hash(&mut hasher); hasher.finish() } @@ -5235,6 +5308,7 @@ impl SetDisks { ctx, capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())), capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)), + tier_free_version_rename_data_source_capable: Arc::new(AtomicBool::new(false)), orphan_purge_backoff: Arc::new(std::sync::Mutex::new(HashMap::new())), #[cfg(test)] storage_class_config_override: Arc::new(std::sync::RwLock::new(None)), @@ -5249,6 +5323,15 @@ impl SetDisks { &self.ctx } + pub(in crate::set_disk) fn tier_free_version_rename_data_source_capable(&self) -> bool { + self.tier_free_version_rename_data_source_capable.load(Ordering::Acquire) + } + + pub(in crate::set_disk) fn observe_tier_free_version_rename_data_source_capability(&self, capable: bool) { + self.tier_free_version_rename_data_source_capable + .store(capable, Ordering::Release); + } + #[cfg(test)] pub(crate) fn set_instance_ctx_for_test(&mut self, ctx: Arc) { self.local_lock_manager = ctx.lock_manager(); @@ -5265,6 +5348,14 @@ impl SetDisks { metadata_sys::get_bucket_incarnation_id_in(&self.ctx, bucket).await } + /// Read a persisted identity without treating lookup failures as legacy absence. + pub async fn find_bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result> { + if crate::bucket::utils::is_meta_bucketname(bucket) { + return Err(Error::other("system metadata bucket has no bucket incarnation")); + } + metadata_sys::find_bucket_incarnation_id_in(&self.ctx, bucket).await + } + /// Admit one short scanner cache publication under this set's instance /// movement fence. The caller must hold the returned guard through its /// final conditional cache write; no scan-round work belongs under it. @@ -7655,6 +7746,21 @@ pub fn e_tag_matches(etag: &str, condition: &str) -> bool { canonicalize_etag(etag) == canonicalize_etag(condition) } +fn weak_etag_matches(etag: &str, condition: &str) -> bool { + let condition = condition.trim(); + if condition == "*" { + return true; + } + fn strip_weak(value: &str) -> &str { + let trimmed = value.trim(); + trimmed + .strip_prefix("W/") + .or_else(|| trimmed.strip_prefix("w/")) + .unwrap_or(trimmed) + } + canonicalize_etag(strip_weak(etag)) == canonicalize_etag(strip_weak(condition)) +} + pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option, if_match: Option) -> bool { let if_none_match = if_none_match .as_deref() @@ -7665,7 +7771,7 @@ pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option, if_m match &oi.etag { Some(etag) => { if let Some(if_none_match) = if_none_match - && e_tag_matches(etag, if_none_match) + && weak_etag_matches(etag, if_none_match) { return true; } @@ -7917,10 +8023,27 @@ mod tests { .await .expect("system metadata identity lookup must not reacquire the held pool metadata lock") .expect_err("system metadata buckets have no user bucket incarnation"); + tokio::time::timeout(Duration::from_secs(30), set.find_bucket_incarnation_id_from_disk(bucket)) + .await + .expect("optional identity lookup must not reacquire the held pool metadata lock") + .expect_err("system metadata buckets must not become a legacy identity"); } drop(_pool_meta_guard); let bucket = "user-incarnation-boundary"; + assert_eq!( + set.find_bucket_incarnation_id_from_disk(bucket) + .await + .expect("a confirmed absent identity is distinct from a read error"), + None, + ); + assert_eq!( + set.bucket_incarnation_id_from_disk(bucket) + .await + .expect_err("required lookup retains its missing-identity error") + .to_string(), + format!("Io error: bucket incarnation metadata is not authoritative: {bucket}"), + ); let incarnation = Uuid::new_v4(); crate::bucket::metadata::save_bucket_incarnation(Arc::clone(&store), bucket, incarnation) .await @@ -7931,6 +8054,28 @@ mod tests { .expect("user bucket identities must still load from the metadata owner"), incarnation, ); + assert_eq!( + set.find_bucket_incarnation_id_from_disk(bucket) + .await + .expect("persisted identity"), + Some(incarnation) + ); + + let identity_key = format!( + "{}/{bucket}/{}", + crate::disk::BUCKET_META_PREFIX, + crate::bucket::metadata::BUCKET_INCARNATION_FILE + ); + for invalid in [Vec::new(), vec![1; 15], Uuid::nil().as_bytes().to_vec()] { + crate::config::com::save_config(store.clone(), &identity_key, invalid) + .await + .expect("persist an invalid identity fixture"); + assert!( + set.find_bucket_incarnation_id_from_disk(bucket).await.is_err(), + "invalid identity bytes must not enable the legacy rebuild path" + ); + assert!(set.bucket_incarnation_id_from_disk(bucket).await.is_err()); + } } async fn make_test_set_disks_with_ctx( @@ -8021,6 +8166,78 @@ mod tests { assert_eq!(Arc::strong_count(&set.set_lock_namespace), before); } + /// Spellings that a Windows disk resolves to the xl.meta of `a/b`. + const BACKSLASH_SPELLINGS_OF_A_B: [&str; 4] = ["a\\b", "a\\\\b", "\\a\\b", "a\\b\\"]; + + async fn ns_write_lock_free_while_held(set: &SetDisks, held: &str, contender: &str) -> bool { + let _held = set + .new_ns_lock("bucket", held) + .await + .expect("namespace lock should be created") + .get_write_lock(Duration::from_secs(1)) + .await + .expect("holder write lock should be acquired"); + set.new_ns_lock("bucket", contender) + .await + .expect("namespace lock should be created") + .get_write_lock(Duration::from_millis(100)) + .await + .is_ok() + } + + async fn ns_lock_test_sets() -> [Arc; 2] { + let local_ctx = Arc::new(InstanceContext::new()); + local_ctx.update_erasure_type(SetupType::Erasure).await; + let dist_ctx = Arc::new(InstanceContext::new()); + dist_ctx.update_erasure_type(SetupType::DistErasure).await; + let manager = Arc::new(rustfs_lock::GlobalLockManager::new()); + let locker: Arc = Arc::new(LocalClient::with_manager(manager)); + [ + make_test_set_disks_with_ctx(Vec::new(), local_ctx).await, + make_test_set_disks_with_ctx(vec![locker], dist_ctx).await, + ] + } + + #[cfg(not(windows))] + #[tokio::test(flavor = "multi_thread")] + async fn new_ns_lock_keeps_backslash_and_slash_keys_distinct_off_windows() { + for set in ns_lock_test_sets().await { + assert!( + !ns_write_lock_free_while_held(&set, "a/b", "a/b").await, + "baseline: one key must conflict" + ); + for spelling in BACKSLASH_SPELLINGS_OF_A_B { + assert!( + ns_write_lock_free_while_held(&set, "a/b", spelling).await, + "off Windows {spelling:?} is a different object and must not wait on `a/b`" + ); + } + } + } + + #[cfg(windows)] + #[tokio::test(flavor = "multi_thread")] + async fn windows_new_ns_lock_serializes_backslash_spellings_of_one_object() { + // Local and distributed lock domains: the distributed key is folded by + // the requester before it reaches the lock client. + for set in ns_lock_test_sets().await { + for spelling in BACKSLASH_SPELLINGS_OF_A_B { + assert!( + !ns_write_lock_free_while_held(&set, "a/b", spelling).await, + "{spelling:?} addresses the xl.meta of `a/b` and must wait on its lock" + ); + assert!( + !ns_write_lock_free_while_held(&set, spelling, "a/b").await, + "`a/b` must wait on the lock held through {spelling:?}" + ); + } + assert!( + ns_write_lock_free_while_held(&set, "a/b", "a/b/c").await, + "folding must not merge distinct objects" + ); + } + } + fn put_object_commit_namespace_lock_wait_sample_count(snapshotter: &metrics_util::debugging::Snapshotter) -> usize { snapshotter .snapshot() @@ -10346,6 +10563,13 @@ mod tests { .expect("orphan part should be written"); let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!( + set.dry_run_reclaim_orphan_data_dirs("bucket", "obj") + .await + .expect("dry-run orphan cleanup"), + 1 + ); + assert!(object_dir.join(orphan.to_string()).exists(), "dry run must retain the orphan payload"); let removed = set .reclaim_orphan_data_dirs("bucket", "obj") .await @@ -10357,6 +10581,67 @@ mod tests { assert!(object_dir.join(STORAGE_FORMAT_FILE).exists(), "metadata must be preserved"); } + #[tokio::test] + async fn reclaim_orphan_data_dirs_drains_empty_discovery_pages() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let orphan = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let live_dir = object_dir.join(live.to_string()); + let orphan_dir = object_dir.join(orphan.to_string()); + fs::create_dir_all(&live_dir).await.expect("create live multipart directory"); + fs::create_dir_all(&orphan_dir).await.expect("create orphan directory"); + fs::write(orphan_dir.join("part.1"), b"stale") + .await + .expect("write orphan payload"); + let parts_dir = live_dir.clone(); + tokio::task::spawn_blocking(move || { + for part in 1..=4200 { + std::fs::write(parts_dir.join(format!("part.{part}")), b"live").expect("write live multipart entry"); + } + }) + .await + .expect("create multipart fixture"); + + let first = disk + .discover_delete_transactions("bucket", Some("obj"), None, 1) + .await + .expect("read first discovery page"); + assert!(first.transactions.is_empty()); + assert!(first.next.is_some(), "ordinary multipart entries must exhaust the first scan budget"); + let last = disk + .discover_delete_transactions("bucket", Some("obj"), first.next, 1) + .await + .expect("finish fixture discovery"); + assert!( + last.transactions.is_empty() && last.next.is_none(), + "the complete inventory has no transaction evidence" + ); + + let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!( + set.dry_run_reclaim_orphan_data_dirs("bucket", "obj") + .await + .expect("dry-run paginated orphan cleanup"), + 1 + ); + assert!(orphan_dir.exists(), "dry-run must retain the orphan"); + assert_eq!( + set.reclaim_orphan_data_dirs("bucket", "obj") + .await + .expect("reclaim after complete discovery"), + 1 + ); + assert!(!orphan_dir.exists(), "both the set preflight and final disk guard must drain empty pages"); + assert_eq!( + fs::read(live_dir.join("part.4200")) + .await + .expect("read referenced multipart tail"), + b"live" + ); + } + async fn recv_abandoned_parts_trace( trace: &mut rustfs_common::trace_bus::TraceSubscription, bucket: &str, @@ -10570,7 +10855,7 @@ mod tests { } #[tokio::test] - async fn reclaim_orphan_data_dirs_recovers_committed_delete_marker_without_meta() { + async fn reclaim_orphan_data_dirs_defers_delete_marker_without_meta() { let (dir, disk) = make_single_local_disk().await; let stale = Uuid::new_v4(); let transaction = Uuid::new_v4(); @@ -10595,8 +10880,105 @@ mod tests { .await .expect("upgrade reclaim should succeed"); - assert_eq!(removed, 1, "the committed delete residue should be reclaimed"); - assert!(!stale_dir.exists(), "the committed stale data dir should be removed"); + assert_eq!(removed, 0, "a local marker is not a set-level commit decision"); + assert_eq!(fs::read(stale_dir.join("part.1")).await.expect("read retained payload"), b"stale"); + } + + #[tokio::test] + async fn reclaim_orphan_data_dirs_preserves_uuid_prefix_children() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let prefix = Uuid::new_v4(); + let child_data = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let child_dir = object_dir.join(prefix.to_string()).join("child"); + write_object_meta_with_data_dirs(&child_dir, "bucket", &format!("obj/{prefix}/child"), &[child_data]).await; + let data_dir = child_dir.join(child_data.to_string()); + fs::create_dir_all(&data_dir).await.expect("create child payload directory"); + fs::write(data_dir.join("part.1"), b"live child payload") + .await + .expect("write child payload"); + let parent_meta = fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read parent metadata"); + let child_meta = fs::read(child_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read child metadata"); + let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!(set.dry_run_reclaim_orphan_data_dirs("bucket", "obj").await.expect("dry run"), 0); + assert_eq!(set.reclaim_orphan_data_dirs("bucket", "obj").await.expect("reclaim"), 0); + assert_eq!( + fs::read(object_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read parent metadata"), + parent_meta + ); + assert_eq!( + fs::read(child_dir.join(STORAGE_FORMAT_FILE)) + .await + .expect("read child metadata"), + child_meta + ); + assert_eq!( + fs::read(data_dir.join("part.1")).await.expect("read child payload"), + b"live child payload" + ); + } + + #[tokio::test] + async fn reclaim_orphan_data_dirs_waits_for_complete_participant_inventory() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let orphan = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let payload_dir = object_dir.join(orphan.to_string()); + fs::create_dir_all(&payload_dir).await.expect("create orphan directory"); + fs::write(payload_dir.join("part.1"), b"potentially referenced payload") + .await + .expect("write payload"); + let set = make_set_disks_with(vec![Some(disk), None]).await; + assert_eq!(set.dry_run_reclaim_orphan_data_dirs("bucket", "obj").await.expect("dry run"), 0); + assert_eq!(set.reclaim_orphan_data_dirs("bucket", "obj").await.expect("reclaim"), 0); + assert_eq!( + fs::read(payload_dir.join("part.1")).await.expect("read retained payload"), + b"potentially referenced payload" + ); + } + + #[tokio::test] + async fn reclaim_orphan_data_dirs_defers_payload_referenced_by_neighboring_journal() { + let (dir, disk) = make_single_local_disk().await; + let live = Uuid::new_v4(); + let old = Uuid::new_v4(); + let transaction = Uuid::new_v4(); + let object_dir = dir.path().join("bucket/obj"); + write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await; + let payload_dir = object_dir.join(old.to_string()); + fs::create_dir_all(&payload_dir).await.expect("create old payload directory"); + fs::write(payload_dir.join("part.1"), b"rollback payload") + .await + .expect("write old payload"); + let transaction_dir = object_dir.join(transaction.to_string()); + write_object_meta_with_data_dirs(&transaction_dir, "bucket", "obj", &[old]).await; + fs::rename( + transaction_dir.join(STORAGE_FORMAT_FILE), + transaction_dir.join(STORAGE_FORMAT_FILE_BACKUP), + ) + .await + .expect("retain rollback metadata"); + let set = make_set_disks_with(vec![Some(disk)]).await; + assert_eq!(set.dry_run_reclaim_orphan_data_dirs("bucket", "obj").await.expect("dry run"), 0); + assert_eq!(set.reclaim_orphan_data_dirs("bucket", "obj").await.expect("reclaim"), 0); + assert_eq!( + fs::read(payload_dir.join("part.1")).await.expect("read rollback payload"), + b"rollback payload" + ); + assert!( + transaction_dir.join(STORAGE_FORMAT_FILE_BACKUP).exists(), + "canonical recovery owns the journal" + ); } // Cross-replica union: a data dir referenced by ANOTHER disk's xl.meta must be @@ -13445,6 +13827,14 @@ mod tests { let if_none_match = Some(String::new()); let if_match = Some(" ".to_string()); assert!(!should_prevent_write(&oi, if_none_match, if_match)); + + let oi = ObjectInfo { + etag: Some("abc".to_string()), + ..Default::default() + }; + assert!(should_prevent_write(&oi, Some("W/\"abc\"".to_string()), None)); + assert!(!should_prevent_write(&oi, Some("W/\"def\"".to_string()), None)); + assert!(should_prevent_write(&oi, None, Some("W/\"abc\"".to_string()))); } #[test] @@ -15992,6 +16382,34 @@ mod tests { Some(StorageError::PreconditionFailed), "data movement must not replace a target delete marker" ); + let ordinary_create_only = ObjectOptions { + versioned: true, + http_preconditions: Some(HTTPPreconditions { + if_none_match: Some("*".to_string()), + ..Default::default() + }), + ..Default::default() + }; + assert_eq!( + set_disks + .check_write_precondition(bucket, object, &ordinary_create_only) + .await, + None, + "ordinary If-None-Match: * must proceed past a top-level delete marker" + ); + let ordinary_if_match = ObjectOptions { + versioned: true, + http_preconditions: Some(HTTPPreconditions { + if_match: Some("*".to_string()), + ..Default::default() + }), + ..Default::default() + }; + assert_eq!( + set_disks.check_write_precondition(bucket, object, &ordinary_if_match).await, + Some(StorageError::ObjectNotFound(bucket.to_string(), object.to_string())), + "ordinary If-Match must treat a top-level delete marker as absent" + ); let err = match set_disks .get_object_reader(bucket, object, None, HeaderMap::new(), &opts) .await diff --git a/crates/ecstore/src/set_disk/ops/bucket.rs b/crates/ecstore/src/set_disk/ops/bucket.rs index 738648df7..d64128e5e 100644 --- a/crates/ecstore/src/set_disk/ops/bucket.rs +++ b/crates/ecstore/src/set_disk/ops/bucket.rs @@ -31,6 +31,7 @@ use crate::disk::DiskAPI; pub(crate) enum BucketInfoQuorum { Read, Write, + HealCandidates, } impl SetDisks { @@ -104,7 +105,9 @@ impl SetDisks { // class. A namespace read must intersect that majority; object // readers still enforce the persisted layout's data-shard quorum. BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count.div_ceil(2).max(1)), - BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count / 2 + 1), + BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => { + reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, disk_count / 2 + 1) + } }; if let Some(err) = error { return Err(err.into()); @@ -126,6 +129,29 @@ impl SetDisks { self.list_bucket_with_quorum(BucketInfoQuorum::Write).await } + pub(crate) async fn bucket_needs_reconstruction(&self, bucket: &str) -> Result { + let results = join_all(self.disk_inventory().await.into_iter().map(|disk| async move { + match disk { + Some(disk) => disk.stat_volume(bucket).await, + None => Err(DiskError::DiskNotFound), + } + })) + .await; + let mut present = false; + let mut missing = false; + for result in results { + match result { + Ok(_) => present = true, + Err(DiskError::VolumeNotFound) => missing = true, + Err(error) => return Err(error.into()), + } + } + if !present { + return Err(Error::VolumeNotFound); + } + Ok(missing) + } + pub(crate) async fn list_bucket_with_quorum(&self, quorum: BucketInfoQuorum) -> Result<(Vec, bool)> { let disks = self.disk_inventory().await; let write_quorum = (disks.len() / 2) + 1; @@ -134,6 +160,7 @@ impl SetDisks { let required = match quorum { BucketInfoQuorum::Read => disks.len().div_ceil(2).max(1), BucketInfoQuorum::Write => write_quorum, + BucketInfoQuorum::HealCandidates => disks.len().max(1), }; let mut futures = Vec::with_capacity(disks.len()); @@ -165,7 +192,9 @@ impl SetDisks { let error = match quorum { BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required), - BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required), + BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => { + reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required) + } }; if let Some(err) = error { return Err(err.into()); @@ -190,10 +219,17 @@ impl SetDisks { } } - topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum); + let minimum_presence = if matches!(quorum, BucketInfoQuorum::HealCandidates) { + // Candidate presence is not mutation authority. Every disk must answer, + // and the heal owner validates the committed generation before recreation. + 1 + } else { + topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum); + required + }; let mut buckets = counts .into_values() - .filter_map(|(count, bucket)| (count >= required).then_some(bucket)) + .filter_map(|(count, bucket)| (count >= minimum_presence).then_some(bucket)) .collect::>(); buckets.sort_by(|left, right| left.name.cmp(&right.name)); Ok((buckets, topology_complete)) diff --git a/crates/ecstore/src/set_disk/ops/delete_recovery.rs b/crates/ecstore/src/set_disk/ops/delete_recovery.rs index 2aef4e65b..2a4eb677c 100644 --- a/crates/ecstore/src/set_disk/ops/delete_recovery.rs +++ b/crates/ecstore/src/set_disk/ops/delete_recovery.rs @@ -20,6 +20,7 @@ use crate::disk::{ DELETE_RECOVERY_PROTOCOL_VERSION, DeleteRecoveryAction, DeleteRecoveryInspection, DeleteRecoveryState, DeleteTransactionId, DiskAPI, DiskStore, }; +use crate::error::StorageError; use crate::set_disk::SetDisks; use futures::future::join_all; use std::collections::{BTreeMap, BTreeSet, HashSet}; @@ -35,7 +36,11 @@ pub(crate) struct DeleteRecoverySummary { pub committed: usize, pub rolled_back: usize, pub blocked: usize, + /// Groups blocked by a `LegacyAmbiguous` residue. Unlike other blocks this + /// is deterministic: the residue cannot be classified by retrying. + pub legacy_ambiguous: usize, pub pending: usize, + pub failed: usize, pub reclaimed: usize, /// Observed removals for this exact key, not aggregated across prefixes. pub removed_versions: HashSet, @@ -52,6 +57,7 @@ pub(crate) struct DeleteRecoveryContext<'a> { /// Only the locked heal path may repair a live marker-creation source. pub repair_marker_source: bool, pub namespace_owner: Option>, + pub recovery_ticket: Option, pub can_commit: &'a (dyn Fn() -> bool + Sync), } @@ -63,6 +69,7 @@ impl<'a> DeleteRecoveryContext<'a> { explicit_delete: false, repair_marker_source: false, namespace_owner: None, + recovery_ticket: None, can_commit, } } @@ -74,9 +81,24 @@ impl DeleteRecoverySummary { self.committed += other.committed; self.rolled_back += other.rolled_back; self.blocked += other.blocked; + self.legacy_ambiguous += other.legacy_ambiguous; self.pending += other.pending; + self.failed += other.failed; self.reclaimed += other.reclaimed; } + + /// The error a heal must return when recovery left this key unresolved. + /// An ambiguous legacy residue is permanent and gets its own error; other + /// pending or blocked groups remain the retryable `SlowDown`. + pub(crate) fn unresolved_heal_error(&self) -> Option { + if self.legacy_ambiguous > 0 { + Some(StorageError::DeleteRecoveryLegacyAmbiguous) + } else if self.pending > 0 || self.blocked > 0 { + Some(StorageError::SlowDown) + } else { + None + } + } } pub(in crate::set_disk) fn unsupported_recovery(error: &DiskError) -> bool { @@ -307,6 +329,9 @@ impl SetDisks { { // An ordinary missing-key no-op requires no recovery decision. // Do not manufacture witness transactions while a drive is offline. + if let Some(ticket) = context.recovery_ticket.as_ref() { + self.ctx.delete_recovery_hints().ack(ticket); + } return Ok(()); } let published = join_all(disks.iter().zip(&states).map(|(disk, state)| async { @@ -336,7 +361,15 @@ impl SetDisks { } // Physical cleanup is retryable after the certified decision. An // unavailable participant retains the decision for its eventual return. - let _ = self.recover_delete_key(bucket, object, context).await; + let ticket = context.recovery_ticket.clone(); + if let Ok(summary) = self.recover_delete_key(bucket, object, context).await + && summary.pending == 0 + && summary.blocked == 0 + && summary.failed == 0 + && let Some(ticket) = ticket.as_ref() + { + self.ctx.delete_recovery_hints().ack(ticket); + } Ok(()) } @@ -363,6 +396,7 @@ impl SetDisks { } let pages = join_all(disks.iter().flatten().map(|disk| discover(disk, bucket, None))).await; let mut keys = BTreeSet::new(); + let mut summary = DeleteRecoverySummary::default(); for page in pages { match page { Ok(ids) => { @@ -373,18 +407,24 @@ impl SetDisks { ); } Err(error) if unavailable(&error) => { - return Ok(DeleteRecoverySummary { - pending: 1, - ..Default::default() - }); + summary.pending += 1; } + Err(DiskError::FileCorrupt) => summary.failed += 1, Err(error) => return Err(error), } } - let mut summary = DeleteRecoverySummary::default(); + // Every key requires complete participation. Avoid taking each key's + // lock and repeating discovery while this sweep already lacks a peer. + if summary.pending > 0 { + return Ok(summary); + } // Lock order: caller's bucket lifecycle fence, then one exact key. for object in keys { - summary.add(self.recover_delete_object(bucket, &object, context.clone()).await?); + match self.recover_delete_object(bucket, &object, context.clone()).await { + Ok(result) => summary.add(result), + Err(DiskError::FileCorrupt) => summary.failed += 1, + Err(error) => return Err(error), + } } Ok(summary) } @@ -428,6 +468,7 @@ impl SetDisks { repair_marker_source, namespace_owner, can_commit, + recovery_ticket: _, } = context; let disks = self.disk_inventory().await; let complete_inventory = disks.len() == self.set_drive_count && disks.iter().all(Option::is_some); @@ -505,7 +546,7 @@ impl SetDisks { None }; if let Some((transaction, source)) = prerequisite { - Box::pin(self.heal_delete_recovery_source(bucket, object, &source, can_commit)).await?; + Box::pin(self.heal_delete_recovery_source(bucket, object, &source, can_commit, namespace_owner.as_ref())).await?; let refreshed = inspect(&disks, bucket, object, transaction) .await .into_iter() @@ -522,13 +563,20 @@ impl SetDisks { } all_states.insert(transaction, refreshed); } - // Legacy snapshots contain no set decision. A fresh authorized removal - // applies only after the entire key is absent and every raw shape is safe. - let legacy_only = explicit_delete + // A fresh authorized removal may discard safe legacy residue beside + // certified absence receipts. Every participant must have no live metadata; + // the receipts remain behind the key-wide barrier until all groups settle. + let explicit_legacy_removal = explicit_delete && all_states.values().flatten().all(|copy| { copy.current_digest.is_none() - && matches!(copy.state, DeleteRecoveryState::Absent | DeleteRecoveryState::LegacyAmbiguous) - && (copy.state != DeleteRecoveryState::LegacyAmbiguous || copy.explicit_delete_safe) + && match copy.state { + DeleteRecoveryState::Absent => true, + DeleteRecoveryState::LegacyAmbiguous => copy.explicit_delete_safe, + DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished => { + copy.journal.as_ref().is_some_and(|journal| journal.target_digest.is_none()) + } + DeleteRecoveryState::NewPrepared => false, + } }); // This is a fresh authorized delete of an absent unversioned key, // rather than evidence about the outcome of a historical transaction. @@ -594,7 +642,7 @@ impl SetDisks { continue; } if copies.iter().any(|state| state.state == DeleteRecoveryState::LegacyAmbiguous) { - if legacy_only { + if explicit_legacy_removal { if dry_run { summary.reclaimed += 1; continue; @@ -620,6 +668,7 @@ impl SetDisks { } } else { summary.blocked += 1; + summary.legacy_ambiguous += 1; unresolved = true; } continue; @@ -925,6 +974,37 @@ fn unavailable(error: &DiskError) -> bool { mod tests { use super::*; use crate::bucket::metadata_sys::{self, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + + #[test] + fn unresolved_heal_error_distinguishes_legacy_ambiguity_from_transient_blocks() { + // rustfs/rustfs#8608: a legacy residue is deterministic and must not be + // reported as request-rate pressure. + let resolved = DeleteRecoverySummary::default(); + assert!(resolved.unresolved_heal_error().is_none()); + let pending = DeleteRecoverySummary { + pending: 1, + ..Default::default() + }; + assert!(matches!(pending.unresolved_heal_error(), Some(StorageError::SlowDown))); + let blocked = DeleteRecoverySummary { + blocked: 1, + ..Default::default() + }; + assert!(matches!(blocked.unresolved_heal_error(), Some(StorageError::SlowDown))); + let ambiguous = DeleteRecoverySummary { + blocked: 1, + legacy_ambiguous: 1, + pending: 1, + ..Default::default() + }; + assert!(matches!( + ambiguous.unresolved_heal_error(), + Some(StorageError::DeleteRecoveryLegacyAmbiguous) + )); + let mut total = DeleteRecoverySummary::default(); + total.add(ambiguous); + assert_eq!(total.legacy_ambiguous, 1); + } use crate::disk::{DeleteOptions, ReadOptions, STORAGE_FORMAT_FILE}; use crate::object_api::{ObjectOptions, PutObjReader}; use crate::storage_api_contracts::bucket::{BucketOperations, DeleteBucketOptions, MakeBucketOptions}; @@ -937,6 +1017,9 @@ mod tests { async fn fixture(bucket: &str) -> (Vec, Arc, Vec) { let (dirs, store) = isolated_store_over_temp_disks().await; + // Manual recovery assertions need sole ownership of the seeded journals. + store.background_cancel_token().expect("fixture background token").cancel(); + tokio::task::yield_now().await; init_bucket_metadata_sys(store.clone(), Vec::new()).await; store .make_bucket(bucket, &MakeBucketOptions::default()) @@ -1054,6 +1137,131 @@ mod tests { assert_eq!(body(&store, bucket, None).await, original); } + #[tokio::test] + async fn delete_recovery_corrupt_bucket_does_not_prevent_other_bucket_rollback() { + let bad_bucket = "recover-corrupt-bucket"; + let good_bucket = "recover-good-bucket"; + let (dirs, store, disks) = fixture(bad_bucket).await; + store + .make_bucket(good_bucket, &MakeBucketOptions::default()) + .await + .expect("create healthy bucket"); + put(&store, bad_bucket, &[61; 4096], false).await; + let bad_transaction = applied(&store, &disks, bad_bucket, None, disks.len()).await; + let good_bytes = vec![62; 4096]; + put(&store, good_bucket, &good_bytes, false).await; + applied(&store, &disks, good_bucket, None, disks.len()).await; + for dir in &dirs { + std::fs::write( + dir.path() + .join(bad_bucket) + .join("object") + .join(bad_transaction.to_string()) + .join("delete-prepared"), + b"{broken", + ) + .expect("inject malformed preparation journal"); + } + + let failed = store + .recover_delete_transactions(bad_bucket, &HealOpts::default()) + .await + .expect("bucket corruption is an explicit partial recovery result"); + assert_eq!(failed.delete_recovery.expect("recovery status").failed, 1); + let error = store + .recover_delete_transactions_at_incarnation(bad_bucket, Some("object"), None, &HealOpts::default()) + .await + .expect_err("exact object recovery must retain typed corruption"); + assert!(matches!(error, crate::error::StorageError::FileCorrupt)); + let recovered = store + .recover_delete_transactions(good_bucket, &HealOpts::default()) + .await + .expect("another bucket must remain recoverable"); + assert!(recovered.detail.contains("rolled_back=1")); + assert!(!recovered.delete_recovery.expect("healthy recovery status").has_unresolved()); + assert_eq!(body(&store, good_bucket, None).await, good_bytes); + } + + #[tokio::test] + async fn delete_recovery_corrupt_key_does_not_prevent_other_key_rollback() { + let bucket = "recover-corrupt-key"; + let (dirs, store, disks) = fixture(bucket).await; + let healthy_bytes = vec![63; 4096]; + put(&store, bucket, &healthy_bytes, false).await; + applied(&store, &disks, bucket, None, disks.len()).await; + let mut reader = PutObjReader::from_vec(vec![64; 4096]); + store + .put_object( + bucket, + "a-corrupt", + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("write corrupt-key fixture"); + let transaction = Uuid::new_v4(); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + for disk in &disks { + disk.delete_version_recoverable( + bucket, + "a-corrupt", + FileInfo { + name: "a-corrupt".to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("prepare corrupt-key fixture"); + } + let damaged = dirs[0] + .path() + .join(bucket) + .join("a-corrupt") + .join(transaction.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP); + std::fs::write(&damaged, b"corrupt metadata").expect("damage one backup while retaining discoverable journal"); + + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("per-key corruption must not abort the remaining prefix"); + + assert_eq!(report.delete_recovery.expect("recovery status").failed, 1); + assert!(report.detail.contains("rolled_back=1")); + assert_eq!(std::fs::read(damaged).expect("corrupt proof is retained"), b"corrupt metadata"); + assert_eq!(body(&store, bucket, None).await, healthy_bytes); + } + + #[tokio::test] + async fn delete_recovery_offline_bucket_reports_pending_without_hiding_exact_object_error() { + let bucket = "recover-offline-status"; + let (_dirs, store, _disks) = fixture(bucket).await; + store.pools[0].disk_set[0].disks.write().await[3] = None; + + let result = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("offline recovery must expose a pending summary even with no discovered transactions"); + let status = result.delete_recovery.expect("offline status is retained"); + assert_eq!(status.pending, 1); + assert_eq!(status.failed, 0); + assert!(status.has_unresolved()); + let error = store + .recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default()) + .await + .expect_err("exact-object workers require a typed retry error"); + assert!(matches!(error, crate::error::StorageError::ErasureReadQuorum)); + } + fn snapshot(root: &std::path::Path) -> BTreeMap> { let mut files = BTreeMap::new(); let mut pending = vec![root.to_path_buf()]; @@ -1276,6 +1484,184 @@ mod tests { .expect("non-force bucket deletion must succeed after proof GC"); } + #[tokio::test] + async fn delete_recovery_committed_generation_finishes_beside_stale_write_backup() { + let bucket = "recover-stale-write-backup"; + let (dirs, store, disks) = fixture(bucket).await; + put(&store, bucket, &vec![81; 1024 * 1024], false).await; + let originals: Vec<_> = dirs + .iter() + .map(|dir| snapshot(&dir.path().join(bucket).join("object"))) + .collect(); + let metadata: Vec<_> = originals + .iter() + .map(|files| { + files + .get(std::path::Path::new(STORAGE_FORMAT_FILE)) + .expect("old generation metadata") + .clone() + }) + .collect(); + let old_data_dir = rustfs_filemeta::FileMeta::load(&metadata[0]) + .expect("decode old generation") + .get_data_dirs() + .expect("old generation data directory") + .into_iter() + .flatten() + .next() + .expect("external payload data directory"); + put(&store, bucket, &vec![82; 1024 * 1024], false).await; + for (index, dir) in dirs.iter().enumerate() { + let root = dir.path().join(bucket).join("object"); + if index == disks.len() - 1 { + tokio::fs::remove_dir_all(&root) + .await + .expect("replace returning member with its old generation"); + } + for (relative, bytes) in &originals[index] { + if index != disks.len() - 1 && !relative.starts_with(old_data_dir.to_string()) { + continue; + } + let path = root.join(relative); + tokio::fs::create_dir_all(path.parent().expect("source file parent")) + .await + .expect("restore old generation directory"); + tokio::fs::write(path, bytes).await.expect("restore old generation bytes"); + } + if index != disks.len() - 1 { + tokio::fs::write(root.join(old_data_dir.to_string()).join("xl.meta.bkp"), &metadata[index]) + .await + .expect("retain stale write backup beside older payload"); + } + } + let transaction = applied(&store, &disks, bucket, None, disks.len()).await; + for disk in &disks { + publish(disk, bucket, transaction).await; + } + let incarnation = store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("recovery bucket identity"); + let before_preview: Vec<_> = dirs + .iter() + .map(|dir| snapshot(&dir.path().join(bucket).join("object"))) + .collect(); + let mut preview_context = DeleteRecoveryContext::new(Some(incarnation), &|| true); + preview_context.explicit_delete = true; + preview_context.dry_run = true; + let preview = store.pools[0].disk_set[0] + .recover_delete_object(bucket, "object", preview_context) + .await + .expect("preview fresh removal beside unfinished certified receipts"); + assert_eq!(preview.reclaimed, 1, "fresh intent covers eligible legacy residue beside QuorumCommitted"); + assert_eq!(preview.committed, 1); + assert_eq!(preview.blocked, 0); + assert_eq!(preview.pending, 0); + for (index, dir) in dirs.iter().enumerate() { + assert_eq!( + snapshot(&dir.path().join(bucket).join("object")), + before_preview[index], + "dry-run member {index}" + ); + } + let report = store.pools[0].disk_set[0] + .recover_delete_object(bucket, "object", DeleteRecoveryContext::new(Some(incarnation), &|| true)) + .await + .expect("an unrelated backup cannot prevent the certified transaction from finishing"); + assert_eq!(report.committed, 1, "the shared transaction finishes on all members"); + assert_eq!(report.pending, 0, "no certified payload remains pending"); + assert_eq!(report.blocked, 1, "the uncertified older backup remains explicit recovery work"); + for (index, disk) in disks.iter().enumerate() { + let state = disk + .inspect_delete_transaction(bucket, "object", transaction) + .await + .expect("inspect completed certified transaction"); + assert_eq!(state.state, DeleteRecoveryState::CommitFinished, "member {index}"); + let root = dirs[index].path().join(bucket).join("object"); + assert!( + !root.join(STORAGE_FORMAT_FILE).exists(), + "deleted metadata remains absent on member {index}" + ); + if index != disks.len() - 1 { + assert_eq!( + tokio::fs::read(root.join(old_data_dir.to_string()).join("xl.meta.bkp")) + .await + .expect("uncertified write backup survives"), + metadata[index], + "member {index} preserves the ambiguous older generation" + ); + } else { + assert!( + !root.join(old_data_dir.to_string()).exists(), + "the certified older member reclaims its payload" + ); + } + } + for (index, dir) in dirs.iter().enumerate().take(disks.len() - 1) { + let source = rustfs_filemeta::FileMeta::load(&metadata[index]).expect("decode backup before protection"); + let mut protected = rustfs_filemeta::FileMeta::new(); + for version in &source.versions { + let mut info = version.into_fileinfo(bucket, "object", true).expect("backup version"); + info.metadata.insert("x-amz-object-lock-legal-hold".into(), "ON".into()); + protected.add_version(info).expect("protect ambiguous backup"); + } + tokio::fs::write( + dir.path() + .join(bucket) + .join("object") + .join(old_data_dir.to_string()) + .join("xl.meta.bkp"), + protected.marshal_msg().expect("encode protected backup"), + ) + .await + .expect("persist protected ambiguous generation"); + } + let protected: Vec<_> = dirs + .iter() + .map(|dir| snapshot(&dir.path().join(bucket).join("object"))) + .collect(); + assert!( + store.delete_object(bucket, "object", ObjectOptions::default()).await.is_err(), + "an existing delete decision cannot authorize removing protected legacy metadata" + ); + for (index, dir) in dirs.iter().enumerate() { + assert_eq!( + snapshot(&dir.path().join(bucket).join("object")), + protected[index], + "protected member {index}" + ); + } + for (index, dir) in dirs.iter().enumerate().take(disks.len() - 1) { + tokio::fs::write( + dir.path() + .join(bucket) + .join("object") + .join(old_data_dir.to_string()) + .join("xl.meta.bkp"), + &metadata[index], + ) + .await + .expect("restore unprotected backup fixture"); + } + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("a fresh authorized delete reclaims safe absent-key legacy residue"); + for disk in &disks { + assert!( + discover(disk, bucket, Some("object")) + .await + .expect("inspect final recovery residue") + .is_empty(), + "the fresh delete finalizes every retained decision" + ); + } + store + .delete_bucket(bucket, &DeleteBucketOptions::default()) + .await + .expect("the fully reclaimed object no longer blocks bucket removal"); + } + #[tokio::test] async fn delete_recovery_set_handles_prepared_subset() { let bucket = "recover-subset"; @@ -2033,13 +2419,18 @@ mod tests { } } let error = store - .recover_delete_transactions(bucket, &HealOpts::default()) + .recover_delete_transactions_at_incarnation(bucket, Some("object"), None, &HealOpts::default()) .await .expect_err("cyclic history cannot settle"); assert!( matches!(error, crate::error::StorageError::FileCorrupt), "typed corruption must reject cyclic evidence: {error:?}" ); + let report = store + .recover_delete_transactions(bucket, &HealOpts::default()) + .await + .expect("a bucket report must retain its corrupt-key status"); + assert_eq!(report.delete_recovery.expect("corrupt-key status").failed, 1); for (path, bytes) in evidence { assert_eq!(std::fs::read(path).expect("all proof retained"), bytes); } @@ -2076,4 +2467,123 @@ mod tests { .await .expect("no stale old body or receipts remain"); } + + #[tokio::test] + async fn heal_mrf_inline_overwrite_does_not_strand_rollback_backup() { + use crate::disk::STORAGE_FORMAT_FILE_BACKUP; + use crate::disk::local::inline_metadata_rollback_dir; + use crate::object_api::ShardIntegrityWriteMode; + use rustfs_filemeta::FileMeta; + use rustfs_heal_contracts::heal_channel::{HealScanMode, LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL}; + + for incoming_inline in [true, false] { + let bucket = if incoming_inline { + "heal-inline-rollback" + } else { + "heal-inline-to-external-rollback" + }; + let (dirs, store, disks) = fixture(bucket).await; + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + let synthetic = inline_metadata_rollback_dir(Uuid::nil(), &FileMeta::new()); + let object = "object"; + let child = format!("{object}/{synthetic}"); + let child_body = b"child bytes must survive parent repair".to_vec(); + let write_options = ObjectOptions { + shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy), + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }; + for (key, body) in [(child.as_str(), child_body.as_slice()), (object, b"inline-gen-1".as_slice())] { + store + .put_object(bucket, key, &mut PutObjReader::from_vec(body.to_vec()), &write_options) + .await + .expect("seed initial inline objects"); + } + let read_options = ReadOptions { + read_data: true, + ..Default::default() + }; + let old_child = disks[3] + .read_version("", bucket, &child, "", &read_options) + .await + .expect("physical child before repair"); + let set = &store.pools[0].disk_set[0]; + set.disks.write().await[3] = None; + let payload = if incoming_inline { + b"inline-gen-2-updated".to_vec() + } else { + vec![0x53; 1024 * 1024] + }; + store + .put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &write_options) + .await + .expect("degraded overwrite with a real stale target"); + set.disks.write().await[3] = Some(disks[3].clone()); + let donor = disks[0] + .read_version("", bucket, object, "", &read_options) + .await + .expect("current donor generation"); + assert_eq!(donor.data.is_some(), incoming_inline, "fixture must use the intended commit branch"); + let heal_opts = HealOpts { + scan_mode: HealScanMode::Normal, + ..Default::default() + }; + let first = store + .heal_mrf_object_at_incarnation(bucket, object, "", incarnation, &heal_opts) + .await + .expect("first MRF repair"); + assert!(first.error.is_none(), "first MRF repair: {:?}", first.error); + assert_eq!(first.item.drives_healed(), Some(1)); + assert!(!first.item.repair_verified); + assert_eq!(first.item.detail, LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL); + let target = disks[3] + .read_version("", bucket, object, "", &read_options) + .await + .expect("repaired target generation"); + assert_eq!(target.mod_time, donor.mod_time); + assert_eq!(target.data_dir, donor.data_dir); + assert_eq!(target.size, donor.size); + assert_eq!(target.version_id, donor.version_id); + let child_after = disks[3] + .read_version("", bucket, &child, "", &read_options) + .await + .expect("physical child survives parent cleanup"); + assert_eq!(child_after.data, old_child.data); + assert!(child_after.data.is_some(), "child fixture must carry its physical inline bytes"); + let child_dir = dirs[3].path().join(bucket).join(object).join(synthetic.to_string()); + assert!(!child_dir.join(STORAGE_FORMAT_FILE_BACKUP).exists()); + assert!(child_dir.join(STORAGE_FORMAT_FILE).exists()); + let second = store + .heal_mrf_object_at_incarnation(bucket, object, "", incarnation, &heal_opts) + .await + .expect("follow-up MRF check must not return SlowDown"); + assert!(second.error.is_none(), "second MRF check: {:?}", second.error); + assert_eq!(second.item.drives_healed(), Some(0)); + assert_eq!(second.item.detail, LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL); + + // EC2+2: force the repaired shard to participate in the exact read quorum. + let inventory = set.disks.read().await.clone(); + *set.disks.write().await = vec![None, None, Some(disks[2].clone()), Some(disks[3].clone())]; + for (key, expected) in [(object, payload), (child.as_str(), child_body)] { + let mut reader = set + .get_object_reader( + bucket, + key, + None, + Default::default(), + &ObjectOptions { + no_lock: true, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("exact-quorum GET using repaired target"); + let mut body = Vec::new(); + reader.read_to_end(&mut body).await.expect("read complete parent/child body"); + assert_eq!(body, expected, "exact-quorum body for {key}"); + } + *set.disks.write().await = inventory; + } + } } diff --git a/crates/ecstore/src/set_disk/ops/heal.rs b/crates/ecstore/src/set_disk/ops/heal.rs index 3979d34e0..19018263c 100644 --- a/crates/ecstore/src/set_disk/ops/heal.rs +++ b/crates/ecstore/src/set_disk/ops/heal.rs @@ -33,6 +33,7 @@ struct ExplicitVersionHeal<'a> { allow_regeneration: bool, preserve_unrepairable: bool, commit_fence: Option<&'a (dyn Fn() -> bool + Sync)>, + namespace_owner: Option<&'a Arc>, retirement: Option<&'a MarkerRetirementContext<'a>>, } @@ -66,6 +67,14 @@ pub(crate) struct AbsenceProofRequest<'a> { pub allow_unversioned: bool, } +fn heal_rollback_backup_dir(response: &disk::RenameDataResp) -> Option { + // A dedicated cleanup directory identifies a current response whose absent + // rollback is intentional. Older peers report only old_data_dir. + response + .rollback_data_dir + .or_else(|| response.old_data_dir.filter(|_| response.cleanup_data_dir.is_none())) +} + fn is_unavailable_heal_rpc(error: &DiskError) -> bool { let DiskError::Io(error) = error else { return false; @@ -783,6 +792,7 @@ impl SetDisks { allow_regeneration: true, preserve_unrepairable: false, commit_fence: None, + namespace_owner: None, retirement: None, }, &mut None, @@ -808,8 +818,15 @@ impl SetDisks { false, true, self.default_parity_count, + ); + #[cfg(feature = "diagnostic-read-version-caller")] + let metadata_read = crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::Heal, + metadata_read, ) .await?; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + let metadata_read = metadata_read.await?; let (parts_metadata, errs, _) = metadata_read.into_legacy(); let (read_quorum, _) = Self::object_quorum_from_meta(&parts_metadata, &errs, self.default_parity_count)?; let read_quorum = usize::try_from(read_quorum).map_err(|_| DiskError::ErasureReadQuorum)?; @@ -922,6 +939,7 @@ impl SetDisks { allow_regeneration: allow_explicit_version_regen, preserve_unrepairable, commit_fence, + namespace_owner, retirement, } = run; trace!( @@ -984,10 +1002,10 @@ impl SetDisks { // Bound, not `_`: this guard must live to the end of the scope. A bare // `_` would drop it here and release the namespace write lock. let _write_lock_guard = if !opts.no_lock && !read_repair_uses_shared_lock { - Some( + Some(Arc::new( self.acquire_heal_object_lock(bucket, object, version_id, HealObjectLockKind::Write, "heal_object") .await?, - ) + )) } else { None }; @@ -1877,7 +1895,7 @@ impl SetDisks { ) .await { - Ok(Some(guard)) => Some(guard), + Ok(Some(guard)) => Some(Arc::new(guard)), Ok(None) => { let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await; return Ok((result, Some(DiskError::Timeout))); @@ -1902,6 +1920,7 @@ impl SetDisks { let mut rename_attempts = 0usize; let mut rename_successes = 0usize; let mut healed_disks = vec![None; out_dated_disks.len()]; + let mut backup_cleanup_owner: Option> = None; for (index, outdated_disk) in out_dated_disks.iter().enumerate() { if let Some(disk) = outdated_disk { if let Some(scope) = &bucket_heal_scope { @@ -1930,59 +1949,113 @@ impl SetDisks { .await }; - if let Err(err) = &rename_result { - warn!( - event = EVENT_HEAL_OBJECT_RENAME, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_HEAL, - bucket, - object, - version_id, - disk_index = index, - endpoint = %disk.endpoint(), - tmp_id, - result = "failed", - error = %err, - "Heal object rename failed" - ); - } else { - rename_successes += 1; - healed_disks[index] = Some(disk.clone()); - if parts_metadata[index].is_remote() { - let rm_data_dir = - parts_metadata[index].data_dir.expect("operation should succeed").to_string(); - - let d_path = Path::new(&encode_dir_object(object)).join(rm_data_dir); - - if let Err(e) = disk - .delete( - bucket, - d_path.to_str().expect("operation should succeed"), - DeleteOptions { - immediate: true, - recursive: true, - ..Default::default() - }, - ) - .await - { - // The healed shard has already been renamed into place; a - // failure cleaning up the old remote data dir must not abort - // the heal and leak the tmp shards (backlog#799 B20). - warn!( - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_HEAL, - bucket, - object, - error = %e, - "Heal remote data-dir cleanup failed" - ); - } + match rename_result { + Err(err) => { + warn!( + event = EVENT_HEAL_OBJECT_RENAME, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_HEAL, + result = "failed", + bucket, + object, + version_id, + disk_index = index, + endpoint = %disk.endpoint(), + tmp_id, + error = %err, + "Heal object rename failed" + ); } + Ok(rename_response) => { + rename_successes += 1; + healed_disks[index] = Some(disk.clone()); + // Heal can reuse a live data directory, and a synthetic + // rollback directory can also contain a child key. + // Retire only this publication's backup while fenced. + if let Some(rollback_dir) = heal_rollback_backup_dir(&rename_response) + && bucket_heal_scope.as_ref().is_none_or(|scope| scope.check().is_ok()) + && commit_fence.is_none_or(|check| check()) + && !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + && !_read_repair_write_lock_guard + .as_ref() + .is_some_and(|guard| guard.is_lock_lost()) + { + let backup_path = format!( + "{}/{rollback_dir}/{}", + rustfs_utils::path::encode_dir_object_ref(object), + disk::STORAGE_FORMAT_FILE_BACKUP, + ); + // The syscall can outlive this future; retain the exact-key + // locks until physical cleanup has finished. + let cleanup_owner = backup_cleanup_owner.get_or_insert_with(|| { + Arc::new(( + namespace_owner.cloned(), + _write_lock_guard.clone(), + _read_repair_write_lock_guard.clone(), + bucket_heal_scope.clone(), + self.ctx.begin_namespace_commit(), + )) + }); + match disk + .delete_with_namespace_owner( + bucket, + &backup_path, + DeleteOptions::default(), + Some(cleanup_owner.clone()), + ) + .await + { + Ok(()) | Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Err(err) => warn!( + event = EVENT_HEAL_OBJECT_RENAME, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_HEAL, + result = "cleanup_failed", + bucket, + object, + version_id, + disk_index = index, + error = %err, + "Heal rollback backup cleanup failed" + ), + } + } + if parts_metadata[index].is_remote() { + let rm_data_dir = + parts_metadata[index].data_dir.expect("operation should succeed").to_string(); - for (i, v) in result.before.drives.iter().enumerate() { - if v.endpoint == disk.endpoint().to_string() { - result.after.drives[i].state = DriveState::Ok.to_string(); + let d_path = Path::new(&encode_dir_object(object)).join(rm_data_dir); + + if let Err(e) = disk + .delete( + bucket, + d_path.to_str().expect("operation should succeed"), + DeleteOptions { + immediate: true, + recursive: true, + ..Default::default() + }, + ) + .await + { + // The healed shard has already been renamed into place; a + // failure cleaning up the old remote data dir must not abort + // the heal and leak the tmp shards (backlog#799 B20). + warn!( + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_HEAL, + bucket, + object, + error = %e, + "Heal remote data-dir cleanup failed" + ); + } + } + + for (i, v) in result.before.drives.iter().enumerate() { + if v.endpoint == disk.endpoint().to_string() { + result.after.drives[i].state = DriveState::Ok.to_string(); + } } } } @@ -3396,6 +3469,7 @@ impl SetDisks { object: &str, source: &crate::disk::DeleteRecoveryVersionIdentity, can_commit: &(dyn Fn() -> bool + Sync), + namespace_owner: Option<&Arc>, ) -> disk::error::Result<()> { if !can_commit() { return Err(DiskError::DiskOngoingReq); @@ -3441,6 +3515,7 @@ impl SetDisks { allow_regeneration: false, preserve_unrepairable: true, commit_fence: Some(can_commit), + namespace_owner, retirement: None, }, &mut None, @@ -3559,8 +3634,8 @@ impl SetDisks { None, )); } - if recovery.pending > 0 || recovery.blocked > 0 { - return Err(StorageError::SlowDown); + if let Some(error) = recovery.unresolved_heal_error() { + return Err(error); } } @@ -3630,6 +3705,8 @@ impl SetDisks { // Pass no_lock=true since we already obtained write lock (or are already called with no_lock=true) let mut inner_opts = *opts; inner_opts.no_lock = true; + let commit_fence = || !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()); + let namespace_owner = _write_lock_guard.as_ref().map(|guard| guard.clone() as Arc); let (result, err) = self .heal_object_with_explicit_version_regen( bucket, @@ -3639,7 +3716,8 @@ impl SetDisks { opts: &inner_opts, allow_regeneration: true, preserve_unrepairable: false, - commit_fence: None, + commit_fence: Some(&commit_fence), + namespace_owner: namespace_owner.as_ref(), retirement: proof.retirement, }, absence, @@ -3661,7 +3739,8 @@ impl SetDisks { opts: &inner_opts, allow_regeneration: true, preserve_unrepairable: false, - commit_fence: None, + commit_fence: Some(&commit_fence), + namespace_owner: namespace_owner.as_ref(), retirement: proof.retirement, }, absence, @@ -3714,10 +3793,11 @@ impl SetDisks { #[cfg(test)] mod heal_result_report_tests { use super::{ - AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, HealWriterFailures, - ReadRepairCommitFingerprint, ReadRepairPauseScope, SetDisks, heal_writer_error_summary, is_retryable_heal_writer_error, + AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, ExplicitVersionHeal, + HealWriterFailures, ReadRepairCommitFingerprint, ReadRepairPauseScope, SetDisks, heal_writer_error_summary, + is_retryable_heal_writer_error, }; - use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope, HealWriterFailureScope}; + use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope, HealWriterFailureScope, heal_rollback_backup_dir}; use crate::disk::endpoint::Endpoint; use crate::disk::error::DiskError; use crate::disk::format::FormatV3; @@ -3730,6 +3810,7 @@ mod heal_result_report_tests { }; use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions}; use crate::storage_api_contracts::heal::HealOperations as _; + use crate::storage_api_contracts::namespace::NamespaceLocking as _; use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _}; use crate::{ config::storageclass, @@ -4673,6 +4754,11 @@ mod heal_result_report_tests { expected.to_string(), "{case}: after.drives must reflect the actual rename outcome at index {index}" ); + let data_path = temp_dirs[index].path().join(&bucket).join(object).join(data_dir.to_string()); + assert!( + !data_path.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP).exists(), + "{case}: heal must retire the backup even when it shares the live data directory" + ); assert_eq!( temp_dirs[index] .path() @@ -5576,6 +5662,331 @@ mod heal_result_report_tests { } } + #[test] + fn heal_rollback_backup_selection_preserves_legacy_peer_semantics() { + let a = Uuid::from_u128(1); + let b = Uuid::from_u128(2); + for (old, rollback, cleanup, expected) in [ + (None, None, None, None), + (Some(a), None, Some(a), None), + (Some(a), Some(a), Some(a), Some(a)), + (Some(a), Some(b), Some(a), Some(b)), + (Some(a), None, None, Some(a)), + ] { + let response = crate::disk::RenameDataResp { + old_data_dir: old, + rollback_data_dir: rollback, + cleanup_data_dir: cleanup, + ..Default::default() + }; + assert_eq!(heal_rollback_backup_dir(&response), expected); + } + let legacy: crate::disk::RenameDataResp = + serde_json::from_str(r#"{"old_data_dir":"00000000-0000-0000-0000-000000000001","sign":null}"#) + .expect("older peer response without rollback or cleanup fields"); + assert_eq!(heal_rollback_backup_dir(&legacy), Some(a)); + } + + #[tokio::test] + async fn non_inline_heal_preserves_backup_after_publication_fence_loss() { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "heal-non-inline-fence"; + let object = "object"; + for disk in &disks { + disk.make_volume(bucket).await.expect("bucket volume"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + set.put_object(bucket, object, &mut PutObjReader::from_vec(vec![0x11; 1024 * 1024]), &options) + .await + .expect("seed external generation"); + let old = disks[0] + .read_version("", bucket, object, "", &ReadOptions::default()) + .await + .expect("old target"); + assert!(old.data.is_none()); + let old_dir = old.data_dir.expect("external old data directory"); + let metadata_path = dirs[0].path().join(bucket).join(object).join(STORAGE_FORMAT_FILE); + let old_metadata = tokio::fs::read(&metadata_path).await.expect("save stale metadata"); + let payload = vec![0x22; 1024 * 1024]; + set.put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &options) + .await + .expect("write external current generation"); + let current = disks[0] + .read_version("", bucket, object, "", &ReadOptions::default()) + .await + .expect("current target"); + assert!(current.data.is_none()); + let part = format!("{object}/{}/part.1", current.data_dir.expect("current external directory")); + let expected_shard = disks[0].read_all(bucket, &part).await.expect("current encoded shard"); + tokio::fs::write(&metadata_path, old_metadata) + .await + .expect("restore stale metadata"); + let backup = metadata_path + .parent() + .expect("object directory") + .join(old_dir.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP); + let can_commit = || !backup.exists(); + let opts = HealOpts { + no_lock: true, + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + let (_, error) = set + .heal_object_with_explicit_version_regen( + bucket, + object, + "", + ExplicitVersionHeal { + opts: &opts, + allow_regeneration: true, + preserve_unrepairable: false, + commit_fence: Some(&can_commit), + namespace_owner: None, + retirement: None, + }, + &mut None, + ) + .await + .expect("report committed external repair"); + assert!( + error.is_none(), + "post-publication fence loss must preserve the committed target: {error:?}" + ); + assert!(backup.exists(), "lost fence must retain the actual rollback witness"); + assert_eq!(disks[0].read_all(bucket, &part).await.expect("repaired physical shard"), expected_shard); + let mut reader = set + .get_object_reader(bucket, object, None, Default::default(), &options) + .await + .expect("committed object remains readable"); + let mut body = Vec::new(); + tokio::io::copy(&mut reader, &mut body).await.expect("complete repaired GET"); + assert_eq!(body, payload); + } + + #[cfg(not(windows))] + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn cancelled_heal_backup_cleanup_retains_key_owner_until_unlink_finishes() { + use crate::disk::os::prepared_publication_test_hooks::{Stage, install_at}; + use std::time::Duration; + + for mode in ["outer", "inner", "read-repair"] { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = format!("heal-cleanup-cancel-{mode}"); + let object = "object"; + for disk in &disks { + disk.make_volume(&bucket).await.expect("create bucket volume"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + set.put_object(&bucket, object, &mut PutObjReader::from_vec(vec![0x11; 1024 * 1024]), &options) + .await + .expect("write old generation on every disk"); + let metadata_path = dirs[0].path().join(&bucket).join(object).join(STORAGE_FORMAT_FILE); + let old_metadata = tokio::fs::read(&metadata_path).await.expect("save stale replica"); + let old = disks[0] + .read_version("", &bucket, object, "", &ReadOptions::default()) + .await + .expect("read old non-inline target"); + assert!(old.data.is_none(), "cancellation fixture must use non-inline publication"); + let rollback_dir = old.data_dir.expect("old external data directory"); + let payload = vec![0x22; 1024 * 1024]; + set.put_object(&bucket, object, &mut PutObjReader::from_vec(payload.clone()), &options) + .await + .expect("write current generation on every disk"); + tokio::fs::write(&metadata_path, old_metadata) + .await + .expect("restore stale target"); + let backup_path = disks[0] + .get_object_path_for_io_if_local( + &bucket, + &format!("{object}/{rollback_dir}/{}", crate::disk::STORAGE_FORMAT_FILE_BACKUP), + ) + .expect("fixture disk is local") + .expect("physical backup path"); + let (arrived_tx, arrived_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let _hook = install_at(Stage::Remove, &backup_path, move || { + let _ = arrived_tx.send(()); + let _ = release_rx.recv_timeout(Duration::from_secs(30)); + }); + let task_set = set.clone(); + let task_bucket = bucket.clone(); + let heal = tokio::spawn(async move { + let opts = HealOpts { + read_repair: mode == "read-repair", + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + if mode == "outer" { + let (_, error) = task_set + .heal_object_with_retirement( + &task_bucket, + object, + "", + &opts, + &mut None, + AbsenceProofRequest { + retirement: None, + allow_unversioned: false, + }, + ) + .await + .expect("production wrapper heal should reach cleanup"); + assert!(error.is_none(), "production wrapper heal error: {error:?}"); + } else { + let (_, error) = task_set + .heal_object(&task_bucket, object, "", &opts) + .await + .expect("inner/read-repair heal should reach cleanup"); + assert!(error.is_none(), "inner/read-repair heal error: {error:?}"); + } + }); + let arrived = tokio::time::timeout(Duration::from_secs(10), arrived_rx).await; + if !matches!(arrived, Ok(Ok(()))) { + heal.abort(); + let outcome = heal.await; + let _ = release_tx.send(()); + panic!("{mode}: cleanup must reach the owned physical unlink hook; task: {outcome:?}"); + } + heal.abort(); + assert!(heal.await.expect_err("heal must be cancelled at the unlink").is_cancelled()); + assert!( + set.ctx.namespace_commits_pending(), + "{mode}: physical cleanup must retain namespace ownership" + ); + let namespace = set.new_ns_lock(&bucket, object).await.expect("successor namespace"); + let mut successor = Box::pin(namespace.get_write_lock(Duration::from_secs(30))); + assert!( + tokio::time::timeout(Duration::from_secs(1), &mut successor).await.is_err(), + "{mode}: a successor must not enter while the old unlink is pending" + ); + release_tx.send(()).expect("release physical cleanup"); + let successor = tokio::time::timeout(Duration::from_secs(10), successor) + .await + .expect("physical completion should release the key lock") + .expect("successor should acquire the released lock"); + assert!(!backup_path.exists(), "{mode}: owned unlink must finish before releasing the key"); + drop(successor); + let mut reader = set + .get_object_reader(&bucket, object, None, Default::default(), &options) + .await + .expect("committed object remains readable after cancellation"); + let mut body = Vec::new(); + tokio::io::copy(&mut reader, &mut body) + .await + .expect("read full committed body"); + assert_eq!(body, payload, "{mode}: cancellation must preserve committed data"); + } + } + + #[tokio::test] + async fn heal_reclaims_published_inline_backup_and_preserves_child_key() { + let (dirs, disks, set) = hermetic_set_disks_isolated(4).await; + let bucket = "heal-inline-rollback"; + let object = "nested/object"; + for disk in &disks { + disk.make_volume(bucket).await.expect("create bucket volume"); + } + let options = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let rollback_dir = crate::disk::local::inline_metadata_rollback_dir(Uuid::nil(), &rustfs_filemeta::FileMeta::new()); + let child = format!("{object}/{rollback_dir}"); + let child_payload = b"a child key must survive parent repair".to_vec(); + set.put_object(bucket, &child, &mut PutObjReader::from_vec(child_payload.clone()), &options) + .await + .expect("write child key in the synthetic rollback directory"); + set.put_object(bucket, object, &mut PutObjReader::from_vec(vec![0x11; 592]), &options) + .await + .expect("write old inline generation on every disk"); + let metadata_path = dirs[0].path().join(bucket).join(object).join(STORAGE_FORMAT_FILE); + let old_metadata = tokio::fs::read(&metadata_path).await.expect("save stale target metadata"); + let payload = vec![0x22; 592]; + set.put_object(bucket, object, &mut PutObjReader::from_vec(payload.clone()), &options) + .await + .expect("write current inline generation on every disk"); + let read_options = ReadOptions { + read_data: true, + ..Default::default() + }; + let expected = disks[0] + .read_version("", bucket, object, "", &read_options) + .await + .expect("read current target generation"); + assert!(expected.data.is_some(), "fixture must exercise inline publication"); + tokio::fs::write(&metadata_path, &old_metadata) + .await + .expect("restore one stale replica without a pre-existing delete transaction"); + let backup_path = metadata_path + .parent() + .expect("object directory") + .join(rollback_dir.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP); + let opts = HealOpts { + no_lock: true, + scan_mode: HealScanMode::Deep, + ..Default::default() + }; + let (_, error) = set + .heal_object(bucket, object, "", &HealOpts { dry_run: true, ..opts }) + .await + .expect("dry-run stale inline repair"); + assert!(error.is_none(), "dry-run error: {error:?}"); + assert_eq!(tokio::fs::read(&metadata_path).await.expect("dry-run target"), old_metadata); + assert!(!backup_path.exists(), "dry-run must not publish a backup"); + + let failure_scope = HealRenameFailureScope::install(bucket, object, &[expected.erasure.distribution[0] - 1]); + let (_, error) = set + .heal_object(bucket, object, "", &opts) + .await + .expect("report failed publication"); + drop(failure_scope); + assert!(error.is_some(), "failed rename must remain retryable"); + assert_eq!(tokio::fs::read(&metadata_path).await.expect("failed target"), old_metadata); + assert!(!backup_path.exists(), "failed publication must not reclaim or manufacture a backup"); + + let (result, error) = set + .heal_object(bucket, object, "", &opts) + .await + .expect("repair stale inline replica"); + assert!(error.is_none(), "repair error: {error:?}"); + assert_eq!(result.drives_healed(), Some(1)); + let repaired = disks[0] + .read_version("", bucket, object, "", &read_options) + .await + .expect("read repaired physical target"); + assert_eq!(repaired.mod_time, expected.mod_time); + assert_eq!(repaired.data_dir, expected.data_dir); + assert_eq!(repaired.data, expected.data); + assert!(!backup_path.exists(), "successful heal must retire its old-generation backup"); + let evidence = disks[0] + .discover_delete_transactions(bucket, Some(object), None, 128) + .await + .expect("discover delete evidence without treating a child key as a transaction"); + assert!(evidence.transactions.is_empty(), "successful repair must leave no delete blocker"); + for (key, expected_body) in [(object, payload), (child.as_str(), child_payload)] { + let mut reader = set + .get_object_reader(bucket, key, None, Default::default(), &options) + .await + .expect("read repaired parent and preserved child"); + let mut body = Vec::new(); + tokio::io::copy(&mut reader, &mut body) + .await + .expect("read complete object body"); + assert_eq!(body, expected_body, "object {key}"); + } + let (healthy, error) = set.heal_object(bucket, object, "", &opts).await.expect("repeat repair"); + assert!(error.is_none(), "repeat repair error: {error:?}"); + assert_eq!(healthy.drives_healed(), Some(0)); + } + #[tokio::test] async fn deep_heal_inline_bitrot_rebuilds_physical_data_and_parity_shards() { use crate::storage_api_contracts::range::HTTPRangeSpec; @@ -5593,12 +6004,13 @@ mod heal_result_report_tests { read_data: true, ..Default::default() }; - let object_options = ObjectOptions { + let mut object_options = ObjectOptions { no_lock: true, ..Default::default() }; for parity in [false, true] { + object_options.versioned = parity; for corrupt_prefix in [false, true] { let object = format!("parity-{parity}-prefix-{corrupt_prefix}.bin"); set.put_object(bucket, &object, &mut PutObjReader::from_vec(payload.clone()), &object_options) @@ -5672,6 +6084,19 @@ mod heal_result_report_tests { assert_eq!(repaired.data, original_meta.data, "heal must restore the exact encoded shard"); assert_eq!(repaired.version_id, original_meta.version_id); assert_eq!(repaired.data_dir, original_meta.data_dir); + let rollback_dir = crate::disk::local::inline_metadata_rollback_dir( + repaired.version_id.unwrap_or_default(), + &rustfs_filemeta::FileMeta::new(), + ); + assert!( + !metadata_path + .parent() + .expect("object directory") + .join(rollback_dir.to_string()) + .join(crate::disk::STORAGE_FORMAT_FILE_BACKUP) + .exists(), + "inline bitrot repair must not retain a rollback backup" + ); } for (range, expected) in [ diff --git a/crates/ecstore/src/set_disk/ops/locking.rs b/crates/ecstore/src/set_disk/ops/locking.rs index 6c3fc112f..df3f8400e 100644 --- a/crates/ecstore/src/set_disk/ops/locking.rs +++ b/crates/ecstore/src/set_disk/ops/locking.rs @@ -28,6 +28,7 @@ use crate::disk::health_state::DriveMembershipSnapshot; use crate::disk::{DiskAPI, new_disk}; use crate::runtime::sources as runtime_sources; use rand::prelude::SliceRandom; +use rustfs_utils::path::object_name_disk_identity; #[cfg(test)] use uuid::Uuid; @@ -63,9 +64,12 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks { NamespaceLock::with_local_manager_shared(self.set_lock_namespace.clone(), self.local_lock_manager.clone()) }; + // Lock the file the disk resolves, not the spelling: on Windows `a\b` + // and `a/b` share one xl.meta, so they must share one lock. The key is + // folded by the requesting node only; lock servers compare it verbatim. let resource = ObjectKey { bucket: Arc::from(bucket), - object: Arc::from(object), + object: Arc::from(object_name_disk_identity(object).as_ref()), version: None, }; diff --git a/crates/ecstore/src/set_disk/ops/multipart.rs b/crates/ecstore/src/set_disk/ops/multipart.rs index e5f9728ee..afd8e25e2 100644 --- a/crates/ecstore/src/set_disk/ops/multipart.rs +++ b/crates/ecstore/src/set_disk/ops/multipart.rs @@ -63,7 +63,6 @@ use crate::crash_inject::{self, CrashPoint}; use crate::disk::DiskAPI; #[cfg(test)] use crate::disk::DiskOption; -#[cfg(test)] use crate::disk::STORAGE_FORMAT_FILE; #[cfg(test)] use crate::disk::new_disk; @@ -231,6 +230,60 @@ pub(crate) struct StaleMultipartCleanupGuard { lock_guard: ObjectLockDiagGuard, } +/// Exact-key recovery is admitted only after every disk proves that the upload +/// metadata is absent. Keep the checked disk generation and upload lock through +/// deletion; a quorum miss is not proof that an upload is an orphan. +/// Final object data, completion replay identity, and old-data cleanup receipts +/// live under the destination bucket/object, never this staging path. This also +/// permits removing an exact orphan from a previous bucket incarnation without +/// claiming that its UploadID belongs to the current incarnation. +pub(crate) struct OrphanMultipartCleanupGuard { + set: Arc, + disks: Vec>, + upload_path: String, + lock_guard: Arc, + exists: bool, +} + +impl OrphanMultipartCleanupGuard { + pub(crate) async fn delete(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result { + fence_commit_on_lock_loss(Some(&self.lock_guard), "orphan_multipart_cleanup", &self.upload_path)?; + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + if !self.exists { + return Ok(false); + } + + let results = join_all(self.disks.iter().map(|disk| async { + let Some(disk) = disk else { + return Err(DiskError::DiskNotFound); + }; + match disk + .delete( + RUSTFS_META_MULTIPART_BUCKET, + &self.upload_path, + crate::disk::DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + { + Err(DiskError::FileNotFound | DiskError::PathNotFound | DiskError::VolumeNotFound) => Ok(()), + result => result, + } + })) + .await; + self.set.record_capacity_scope_if_needed(None, &self.disks); + let errors = results.into_iter().map(|result| result.err()).collect::>(); + if let Some(err) = reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, self.disks.len()) { + return Err(to_object_err(err.into(), vec![bucket, object])); + } + fence_commit_on_lock_loss(Some(&self.lock_guard), "orphan_multipart_cleanup", &self.upload_path)?; + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + Ok(true) + } +} + pub(crate) struct DataMovementMultipartAbortGuard { upload_path: String, write_quorum: Option, @@ -557,6 +610,34 @@ fn multipart_bucket_incarnation_matches(metadata: &HashMap, expe matches!(multipart_bucket_incarnation_id(metadata), Ok(Some(actual)) if actual == expected) } +fn orphan_multipart_upload_id_is_canonical(upload_id: &str) -> bool { + let Ok(decoded) = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes()) else { + return false; + }; + let Ok(decoded) = std::str::from_utf8(&decoded) else { + return false; + }; + let Some((_, raw)) = decoded.split_once('.') else { + return false; + }; + let Some((uuid, timestamp)) = raw.split_once('x') else { + return false; + }; + let Ok(id) = Uuid::parse_str(uuid) else { + return false; + }; + if id.is_nil() || id.to_string() != uuid { + return false; + } + let Ok(timestamp_ns) = timestamp.parse::() else { + return false; + }; + if timestamp_ns.to_string() != timestamp { + return false; + } + OffsetDateTime::from_unix_timestamp_nanos(timestamp_ns).is_ok() +} + fn validate_multipart_bucket_incarnation( metadata: &HashMap, bucket: &str, @@ -1246,6 +1327,114 @@ impl SetDisks { }) } + pub(crate) async fn lock_orphan_multipart_cleanup( + self: &Arc, + bucket: &str, + object: &str, + upload_id: &str, + opts: &ObjectOptions, + held_guards: &[OrphanMultipartCleanupGuard], + ) -> Result { + let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()); + if opts.data_movement + || opts.no_lock + || self.local_lock_manager.is_disabled() + || !orphan_multipart_upload_id_is_canonical(upload_id) + { + return Err(invalid_upload()); + } + // This path is derived from the caller's exact bucket/key and a canonical + // UUID/timestamp upload identity, never from the shared MPU directory list. + let upload_path = Self::get_upload_id_dir(bucket, object, upload_id); + // Pool labels do not partition the raw ObjectKey sent to lock clients. + // Reuse the actual owner only for this exact path and proven backend; + // independent distributed domains still need their own guards. + let distributed = self.ctx.is_dist_erasure().await; + let mut lock_guard = None; + for held in held_guards { + if held.upload_path != upload_path { + continue; + } + let shared_domain = match (distributed, held.set.ctx.is_dist_erasure().await) { + (false, false) => self.shares_namespace_lock_domain(&held.set).await, + (true, true) => { + let same = super::super::same_distributed_lock_domain(&self.lockers, &held.set.lockers); + if !same + && self + .lockers + .iter() + .any(|client| held.set.lockers.iter().any(|held_client| Arc::ptr_eq(client, held_client))) + { + // A guard for only part of another quorum is not proof + // of ownership there, and acquiring it again may wait + // on ourselves. Preserve all staging in this topology. + return Err(Error::other("orphan multipart cleanup cannot cover partially overlapping lock domains")); + } + same + } + _ => false, + }; + if shared_domain { + lock_guard = Some(Arc::clone(&held.lock_guard)); + break; + } + } + let lock_guard = match lock_guard { + Some(guard) => guard, + None => Arc::new( + self.acquire_write_lock_diag("orphan_multipart_cleanup", RUSTFS_META_MULTIPART_BUCKET, &upload_path) + .await?, + ), + }; + fence_commit_on_lock_loss(Some(&lock_guard), "orphan_multipart_cleanup", &upload_path)?; + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + let disks = self.disks.read().await.clone(); + if disks.len() != self.set_drive_count || disks.is_empty() { + return Err(Error::ErasureReadQuorum); + } + // Raw full fan-out cannot quorum-short-circuit or interpret an empty + // version list as absence of the physical metadata file. + let (_, errors) = Self::read_all_raw_file_info(&disks, RUSTFS_META_MULTIPART_BUCKET, &upload_path, false).await; + if errors.len() != disks.len() { + return Err(Error::ErasureReadQuorum); + } + for error in errors { + match error { + Some(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + // Even one valid metadata copy may still be recoverable. Never + // convert a partial, corrupt, or unreachable upload into debris. + None => return Err(invalid_upload()), + Some(error) => return Err(to_object_err(error.into(), vec![bucket, object, upload_id])), + } + } + let mut exists = false; + for disk in &disks { + let Some(disk) = disk else { + return Err(Error::ErasureReadQuorum); + }; + match disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await { + Ok(entries) => { + // Raw metadata reads report a directory named xl.meta as + // FileNotFound. Its physical presence is unknown/corrupt + // metadata, not proof that this staging can be deleted. + if entries.iter().any(|entry| entry.trim_end_matches('/') == STORAGE_FORMAT_FILE) { + return Err(to_object_err(DiskError::FileCorrupt.into(), vec![bucket, object, upload_id])); + } + exists = true; + } + Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Err(error) => return Err(to_object_err(error.into(), vec![bucket, object, upload_id])), + } + } + Ok(OrphanMultipartCleanupGuard { + set: self.clone(), + disks, + upload_path, + lock_guard, + exists, + }) + } + #[allow(clippy::too_many_arguments)] pub(crate) async fn list_multipart_uploads_for_incarnation( &self, @@ -1678,6 +1867,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { drop(admission_guard); } + let mut commit_owns_cleanup = false; let result: Result = async { let erasure = Arc::new(coding::Erasure::try_new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size) @@ -1881,120 +2071,171 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { .await?; (Some(upload_guard), Some(part_guard)) }; - let (commit_fi, _) = self - .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts) - .await?; - let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?; - if commit_size_limit != multipart_size_limit { - return Err(Error::InvalidArgument( - "multipart upload".to_string(), - SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(), - "size limit metadata changed or is missing".to_string(), - )); - } - ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?; - ensure_multipart_bucket_incarnation( - &self.ctx, - &commit_fi, - bucket, - object, - upload_id, - opts.expected_bucket_incarnation_id, - ) - .await?; - #[cfg(test)] - pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost).await; - fence_commit_on_lock_loss(_upload_commit_guard.as_ref(), "put_object_part_commit", &upload_id_path)?; - fence_commit_on_lock_loss(_part_commit_guard.as_ref(), "put_object_part_commit", &part_lock_path)?; - if opts - .namespace_lock_fence - .as_ref() - .is_some_and(NamespaceLockFence::is_lock_lost) - { - return Err(StorageError::NamespaceLockQuorumUnavailable { - mode: "put_object_part_outer_lock", - bucket: bucket.to_string(), - object: object.to_string(), - required: 1, - achieved: 0, - }); - } - ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; - - if let Some(limit) = commit_size_limit { - let current_size = self - .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id) - .await?; - let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?; - admitted_multipart_size(current_size, candidate_size, limit)?; - } - - let decommission_capacity_guard = if let Some(store) = opts.decommission_capacity_admission.as_ref() { - Some( - store - .acquire_external_decommission_capacity_fence( - &[self.pool_index], - DecommissionCapacityAdmission::ExistingMultipart, - ) - .await?, - ) - } else { - None - }; - #[cfg(test)] - pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterCapacityAdmission).await; - if decommission_capacity_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) { - return Err(StorageError::NamespaceLockQuorumUnavailable { - mode: "put_object_part_decommission_capacity", - bucket: bucket.to_string(), - object: object.to_string(), - required: 1, - achieved: 0, - }); - } - let rename_result = self - .rename_part( - &shuffle_disks, - RUSTFS_META_TMP_BUCKET, - &tmp_part_path, - RUSTFS_META_MULTIPART_BUCKET, - &part_path, - part_info_buff.into(), - write_quorum, - Some(MultipartWriteQuorumContext { - stage: MULTIPART_WRITE_QUORUM_RENAME_PART, + // Only guards owned by this operation may outlive its caller. Internal + // data-movement and borrowed-lock callers retain their synchronous + // lifetime; a fence by itself cannot transfer the outer lock lease. + let detach_commit_owner = _upload_commit_guard.is_some() + && opts.namespace_lock_fence.is_none() + && (opts.bucket_lifecycle_lock_fence.is_none() || opts.bucket_lifecycle_lock_owner.is_some()); + let commit_set = self.clone(); + let commit_bucket = bucket.to_owned(); + let commit_object = object.to_owned(); + let commit_upload_id = upload_id.to_owned(); + let commit_upload_id_path = upload_id_path.clone(); + let commit_tmp_part = tmp_part.clone(); + let commit_opts = opts.clone(); + commit_owns_cleanup = true; + // The body is already staged. Keep every commit guard through the + // prepare, publish, settlement/rollback and temporary cleanup I/O, + // even when the HTTP request stops awaiting this task. + let commit = async move { + let _upload_commit_guard = _upload_commit_guard; + let _part_commit_guard = _part_commit_guard; + let _capped_staging_guard = _capped_staging_guard; + let bucket = commit_bucket.as_str(); + let object = commit_object.as_str(); + let upload_id = commit_upload_id.as_str(); + let upload_id_path = commit_upload_id_path; + let opts = &commit_opts; + let result: Result = async { + let (commit_fi, _) = commit_set + .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts) + .await?; + let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?; + if commit_size_limit != multipart_size_limit { + return Err(Error::InvalidArgument( + "multipart upload".to_string(), + SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(), + "size limit metadata changed or is missing".to_string(), + )); + } + ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?; + ensure_multipart_bucket_incarnation( + &commit_set.ctx, + &commit_fi, bucket, object, upload_id, - part_number: Some(part_id), - }), - ) + opts.expected_bucket_incarnation_id, + ) + .await?; + #[cfg(test)] + pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost).await; + fence_commit_on_lock_loss(_upload_commit_guard.as_ref(), "put_object_part_commit", &upload_id_path)?; + fence_commit_on_lock_loss(_part_commit_guard.as_ref(), "put_object_part_commit", &part_lock_path)?; + if opts + .namespace_lock_fence + .as_ref() + .is_some_and(NamespaceLockFence::is_lock_lost) + { + return Err(StorageError::NamespaceLockQuorumUnavailable { + mode: "put_object_part_outer_lock", + bucket: bucket.to_string(), + object: object.to_string(), + required: 1, + achieved: 0, + }); + } + ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?; + + if let Some(limit) = commit_size_limit { + let current_size = commit_set + .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id) + .await?; + let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?; + admitted_multipart_size(current_size, candidate_size, limit)?; + } + + let decommission_capacity_guard = if let Some(store) = opts.decommission_capacity_admission.as_ref() { + Some( + store + .acquire_external_decommission_capacity_fence( + &[commit_set.pool_index], + DecommissionCapacityAdmission::ExistingMultipart, + ) + .await?, + ) + } else { + None + }; + #[cfg(test)] + pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterCapacityAdmission).await; + if decommission_capacity_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) { + return Err(StorageError::NamespaceLockQuorumUnavailable { + mode: "put_object_part_decommission_capacity", + bucket: bucket.to_string(), + object: object.to_string(), + required: 1, + achieved: 0, + }); + } + let rename_result = commit_set + .rename_part( + &shuffle_disks, + RUSTFS_META_TMP_BUCKET, + &tmp_part_path, + RUSTFS_META_MULTIPART_BUCKET, + &part_path, + part_info_buff.into(), + write_quorum, + Some(MultipartWriteQuorumContext { + stage: MULTIPART_WRITE_QUORUM_RENAME_PART, + bucket, + object, + upload_id, + part_number: Some(part_id), + }), + ) + .await; + drop(decommission_capacity_guard); + let _ = rename_result?; + #[cfg(test)] + observe_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost); + + #[cfg(test)] + pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterRename).await; + + let ret: PartInfo = PartInfo { + etag: Some(etag.clone()), + part_num: part_id, + last_mod: Some(OffsetDateTime::now_utc()), + size: w_size, + actual_size, + }; + + // error!("put_object_part ret {:?}", &ret); + + Ok(ret) + } .await; - drop(decommission_capacity_guard); - let _ = rename_result?; - #[cfg(test)] - observe_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockLost); - - #[cfg(test)] - pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartAfterRename).await; - drop(_part_commit_guard); - drop(_upload_commit_guard); - - let ret: PartInfo = PartInfo { - etag: Some(etag.clone()), - part_num: part_id, - last_mod: Some(OffsetDateTime::now_utc()), - size: w_size, - actual_size, + if result.is_err() + && let Err(err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_part).await + { + warn!( + event = EVENT_SET_DISK_MULTIPART, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + result = "cleanup_failed", + bucket, + object, + error = ?err, + "failed to clean multipart commit staging" + ); + } + result }; - - // error!("put_object_part ret {:?}", &ret); - - Ok(ret) + if detach_commit_owner { + tokio::spawn(commit) + .await + .map_err(|err| Error::other_with_context("put_object_part commit task failed", err))? + } else { + commit.await + } } .await; - if result.is_err() + if !commit_owns_cleanup + && result.is_err() && let Err(err) = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_part).await { warn!(tmp_part = %tmp_part, error = ?err, "failed to cleanup multipart temporary data"); @@ -3385,7 +3626,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { #[cfg(any(test, feature = "test-util"))] pause_multipart_commit(&commit_bucket, &commit_object, MultipartCommitPause::BeforeQuotaRename).await; if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _decommission_object_lock_guard @@ -3440,7 +3680,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { verify_object_transaction_epoch_fence(&commit_set, &commit_bucket, &commit_object, expected).await?; } if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _decommission_object_lock_guard @@ -4233,6 +4472,300 @@ mod tests { } } + #[test] + fn orphan_multipart_identity_requires_canonical_upload_id() { + let created = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(10); + let id = Uuid::new_v4(); + let timestamp = (created + time::Duration::seconds(1)).unix_timestamp_nanos(); + let encode = |raw: &str| runtime_sources::deployment_upload_id(raw); + let upload_id = encode(&format!("{id}x{timestamp}")); + assert!(orphan_multipart_upload_id_is_canonical(&upload_id)); + for raw in [ + format!("{}x{timestamp}", Uuid::nil()), + format!("{}x{timestamp}", id.simple()), + format!("{id}x0{timestamp}"), + format!("{id}x+{timestamp}"), + format!("{id}x{timestamp}/../another-upload"), + format!("../{id}x{timestamp}"), + format!("{id}x{}", i128::MAX), + ] { + assert!( + !orphan_multipart_upload_id_is_canonical(&encode(&raw)), + "noncanonical upload identity must be rejected: {raw}" + ); + } + assert!(!orphan_multipart_upload_id_is_canonical("not-base64")); + } + + #[tokio::test] + async fn orphan_multipart_cleanup_requires_every_disk_to_prove_metadata_absence() { + let (_temp_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "orphan-multipart-proof"; + let object = "object"; + make_bucket_on_all(&disks, bucket).await; + let upload = set + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("create upload"); + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id); + let metadata_path = format!("{upload_path}/{STORAGE_FORMAT_FILE}"); + let leftover_path = format!("{upload_path}/part.late"); + for disk in &disks { + disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path, Bytes::from_static(b"late part")) + .await + .expect("write late part fixture"); + } + for disk in &disks[..3] { + disk.delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default()) + .await + .expect("remove majority metadata"); + } + let opts = ObjectOptions::default(); + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .is_err(), + "one surviving metadata copy must block orphan recovery" + ); + disks[3] + .write_all(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Bytes::from_static(b"corrupt metadata")) + .await + .expect("corrupt final metadata copy"); + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .is_err(), + "corrupt metadata must not be treated as absence" + ); + disks[3] + .delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default()) + .await + .expect("remove final metadata copy"); + set.disks.write().await[3] = None; + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .is_err(), + "an unreachable disk must not be hidden by absence quorum" + ); + set.disks.write().await[3] = Some(disks[3].clone()); + for disk in &disks { + assert_eq!( + disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path) + .await + .expect("failed preflight must preserve every part"), + Bytes::from_static(b"late part") + ); + } + + let guard = set + .lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .expect("all disks now prove metadata absence"); + assert!(guard.delete(bucket, object, &opts).await.expect("recover exact staging")); + drop(guard); + for disk in &disks { + assert!(matches!( + disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await, + Err(DiskError::FileNotFound | DiskError::VolumeNotFound) + )); + } + let guard = set + .lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &opts, &[]) + .await + .expect("already absent staging remains provably absent"); + assert!( + !guard + .delete(bucket, object, &opts) + .await + .expect("absence does not report recovery") + ); + } + + #[tokio::test] + async fn orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk() { + let (temp_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "orphan-multipart-directory"; + let object = "object"; + make_bucket_on_all(&disks, bucket).await; + let upload = set + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("create upload"); + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id); + let metadata_path = format!("{upload_path}/{STORAGE_FORMAT_FILE}"); + let leftover_path = format!("{upload_path}/part.late"); + for disk in &disks { + disk.delete(RUSTFS_META_MULTIPART_BUCKET, &metadata_path, Default::default()) + .await + .expect("remove upload metadata"); + disk.write_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path, Bytes::from_static(b"late part")) + .await + .expect("write leftover part"); + } + let metadata_directory = temp_dirs[3].path().join(RUSTFS_META_MULTIPART_BUCKET).join(&metadata_path); + tokio::fs::create_dir_all(&metadata_directory) + .await + .expect("replace one missing metadata file with a directory"); + let (_, errors) = + SetDisks::read_all_raw_file_info(&set.disks.read().await, RUSTFS_META_MULTIPART_BUCKET, &upload_path, false).await; + assert!( + errors.iter().all(|error| matches!(error, Some(DiskError::FileNotFound))), + "raw metadata reads alone must reproduce the directory-as-absence case: {errors:?}" + ); + assert!( + set.lock_orphan_multipart_cleanup(bucket, object, &upload.upload_id, &ObjectOptions::default(), &[]) + .await + .is_err(), + "one physical metadata directory must block cleanup on every disk" + ); + assert!(metadata_directory.is_dir(), "failed preflight must preserve the metadata directory"); + for disk in &disks { + assert_eq!( + disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &leftover_path) + .await + .expect("failed preflight must preserve every leftover part"), + Bytes::from_static(b"late part") + ); + } + } + + #[derive(Clone, Copy, PartialEq, Eq)] + enum OrphanMultipartLockDomainCase { + Shared, + Independent, + PartiallyOverlapping, + } + + async fn check_orphan_multipart_distributed_lock_owners(case: OrphanMultipartLockDomainCase) { + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + ctx.update_erasure_type(SetupType::DistErasure).await; + let first_client: Arc = + Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new()))); + let second_client: Arc = + Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new()))); + let second_clients = match case { + OrphanMultipartLockDomainCase::Shared => vec![first_client.clone()], + OrphanMultipartLockDomainCase::Independent => vec![second_client], + OrphanMultipartLockDomainCase::PartiallyOverlapping => vec![first_client.clone(), second_client], + }; + let (first_dirs, first_disks, first_set) = + hermetic_set_disks_with_lockers_and_ctx(4, 0, 2, vec![first_client], ctx.clone()).await; + let (second_dirs, second_disks, second_set) = hermetic_set_disks_with_lockers_and_ctx(4, 1, 2, second_clients, ctx).await; + let bucket = "orphan-multipart-dist-locks"; + let object = "object"; + make_bucket_on_all(&first_disks, bucket).await; + make_bucket_on_all(&second_disks, bucket).await; + let upload_id = runtime_sources::deployment_upload_id(&format!( + "{}x{}", + Uuid::new_v4(), + OffsetDateTime::now_utc().unix_timestamp_nanos() + )); + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id); + for directory in first_dirs.iter().chain(second_dirs.iter()) { + let staging = directory.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path); + tokio::fs::create_dir_all(&staging) + .await + .expect("create exact orphan staging"); + tokio::fs::write(staging.join("part.late"), b"late part") + .await + .expect("write orphan part"); + } + let opts = ObjectOptions::default(); + let first_guard = first_set + .lock_orphan_multipart_cleanup(bucket, object, &upload_id, &opts, &[]) + .await + .expect("first domain has an orphan"); + let second_result = tokio::time::timeout( + Duration::from_secs(10), + second_set.lock_orphan_multipart_cleanup(bucket, object, &upload_id, &opts, std::slice::from_ref(&first_guard)), + ) + .await + .expect("lock-domain admission must not wait on a guard it already owns"); + if case == OrphanMultipartLockDomainCase::PartiallyOverlapping { + let error = match second_result { + Ok(_) => panic!("one existing quorum cannot authorize a partially overlapping domain"), + Err(error) => error, + }; + assert!(error.to_string().contains("partially overlapping lock domains")); + for directory in first_dirs.iter().chain(second_dirs.iter()) { + assert!( + directory + .path() + .join(RUSTFS_META_MULTIPART_BUCKET) + .join(&upload_path) + .join("part.late") + .exists(), + "uncertain lock coverage must preserve every physical part" + ); + } + return; + } + let second_guard = second_result.expect("the second pool has a proven lock domain"); + assert_eq!( + Arc::ptr_eq(&first_guard.lock_guard, &second_guard.lock_guard), + case == OrphanMultipartLockDomainCase::Shared, + "only the same exact backend may share a real upload guard across pool labels" + ); + let second_namespace = second_set + .new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_path) + .await + .unwrap(); + let mut second_writer = Box::pin(second_namespace.get_write_lock(Duration::from_secs(10))); + assert!(futures::poll!(&mut second_writer).is_pending()); + if case == OrphanMultipartLockDomainCase::Shared { + drop(first_guard); + assert!( + futures::poll!(&mut second_writer).is_pending(), + "the second pool must retain the shared real owner after the first guard drops" + ); + drop(second_guard); + drop( + tokio::time::timeout(Duration::from_secs(10), second_writer) + .await + .expect("the shared backend should release after the last owner") + .expect("shared-domain writer resumes"), + ); + return; + } + let first_namespace = first_set + .new_ns_lock(RUSTFS_META_MULTIPART_BUCKET, &upload_path) + .await + .unwrap(); + let mut first_writer = Box::pin(first_namespace.get_write_lock(Duration::from_secs(10))); + assert!(futures::poll!(&mut first_writer).is_pending()); + drop(first_guard); + drop( + tokio::time::timeout(Duration::from_secs(10), first_writer) + .await + .expect("first domain should release independently") + .expect("first writer resumes"), + ); + assert!(futures::poll!(&mut second_writer).is_pending(), "the second real owner must remain held"); + drop(second_guard); + drop( + tokio::time::timeout(Duration::from_secs(10), second_writer) + .await + .expect("second domain should release with its owner") + .expect("second writer resumes"), + ); + } + + #[tokio::test] + async fn orphan_multipart_cleanup_keeps_separate_distributed_lock_owners() { + check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::Independent).await; + } + + #[tokio::test] + async fn orphan_multipart_cleanup_reuses_shared_distributed_owner_across_pools() { + check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::Shared).await; + } + + #[tokio::test] + async fn orphan_multipart_cleanup_rejects_partially_overlapping_distributed_domains() { + check_orphan_multipart_distributed_lock_owners(OrphanMultipartLockDomainCase::PartiallyOverlapping).await; + } + async fn stage_upload_with_create_opts( set_disks: &Arc, bucket: &str, @@ -6370,6 +6903,76 @@ mod tests { .expect("abort should delete the upload after UploadPart releases the lock"); } + #[tokio::test(flavor = "multi_thread")] + #[serial] + async fn cancelled_put_object_part_keeps_upload_lock_through_commit() { + for pause in [ + MultipartCommitPause::PutPartAfterCapacityAdmission, + MultipartCommitPause::PutPartAfterRename, + ] { + let manager = Arc::new(rustfs_lock::GlobalLockManager::new()); + let signaling = Arc::new(SignalingLockClient::new(Arc::new(LocalClient::with_manager(manager)))); + let lockers: Vec> = vec![signaling.clone()]; + let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks_with_lockers(4, 0, 2, lockers).await; + let bucket = "multipart-cancelled-part-commit-bucket"; + let object = "object"; + make_bucket_on_all(&disk_stores, bucket).await; + let upload = set_disks + .new_multipart_upload(bucket, object, &ObjectOptions::default()) + .await + .expect("multipart upload should be created"); + let upload_id = upload.upload_id; + let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id); + signaling.set_target(rustfs_lock::ObjectKey::new(RUSTFS_META_MULTIPART_BUCKET, upload_path.clone())); + let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await; + let barrier = MultipartCommitBarrier::install(bucket, object, pause); + + let put_store = set_disks.clone(); + let put_upload_id = upload_id.clone(); + let put = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(vec![0x45; 4096]); + put_store + .put_object_part(bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + }); + barrier.wait_until_paused().await; + put.abort(); + assert!( + put.await + .expect_err("the request waiter should remain cancellable") + .is_cancelled() + ); + + let abort_store = set_disks.clone(); + let abort_upload_id = upload_id.clone(); + let mut abort = tokio::spawn(async move { + abort_store + .abort_multipart_upload(bucket, object, &abort_upload_id, &ObjectOptions::default()) + .await + }); + signaling.wait_for_attempts(2).await; + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut abort).await.is_err(), + "Abort must not acquire the upload lock while a cancelled request's commit is paused at {pause:?}" + ); + barrier.release(); + tokio::time::timeout(Duration::from_secs(10), abort) + .await + .expect("Abort should finish after the commit drains") + .expect("Abort task should not panic") + .expect("Abort should delete the committed part staging"); + for disk in &disk_stores { + assert!( + matches!( + disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, -1).await, + Err(DiskError::FileNotFound) + ), + "the upload directory must be absent after the commit and Abort finish" + ); + } + } + } + #[tokio::test(flavor = "multi_thread")] #[serial] async fn put_object_part_different_part_numbers_commit_concurrently() { @@ -9799,6 +10402,18 @@ mod tests { "the post-commit crash must leave the upload listable for reclamation" ); + let cleanup = set_disks + .lock_orphan_multipart_cleanup(bucket, object, &u_new, &ObjectOptions::default(), &[]) + .await + .expect("the committed upload has no staging metadata on any disk"); + assert!( + cleanup + .delete(bucket, object, &ObjectOptions::default()) + .await + .expect("recover completed upload staging") + ); + drop(cleanup); + // The commit rename already consumed the upload's xl.meta, so the // retry cannot read staging. It must still return the committed // object: AWS CompleteMultipartUpload is idempotent for the same @@ -9817,21 +10432,6 @@ mod tests { !upload_is_listed(&set_disks, bucket, object, &u_new).await, "the idempotent retry must reclaim staging left behind by the crash" ); - - // Reclaim the leftover exactly as the production tail does: delete_all - // on the upload path (abort_multipart_upload cannot — the upload's - // metadata is gone, so it too answers InvalidUploadID). - let upload_dir = SetDisks::get_upload_id_dir(bucket, object, &u_new); - set_disks - .delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_dir) - .await - .expect("sweeping the leftover upload staging should succeed"); - assert!( - !upload_is_listed(&set_disks, bucket, object, &u_new).await, - "reclaiming the upload must drop it from the listing" - ); - let (body_after, _) = read_object(&set_disks, bucket, object).await; - assert_eq!(body_after, new, "reclaiming the leftover upload must not disturb the committed object"); } #[tokio::test] @@ -9878,6 +10478,18 @@ mod tests { .expect("the failed post-commit completion should release its object guard"), ); + let cleanup = set_disks + .lock_orphan_multipart_cleanup(bucket, object, &u_new, &ObjectOptions::default(), &[]) + .await + .expect("committed staging has no upload metadata"); + assert!( + cleanup + .delete(bucket, object, &ObjectOptions::default()) + .await + .expect("orphan recovery must only remove upload staging") + ); + drop(cleanup); + let (body, _) = read_object(&set_disks, bucket, object).await; assert_eq!(body, new, "the committed replacement must remain readable after the crash"); let mut receipts = 0; diff --git a/crates/ecstore/src/set_disk/ops/object.rs b/crates/ecstore/src/set_disk/ops/object.rs index 6f5ae956e..ff6e5ebae 100644 --- a/crates/ecstore/src/set_disk/ops/object.rs +++ b/crates/ecstore/src/set_disk/ops/object.rs @@ -20,6 +20,7 @@ //! SetDisks core (io_primitives) via inherent calls. use crate::core::pools::DecommissionCapacityAdmission; +use crate::disk::cleanup_runtime::{self, OLD_DATA_CLEANUP_RECEIPT_FILE}; use rustfs_common::mrf_channel::MrfDeleteMarkerPurge; use rustfs_filemeta::metadata_keys; @@ -36,9 +37,9 @@ use super::super::{ GET_STAGE_LOCK_ACQUIRE, GET_STAGE_METADATA, GET_STAGE_OBJECT_INFO, GET_STAGE_PATH_DECISION, GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetCodecStreamingFallbackReason, GetDirectMemoryDecision, GetObjectReader, HTTPRangeSpec, HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext, Instant, - LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey, - ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin, - PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision, + LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, LateMetadataFanout, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, + ObjectKey, ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, + Ordering, Pin, PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision, ReplicationObjectBridge, Result, SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks, SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, adaptive_duplex_buffer_size, @@ -93,6 +94,392 @@ impl<'a> DeleteCommitContext<'a> { } } +const TIER_FREE_VERSION_SOURCE_LOOKUP_COUNTERFACTUAL_SKIP_ENV: &str = + "RUSTFS_PUT_TIER_FREE_VERSION_SOURCE_LOOKUP_COUNTERFACTUAL_SKIP"; +const TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV: &str = "RUSTFS_PUT_TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA"; +const TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ASSUME_HOMOGENEOUS_ENV: &str = + "RUSTFS_PUT_TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ASSUME_HOMOGENEOUS"; +const RENAME_TAIL_CLEANUP_PER_DISK_LIMIT_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_PER_DISK_LIMIT"; +const RENAME_TAIL_CLEANUP_PER_DISK_LANES_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_PER_DISK_LANES"; +const RENAME_TAIL_CLEANUP_LANE_ENABLE_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ENABLE"; +const RENAME_TAIL_CLEANUP_LANE_CAPACITY_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_CAPACITY"; +const RENAME_TAIL_CLEANUP_LANE_BATCH_MAX_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_BATCH_MAX"; +const RENAME_TAIL_CLEANUP_LANE_BATCH_WINDOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_BATCH_WINDOW_MS"; +const RENAME_TAIL_CLEANUP_LANE_ACTIVE_WINDOW_MS_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ACTIVE_WINDOW_MS"; +const RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP"; +const DEFAULT_RENAME_TAIL_CLEANUP_PER_DISK_LANES: usize = 1024; +const DEFAULT_RENAME_TAIL_CLEANUP_LANE_CAPACITY: usize = 1024; +const DEFAULT_RENAME_TAIL_CLEANUP_LANE_BATCH_MAX: usize = 1; + +fn tier_free_version_source_lookup_counterfactual_skip_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(TIER_FREE_VERSION_SOURCE_LOOKUP_COUNTERFACTUAL_SKIP_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_zero_target_tmp_delete_skip_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn tier_free_version_source_reuse_rename_data_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn tier_free_version_source_reuse_rename_data_assume_homogeneous_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ASSUME_HOMOGENEOUS_ENV) + .ok() + .as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_per_disk_limit() -> usize { + static LIMIT: std::sync::OnceLock = std::sync::OnceLock::new(); + *LIMIT.get_or_init(|| rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_PER_DISK_LIMIT_ENV, 0)) +} + +fn rename_tail_cleanup_per_disk_lanes() -> usize { + static LANES: std::sync::OnceLock = std::sync::OnceLock::new(); + *LANES.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_PER_DISK_LANES_ENV, DEFAULT_RENAME_TAIL_CLEANUP_PER_DISK_LANES).max(1) + }) +} + +fn rename_tail_cleanup_foreground_yield() -> Duration { + static YIELD: std::sync::OnceLock = std::sync::OnceLock::new(); + *YIELD + .get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64("RUSTFS_PUT_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD_MS", 0))) +} + +fn rename_tail_cleanup_foreground_active_yield() -> Duration { + static YIELD: std::sync::OnceLock = std::sync::OnceLock::new(); + *YIELD.get_or_init(|| { + Duration::from_millis(rustfs_utils::get_env_u64("RUSTFS_PUT_RENAME_TAIL_CLEANUP_FOREGROUND_ACTIVE_YIELD_MS", 0)) + }) +} + +fn rename_tail_cleanup_lane_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| { + matches!( + std::env::var(RENAME_TAIL_CLEANUP_LANE_ENABLE_ENV).ok().as_deref(), + Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON") + ) + }) +} + +fn rename_tail_cleanup_lane_capacity() -> usize { + static CAPACITY: std::sync::OnceLock = std::sync::OnceLock::new(); + *CAPACITY.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_LANE_CAPACITY_ENV, DEFAULT_RENAME_TAIL_CLEANUP_LANE_CAPACITY).max(1) + }) +} + +fn rename_tail_cleanup_lane_batch_max() -> usize { + static BATCH_MAX: std::sync::OnceLock = std::sync::OnceLock::new(); + *BATCH_MAX.get_or_init(|| { + rustfs_utils::get_env_usize(RENAME_TAIL_CLEANUP_LANE_BATCH_MAX_ENV, DEFAULT_RENAME_TAIL_CLEANUP_LANE_BATCH_MAX).max(1) + }) +} + +fn rename_tail_cleanup_lane_batch_window() -> Duration { + static WINDOW: std::sync::OnceLock = std::sync::OnceLock::new(); + *WINDOW.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_LANE_BATCH_WINDOW_MS_ENV, 0))) +} + +fn rename_tail_cleanup_lane_active_window() -> Duration { + static WINDOW: std::sync::OnceLock = std::sync::OnceLock::new(); + *WINDOW.get_or_init(|| Duration::from_millis(rustfs_utils::get_env_u64(RENAME_TAIL_CLEANUP_LANE_ACTIVE_WINDOW_MS_ENV, 0))) +} + +async fn rename_tail_cleanup_per_disk_permit(disk_index: usize) -> Option { + let limit = rename_tail_cleanup_per_disk_limit(); + if limit == 0 { + return None; + } + static LANES: std::sync::OnceLock>> = std::sync::OnceLock::new(); + let lanes = LANES.get_or_init(|| { + (0..rename_tail_cleanup_per_disk_lanes()) + .map(|_| Arc::new(tokio::sync::Semaphore::new(limit))) + .collect() + }); + let semaphore = Arc::clone(&lanes[disk_index % lanes.len()]); + let wait_started = rustfs_io_metrics::put_stage_timer(); + let permit = semaphore.clone().acquire_owned().await.ok(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_PERMIT_WAIT, + wait_started, + ); + if let Some(semaphore_permit) = permit { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_INFLIGHT, + limit.saturating_sub(semaphore.available_permits()) as f64, + ); + Some(semaphore_permit) + } else { + None + } +} + +type RenameTailCleanupLaneFuture = Pin + Send + 'static>>; + +struct RenameTailCleanupLaneJob { + disk_index: usize, + future: RenameTailCleanupLaneFuture, + done: tokio::sync::oneshot::Sender<()>, +} + +struct RenameTailCleanupLane { + sender: tokio::sync::mpsc::Sender, + queued: Arc, + oldest_enqueued_at: Arc>>, +} + +struct RenameTailCleanupLaneQueue { + lanes: Vec, +} + +async fn run_rename_tail_cleanup_lane_job(job: RenameTailCleanupLaneJob) { + let service_started = rustfs_io_metrics::put_stage_timer(); + job.future.await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_SERVICE_TIME, + service_started, + ); + let _ = job.done.send(()); +} + +impl RenameTailCleanupLaneQueue { + fn new() -> Self { + let lanes = (0..rename_tail_cleanup_per_disk_lanes()) + .map(|_| { + let (sender, mut receiver) = + tokio::sync::mpsc::channel::(rename_tail_cleanup_lane_capacity()); + let queued = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let active_workers = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let oldest_enqueued_at = Arc::new(tokio::sync::Mutex::new(None::)); + let worker_queued = Arc::clone(&queued); + let worker_active = Arc::clone(&active_workers); + let worker_oldest_enqueued_at = Arc::clone(&oldest_enqueued_at); + tokio::spawn(async move { + loop { + let queued_before_poll = worker_queued.load(Ordering::Relaxed); + let worker_poll_started = rustfs_io_metrics::put_stage_timer(); + let Some(job) = receiver.recv().await else { + return; + }; + if queued_before_poll == 0 { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_IDLE_WAIT, + worker_poll_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_POLL_WAIT, + worker_poll_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_JOB_WAIT, + worker_poll_started, + ); + } + let mut batch = vec![job]; + let batch_window = rename_tail_cleanup_lane_batch_window(); + let batch_max = rename_tail_cleanup_lane_batch_max(); + if batch_max > 1 && !batch_window.is_zero() { + tokio::time::sleep(batch_window).await; + } + while batch.len() < batch_max { + let Some(next) = receiver.try_recv().ok() else { + break; + }; + batch.push(next); + } + let queued_after_dequeue = worker_queued + .fetch_sub(batch.len(), Ordering::Relaxed) + .saturating_sub(batch.len()); + if queued_after_dequeue == 0 { + *worker_oldest_enqueued_at.lock().await = None; + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH, + queued_after_dequeue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_dequeue as f64, + ); + if let Some(oldest) = *worker_oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BATCH_SIZE, + batch.len() as f64, + ); + if batch.len() > 1 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BURST_BATCH_SIZE, + batch.len() as f64, + ); + } + let active = worker_active.fetch_add(1, Ordering::Relaxed) + 1; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_ACTIVE, + active as f64, + ); + let active_window = rename_tail_cleanup_lane_active_window(); + if !active_window.is_zero() { + let mut deferred = Vec::new(); + let mut ready = Vec::new(); + for job in batch { + let active = rename_foreground_active_count(job.disk_index); + if active > 0 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_FOREGROUND_ACTIVE_HIT, + active as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_REQUEUE, + 1.0, + ); + deferred.push(job); + } else { + ready.push(job); + } + } + for job in ready { + run_rename_tail_cleanup_lane_job(job).await; + } + if !deferred.is_empty() { + tokio::time::sleep(active_window).await; + for job in deferred { + run_rename_tail_cleanup_lane_job(job).await; + } + } + } else { + for job in batch { + run_rename_tail_cleanup_lane_job(job).await; + } + } + worker_active.fetch_sub(1, Ordering::Relaxed); + } + }); + RenameTailCleanupLane { + sender, + queued, + oldest_enqueued_at, + } + }) + .collect(); + Self { lanes } + } + + async fn enqueue( + &self, + disk_index: usize, + future: RenameTailCleanupLaneFuture, + ) -> Option> { + let lane = &self.lanes[disk_index % self.lanes.len()]; + let enqueued_at = Instant::now(); + let queue_push_started = rustfs_io_metrics::put_stage_timer(); + { + let mut oldest = lane.oldest_enqueued_at.lock().await; + if oldest.is_none() { + *oldest = Some(enqueued_at); + } + } + let queued_after_enqueue = lane.queued.fetch_add(1, Ordering::Relaxed) + 1; + let (done, done_rx) = tokio::sync::oneshot::channel(); + let job = RenameTailCleanupLaneJob { + disk_index, + future, + done, + }; + match lane.sender.try_send(job) { + Ok(()) => { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH, + queued_after_enqueue as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + queued_after_enqueue as f64, + ); + if let Some(oldest) = *lane.oldest_enqueued_at.lock().await { + let oldest_age_ms = oldest.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE, + oldest_age_ms, + ); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE, + oldest_age_ms, + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH, + queue_push_started, + ); + Some(done_rx) + } + Err(err) => { + lane.queued.fetch_sub(1, Ordering::Relaxed); + if lane.queued.load(Ordering::Relaxed) == 0 { + *lane.oldest_enqueued_at.lock().await = None; + } + let job = err.into_inner(); + run_rename_tail_cleanup_lane_job(job).await; + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + lane.queued.load(Ordering::Relaxed) as f64, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH, + queue_push_started, + ); + None + } + } + } +} + +fn rename_tail_cleanup_lane_queue() -> &'static RenameTailCleanupLaneQueue { + static QUEUE: std::sync::OnceLock = std::sync::OnceLock::new(); + QUEUE.get_or_init(RenameTailCleanupLaneQueue::new) +} + fn delete_marker_purge_candidate( file_info: &FileInfo, expected_bucket_incarnation_id: Option, @@ -358,7 +745,9 @@ use crate::object_api::{ }; use crate::services::notification_sys::RemoteVersionStateFleetProofToken; use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease, tier_destination_id_from_metadata}; -use crate::set_disk::core::io_primitives::{RenameRollbackReceipt, RenameTailCleanup, finish_rename_tail_heal}; +use crate::set_disk::core::io_primitives::{ + RenameRollbackReceipt, RenameTailCleanup, finish_rename_tail_heal, rename_foreground_active_count, +}; #[cfg(test)] use crate::storage_api_contracts::namespace::NamespaceLocking; #[cfg(test)] @@ -375,6 +764,8 @@ use rustfs_filemeta::ObjectPartInfo; use rustfs_utils::path::decode_dir_object; #[cfg(test)] use rustfs_utils::path::encode_dir_object; +use rustfs_utils::path::object_name_disk_identity; +use std::borrow::Cow; use std::future::Future; use std::task::{Context, Poll}; use std::time::Duration; @@ -528,7 +919,6 @@ async fn acquire_single_tier_delete_lease(opts: &ObjectOptions, source: &ObjectI Ok(Some(lease)) } -const OLD_DATA_CLEANUP_RECEIPT_FILE: &str = ".rustfs-old-data-cleanup-receipt.json"; const SCANNER_PUBLICATION_LEASE_FENCE_MAX_BYTES: usize = 64 * 1024; const SCANNER_PUBLICATION_LEASE_FENCE_MAX_ENTRIES: usize = 256; @@ -1009,6 +1399,42 @@ mod tier_free_version_receipt_eligibility_tests { } } +// Own only completed staging, before the commit owner can rename or roll back +// shards. Borrow the request state; clone it only if cancellation needs cleanup. +struct PutObjectStagingCleanup<'a> { + set: &'a SetDisks, + tmp_dir: &'a str, + reservation: &'a Option, + runtime: Option, +} + +impl Drop for PutObjectStagingCleanup<'_> { + fn drop(&mut self) { + let Some(runtime) = self.runtime.take() else { + return; + }; + let set = self.set.clone(); + let tmp_dir = self.tmp_dir.to_owned(); + let reservation = self.reservation.clone(); + // A request future may be dropped outside its polling runtime. Cleanup + // remains best-effort on runtime shutdown, like the existing PUT tail. + let _enter = runtime.enter(); + cleanup_runtime::spawn(reservation, async move { + if let Err(err) = set.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await { + warn!( + event = EVENT_SET_DISK_WRITE, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + state = "cancelled_staging_cleanup_failed", + tmp_dir = %tmp_dir, + error = ?err, + "Cancelled PUT staging cleanup failed" + ); + } + }); + } +} + struct PutObjectCommitCancellation { token: CancellationToken, armed: bool, @@ -2809,9 +3235,9 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { let object_size = usize::try_from(object_info.size) .map_err(|_| to_object_err(Error::other("two-phase GET object size is invalid"), vec![bucket, object]))?; let mut output = Vec::with_capacity(object_size); - let (fi, files, disks, late_metadata_fanout_disks) = snapshot.into_owned_with_late_metadata_fanout(); + let (fi, files, disks, late_metadata_fanout) = snapshot.into_owned_with_late_metadata_fanout(); let expected_identity = super::super::read::LateMetadataIdentity::from_file_info(&fi); - let late_metadata_fanout_disks = late_metadata_fanout_disks.ok_or_else(|| { + let late_metadata_fanout = late_metadata_fanout.ok_or_else(|| { to_object_err(Error::other("two-phase GET fallback context is missing"), vec![bucket, object]) })?; let initial_result = Self::get_object_with_fileinfo( @@ -2836,6 +3262,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks { ) .await; if prepare_late_materialized_retry(&initial_result, &mut output, object_size) { + let late_metadata_fanout_disks = match late_metadata_fanout { + LateMetadataFanout::CapturedDisks(disks) => disks, + LateMetadataFanout::RefreshCurrentSet => self.get_disks_for_data_read().await, + }; #[cfg(test)] super::super::rename_fanout_barrier::checkpoint( object, @@ -3355,7 +3785,26 @@ pub(in crate::set_disk) fn merge_replication_metadata_lww( changed } +struct CommittedPutOutcome { + // Namespace publication succeeded even if its writer must retain repair + // responsibility. Completion errors must not select rollback cleanup. + completion: Result<(ObjectInfo, Option)>, +} + impl SetDisks { + /// Failure cleanup stays awaited, but shares the same execution budget as + /// successful PUT continuations when isolation is enabled. + async fn delete_put_tmp_with_budget(&self, tmp_dir: &str, reservation: &Option) -> Result<()> { + if reservation.is_none() { + return self.delete_all(RUSTFS_META_TMP_BUCKET, tmp_dir).await; + } + let set = self.clone(); + let tmp_dir = tmp_dir.to_owned(); + cleanup_runtime::spawn(reservation.clone(), async move { set.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await }) + .await + .map_err(Error::other)? + } + async fn reclaim_missing_delete_residue( &self, bucket: &str, @@ -3434,26 +3883,107 @@ impl SetDisks { committed_data_dir: Option, epoch: Option, ) { + if rename_tail_cleanup_lane_enabled() { + let bucket = bucket.to_string(); + let object = object.to_string(); + let mut completions = Vec::with_capacity(targets.len()); + for target in targets { + let cleanup_set = self.clone(); + let cleanup_bucket = bucket.clone(); + let cleanup_object = object.clone(); + let disk_index = target.disk_index; + let future = Box::pin(async move { + cleanup_set + .cleanup_rename_tail_target(target, &cleanup_bucket, &cleanup_object, committed_data_dir, epoch) + .await; + }); + if let Some(done) = rename_tail_cleanup_lane_queue().enqueue(disk_index, future).await { + completions.push(done); + } + } + for done in completions { + let _ = done.await; + } + return; + } + for target in targets { - let mut disks = vec![None; self.set_drive_count]; - disks[target.disk_index] = Some(target.disk); - self.persist_old_data_cleanup_receipts(&disks, bucket, object, target.old_data_dir, committed_data_dir, epoch) - .await; - let cleanup = self - .commit_rename_data_dir_and_mark_capacity( - &disks, - bucket, - object, - &target.old_data_dir.to_string(), - &committed_data_dir.unwrap_or_default().to_string(), - 1, - ) - .await; - self.report_old_data_dir_cleanup(bucket, object, &target.old_data_dir.to_string(), &cleanup) + self.cleanup_rename_tail_target(target, bucket, object, committed_data_dir, epoch) .await; } } + async fn cleanup_rename_tail_target( + &self, + target: RenameTailCleanup, + bucket: &str, + object: &str, + committed_data_dir: Option, + epoch: Option, + ) { + let foreground_yield = rename_tail_cleanup_foreground_yield(); + if !foreground_yield.is_zero() { + let yield_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(foreground_yield).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD, + yield_started, + ); + } + let active_yield = rename_tail_cleanup_foreground_active_yield(); + if !active_yield.is_zero() { + let active = rename_foreground_active_count(target.disk_index); + if active > 0 { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFERRED_FOREGROUND_ACTIVE, + active as f64, + ); + let yield_started = rustfs_io_metrics::put_stage_timer(); + tokio::time::sleep(active_yield).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD, + yield_started, + ); + } + } + let _cleanup_per_disk_permit = rename_tail_cleanup_per_disk_permit(target.disk_index).await; + let mut disks = vec![None; self.set_drive_count]; + disks[target.disk_index] = Some(target.disk); + let receipt_started = rustfs_io_metrics::put_stage_timer(); + self.persist_old_data_cleanup_receipts(&disks, bucket, object, target.old_data_dir, committed_data_dir, epoch) + .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_RECEIPT, + receipt_started, + ); + let commit_capacity_started = rustfs_io_metrics::put_stage_timer(); + let cleanup = self + .commit_rename_data_dir_and_mark_capacity( + &disks, + bucket, + object, + &target.old_data_dir.to_string(), + &committed_data_dir.unwrap_or_default().to_string(), + 1, + ) + .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COMMIT_CAPACITY, + commit_capacity_started, + ); + let report_started = rustfs_io_metrics::put_stage_timer(); + self.report_old_data_dir_cleanup(bucket, object, &target.old_data_dir.to_string(), &cleanup) + .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_REPORT, + report_started, + ); + } + + fn needs_old_data_cleanup_receipt(old_data_dir: Uuid, committed_data_dir: Option, epoch: Option) -> bool { + epoch.is_some() && committed_data_dir != Some(old_data_dir) + } + pub(in crate::set_disk) async fn persist_old_data_cleanup_receipts( &self, disks: &[Option], @@ -3463,10 +3993,10 @@ impl SetDisks { committed_data_dir: Option, epoch: Option, ) { - let Some(epoch) = epoch else { return }; - if committed_data_dir == Some(old_data_dir) { + if !Self::needs_old_data_cleanup_receipt(old_data_dir, committed_data_dir, epoch) { return; } + let Some(epoch) = epoch else { return }; let receipt = OldDataCleanupReceipt::new(epoch, old_data_dir, committed_data_dir); let Ok(encoded) = receipt.encode() else { return; @@ -3652,6 +4182,23 @@ impl SetDisks { .scanner_publication_commit_scope .clone() .map(ScannerPublicationCommitScopeGuard::new); + // Reserve before staging. Some callers already hold external locks, + // so admission must not wait for another PUT to release its budget. + // Commit/cleanup owners retain clones across cancellation and early ACK. + if opts + .put_object_cancellation + .as_ref() + .is_some_and(CancellationToken::is_cancelled) + { + return Err(StorageError::OperationCanceled); + } + let cleanup_reservation = cleanup_runtime::reserve().map_err(|err| { + if err.kind() == std::io::ErrorKind::WouldBlock { + StorageError::SlowDown + } else { + Error::other(err) + } + })?; let storage_class_config = self.storage_class_config_snapshot(); self.invalidate_get_object_metadata_cache(bucket, object).await; @@ -3770,8 +4317,20 @@ impl SetDisks { (None, None, None) }; let mut tmp_cleanup_owned = false; + let mut staging_cleanup = PutObjectStagingCleanup { + set: self, + tmp_dir: &tmp_dir, + reservation: &cleanup_reservation, + runtime: None, + }; let rollback_receipt = RenameRollbackReceipt::default(); + let operation_first_poll_delay_started = rustfs_io_metrics::put_stage_timer(); let operation = async { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY, + operation_first_poll_delay_started, + ); + let operation_body_to_commit_submit_started = rustfs_io_metrics::put_stage_timer(); let erasure = Arc::new(erasure_from_file_info(&fi, false)?); let put_object_size = known_put_object_storage_size(data.size()); @@ -4128,6 +4687,12 @@ impl SetDisks { drop(writers); // drop writers to close all files, this is to prevent FileAccessDenied errors when renaming data + // File creation, writes and integrity sidecars must finish before + // cancellation can remove staging without racing pending writers. + if !direct_inline_commit { + staging_cleanup.runtime = Some(tokio::runtime::Handle::current()); + } + if parity_blocks == 0 { let written_size = i64::try_from(w_size).map_err(|_| Error::other("put_object written size overflows i64"))?; let logical_shard_size = usize::try_from(erasure.shard_file_size(written_size)) @@ -4209,6 +4774,8 @@ impl SetDisks { } #[cfg(any(test, feature = "test-util"))] pause_put_object_commit(bucket, object, PutObjectCommitPause::AfterNamespace).await; + let namespace_lock_acquired_to_commit_ready_started = rustfs_io_metrics::put_stage_timer(); + let commit_ready_precondition_and_timestamp_started = rustfs_io_metrics::put_stage_timer(); if opts.http_preconditions.is_some() && let Some(err) = self.check_write_precondition(bucket, object, opts).await @@ -4229,8 +4796,13 @@ impl SetDisks { } } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_PRECONDITION_AND_TIMESTAMP, + commit_ready_precondition_and_timestamp_started, + ); if let Some(expected) = opts.expected_current_version_id.as_deref() { + let expected_version_lookup_started = rustfs_io_metrics::put_stage_timer(); let current = self .get_object_info( bucket, @@ -4253,11 +4825,16 @@ impl SetDisks { if current.version_id.map(|version| version.to_string()).as_deref() != Some(expected) { return Err(StorageError::PreconditionFailed); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPECTED_VERSION_LOOKUP, + expected_version_lookup_started, + ); } if let Some(version_id) = opts.version_id.as_deref() && !is_meta_bucketname(bucket) { + let explicit_version_lookup_started = rustfs_io_metrics::put_stage_timer(); let current = self .get_object_info( bucket, @@ -4310,13 +4887,23 @@ impl SetDisks { Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {} Err(err) => return Err(err), } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPLICIT_VERSION_LOOKUP, + explicit_version_lookup_started, + ); } + let restore_verify_started = rustfs_io_metrics::put_stage_timer(); self.require_current_restore_operation_id(bucket, object, opts, expected_restore_operation_id, "put_object_commit") .await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_RESTORE_VERIFY, + restore_verify_started, + ); // Fence every commit-time read before entering rename_data. Once // rename_data returns Ok the write is durable and must not be aborted. + let fence_verify_started = rustfs_io_metrics::put_stage_timer(); if object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || publication_commit_guard .as_ref() @@ -4360,6 +4947,7 @@ impl SetDisks { }); } + let decommission_capacity_fence_started = rustfs_io_metrics::put_stage_timer(); if decommission_capacity_guard.is_none() && let Some(store) = opts.decommission_capacity_admission.as_ref() { @@ -4369,6 +4957,10 @@ impl SetDisks { .await?, ); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_DECOMMISSION_CAPACITY_FENCE, + decommission_capacity_fence_started, + ); // The object namespace is acquired above, after the input stream // has been fully staged. Only then admit the local publication @@ -4386,21 +4978,36 @@ impl SetDisks { achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_FENCE_VERIFY, + fence_verify_started, + ); let transaction_fencing_proof = object_transaction_fencing_fleet_proof(); if object_transaction_fencing_requested() && transaction_fencing_proof.is_none() { return Err(Error::other("object transaction fencing requires a live fleet capability proof")); } + let transaction_epoch_fence_read_started = rustfs_io_metrics::put_stage_timer(); let transaction_epoch_fence = if transaction_fencing_proof.is_some() { Some(read_object_transaction_epoch_fence(self, bucket, object).await?) } else { None }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_TRANSACTION_EPOCH_FENCE_READ, + transaction_epoch_fence_read_started, + ); + let quota_begin_started = rustfs_io_metrics::put_stage_timer(); let quota_context = reservation::begin(&self.ctx, bucket, object, opts, self.pool_index, self.set_index).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_BEGIN, + quota_begin_started, + ); let quota_mutation_fence = quota_context.is_enforced() || opts.quota_admission.is_some(); let mut replication_quota_size = None; + let metadata_adjust_started = rustfs_io_metrics::put_stage_timer(); if opts.replication_request { if quota_context.is_enforced() && opts.preserve_ciphertext { return Err(Error::PartMissingOrCorrupt); @@ -4464,7 +5071,12 @@ impl SetDisks { } } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_METADATA_ADJUST, + metadata_adjust_started, + ); + let quota_reserve_started = rustfs_io_metrics::put_stage_timer(); let (quota_old_size, quota_new_size) = if quota_context.is_enforced() { let new_size = match replication_quota_size { Some(size) => size, @@ -4482,8 +5094,13 @@ impl SetDisks { (0, 0) }; let quota_reservation = quota_context.reserve(quota_old_size, quota_new_size).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_RESERVE, + quota_reserve_started, + ); let (commit_disks, quota_fence_tokens) = if quota_mutation_fence { - match Self::prepare_quota_mutation_fences(&shuffle_disks, bucket, object, write_quorum).await { + let quota_mutation_fence_prepare_started = rustfs_io_metrics::put_stage_timer(); + let prepared = match Self::prepare_quota_mutation_fences(&shuffle_disks, bucket, object, write_quorum).await { Ok((disks, tokens)) => { for (metadata, token) in parts_metadatas.iter_mut().zip(tokens.iter().copied()) { if let Some(token) = token { @@ -4500,10 +5117,16 @@ impl SetDisks { quota_reservation.abort().await; return Err(err); } - } + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_MUTATION_FENCE_PREPARE, + quota_mutation_fence_prepare_started, + ); + prepared } else { (shuffle_disks.clone(), vec![None; shuffle_disks.len()]) }; + let commit_context_prepare_started = rustfs_io_metrics::put_stage_timer(); let transaction_epoch = transaction_epoch_fence.map(|_| assign_object_transaction_epoch(&commit_disks, &mut parts_metadatas)); @@ -4526,7 +5149,8 @@ impl SetDisks { || commit_object_lock_guard.is_some() || commit_decommission_object_lock_guard.is_some() || commit_publication_guard.is_some(); - let commit_allows_early_ack = write_context.allows_early_ack(opts.write_completion) + let commit_write_completion = opts.write_completion; + let commit_allows_early_ack = write_context.allows_early_ack(commit_write_completion) && !(opts.data_movement && opts.has_decommission_capacity_reservation()) && commit_owns_namespace_guard && commit_scanner_publication_scope.is_none(); @@ -4550,10 +5174,29 @@ impl SetDisks { let commit_tier_free_version_receipt_sink = opts.tier_free_version_receipt_sink.clone(); let commit_skip_free_version = opts.skip_free_version; let request_cancellation = operation_cancellation.clone(); + let commit_cleanup_reservation = cleanup_reservation.clone(); tmp_cleanup_owned = true; + // No await separates this handoff from starting the commit owner. + // Its rollback rules, not request Drop, now decide what to retain. + staging_cleanup.runtime = None; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_READY_CONTEXT_PREPARE, + commit_context_prepare_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY, + namespace_lock_acquired_to_commit_ready_started, + ); - let commit = move |cancellation: Option| async move { + let commit = move |commit_submit_to_closure_enter_started: Option, + cancellation: Option| async move { let write_context = write_context; + let commit_closure_total_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER, + commit_submit_to_closure_enter_started, + ); + let commit_closure_enter_to_first_await_started = rustfs_io_metrics::put_stage_timer(); let mut _object_lock_guard = commit_object_lock_guard; let mut _decommission_object_lock_guard = commit_decommission_object_lock_guard; let mut _publication_guard = commit_publication_guard; @@ -4561,14 +5204,21 @@ impl SetDisks { let mut _decommission_capacity_guard = commit_decommission_capacity_guard; let mut quota_reservation = quota_reservation; let rename_stage_start = Instant::now(); + let commit_before_rename_started = rustfs_io_metrics::put_stage_timer(); + let lock_held_pre_rename_prepare_started = rustfs_io_metrics::put_stage_timer(); let pre_rename = async { #[cfg(any(test, feature = "test-util"))] pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::AfterQuotaReservation).await; + let quota_mark_commit_started = rustfs_io_metrics::put_stage_timer(); quota_reservation.mark_commit_started().await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_QUOTA_MARK_COMMIT_STARTED, + quota_mark_commit_started, + ); #[cfg(any(test, feature = "test-util"))] pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::BeforeQuotaRename).await; + let first_fence_check_started = rustfs_io_metrics::put_stage_timer(); if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _decommission_object_lock_guard @@ -4596,6 +5246,10 @@ impl SetDisks { achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_FIRST_FENCE_CHECK, + first_fence_check_started, + ); let restore_opts = ObjectOptions { version_id: commit_version_id.clone(), versioned: commit_versioned, @@ -4603,6 +5257,7 @@ impl SetDisks { no_lock: true, ..Default::default() }; + let restore_verify_started = rustfs_io_metrics::put_stage_timer(); commit_set .require_current_restore_operation_id( &commit_bucket, @@ -4612,6 +5267,10 @@ impl SetDisks { "put_object_quota_reservation", ) .await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_RESTORE_VERIFY, + restore_verify_started, + ); if let Some(proof) = transaction_fencing_proof.as_ref() && !object_transaction_fencing_fleet_proof_matches(proof) { @@ -4625,10 +5284,15 @@ impl SetDisks { PutObjectCommitPause::BeforeTransactionEpochVerify, ) .await; + let transaction_fence_verify_started = rustfs_io_metrics::put_stage_timer(); verify_object_transaction_epoch_fence(&commit_set, &commit_bucket, &commit_object, expected).await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TRANSACTION_FENCE_VERIFY, + transaction_fence_verify_started, + ); } + let second_fence_check_started = rustfs_io_metrics::put_stage_timer(); if quota_reservation.is_lock_lost() - || !quota_reservation.capability_proof_matches() || write_context.is_lock_lost() || _object_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) || _decommission_object_lock_guard @@ -4656,11 +5320,20 @@ impl SetDisks { achieved: 0, }); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SECOND_FENCE_CHECK, + second_fence_check_started, + ); Ok(()) }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT, + commit_closure_enter_to_first_await_started, + ); + let pre_rename_await_total_started = rustfs_io_metrics::put_stage_timer(); let mut pre_rename_result = if cancellation.is_some() || request_cancellation.is_some() { tokio::select! { - biased; + biased; _ = wait_for_put_object_commit_cancellation(cancellation.as_ref(), request_cancellation.as_ref()) => { Err(StorageError::OperationCanceled) }, @@ -4669,6 +5342,12 @@ impl SetDisks { } else { pre_rename.await }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_AWAIT_TOTAL, + pre_rename_await_total_started, + ); + let pre_rename_post_await_to_rename_started = rustfs_io_metrics::put_stage_timer(); + let scanner_scope_check_started = rustfs_io_metrics::put_stage_timer(); if pre_rename_result.is_ok() && let Some(scope) = commit_scanner_publication_scope.as_ref() && let Err(err) = scope.try_begin() @@ -4683,6 +5362,10 @@ impl SetDisks { let _ = scope.mark_indeterminate(); pre_rename_result = Err(StorageError::OperationCanceled); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SCANNER_SCOPE_CHECK, + scanner_scope_check_started, + ); if let Err(err) = pre_rename_result { SetDisks::abort_quota_reservation_after_fence( quota_reservation, @@ -4694,7 +5377,10 @@ impl SetDisks { quota_mutation_fence, ) .await; - if let Err(cleanup_err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_dir).await { + if let Err(cleanup_err) = commit_set + .delete_put_tmp_with_budget(&commit_tmp_dir, &commit_cleanup_reservation) + .await + { warn!(tmp_dir = %commit_tmp_dir, error = ?cleanup_err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { warn!( @@ -4707,43 +5393,76 @@ impl SetDisks { } return Err(err); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_PRE_RENAME_PREPARE, + lock_held_pre_rename_prepare_started, + ); - let put_tier_free_version_source = - if commit_put_tier_free_version_id.is_some() && commit_tier_free_version_receipt_sink.is_some() { - match commit_set - .get_object_info( - &commit_bucket, - &commit_object, - &ObjectOptions { - no_lock: true, - metadata_cache_safe: false, - versioned: commit_versioned, - version_suspended: commit_version_suspended, - ..Default::default() - }, - ) - .await - { - Ok(source) => Some(source), - Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None, - Err(err) => { - debug!( - event = EVENT_LIFECYCLE_TRANSITIONED_DELETE_CLEANUP_OWNER, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_SET_DISK, - bucket = %commit_bucket, - object = %commit_object, - error = ?err, - "Skipped opportunistic tier free-version receipt source capture" - ); - None - } + let tier_free_version_source_lookup_started = rustfs_io_metrics::put_stage_timer(); + let reuse_rename_data_source_for_tier_free_version = tier_free_version_source_reuse_rename_data_enabled() + && (commit_set.tier_free_version_rename_data_source_capable() + || tier_free_version_source_reuse_rename_data_assume_homogeneous_enabled()); + let mut put_tier_free_version_source = if commit_put_tier_free_version_id.is_some() + && commit_tier_free_version_receipt_sink.is_some() + && !tier_free_version_source_lookup_counterfactual_skip_enabled() + && !reuse_rename_data_source_for_tier_free_version + { + #[cfg(feature = "diagnostic-read-version-caller")] + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::TierFreeVersionSourceLookup, + ); + match commit_set + .get_object_info( + &commit_bucket, + &commit_object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + versioned: commit_versioned, + version_suspended: commit_version_suspended, + ..Default::default() + }, + ) + .await + { + Ok(source) => Some(source), + Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None, + Err(err) => { + debug!( + event = EVENT_LIFECYCLE_TRANSITIONED_DELETE_CLEANUP_OWNER, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + bucket = %commit_bucket, + object = %commit_object, + error = ?err, + "Skipped opportunistic tier free-version receipt source capture" + ); + None } - } else { - None - }; + } + } else { + None + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TIER_FREE_VERSION_SOURCE_LOOKUP, + tier_free_version_source_lookup_started, + ); + let assign_rename_indexes_started = rustfs_io_metrics::put_stage_timer(); Self::assign_rename_data_indexes(&mut parts_metadatas); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_ASSIGN_RENAME_INDEXES, + assign_rename_indexes_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_POST_AWAIT_TO_RENAME, + pre_rename_post_await_to_rename_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME, + commit_before_rename_started, + ); + let lock_held_rename_started = rustfs_io_metrics::put_stage_timer(); let mut rename_result = SetDisks::rename_data_owned_with_fence( &commit_disks, (RUSTFS_META_TMP_BUCKET, commit_tmp_dir.as_str()), @@ -4761,6 +5480,12 @@ impl SetDisks { ), ) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME, + lock_held_rename_started, + ); + let commit_after_rename_to_return_started = rustfs_io_metrics::put_stage_timer(); + let lock_held_rename_result_to_tail_handoff_started = rustfs_io_metrics::put_stage_timer(); if let Some(scope) = commit_scanner_publication_scope.as_ref() { if rename_result.is_ok() { let _ = scope.mark_committed(); @@ -4775,7 +5500,15 @@ impl SetDisks { let mut rename_guard_release = None; let mut needs_immediate_heal = false; let mut tail_owns_tmp_cleanup = false; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME_RESULT_TO_TAIL_HANDOFF, + lock_held_rename_result_to_tail_handoff_started, + ); + let lock_held_tail_handoff_started = rustfs_io_metrics::put_stage_timer(); if let Ok(rename_commit) = rename_result.as_mut() { + commit_set.observe_tier_free_version_rename_data_source_capability( + rename_commit.tail_drain.is_none() && rename_commit.old_current_source_all_successful_checked, + ); commit_set.record_capacity_scope_if_needed(commit_capacity_scope_token, &rename_commit.capacity_disks); // Install the tail watcher before any post-commit await. The // latch keeps namespace guards through their prior handoff point. @@ -4810,6 +5543,7 @@ impl SetDisks { let fence_object = commit_object.clone(); let (guard_release_tx, guard_release_rx) = tokio::sync::oneshot::channel(); rename_guard_release = Some(guard_release_tx); + let cleanup_reservation = commit_cleanup_reservation.clone(); tokio::spawn(finish_rename_tail_heal( rename_tail_drain, guard_release_rx, @@ -4842,36 +5576,72 @@ impl SetDisks { decommission_object_lock_guard, decommission_capacity_guard, ), - targets| async move { + targets| { drop(object_lock_guard); drop(write_context); drop(publication_guard); drop(bucket_lifecycle_guard); - cleanup_set - .cleanup_rename_tail( - targets, - &cleanup_bucket, - &cleanup_object, - committed_data_dir, - transaction_epoch, - ) - .await; - if let Err(err) = cleanup_set.delete_all(RUSTFS_META_TMP_BUCKET, &cleanup_tmp_dir).await { - warn!(tmp_dir = %cleanup_tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); - } else if issue3031_diag_enabled() { - warn!( - target: "rustfs_ecstore::set_disk", - tmp_dir = %cleanup_tmp_dir, - "issue3031_put_object_tmp_cleanup_done" + cleanup_runtime::run(cleanup_reservation, async move { + let zero_target_cleanup = targets.is_empty(); + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COUNT, + targets.len() as f64, ); - } - drop(decommission_object_lock_guard); - drop(decommission_capacity_guard); + if zero_target_cleanup { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_ZERO_TARGET, + 1.0, + ); + } + cleanup_set + .cleanup_rename_tail( + targets, + &cleanup_bucket, + &cleanup_object, + committed_data_dir, + transaction_epoch, + ) + .await; + let tmp_delete_started = rustfs_io_metrics::put_stage_timer(); + if zero_target_cleanup && rename_tail_cleanup_zero_target_tmp_delete_skip_enabled() { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP, + 1.0, + ); + } else if let Err(err) = + cleanup_set.delete_all(RUSTFS_META_TMP_BUCKET, &cleanup_tmp_dir).await + { + warn!(tmp_dir = %cleanup_tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); + } else if issue3031_diag_enabled() { + warn!( + target: "rustfs_ecstore::set_disk", + tmp_dir = %cleanup_tmp_dir, + "issue3031_put_object_tmp_cleanup_done" + ); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE, + tmp_delete_started, + ); + let guard_drop_started = rustfs_io_metrics::put_stage_timer(); + drop(decommission_object_lock_guard); + drop(decommission_capacity_guard); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DECOMMISSION_GUARD_DROP, + guard_drop_started, + ); + }) + }, + move |request| async move { + let _ = heal_set.submit_rename_tail_heal(request, commit_write_completion).await; }, - |request| async move { heal_set.submit_rename_tail_heal(request).await }, )); } } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_TAIL_HANDOFF, + lock_held_tail_handoff_started, + ); if !tail_owns_tmp_cleanup { drop(_decommission_capacity_guard.take()); } @@ -4880,6 +5650,11 @@ impl SetDisks { pause_put_object_commit(&commit_bucket, &commit_object, PutObjectCommitPause::AfterRenameHandoff).await; } if quota_mutation_fence && !tail_owns_tmp_cleanup { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_TRIGGERED, + 1.0, + ); + let quota_fence_release_started = rustfs_io_metrics::put_stage_timer(); let _ = SetDisks::release_quota_mutation_fences( &commit_disks, "a_fence_tokens, @@ -4888,9 +5663,23 @@ impl SetDisks { write_quorum, ) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE, + quota_fence_release_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_SKIPPED, + 1.0, + ); } if rename_result.is_ok() { + let quota_commit_started = rustfs_io_metrics::put_stage_timer(); quota_reservation.commit().await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_COMMIT, + quota_commit_started, + ); } let rename_commit = match rename_result { Ok(commit) => commit, @@ -4900,7 +5689,10 @@ impl SetDisks { // rollback backup for recovery; cleanup is unsafe. return Err(err.into()); } - if let Err(cleanup_err) = commit_set.delete_all(RUSTFS_META_TMP_BUCKET, &commit_tmp_dir).await { + if let Err(cleanup_err) = commit_set + .delete_put_tmp_with_budget(&commit_tmp_dir, &commit_cleanup_reservation) + .await + { warn!(tmp_dir = %commit_tmp_dir, error = ?cleanup_err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { warn!( @@ -4918,7 +5710,17 @@ impl SetDisks { let op_old_dir = rename_commit.data_dir; let cleanup_disks = rename_commit.cleanup_disks; let old_current_size = rename_commit.old_current_size; + if put_tier_free_version_source.is_none() + && reuse_rename_data_source_for_tier_free_version + && rename_commit.old_current_source_checked + { + put_tier_free_version_source = rename_commit + .old_current_source + .as_ref() + .map(|source| build_get_object_info(source, &commit_bucket, &commit_object, commit_versioned)); + } let mut fi = rename_commit.committed_file_info; + let lock_held_post_rename_started = rustfs_io_metrics::put_stage_timer(); if let (Some(source), Some(free_version_id)) = (put_tier_free_version_source.as_ref(), commit_put_tier_free_version_id) && transitioned_delete_publishes_free_version(source, &fi, commit_skip_free_version) @@ -4933,7 +5735,12 @@ impl SetDisks { ); } - if needs_immediate_heal { + let repair_result = if needs_immediate_heal { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_TRIGGERED, + 1.0, + ); + let heal_submit_started = rustfs_io_metrics::put_stage_timer(); let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options( commit_bucket.clone(), Some(commit_object.clone()), @@ -4946,29 +5753,86 @@ impl SetDisks { .or_else(|| commit_version_suspended.then(Uuid::nil)) .map(|version_id| version_id.to_string()); request.expected_bucket_incarnation_id = source_bucket_incarnation_id; - commit_set.submit_rename_tail_heal(request).await; - } + let result = commit_set.submit_rename_tail_heal(request, commit_write_completion).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT, + heal_submit_started, + ); + result + } else { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_SKIPPED, + 1.0, + ); + Ok(()) + }; let rename_stage_elapsed = rename_stage_start.elapsed(); let rename_stage_ms = rename_stage_elapsed.as_millis() as u64; - if let Some(old_dir) = op_old_dir { - commit_set - .persist_old_data_cleanup_receipts( - &cleanup_disks, - &commit_bucket, - &commit_object, - old_dir, - committed_data_dir, - transaction_epoch, - ) + if let Some(old_dir) = op_old_dir + && Self::needs_old_data_cleanup_receipt(old_dir, committed_data_dir, transaction_epoch) + { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_TRIGGERED, + 1.0, + ); + let cleanup_receipt_started = rustfs_io_metrics::put_stage_timer(); + if commit_cleanup_reservation.is_none() { + commit_set + .persist_old_data_cleanup_receipts( + &cleanup_disks, + &commit_bucket, + &commit_object, + old_dir, + committed_data_dir, + transaction_epoch, + ) + .await; + } else { + let cleanup_set = commit_set.clone(); + let receipt_disks = cleanup_disks.clone(); + let receipt_bucket = commit_bucket.clone(); + let receipt_object = commit_object.clone(); + cleanup_runtime::run(commit_cleanup_reservation.clone(), async move { + cleanup_set + .persist_old_data_cleanup_receipts( + &receipt_disks, + &receipt_bucket, + &receipt_object, + old_dir, + committed_data_dir, + transaction_epoch, + ) + .await; + }) .await; + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT, + cleanup_receipt_started, + ); + } else { + rustfs_io_metrics::record_put_object_stage_duration( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_SKIPPED, + 1.0, + ); } + let metadata_invalidate_started = rustfs_io_metrics::put_stage_timer(); commit_set .invalidate_get_object_metadata_cache(&commit_bucket, &commit_object) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_METADATA_INVALIDATE, + metadata_invalidate_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_RENAME, + lock_held_post_rename_started, + ); + let lock_held_guard_release_started = rustfs_io_metrics::put_stage_timer(); if let Some(release) = rename_guard_release.take() { let _ = release.send(true); } @@ -4977,6 +5841,10 @@ impl SetDisks { drop(_object_lock_guard.take()); drop(_publication_guard.take()); drop(_bucket_lifecycle_guard.take()); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_GUARD_RELEASE, + lock_held_guard_release_started, + ); rustfs_io_metrics::record_put_object_stage_duration("set_disk_rename", duration_millis_f64(rename_stage_elapsed)); if (rename_stage_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS { @@ -5004,16 +5872,41 @@ impl SetDisks { // here must not negate an already-committed, durable write, so we // deliberately do NOT `?`-propagate it into a 503. On residue the // report path emits the leak metric and enqueues a heal. - let cleanup = commit_set - .commit_rename_data_dir_and_mark_capacity( - &cleanup_disks, - &commit_bucket, - &commit_object, - &old_dir.to_string(), - &committed_dir, - write_quorum, - ) + if commit_cleanup_reservation.is_none() { + let cleanup = commit_set + .commit_rename_data_dir_and_mark_capacity( + &cleanup_disks, + &commit_bucket, + &commit_object, + &old_dir.to_string(), + &committed_dir, + write_quorum, + ) + .await; + commit_set + .report_old_data_dir_cleanup(&commit_bucket, &commit_object, &old_dir.to_string(), &cleanup) + .await; + } else { + let cleanup_set = commit_set.clone(); + let cleanup_bucket = commit_bucket.clone(); + let cleanup_object = commit_object.clone(); + cleanup_runtime::run(commit_cleanup_reservation.clone(), async move { + let cleanup = cleanup_set + .commit_rename_data_dir_and_mark_capacity( + &cleanup_disks, + &cleanup_bucket, + &cleanup_object, + &old_dir.to_string(), + &committed_dir, + write_quorum, + ) + .await; + cleanup_set + .report_old_data_dir_cleanup(&cleanup_bucket, &cleanup_object, &old_dir.to_string(), &cleanup) + .await; + }) .await; + } let cleanup_elapsed = cleanup_stage_start.elapsed(); let cleanup_ms = cleanup_elapsed.as_millis() as u64; cleanup_stage_ms = Some(cleanup_ms); @@ -5021,9 +5914,6 @@ impl SetDisks { "set_disk_old_data_cleanup", duration_millis_f64(cleanup_elapsed), ); - commit_set - .report_old_data_dir_cleanup(&commit_bucket, &commit_object, &old_dir.to_string(), &cleanup) - .await; if (cleanup_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS { warn!( event = EVENT_SET_DISK_COMMIT_TAIL_SLOW, @@ -5085,6 +5975,7 @@ impl SetDisks { event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, component = LOG_COMPONENT_ECSTORE, subsystem = LOG_SUBSYSTEM_SET_DISK, + result = if repair_result.is_ok() { "success" } else { "repair_required" }, bucket = %commit_bucket, object = %commit_object, write_quorum, @@ -5095,7 +5986,6 @@ impl SetDisks { cleanup_ms = cleanup_stage_ms.unwrap_or_default(), cleanup_present = cleanup_stage_ms.is_some(), commit_tail_ms = total_commit_tail_ms as u64, - result = "success", "SetDisk put_object stage summary" ); } @@ -5103,7 +5993,7 @@ impl SetDisks { if !tail_owns_tmp_cleanup { let cleanup_set = commit_set.clone(); let cleanup_tmp_dir = commit_tmp_dir.clone(); - tokio::spawn(async move { + cleanup_runtime::spawn(commit_cleanup_reservation.clone(), async move { if let Err(err) = cleanup_set.delete_all(RUSTFS_META_TMP_BUCKET, &cleanup_tmp_dir).await { warn!(tmp_dir = %cleanup_tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { @@ -5116,15 +6006,38 @@ impl SetDisks { }); } - Ok(( - ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned), - old_current_size, - )) + let completion = repair_result.map(|()| { + let object_info_build_started = rustfs_io_metrics::put_stage_timer(); + let object_info = ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_OBJECT_INFO_BUILD, + object_info_build_started, + ); + let post_object_info_to_return_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_OBJECT_INFO_TO_RETURN, + post_object_info_to_return_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN, + commit_after_rename_to_return_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL, + commit_closure_total_started, + ); + (object_info, old_current_size) + }); + Ok(CommittedPutOutcome { completion }) }; if let Some(handoff) = commit_cancellation_handoff_tx.take() { let _ = handoff.send(()); } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT, + operation_body_to_commit_submit_started, + ); if detach_commit_owner { if let Some(scope_outcome_guard) = scope_outcome_guard.as_mut() { // The spawned commit closure owns the scope clone and is @@ -5133,17 +6046,53 @@ impl SetDisks { } let mut cancellation = PutObjectCommitCancellation::new(); let child_token = cancellation.child_token(); - let result = tokio::spawn(async move { Box::pin(commit(Some(child_token))).await }) - .await - .map_err(|err| Error::other(format!("put_object commit task failed: {err}")))?; + let commit_task_wait_started = rustfs_io_metrics::put_stage_timer(); + let commit_submit_to_closure_enter_started = rustfs_io_metrics::put_stage_timer(); + #[cfg(feature = "diagnostic-read-version-caller")] + let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::PutCommitOwner, + ); + let result = tokio::spawn(async move { + let commit = Box::pin(commit(commit_submit_to_closure_enter_started, Some(child_token))); + #[cfg(feature = "diagnostic-read-version-caller")] + return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::PutCommitOwner, + commit, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + commit.await + }) + .await + .map_err(|err| Error::other(format!("put_object commit task failed: {err}")))?; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT, + commit_task_wait_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL, + commit_task_wait_started, + ); cancellation.disarm(); result } else { - Box::pin(commit(None)).await + let commit_task_wait_started = rustfs_io_metrics::put_stage_timer(); + let commit_submit_to_closure_enter_started = rustfs_io_metrics::put_stage_timer(); + let result = Box::pin(commit(commit_submit_to_closure_enter_started, None)).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT, + commit_task_wait_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL, + commit_task_wait_started, + ); + result } }; - let result: Result<(ObjectInfo, Option)> = if let Some((cancellation, mut handoff_rx)) = cancellation_wait - { + let operation_select_wait_started = rustfs_io_metrics::put_stage_timer(); + let result: Result = if let Some((cancellation, mut handoff_rx)) = cancellation_wait { tokio::pin!(operation); tokio::select! { biased; @@ -5154,27 +6103,22 @@ impl SetDisks { } else { operation.await }; - - if issue3031_diag_enabled() - && let Err(err) = &result - { - let stage_hint = if err.to_string().contains("not enough disks to write") { - "writer_setup_or_quorum" - } else { - "unknown" - }; - warn!( - event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, - component = LOG_COMPONENT_ECSTORE, - subsystem = LOG_SUBSYSTEM_SET_DISK, - bucket = %bucket, - object = %object, - result = "error", - stage_hint, - error = %err, - "SetDisk put_object stage summary" - ); - } + // Normal completion keeps the existing cleanup/error semantics below. + staging_cleanup.runtime = None; + drop(staging_cleanup); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT, + operation_select_wait_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL, + operation_select_wait_started, + ); + let operation_result_return_started = rustfs_io_metrics::put_stage_timer(); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN, + operation_result_return_started, + ); if issue3031_diag_enabled() { warn!( @@ -5201,7 +6145,7 @@ impl SetDisks { // entry is reclaimed by cleanup_stale_tmp_objects (24h expiry, // 5-minute background loop). let set_disks = self.clone(); - tokio::spawn(async move { + cleanup_runtime::spawn(cleanup_reservation.clone(), async move { if let Err(err) = set_disks.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await { warn!(tmp_dir = %tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { @@ -5216,7 +6160,7 @@ impl SetDisks { // Failure path (quorum loss / rollback): keep the cleanup inline so // a failed PUT never returns while its tmp shards are still on disk // (state-residue hardening tracked by backlog#864 / backlog#898). - if let Err(err) = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_dir).await { + if let Err(err) = self.delete_put_tmp_with_budget(&tmp_dir, &cleanup_reservation).await { warn!(tmp_dir = %tmp_dir, error = ?err, "failed to cleanup put_object temporary data"); } else if issue3031_diag_enabled() { warn!( @@ -5229,6 +6173,29 @@ impl SetDisks { } } + let result = result.and_then(|committed| committed.completion); + + if issue3031_diag_enabled() + && let Err(err) = &result + { + let stage_hint = if err.to_string().contains("not enough disks to write") { + "writer_setup_or_quorum" + } else { + "unknown" + }; + warn!( + event = EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_SET_DISK, + result = "error", + bucket = %bucket, + object = %object, + stage_hint, + error = %err, + "SetDisk put_object stage summary" + ); + } + result } } @@ -7840,6 +8807,21 @@ impl SetDisks { // quorum-minus-one delete into an apparent success. let write_quorum = self.set_drive_count / 2 + 1; let rollback_dir = Uuid::new_v4(); + // Register before preparation: cancellation and response loss can + // leave a durable journal even when publication never returns. + let recovery_ticket = (!is_meta_bucketname(bucket)) + .then(|| { + self.ctx + .delete_recovery_hints() + .register(crate::store::delete_recovery_runtime::DeleteRecoveryKey { + pool: self.pool_index, + set: self.set_index, + bucket: bucket.to_owned(), + object: object.to_owned(), + incarnation: source_bucket_incarnation_id, + }) + }) + .flatten(); let mut futures = Vec::with_capacity(disks.len()); let mut errs = Vec::with_capacity(disks.len()); @@ -7910,6 +8892,7 @@ impl SetDisks { &disks, write_quorum, DeleteRecoveryContext { + recovery_ticket, namespace_owner: namespace_owner.clone().map(|owner| owner as Arc), ..DeleteRecoveryContext::new(source_bucket_incarnation_id, &|| commit_context.can_commit(bucket, object)) }, @@ -8070,6 +9053,41 @@ impl SetDisks { } } +impl SetDisks { + /// Read current metadata without cache admission or read-side repair. The + /// outer batch owner already holds the sorted namespace write-lock set. + pub(crate) async fn conditional_delete_current_snapshot( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + ) -> Result<(ObjectInfo, crate::disk::CurrentDeleteGeneration)> { + let snapshot = self.get_object_fileinfo(bucket, object, opts, false, false).await?; + let file_info = snapshot.fi(); + let generation = crate::disk::CurrentDeleteGeneration::from_file_info(file_info) + .map_err(|error| to_object_err(error.into(), vec![bucket, object]))?; + Ok(( + ObjectInfo::from_file_info(file_info, bucket, object, opts.versioned || opts.version_suspended), + generation, + )) + } + + /// Mixed-version peers must reject the operation before any group mutates. + pub(crate) async fn preflight_guarded_delete_versions(&self) -> Result<()> { + let disks = self.disks.read().await.clone(); + guarded_delete_capability_barrier(disks.iter().flatten().map(|disk| disk.supports_guarded_delete_versions())).await + } +} + +async fn guarded_delete_capability_barrier(probes: impl IntoIterator) -> Result<()> +where + F: std::future::Future>, +{ + // Capability probes are read-only, so a failure can cancel the remaining + // probes. Admission still requires every probe to complete successfully. + futures::future::try_join_all(probes).await.map(|_| ()).map_err(Error::from) +} + #[async_trait::async_trait] impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { type Error = Error; @@ -8397,6 +9415,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { objects: Vec, opts: ObjectOptions, ) -> (Vec, Vec>, Vec>) { + let current_generations = opts.conditional_delete_generations.as_deref(); let mut del_objects = vec![DeletedObject::default(); objects.len()]; let mut accounting = vec![None; objects.len()]; let delete_config_snapshot = opts @@ -8408,14 +9427,31 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { self.invalidate_get_object_metadata_cache(bucket, &object.object_name).await; } - let mut del_errs = (0..objects.len()).map(|_| None).collect::>(); + let mut del_errs = objects + .iter() + .map(|object| { + object.delete_condition.as_ref().map(|_| { + StorageError::InvalidArgument( + bucket.to_owned(), + object.object_name.clone(), + "conditional delete was not admitted by the store preflight".to_owned(), + ) + }) + }) + .collect::>(); // Acquire locks in batch mode (best effort, matching previous behavior) let mut batch = rustfs_lock::BatchLockRequest::new(self.locker_owner.as_str()).with_all_or_nothing(false); + // Keyed by the same disk identity `new_ns_lock` uses, so aliases such + // as `a\b` and `a/b` on Windows share one lock (and one entry here). + let lock_names: Vec> = objects + .iter() + .map(|dobj| object_name_disk_identity(&dobj.object_name)) + .collect(); let mut unique_objects: HashSet = HashSet::new(); - for dobj in &objects { - if unique_objects.insert(dobj.object_name.clone()) { - batch = batch.add_write_lock(ObjectKey::new(bucket, dobj.object_name.clone())); + for lock_name in &lock_names { + if unique_objects.insert(lock_name.to_string()) { + batch = batch.add_write_lock(ObjectKey::new(bucket, lock_name.to_string())); } } let unique_lock_count = batch.requests.len(); @@ -8468,8 +9504,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } // Mark failures for objects that could not be locked - for (i, dobj) in objects.iter().enumerate() { - if let Some(err) = failed_map.get(&(bucket.to_string(), dobj.object_name.clone())) { + for (i, lock_name) in lock_names.iter().enumerate() { + if let Some(err) = failed_map.get(&(bucket.to_string(), lock_name.to_string())) { del_errs[i] = Some(Error::other(err.to_string())); } } @@ -8484,7 +9520,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { Err(err) => { let message = err.to_string(); for (index, item) in del_errs.iter_mut().enumerate() { - if locked_objects.contains(&objects[index].object_name) { + if locked_objects.contains(lock_names[index].as_ref()) { *item = Some(Error::other(message.clone())); } } @@ -8505,6 +9541,15 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { continue; } + let guarded = current_generations.is_some_and(|generations| generations.contains_key(&dobj.object_name)); + if guarded && dobj.version_id.is_some() { + del_errs[i] = Some(StorageError::InvalidArgument( + bucket.to_owned(), + dobj.object_name.clone(), + "current-object generation guards cannot protect explicit-version deletes".to_owned(), + )); + continue; + } let replication_object_name = decode_dir_object(&dobj.object_name); let explicit_null_version = is_explicit_null_version(dobj.version_id); let version_id = delete_file_info_version_id(dobj.version_id); @@ -8518,6 +9563,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { object_lock_delete: opts.object_lock_delete.clone(), object_lock_config_snapshot: Some(Arc::clone(&object_lock_config)), no_lock: true, + suppress_read_repair: guarded, ..Default::default() }; // A missing bucket Object Lock configuration does not prove that @@ -8549,6 +9595,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { && !opts.data_movement && !opts.replication_request && opts.http_preconditions.is_none() + && !guarded { let recovery = self .recover_delete_key( @@ -8607,6 +9654,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { && !versioned && !version_suspended && opts.http_preconditions.is_none() + && !guarded { missing_delete_retry_indices.push(i); } @@ -8762,20 +9810,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { vr.set_delete_marker_incarnation(incarnation); } - let v = { - if vers_map.contains_key(&dobj.object_name) { - let val = vers_map.get_mut(&dobj.object_name).unwrap(); - val.versions.push(vr.clone()); - val.clone() - } else { - FileInfoVersions { - name: vr.name.clone(), - versions: vec![vr.clone()], - ..Default::default() - } - } - }; - if vr.deleted || removed_delete_marker { del_objects[i] = DeletedObject { delete_marker: true, @@ -8829,8 +9863,15 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } // Only add to vers_map if we hold the lock - if locked_objects.contains(&dobj.object_name) { - vers_map.insert(&dobj.object_name, v); + if locked_objects.contains(lock_names[i].as_ref()) { + vers_map + .entry(&dobj.object_name) + .or_insert_with(|| FileInfoVersions { + name: vr.name.clone(), + ..Default::default() + }) + .versions + .push(vr); } } @@ -8920,6 +9961,24 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { fi_vers.versions.retain(|fi| del_errs[fi.idx].is_none()); } vers.retain(|fi_vers| !fi_vers.versions.is_empty()); + let recovery_tickets: Vec<_> = vers + .iter() + .map(|group| { + (!is_meta_bucketname(bucket)) + .then(|| { + self.ctx + .delete_recovery_hints() + .register(crate::store::delete_recovery_runtime::DeleteRecoveryKey { + pool: self.pool_index, + set: self.set_index, + bucket: bucket.to_owned(), + object: group.name.clone(), + incarnation: opts.expected_bucket_incarnation_id, + }) + }) + .flatten() + }) + .collect(); let mut futures = Vec::with_capacity(disks.len()); let lock_lost_during_commit = Arc::new(std::sync::atomic::AtomicBool::new(false)); @@ -8941,7 +10000,56 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { delete_recovery_bucket_incarnation: opts.expected_bucket_incarnation_id, ..Default::default() }; - if is_meta_bucketname(bucket) { + if current_generations + .is_some_and(|generations| vers.iter().any(|group| generations.contains_key(&group.name))) + { + let mut guarded = Vec::new(); + let mut ordinary = Vec::new(); + let mut guarded_indices = Vec::new(); + let mut ordinary_indices = Vec::new(); + let count = vers.len(); + for (index, group) in vers.into_iter().enumerate() { + if let Some(generation) = current_generations.and_then(|generations| generations.get(&group.name)) { + guarded_indices.push(index); + guarded.push(crate::disk::GuardedFileInfoVersions { + group, + expected_current: generation.clone(), + }); + } else { + ordinary_indices.push(index); + ordinary.push(group); + } + } + let mut results = vec![None; count]; + let guarded_results = disk + .delete_versions_guarded_recoverable_with_namespace_owner( + bucket, + guarded, + options.clone(), + disk_namespace_owner.clone(), + ) + .await; + for (index, error) in guarded_indices.into_iter().zip(guarded_results) { + results[index] = error; + } + if !ordinary.is_empty() { + let errors = if is_meta_bucketname(bucket) { + disk.delete_versions(bucket, ordinary, options).await + } else { + disk.delete_versions_recoverable_with_namespace_owner( + bucket, + ordinary, + options, + disk_namespace_owner, + ) + .await + }; + for (index, error) in ordinary_indices.into_iter().zip(errors) { + results[index] = error; + } + } + results + } else if is_meta_bucketname(bucket) { disk.delete_versions(bucket, vers, options).await } else { disk.delete_versions_recoverable_with_namespace_owner(bucket, vers, options, disk_namespace_owner) @@ -8983,6 +10091,13 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } let mut has_err = reduce_write_quorum_errs(&disk_err, OBJECT_OP_IGNORED_ERRS, self.set_drive_count / 2 + 1); + if disk_err + .iter() + .flatten() + .any(|error| matches!(error, DiskError::GuardedDeleteUnsupported)) + { + has_err = Some(DiskError::GuardedDeleteUnsupported); + } if !is_meta_bucketname(bucket) && disk_err.iter().flatten().any(super::delete_recovery::unsupported_recovery) { has_err = Some(DiskError::UnsupportedDisk); } @@ -9044,6 +10159,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { published_groups.extend(groups.iter().map(|index| vers[*index].name.clone())); let publications = futures::stream::iter(groups.into_iter().map(|index: usize| { let group = &vers[index]; + let recovery_ticket = recovery_tickets[index].clone(); let namespace_owner = namespace_owner.clone(); let disks = &disks; let opts = &opts; @@ -9063,6 +10179,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { disks, self.set_drive_count / 2 + 1, DeleteRecoveryContext { + recovery_ticket, namespace_owner, ..DeleteRecoveryContext::new(opts.expected_bucket_incarnation_id, &can_commit) }, @@ -9829,10 +10946,18 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { // Use the same full xl.meta read path as GetObject metadata resolution. // This avoids HEAD/GetObject metadata visibility skew immediately after // PutObject/CompleteMultipartUpload. - let snapshot = self - .get_object_fileinfo(bucket, object, opts, true, false) - .await - .map_err(|e| to_object_err(e, vec![bucket, object]))?; + let read = self.get_object_fileinfo(bucket, object, opts, true, false); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ObjectInfoRead, + async { + crate::cluster::rpc::runtime_sources::observe_read_version_context_boundary( + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ObjectInfoRead, + ); + read.await + }, + ); + let snapshot = read.await.map_err(|e| to_object_err(e, vec![bucket, object]))?; let oi = ObjectInfo::from_file_info(snapshot.fi(), bucket, object, opts.versioned || opts.version_suspended); @@ -11173,6 +12298,232 @@ mod object_encryption_resolver_wiring_tests { } } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_put_owner_preserves_parent_context_at_tier_free_version_lookup() { + use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry}; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context}; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let (_dirs, disks, set) = runtime.block_on(crate::set_disk::hermetic_set_disks_isolated(4)); + let bucket = "diagnostic-put-owner"; + runtime.block_on(async { + for disk in &disks { + disk.make_volume(bucket).await.expect("make test bucket volume"); + } + for index in 0..4 { + set.put_object( + bucket, + &format!("object-{index}"), + &mut PutObjReader::from_vec(b"old body".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("seed source object"); + } + }); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut request = S3HttpRequestGuard::new("PUT"); + let put = request.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject); + current_s3_http_operation_context().expect("captured PutObject") + }); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + let mut index = 0; + for earlier in [None, Some(ReadVersionEntrySource::ScannerConfigWrite)] { + for api in [None, Some(put)] { + let object = format!("object-{index}"); + index += 1; + let write = async { + let mut reader = PutObjReader::from_vec(b"new body".to_vec()); + let info = set + .put_object( + bucket, + &object, + &mut reader, + &ObjectOptions { + tier_free_version_receipt_sink: Some(crate::object_api::TierFreeVersionReceiptSink::new()), + ..Default::default() + }, + ) + .await + .expect("replace source object"); + assert_eq!(info.size, 8); + }; + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, write).await, + None => write.await, + } + }) + .await; + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + } + }); + }); + let rows = snapshots.snapshot().into_vec(); + for source in ["scanner_config_write", "put_commit_owner"] { + for boundary in ["put_commit_owner", "tier_free_version_source_lookup"] { + for (api, state) in [("unattributed", "absent"), ("s3:PutObject", "preserved")] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&boundary) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(count) if *count > 0) + }), + "actual detached owner and lookup must retain context: {source}/{boundary}/{api}" + ); + } + } + } + runtime.block_on(async { + for index in 0..4 { + let mut reader = set + .get_object_reader(bucket, &format!("object-{index}"), None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("committed object remains readable"); + let mut body = Vec::new(); + tokio::io::AsyncReadExt::read_to_end(&mut reader.stream, &mut body) + .await + .expect("read committed body"); + assert_eq!(body, b"new body"); + } + }); + } + + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + fn diagnostic_write_precondition_and_object_info_keep_origin_and_api() { + use crate::cluster::rpc::runtime_sources::{current_read_version_entry_source, with_read_version_entry}; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::internode_metrics::ReadVersionEntrySource; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_context}; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let recorder = DebuggingRecorder::new(); + let snapshots = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut guard = S3HttpRequestGuard::new("PUT"); + let put = guard.in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::PutObject); + current_s3_http_operation_context().expect("captured PutObject") + }); + let (_dirs, disks, set) = runtime.block_on(crate::set_disk::hermetic_set_disks_isolated(4)); + let bucket = "diagnostic-precondition"; + let object = "object"; + let original = runtime.block_on(async { + for disk in &disks { + disk.make_volume(bucket).await.expect("create test bucket volume"); + } + set.put_object(bucket, object, &mut PutObjReader::from_vec(b"body".to_vec()), &ObjectOptions::default()) + .await + .expect("seed object") + }); + let etag = original.etag.expect("seed ETag"); + metrics::with_local_recorder(&recorder, || { + runtime.block_on(async { + for earlier in [None, Some(ReadVersionEntrySource::ConfigWrite)] { + for api in [None, Some(put)] { + let read = async { + assert_eq!( + set.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .expect("read object info") + .etag, + Some(etag.clone()) + ); + for (expected, condition) in [(false, etag.clone()), (true, "other-etag".to_string())] { + let result = set + .check_write_precondition( + bucket, + object, + &ObjectOptions { + http_preconditions: Some(HTTPPreconditions { + if_match: Some(condition), + ..Default::default() + }), + ..Default::default() + }, + ) + .await; + assert_eq!(result, expected.then_some(Error::PreconditionFailed)); + assert_eq!(current_s3_http_operation_context(), api); + } + }; + rustfs_io_metrics::s3_http_metrics::with_isolated_s3_http_operation_context(api, async { + match earlier { + Some(source) => with_read_version_entry(source, read).await, + None => read.await, + } + }) + .await; + assert_eq!(current_read_version_entry_source(), None); + assert_eq!(current_s3_http_operation_context(), None); + } + } + }); + }); + let rows = snapshots.snapshot().into_vec(); + for (source, boundary) in [ + ("write_precondition_read", "write_precondition_lookup"), + ("write_precondition_read", "object_info_read"), + ("write_precondition_read", "full_wait"), + ("object_info_read", "object_info_read"), + ("config_write", "write_precondition_lookup"), + ("config_write", "object_info_read"), + ("config_write", "full_wait"), + ] { + for (api, state) in [("unattributed", "absent"), ("s3:PutObject", "preserved")] { + assert!( + rows.iter().any(|(key, _, _, value)| { + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + key.key().name() == "rustfs_storage_diagnostic_context_transitions_total" + && labels.get("entry_source") == Some(&source) + && labels.get("boundary") == Some(&boundary) + && labels.get("parent_api_operation") == Some(&api) + && labels.get("child_api_operation") == Some(&api) + && labels.get("api_context") == Some(&state) + && matches!(value, DebugValue::Counter(count) if *count > 0) + }), + "actual lookup must preserve origin and API: {source}/{boundary}/{api}" + ); + } + } + assert_eq!( + runtime + .block_on(set.get_object_info(bucket, object, &ObjectOptions::default())) + .expect("object retained") + .etag, + Some(etag) + ); + } + #[tokio::test] async fn get_object_reader_forwards_instance_resolver() { let resolver = Arc::new(CountingResolver { @@ -19412,6 +20763,73 @@ mod put_object_tmp_cleanup_tests { ); } + #[tokio::test] + async fn put_cleanup_admission_rejects_before_staging_with_an_external_lock() { + const CHILD: &str = "RUSTFS_CLEANUP_ADMISSION_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "set_disk::ops::object::put_object_tmp_cleanup_tests::put_cleanup_admission_rejects_before_staging_with_an_external_lock", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING", "1") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS", "1") + .env("RUSTFS_CLEANUP_CPUS", "none") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .output() + .unwrap() + }) + .await + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + let (dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "cleanup-admission-bucket"; + let object = "externally-locked"; + for disk in &disks { + disk.make_volume(bucket).await.unwrap(); + } + let occupied = cleanup_runtime::reserve().unwrap().expect("isolation enabled in child"); + let external_lock = set.acquire_write_lock_diag("admission_test", bucket, object).await.unwrap(); + let mut reader = PutObjReader::from_vec(vec![7u8; TEST_OBJECT_SIZE]); + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let result = tokio::time::timeout(Duration::from_secs(20), set.put_object(bucket, object, &mut reader, &opts)) + .await + .expect("admission must not wait while the caller owns locks"); + assert!(matches!(result, Err(StorageError::SlowDown))); + assert!(non_trash_tmp_entries(&dirs).await.is_empty(), "rejection must not stage shards"); + drop(external_lock); + drop(occupied); + let mut reader = PutObjReader::from_vec(vec![9u8; TEST_OBJECT_SIZE]); + set.put_object(bucket, object, &mut reader, &ObjectOptions::default()) + .await + .unwrap(); + wait_for_tmp_workspace_to_drain(&dirs, "admitted PUT must eventually clean staging").await; + let mut actual = set + .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .unwrap(); + let mut body = Vec::new(); + actual.stream.read_to_end(&mut body).await.unwrap(); + assert_eq!(body, vec![9u8; TEST_OBJECT_SIZE]); + } + #[tokio::test] async fn put_object_success_eventually_cleans_tmp_workspace() { let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; @@ -20720,6 +22138,294 @@ mod put_object_tmp_cleanup_tests { .await; } + #[tokio::test] + async fn system_metadata_repair_error_does_not_wait_for_committed_tmp_cleanup() { + const CHILD: &str = "RUSTFS_METADATA_REPAIR_CLEANUP_TEST_CHILD"; + if std::env::var_os(CHILD).is_none() { + let output = tokio::task::spawn_blocking(|| { + std::process::Command::new(std::env::current_exe().expect("test executable must be available")) + .args([ + "--exact", + "set_disk::ops::object::put_object_tmp_cleanup_tests::system_metadata_repair_error_does_not_wait_for_committed_tmp_cleanup", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING", "2") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS", "1") + .env("RUSTFS_CLEANUP_CPUS", "none") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP") + .output() + .expect("the isolated cleanup regression child must run") + }) + .await + .expect("cleanup regression child launcher must join"); + assert!( + output.status.success(), + "cleanup regression failed:\n{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + return; + } + + let (dirs, _disks, set) = hermetic_set_disks(4).await; + let occupied = cleanup_runtime::reserve() + .expect("the cleanup blocker must be admitted") + .expect("the child must enable cleanup isolation"); + let (ready, started) = tokio::sync::oneshot::channel(); + let (release, released) = tokio::sync::oneshot::channel(); + let blocker = cleanup_runtime::spawn(Some(occupied), async move { + ready.send(()).expect("the fixture must observe the cleanup worker"); + let _ = released.await; + }); + tokio::time::timeout(Duration::from_secs(30), started) + .await + .expect("the cleanup blocker must acquire the execution worker") + .expect("the blocker readiness sender must stay alive"); + + let object = "pool.bin.identity-cleanup-owner"; + let body = b"committed metadata with pending repair".to_vec(); + let fault = rename_fault_injection::fail_rename_on(object, &[0]); + let response = tokio::time::timeout( + Duration::from_secs(30), + crate::config::com::save_config_with_opts( + Arc::clone(&set), + object, + body.clone(), + &ObjectOptions { + max_parity: true, + write_completion: WriteCompletion::TailDrainedWithOwnerRepair, + ..Default::default() + }, + ), + ) + .await; + release.send(()).expect("the cleanup blocker must stay alive until released"); + tokio::time::timeout(Duration::from_secs(30), blocker) + .await + .expect("the released cleanup blocker must finish") + .expect("the cleanup blocker task must join"); + let error = response + .expect("repair-required response must not wait for the committed temporary cleanup owner") + .expect_err("the committed minority failure must reach the metadata writer"); + assert!( + matches!(error, Error::Io(ref inner) if inner.get_ref().is_some_and(|source| source.is::())) + ); + drop(fault); + wait_for_tmp_workspace_to_drain(&dirs, "the retained cleanup owner must eventually remove staged metadata").await; + let committed = crate::config::com::read_config(Arc::clone(&set), object) + .await + .expect("the repair-required metadata quorum must remain readable"); + assert_eq!(committed, body); + } + + #[tokio::test] + async fn rename_tail_heal_keeps_system_authority_separate_from_user_incarnation() { + let (_dirs, _disks, set) = hermetic_set_disks(4).await; + let mut heals = set.capture_test_rename_tail_heals(); + let incarnation = Uuid::new_v4(); + for bucket in [RUSTFS_META_BUCKET, ".minio.sys"] { + for expected in [None, Some(incarnation)] { + let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options( + bucket.to_string(), + Some("pool.bin.identity".to_string()), + false, + Some(HealChannelPriority::Normal), + Some(0), + Some(0), + ); + request.expected_bucket_incarnation_id = expected; + let error = set + .submit_rename_tail_heal(request, WriteCompletion::TailDrainedWithOwnerRepair) + .await + .expect_err("system repair must remain with its writer"); + assert!( + matches!(error, Error::Io(ref inner) if inner.get_ref().is_some_and(|source| source.is::())) + ); + assert!( + heals.try_recv().is_err(), + "even a supplied user incarnation cannot authorize system metadata heal" + ); + } + } + let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options( + "ordinary-bucket".to_string(), + Some("ordinary-object".to_string()), + false, + Some(HealChannelPriority::Normal), + Some(0), + Some(0), + ); + request.expected_bucket_incarnation_id = Some(incarnation); + set.submit_rename_tail_heal(request, WriteCompletion::Quorum) + .await + .expect("ordinary incarnation-bound fallback must remain available"); + let admitted = heals + .try_recv() + .expect("the ordinary fallback must actually reach the channel"); + assert_eq!(admitted.expected_bucket_incarnation_id, Some(incarnation)); + assert_eq!(admitted.source, rustfs_heal_contracts::heal_channel::HealRequestSource::Mrf); + assert_eq!(admitted.object_prefix.as_deref(), Some("ordinary-object")); + } + + #[tokio::test] + #[serial_test::serial(capacity_dirty_scope)] + async fn system_metadata_partial_write_respects_completion_contract() { + temp_env::async_with_vars([(ENV_RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE, Some("true"))], async { + for completion in [ + WriteCompletion::Quorum, + WriteCompletion::TailDrained, + WriteCompletion::TailDrainedWithOwnerRepair, + ] { + let (dirs, disks, set) = hermetic_set_disks(4).await; + let object = format!("pool.bin.identity-{completion:?}"); + let original = b"original deployment identity".to_vec(); + let fixture_options = ObjectOptions { + max_parity: true, + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }; + crate::config::com::save_config_with_opts(Arc::clone(&set), &object, original.clone(), &fixture_options) + .await + .expect("healthy system metadata write must succeed without a bucket incarnation"); + let original_info = set + .get_object_info(RUSTFS_META_BUCKET, &object, &ObjectOptions::default()) + .await + .expect("original metadata must be readable before the partial overwrite"); + let original_etag = original_info.etag.expect("original metadata must have an ETag"); + let updated = b"updated deployment identity".to_vec(); + let mut heals = set.capture_test_rename_tail_heals(); + let tasks = rename_fanout_barrier::observe_tasks(&object); + let barrier = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_RENAME); + let fault = rename_fault_injection::fail_rename_on(&object, &[0]); + let writer = Arc::clone(&set); + let write_object = object.clone(); + let write_body = updated.clone(); + let write = tokio::spawn(async move { + crate::config::com::save_config_with_opts( + writer, + &write_object, + write_body, + &ObjectOptions { + max_parity: true, + write_completion: completion, + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused()) + .await + .expect("the metadata rename tail must reach the failure barrier"); + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let mut updated_copies = 0; + for disk in &disks { + let info = disk + .read_version("", RUSTFS_META_BUCKET, &object, "", &ReadOptions::default()) + .await + .expect("each disk must retain either the original or updated metadata"); + if info.metadata.get("etag").is_some_and(|etag| etag != &original_etag) { + updated_copies += 1; + } + } + if updated_copies == 3 { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("the metadata overwrite must reach quorum while the last disk is paused"); + if completion != WriteCompletion::Quorum { + assert!( + !write.is_finished(), + "drained completion must wait for the rename tail before acknowledging" + ); + } + barrier.release(); + let result = tokio::time::timeout(Duration::from_secs(30), write) + .await + .expect("the metadata tail must drain") + .expect("the metadata writer task must join"); + if completion == WriteCompletion::TailDrainedWithOwnerRepair { + let error = result.expect_err("the pool metadata owner must receive repair responsibility"); + let Error::Io(error) = error else { + panic!("system metadata repair must retain its typed error"); + }; + assert!( + error + .get_ref() + .is_some_and(|source| source.is::()), + "the config writer must retain the exact repair-required cause: {error}" + ); + } else { + result.expect("ordinary system metadata must preserve quorum-success semantics"); + } + tokio::time::timeout(Duration::from_secs(30), async { + while tasks.running() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("the committed rename tail must eventually finish for every completion mode"); + assert_eq!(tasks.running(), 0, "repair failure must be returned after all rename tasks drain"); + assert!(heals.try_recv().is_err(), "system metadata must never enqueue an unfenced MRF heal"); + let mut stale_copies = 0; + let mut committed_etags = Vec::new(); + for disk in &disks { + let info = disk + .read_version("", RUSTFS_META_BUCKET, &object, "", &ReadOptions::default()) + .await + .expect("every physical disk must retain its metadata after the partial rename"); + let etag = info.metadata.get("etag").expect("each metadata copy must retain an ETag"); + if etag == &original_etag { + stale_copies += 1; + } else { + committed_etags.push(etag.clone()); + } + } + assert_eq!(stale_copies, 1, "exactly the failed rename must retain the old metadata"); + assert_eq!(committed_etags.len(), 3, "the new metadata must retain its write quorum"); + assert!(committed_etags.iter().all(|etag| etag == &committed_etags[0])); + let committed = + tokio::time::timeout(Duration::from_secs(30), crate::config::com::read_config(Arc::clone(&set), &object)) + .await + .expect("the committed metadata read must finish after the rename tail drains") + .expect("a repair-required result must not roll back the committed metadata quorum"); + assert_eq!(committed, updated, "the read must see the new bytes after cache invalidation"); + tokio::time::timeout(Duration::from_secs(30), async { + while !non_trash_tmp_entries(&dirs).await.is_empty() { + tokio::task::yield_now().await; + } + }) + .await + .expect("repair-required results must still schedule committed temporary-data cleanup"); + drop(fault); + drop(barrier); + tokio::time::timeout( + Duration::from_secs(30), + crate::config::com::save_config_with_opts(Arc::clone(&set), &object, updated.clone(), &fixture_options), + ) + .await + .expect("the reconciliation must finish after the fault and barrier are disarmed") + .expect("the owning writer must be able to reconcile after the failed disk recovers"); + for disk in &disks { + let info = disk + .read_version("", RUSTFS_META_BUCKET, &object, "", &ReadOptions::default()) + .await + .expect("the reconciled metadata must exist on every disk"); + assert_ne!(info.metadata.get("etag"), Some(&original_etag)); + } + } + }) + .await; + } + #[tokio::test] #[serial_test::serial(capacity_dirty_scope)] async fn tail_drained_put_preserves_quorum_success_without_unfenced_fallback_heal() { @@ -20995,6 +22701,42 @@ mod put_object_tmp_cleanup_tests { )); } + async fn borrowed_write_context_accepts(set: &SetDisks, bucket: &str, held: &str, target: &str) -> bool { + let guard = crate::object_api::WriteCommitGuard::acquire( + &set.new_ns_lock(bucket, held).await.expect("namespace wrapper"), + get_lock_acquire_timeout(), + ) + .await + .expect("outer write lock"); + let mut opts = ObjectOptions::default(); + opts.add_write_commit_guard(&guard); + matches!( + WriteCommitContext::from_options(&opts, bucket, target, false), + Ok(WriteCommitContext::Borrowed(_)) + ) + } + + #[cfg(not(windows))] + #[tokio::test] + async fn borrowed_write_context_keeps_backslash_spelling_distinct_off_windows() { + let (_dirs, _disks, set) = hermetic_set_disks(4).await; + assert!(borrowed_write_context_accepts(&set, "bucket", "a/b", "a/b").await); + assert!(!borrowed_write_context_accepts(&set, "bucket", "a/b", "a\\b").await); + assert!(!borrowed_write_context_accepts(&set, "bucket", "a\\b", "a/b").await); + } + + #[cfg(windows)] + #[tokio::test] + async fn windows_borrowed_write_context_accepts_backslash_spelling_of_held_object() { + // The guard records the folded lock key; a self-copy through `a\b` + // must still prove the namespace it commits into. + let (_dirs, _disks, set) = hermetic_set_disks(4).await; + assert!(borrowed_write_context_accepts(&set, "bucket", "a/b", "a\\b").await); + assert!(borrowed_write_context_accepts(&set, "bucket", "a\\b", "a\\b").await); + assert!(borrowed_write_context_accepts(&set, "bucket", "a\\b\\", "a/b").await); + assert!(!borrowed_write_context_accepts(&set, "bucket", "a/b", "a/b/c").await); + } + #[tokio::test] #[serial_test::serial(capacity_dirty_scope)] async fn borrowed_tail_drained_put_keeps_owner_after_caller_cancel() { @@ -22334,6 +24076,387 @@ mod delete_objects_lock_gating_tests { .expect("plain object should be written"); } + #[tokio::test] + #[serial_test::serial(shard_integrity_rollout)] + async fn conditional_delete_snapshot_preserves_multipart_etag() { + use crate::storage_api_contracts::multipart::MultipartOperations as _; + let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "conditional-multipart-etag"; + let object = "multipart.bin"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("create bucket"); + } + let options = ObjectOptions::default(); + let upload = set_disks + .new_multipart_upload(bucket, object, &options) + .await + .expect("create multipart upload"); + let part = set_disks + .put_object_part( + bucket, + object, + &upload.upload_id, + 1, + &mut PutObjReader::from_vec(vec![11; 4097]), + &options, + ) + .await + .expect("upload the final part"); + let committed = set_disks + .clone() + .complete_multipart_upload( + bucket, + object, + &upload.upload_id, + vec![CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }], + &options, + ) + .await + .expect("complete multipart object"); + let (snapshot, generation) = set_disks + .conditional_delete_current_snapshot( + bucket, + object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("read current multipart snapshot"); + let etag = committed.etag.as_deref().expect("multipart ETag"); + assert!(etag.ends_with("-1")); + assert_eq!(snapshot.etag, committed.etag); + let condition = crate::storage_api_contracts::object::DeleteObjectCondition::parse(&format!("\"{etag}\"")) + .expect("quoted response ETag"); + assert!(condition.matches(snapshot.etag.as_deref().expect("snapshot ETag"))); + assert!(matches!(generation, crate::disk::CurrentDeleteGeneration::Present { etag: stored, .. } if stored == etag)); + } + + #[tokio::test] + async fn guarded_delete_drive_capability_barrier_waits_for_success_and_cancels_on_error() { + use std::sync::atomic::{AtomicBool, AtomicUsize}; + use tokio_util::sync::CancellationToken; + + struct ProbeLifetime { + active: Arc, + dropped: Arc, + } + impl Drop for ProbeLifetime { + fn drop(&mut self) { + self.active.fetch_sub(1, Ordering::SeqCst); + self.dropped.fetch_add(1, Ordering::SeqCst); + } + } + + for unsupported in [false, true] { + let entered = Arc::new(AtomicUsize::new(0)); + let active = Arc::new(AtomicUsize::new(0)); + let dropped = Arc::new(AtomicUsize::new(0)); + let all_entered = Arc::new(tokio::sync::Notify::new()); + let completed = Arc::new(AtomicUsize::new(0)); + let two_completed = Arc::new(tokio::sync::Notify::new()); + let releases: Vec<_> = (0..3).map(|_| CancellationToken::new()).collect(); + let mutation_started = Arc::new(AtomicBool::new(false)); + let probes: Vec<_> = releases + .iter() + .enumerate() + .map(|(index, release)| { + let entered = Arc::clone(&entered); + let active = Arc::clone(&active); + let dropped = Arc::clone(&dropped); + let all_entered = Arc::clone(&all_entered); + let completed = Arc::clone(&completed); + let two_completed = Arc::clone(&two_completed); + let release = release.clone(); + async move { + active.fetch_add(1, Ordering::SeqCst); + let _lifetime = ProbeLifetime { active, dropped }; + if entered.fetch_add(1, Ordering::SeqCst) + 1 == 3 { + all_entered.notify_one(); + } + release.cancelled().await; + if completed.fetch_add(1, Ordering::SeqCst) + 1 == 2 { + two_completed.notify_one(); + } + if unsupported && index == 0 { + Err(DiskError::GuardedDeleteUnsupported) + } else { + Ok(()) + } + } + }) + .collect(); + let mutated = Arc::clone(&mutation_started); + let operation = tokio::spawn(async move { + guarded_delete_capability_barrier(probes).await?; + mutated.store(true, Ordering::SeqCst); + Ok::<(), StorageError>(()) + }); + tokio::time::timeout(std::time::Duration::from_secs(5), all_entered.notified()) + .await + .expect("all drive probes start concurrently"); + assert_eq!(entered.load(Ordering::SeqCst), 3); + assert_eq!(active.load(Ordering::SeqCst), 3); + assert!(!mutation_started.load(Ordering::SeqCst)); + releases[0].cancel(); + if !unsupported { + releases[1].cancel(); + tokio::time::timeout(std::time::Duration::from_secs(5), two_completed.notified()) + .await + .expect("both released probes complete"); + assert_eq!(completed.load(Ordering::SeqCst), 2); + assert!(!operation.is_finished(), "the last successful probe is required before admission"); + assert!(!mutation_started.load(Ordering::SeqCst), "unknown capability prevents mutation"); + releases[2].cancel(); + } + let result = tokio::time::timeout(std::time::Duration::from_secs(5), operation) + .await + .expect("a failed probe does not wait for unreleased peers") + .expect("probe task joins"); + assert_eq!(active.load(Ordering::SeqCst), 0, "no probe lifetime leaks after the barrier returns"); + assert_eq!(dropped.load(Ordering::SeqCst), 3, "completed and canceled probes release their resources"); + if unsupported { + assert_eq!(result, Err(StorageError::GuardedDeleteUnsupported)); + assert_eq!(completed.load(Ordering::SeqCst), 1, "unfinished peers are canceled rather than drained"); + assert!( + !releases[1].is_cancelled() && !releases[2].is_cancelled(), + "the error returns while other peers remain blocked" + ); + assert!(!mutation_started.load(Ordering::SeqCst), "unsupported capability prevents mutation"); + } else { + assert!(result.is_ok()); + assert_eq!(completed.load(Ordering::SeqCst), 3); + assert!(mutation_started.load(Ordering::SeqCst), "all successful probes admit mutation"); + } + } + } + + #[tokio::test] + async fn conditional_batch_delete_preserves_write_quorum_with_one_faulty_local_drive() { + use crate::disk::{CurrentDeleteGeneration, GuardedFileInfoVersions}; + let (_dirs, disks, set) = hermetic_set_disks(4).await; + let bucket = "guarded-faulty-quorum"; + let object = "object"; + for disk in &disks { + disk.make_volume(bucket).await.expect("create fixture volume"); + } + put_plain_object(&set, bucket, object).await; + let disk_info = disks[0] + .read_version(bucket, bucket, object, "", &crate::disk::ReadOptions::default()) + .await + .expect("read authoritative disk metadata"); + let (_, generation) = set + .conditional_delete_current_snapshot( + bucket, + object, + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("capture guarded generation"); + assert_eq!( + generation, + CurrentDeleteGeneration::from_file_info(&disk_info).expect("disk and set share a generation") + ); + let metadata_path = disks[0] + .path() + .join(bucket) + .join(object) + .join(crate::disk::STORAGE_FORMAT_FILE); + let before = tokio::fs::read(&metadata_path) + .await + .expect("read faulty drive's original metadata"); + disks[0].force_offline_for_test(); + set.preflight_guarded_delete_versions() + .await + .expect("local capability is known independently of drive health"); + let rejected_transaction = Uuid::new_v4(); + let rejected = disks[0] + .delete_versions_guarded_recoverable( + bucket, + vec![GuardedFileInfoVersions { + group: FileInfoVersions { + name: object.to_owned(), + versions: vec![FileInfo { + name: object.to_owned(), + version_id: disk_info.version_id, + ..Default::default() + }], + ..Default::default() + }, + expected_current: generation.clone(), + }], + DeleteOptions { + old_data_dir: Some(rejected_transaction), + ..Default::default() + }, + ) + .await; + assert_eq!( + rejected, + vec![Some(DiskError::FaultyDisk)], + "a known capability does not allow mutation on a faulty drive" + ); + let (_, errors) = set + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: object.to_owned(), + ..Default::default() + }], + ObjectOptions { + conditional_delete_generations: Some(Arc::new(HashMap::from([(object.to_owned(), generation)]))), + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert!( + errors[0].is_none(), + "three healthy drives satisfy the configured write quorum: {errors:?}" + ); + assert_eq!(tokio::fs::read(&metadata_path).await.expect("faulty drive is not mutated"), before); + assert!( + !metadata_path + .parent() + .expect("object directory") + .join(rejected_transaction.to_string()) + .exists() + ); + for disk in &disks[1..] { + assert!( + !disk + .path() + .join(bucket) + .join(object) + .join(crate::disk::STORAGE_FORMAT_FILE) + .exists(), + "healthy quorum drives apply the guarded deletion" + ); + } + assert!( + set.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)), + "the healthy read quorum observes the completed delete" + ); + } + + #[tokio::test] + async fn conditional_batch_delete_rejects_unadmitted_conditions() { + let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "conditional-unadmitted"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("create bucket"); + } + put_plain_object(&set_disks, bucket, "protected").await; + let (_, errors) = set_disks + .delete_objects( + bucket, + vec![ObjectToDelete { + object_name: "protected".to_owned(), + delete_condition: Some(crate::storage_api_contracts::object::DeleteObjectCondition::Exists), + ..Default::default() + }], + ObjectOptions { + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert!(matches!(&errors[0], Some(StorageError::InvalidArgument(_, _, _)))); + set_disks + .get_object_info(bucket, "protected", &ObjectOptions::default()) + .await + .expect("unadmitted condition must not delete"); + } + + #[tokio::test] + async fn conditional_batch_delete_disk_generation_rejects_overwrite_in_mixed_batch() { + let (_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "conditional-generation"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("create bucket"); + } + put_plain_object(&set_disks, bucket, "protected").await; + put_plain_object(&set_disks, bucket, "ordinary").await; + let (_, generation) = set_disks + .conditional_delete_current_snapshot( + bucket, + "protected", + &ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }, + ) + .await + .expect("snapshot original generation"); + let current = set_disks + .put_object( + bucket, + "protected", + &mut PutObjReader::from_vec(b"new-current".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("overwrite after snapshot"); + let (_, errors) = set_disks + .delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: "protected".to_owned(), + ..Default::default() + }, + ObjectToDelete { + object_name: "ordinary".to_owned(), + ..Default::default() + }, + ], + ObjectOptions { + conditional_delete_generations: Some(Arc::new(HashMap::from([("protected".to_owned(), generation)]))), + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + ) + .await; + assert_eq!(errors[0], Some(StorageError::ConditionalDeleteConflict)); + assert!(errors[1].is_none(), "ordinary group still succeeds: {errors:?}"); + let after = set_disks + .get_object_info(bucket, "protected", &ObjectOptions::default()) + .await + .expect("new data survives stale delete"); + assert_eq!(after.data_dir, current.data_dir); + assert_eq!(after.etag, current.etag); + assert!( + set_disks + .get_object_info(bucket, "ordinary", &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + #[tokio::test] async fn delete_objects_reports_mixed_results_per_key() { let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; @@ -23049,6 +25172,73 @@ mod delete_objects_lock_gating_tests { } drop(outer_guard); } + + #[cfg(windows)] + #[tokio::test] + async fn windows_delete_objects_batch_waits_on_backslash_alias_lock() { + let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await; + let bucket = "batch-alias-lock-bucket"; + for disk in &disk_stores { + disk.make_volume(bucket).await.expect("bucket volume should be created"); + } + put_plain_object(&set_disks, bucket, "a/b").await; + + let outer_guard = set_disks + .new_ns_lock(bucket, "a/b") + .await + .expect("namespace lock should be created") + .get_write_lock(Duration::from_secs(1)) + .await + .expect("outer write lock should be acquired"); + let lock_manager = set_disks + .local_lock_manager + .as_fast_lock_manager() + .expect("isolated fixture should use the local lock manager"); + assert_eq!(lock_manager.current_lock_counts(), (1, 0)); + + let objects = vec![ObjectToDelete { + object_name: "a\\b".to_string(), + ..Default::default() + }]; + let mut delete = Box::pin(set_disks.delete_objects( + bucket, + objects, + ObjectOptions { + object_lock_config_snapshot: Some(Arc::new(ObjectLockConfigSnapshot::new( + ObjectLockConfigState::ConfirmedAbsent, + ))), + ..Default::default() + }, + )); + // `a\b` is the xl.meta of `a/b`: the batch must queue behind the outer + // holder instead of taking a second exclusive lock beside it. + let blocked = tokio::time::timeout( + Duration::from_millis(300), + futures::future::poll_fn(|cx| { + let result = std::future::Future::poll(delete.as_mut(), cx); + // Holders only: the queued batch shows up as a waiter. + assert_eq!( + lock_manager.current_lock_counts().0, + 1, + "batch delete through `a\\b` must not lock beside the `a/b` holder" + ); + result + }), + ) + .await; + assert!(blocked.is_err(), "batch delete through `a\\b` must wait for the `a/b` holder"); + + drop(outer_guard); + let (deleted, errs) = tokio::time::timeout(Duration::from_secs(30), delete) + .await + .expect("batch delete should finish once the holder releases"); + assert!(errs[0].is_none(), "batch delete should succeed after the holder releases: {:?}", errs[0]); + assert!(deleted[0].found, "`a\\b` must delete the object stored as `a/b`"); + set_disks + .get_object_info(bucket, "a/b", &ObjectOptions::default()) + .await + .expect_err("`a/b` must be gone after deleting `a\\b`"); + } } #[cfg(test)] @@ -24239,6 +26429,22 @@ mod single_delete_namespace_owner_tests { } } +#[cfg(test)] +mod cleanup_receipt_admission_tests { + use super::*; + + #[test] + fn receipt_admission_skips_only_work_that_the_writer_would_skip() { + let old = Uuid::new_v4(); + let current = Uuid::new_v4(); + let epoch = Uuid::new_v4(); + assert!(!SetDisks::needs_old_data_cleanup_receipt(old, Some(current), None)); + assert!(!SetDisks::needs_old_data_cleanup_receipt(old, Some(old), Some(epoch))); + assert!(SetDisks::needs_old_data_cleanup_receipt(old, Some(current), Some(epoch))); + assert!(SetDisks::needs_old_data_cleanup_receipt(old, None, Some(epoch))); + } +} + #[cfg(test)] mod crash_delete_retry_tests { use super::*; diff --git a/crates/ecstore/src/set_disk/read.rs b/crates/ecstore/src/set_disk/read.rs index 54b31d386..3b8820056 100644 --- a/crates/ecstore/src/set_disk/read.rs +++ b/crates/ecstore/src/set_disk/read.rs @@ -15,9 +15,9 @@ use super::{ Arc, Bytes, DiskError, DiskOption, DiskStore, Endpoint, ErasureCache, Error, FileInfo, GetCodecStreamingFallbackReason, GetObjectFileInfo, GetObjectMetadataCacheCapability, GetObjectMetadataCacheEntry, GetObjectMetadataCacheGeneration, - GetObjectMetadataCacheKey, GetObjectReadPolicy, HashAlgorithm, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, - OBJECT_OP_IGNORED_ERRS, ObjectInfo, ObjectOptions, RUSTFS_META_BUCKET, ReadOptions, Result, SetDisks, StorageError, - adaptive_duplex_buffer_size, build_get_codec_streaming_decode_engine, build_inline_bitrot_readers_from_refs, + GetObjectMetadataCacheKey, GetObjectMetadataCacheLookupKey, GetObjectReadPolicy, HashAlgorithm, LOG_COMPONENT_ECSTORE, + LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectInfo, ObjectOptions, RUSTFS_META_BUCKET, ReadOptions, Result, SetDisks, + StorageError, adaptive_duplex_buffer_size, build_get_codec_streaming_decode_engine, build_inline_bitrot_readers_from_refs, collect_inline_data_shard_fileinfos_by_index, debug, error, get_codec_streaming_metrics_path, get_codec_streaming_multipart_max_parts, get_object_read_policy, is_codec_streaming_multipart_enabled, is_multipart_reader_setup_prefetch_enabled, object_fits_single_block, reduce_read_quorum_errs, to_object_err, @@ -208,6 +208,16 @@ use tokio::time::timeout; #[cfg(test)] use uuid::Uuid; +fn metadata_quorum_outcome(error: &StorageError) -> &'static str { + if error.is_not_found() { + "not_found" + } else if error.is_quorum_error() { + "not_reached" + } else { + "unavailable" + } +} + pub(super) struct GetObjectDownstreamWriter { inner: W, } @@ -400,7 +410,7 @@ impl SetDisks { let Some(generation) = self.get_object_metadata_cache_generation(bucket, object) else { return MetadataCacheLookup::Miss; }; - let key = GetObjectMetadataCacheKey::new(bucket, object, generation); + let key = GetObjectMetadataCacheLookupKey::new(bucket, object, generation); // moka handles TTL expiry automatically; no is_fresh() check needed let Some(entry) = self.get_object_metadata_cache.get(&key).await else { return MetadataCacheLookup::Miss; @@ -462,14 +472,14 @@ impl SetDisks { entry: Arc, after_insert: impl FnOnce(), ) { - // Every publisher shares Moka's per-key compute lock. A conditional - // snapshot cannot downgrade a full layout or extend its original TTL. + // Every publisher shares Moka's per-key compute lock. A partial snapshot + // cannot downgrade a full layout or extend its original TTL. self.get_object_metadata_cache .entry(key.clone()) .and_compute_with(|current| { std::future::ready( if !self.is_get_object_metadata_cache_generation_current(generation) - || (entry.capability == GetObjectMetadataCacheCapability::ConditionOnly + || (entry.capability != GetObjectMetadataCacheCapability::Full && current .as_ref() .is_some_and(|existing| existing.value().capability == GetObjectMetadataCacheCapability::Full)) @@ -509,6 +519,10 @@ impl SetDisks { let object: Arc = Arc::from(object); let version_id: Arc = Arc::from(version_id); let opts = *opts; + #[cfg(feature = "diagnostic-read-version-caller")] + let api_operation = rustfs_io_metrics::get_stage_metrics_enabled() + .then(rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context) + .flatten(); let processor = runtime_sources::batch_processors().read_processor(); let tasks: Vec<_> = disks @@ -520,8 +534,28 @@ impl SetDisks { let object = object.clone(); let version_id = version_id.clone(); let task_opts = opts; + #[cfg(feature = "diagnostic-read-version-caller")] + #[allow(clippy::redundant_locals)] + let api_operation = api_operation; - async move { disk.read_version(&bucket, &bucket, &object, &version_id, &task_opts).await } + async move { + let read_version = + async move { disk.read_version(&bucket, &bucket, &object, &version_id, &task_opts).await }; + #[cfg(feature = "diagnostic-read-version-caller")] + let read_version = crate::cluster::rpc::runtime_sources::with_read_version_request_caller_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::SetDiskRead, + read_version, + ); + #[cfg(feature = "diagnostic-read-version-caller")] + if let Some(api_operation) = api_operation { + rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(api_operation), read_version) + .await + } else { + read_version.await + } + #[cfg(not(feature = "diagnostic-read-version-caller"))] + read_version.await + } }) }) .collect(); @@ -602,6 +636,11 @@ impl SetDisks { purpose: MetadataReadPurpose, ) -> Result { let vid = opts.version_id.clone().unwrap_or_default(); + let metadata_metrics_path = if crate::bucket::utils::is_meta_bucketname(bucket) { + GET_OBJECT_PATH_INTERNAL_META + } else { + GET_OBJECT_PATH_LEGACY_DUPLEX + }; let stage_metrics_enabled = rustfs_io_metrics::get_stage_metrics_enabled(); let metadata_cache_lookup_start = get_stage_timer_if_enabled(stage_metrics_enabled); @@ -617,12 +656,17 @@ impl SetDisks { match self.lookup_cached_get_object_fileinfo(bucket, object).await { MetadataCacheLookup::Hit(cached) if cached.capability == GetObjectMetadataCacheCapability::Full + || (cached.capability == GetObjectMetadataCacheCapability::QuorumSelected + && read_data + && allow_early_stop) || (purpose == MetadataReadPurpose::GetObjectReader + && cached.capability == GetObjectMetadataCacheCapability::ConditionOnly && allow_early_stop && crate::object_api::get_object_read_condition_is_terminal( crate::object_api::GetObjectReadMetadata::from_file_info(bucket, object, &cached.fi), )) => { + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, "reached"); rustfs_io_metrics::record_get_object_metadata_cache_decision( GET_OBJECT_PATH_SET_DISK, GET_METADATA_CACHE_DECISION_HIT, @@ -633,7 +677,15 @@ impl SetDisks { GET_STAGE_METADATA_CACHE_LOOKUP, metadata_cache_lookup_start, ); - return Ok(GetObjectFileInfo::shared(cached)); + let snapshot = if cached.capability == GetObjectMetadataCacheCapability::QuorumSelected + && non_inline_data_read_early_stop_allowed(read_data, bucket, object) + && late_materialization_candidate_is_safe(&cached.fi) + { + GetObjectFileInfo::shared_with_late_metadata_refresh(cached) + } else { + GetObjectFileInfo::shared(cached) + }; + return Ok(snapshot); } MetadataCacheLookup::Hit(_) => { // Keep the conditional entry for other requests; this read @@ -683,42 +735,56 @@ impl SetDisks { let allow_read_version_coalescing = purpose == MetadataReadPurpose::GetObjectReader && !crate::bucket::utils::is_meta_bucketname(bucket) && crate::runtime::global::get_metadata_read_version_coalescing_service_ready(); - let metadata_read = if allow_read_version_coalescing { - Self::read_metadata_for_get_object( - &disks, - "", - bucket, - object, - vid.as_str(), - read_data, - opts.incl_free_versions, - allow_early_stop, - self.default_parity_count, - ) - .await? - } else { - Self::read_metadata_observed( - &disks, - "", - bucket, - object, - vid.as_str(), - read_data, - false, - opts.incl_free_versions, - allow_early_stop, - self.default_parity_count, - ) - .await? + let metadata_read = async { + if allow_read_version_coalescing { + Self::read_metadata_for_get_object( + &disks, + "", + bucket, + object, + vid.as_str(), + read_data, + opts.incl_free_versions, + allow_early_stop, + self.default_parity_count, + ) + .await + } else { + Self::read_metadata_observed( + &disks, + "", + bucket, + object, + vid.as_str(), + read_data, + false, + opts.incl_free_versions, + allow_early_stop, + self.default_parity_count, + ) + .await + } + }; + #[cfg(feature = "diagnostic-read-version-caller")] + let metadata_read_result = crate::cluster::rpc::runtime_sources::with_read_version_caller_and_entry_if_absent( + rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::Object, + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ObjectRead, + metadata_read, + ) + .await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + let metadata_read_result = metadata_read.await; + let metadata_read = match metadata_read_result { + Ok(metadata_read) => metadata_read, + Err(err) => { + let err = StorageError::from(err); + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, metadata_quorum_outcome(&err)); + return Err(err); + } }; let metadata_fanout_complete = metadata_read.is_complete(); let condition_stopped = metadata_read.condition_stopped; let (mut parts_metadata, errs, metadata_fanout_diagnostics) = metadata_read.into_legacy(); - let metadata_metrics_path = if crate::bucket::utils::is_meta_bucketname(bucket) { - GET_OBJECT_PATH_INTERNAL_META - } else { - GET_OBJECT_PATH_LEGACY_DUPLEX - }; metadata_fanout_diagnostics.record(metadata_metrics_path); // warn!("get_object_fileinfo parts_metadata {:?}", &parts_metadata); // warn!("get_object_fileinfo {}/{} errs {:?}", bucket, object, &errs); @@ -731,6 +797,7 @@ impl SetDisks { { Ok(v) => v, Err(e) => { + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, metadata_quorum_outcome(&e)); // error!("Self::object_quorum_from_meta: {:?}, bucket: {}, object: {}", &e, bucket, object); record_get_stage_duration_if_enabled( GET_OBJECT_PATH_SET_DISK, @@ -745,6 +812,7 @@ impl SetDisks { let write_quorum = usize::try_from(write_quorum) .map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![bucket, object]))?; if let Some(err) = reduce_read_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, read_quorum) { + rustfs_io_metrics::record_get_object_metadata_quorum_result(metadata_metrics_path, "not_reached"); error!("reduce_read_quorum_errs: {:?}, bucket: {}, object: {}", &err, bucket, object); record_get_stage_duration_if_enabled( GET_OBJECT_PATH_SET_DISK, @@ -755,7 +823,17 @@ impl SetDisks { } let (op_online_disks, mut fi, fileinfo_selection_quorum) = - Self::select_valid_fileinfo(&disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum)?; + match Self::select_valid_fileinfo(&disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum) { + Ok(selection) => selection, + Err(err) => { + let err = StorageError::from(err); + rustfs_io_metrics::record_get_object_metadata_quorum_result( + metadata_metrics_path, + metadata_quorum_outcome(&err), + ); + return Err(err); + } + }; let include_part_checksums = opts.include_part_checksums || opts.part_number.is_some() || opts.data_movement || opts.raw_data_movement_read; if include_part_checksums { @@ -778,7 +856,16 @@ impl SetDisks { "metadata_read_error", ) .await; - } else if use_metadata_cache && (metadata_fanout_complete || condition_stopped) { + } else if use_metadata_cache + && (metadata_fanout_complete + || condition_stopped + || (read_data + && allow_early_stop + && metadata_fanout_diagnostics.valid_responses() >= read_quorum + && errs.iter().all(Option::is_none) + && fileinfo_selection_quorum >= read_quorum + && op_online_disks.iter().filter(|disk| disk.is_some()).count() >= read_quorum)) + { #[cfg(test)] metadata_cache_tests::wait_before_metadata_cache_publish(bucket, object).await; self.cache_get_object_fileinfo( @@ -788,8 +875,10 @@ impl SetDisks { &fi, if metadata_fanout_complete { GetObjectMetadataCacheCapability::Full - } else { + } else if condition_stopped { GetObjectMetadataCacheCapability::ConditionOnly + } else { + GetObjectMetadataCacheCapability::QuorumSelected }, ), &parts_metadata, @@ -2653,7 +2742,9 @@ fn is_get_object_metadata_cache_request_eligible(bucket: &str, opts: &ObjectOpti #[cfg(test)] mod metadata_cache_tests { use super::*; + use crate::set_disk::core::io_primitives::disk_call_counters; use rustfs_heal_contracts::heal_channel::HealAdmissionDropReason; + use rustfs_utils::path::object_name_disk_identity; use serial_test::serial; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::{Mutex, OnceLock}; @@ -4011,8 +4102,65 @@ mod metadata_cache_tests { assert_eq!(set.get_object_metadata_cache.entry_count(), 0); } + async fn cache_valid_test_fileinfo(set: &SetDisks, object: &str) { + let fi = valid_test_fileinfo(object); + let generation = set.get_object_metadata_cache_generation("bucket", object); + set.cache_get_object_fileinfo( + ("bucket", object), + generation, + (&fi, GetObjectMetadataCacheCapability::Full), + std::slice::from_ref(&fi), + &[], + 0, + ) + .await; + assert!(set.cached_get_object_fileinfo("bucket", object).await.is_some()); + } + + #[cfg(not(windows))] #[tokio::test] - #[serial(metadata_cache_publish_barrier)] + async fn metadata_cache_keeps_backslash_and_slash_keys_distinct_off_windows() { + assert_eq!(object_name_disk_identity("a\\b"), "a\\b"); + let set = new_metadata_cache_test_set().await; + cache_valid_test_fileinfo(&set, "a/b").await; + cache_valid_test_fileinfo(&set, "a\\b").await; + + set.invalidate_get_object_metadata_cache("bucket", "a\\b").await; + + assert!( + set.cached_get_object_fileinfo("bucket", "a/b").await.is_some(), + "off Windows `a\\b` is a different object and must not retire `a/b`" + ); + assert!(set.cached_get_object_fileinfo("bucket", "a\\b").await.is_none()); + } + + #[cfg(windows)] + #[tokio::test] + async fn windows_metadata_cache_shares_backslash_and_slash_spellings() { + assert_eq!(object_name_disk_identity("a\\b"), "a/b"); + let set = new_metadata_cache_test_set().await; + cache_valid_test_fileinfo(&set, "a/b").await; + assert!( + set.cached_get_object_fileinfo("bucket", "a\\b").await.is_some(), + "both spellings address the same xl.meta and must share one cache entry" + ); + + set.invalidate_get_object_metadata_cache("bucket", "a\\b").await; + assert!( + set.cached_get_object_fileinfo("bucket", "a/b").await.is_none(), + "a mutation through `a\\b` must retire metadata cached for `a/b`" + ); + + cache_valid_test_fileinfo(&set, "a\\b").await; + set.invalidate_get_object_metadata_cache("bucket", "a/b").await; + assert!( + set.cached_get_object_fileinfo("bucket", "a\\b").await.is_none(), + "a mutation through `a/b` must retire metadata cached for `a\\b`" + ); + } + + #[tokio::test] + #[serial(metadata_cache_early_stop)] async fn metadata_cache_production_fanout_cannot_publish_after_invalidation() { // Isolated context: an ambient DistErasure window (another test's // SetupTypeGuard) would bypass metadata-cache publication entirely @@ -4066,6 +4214,191 @@ mod metadata_cache_tests { ); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial(metadata_cache_early_stop)] + async fn metadata_cache_publishes_quorum_selected_early_stop_result() { + use tokio::io::AsyncReadExt; + + let isolated_ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + isolated_ctx + .update_erasure_type(crate::layout::endpoints::SetupType::Erasure) + .await; + let (dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(4, 2, isolated_ctx).await; + let bucket = "metadata-cache-early-stop"; + let object = "objects/repeated-key"; + let disks = set.disks.read().await.clone(); + for disk in disks.iter().flatten() { + disk.make_volume(bucket) + .await + .expect("test bucket should be created on each disk"); + } + let expected_body = vec![0x5a; 1024 * 1024]; + set.put_object( + bucket, + object, + &mut PutObjReader::from_vec(expected_body.clone()), + &ObjectOptions::default(), + ) + .await + .expect("test object should be written before enabling the slow-tail fault"); + + let calls = disk_call_counters::observe(object); + temp_env::async_with_vars( + [ + ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("false")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS", Some("500")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS", Some("3")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET", Some(bucket)), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_OBJECT_PREFIX", Some("objects/")), + ], + async { + let first = tokio::time::timeout( + Duration::from_millis(400), + set.get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, true), + ) + .await + .expect("first metadata read should stop before the delayed peer") + .expect("read quorum should provide valid object metadata"); + assert!(first.owned.is_some(), "the first lookup should resolve through disk metadata"); + assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), 4); + assert_eq!(calls.total(disk_call_counters::KIND_METADATA_SLOWTAIL_FAULT), 1); + + let second = set + .get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, true) + .await + .expect("second lookup should use the quorum-published metadata cache entry"); + assert!(second.shared_entry().is_some(), "the repeated GET should hit the published cache entry"); + assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), 4); + + let cached = second.shared_entry().expect("second lookup should share cached metadata"); + assert_eq!(cached.fi.name, object); + let cached_online_count = cached.online_disks.iter().filter(|disk| disk.is_some()).count(); + assert_eq!(cached_online_count, 3, "the warm cache should contain three of four disks"); + assert_eq!(cached.capability, GetObjectMetadataCacheCapability::QuorumSelected); + + let data_dir = cached + .fi + .data_dir + .expect("quorum cache retains the object data directory") + .to_string(); + let selected_disk_indices = cached + .online_disks + .iter() + .enumerate() + .filter_map(|(index, disk)| disk.as_ref().map(|_| index)) + .take(2) + .collect::>(); + assert_eq!(selected_disk_indices.len(), 2, "the warmed cache should name enough selected shards"); + for index in selected_disk_indices { + tokio::fs::remove_file(dirs[index].path().join(bucket).join(object).join(&data_dir).join("part.1")) + .await + .expect("simulate loss of two cached selected shards"); + } + + let mut reader = set + .get_object_reader(bucket, object, None, http::HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("a cached partial layout must recover through omitted disks"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("the GET should decode after refreshing the omitted disks"); + assert_eq!(body, expected_body, "recovered body must be byte-exact"); + }, + ) + .await; + + let full_layout = set + .get_object_fileinfo_gated(bucket, object, &ObjectOptions::default(), false, false) + .await + .expect("read-before-write consumers must fetch a complete layout"); + assert!( + full_layout.shared_entry().is_none(), + "a quorum-selected partial layout cannot satisfy mutation consumers" + ); + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) > 4, + "mutation consumers must independently fan out after a quorum-only cache hit" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[serial(metadata_cache_early_stop)] + async fn metadata_cache_early_stop_publication_is_fenced_by_invalidation() { + let isolated_ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + isolated_ctx + .update_erasure_type(crate::layout::endpoints::SetupType::Erasure) + .await; + let (_dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks_with_ctx(4, 2, isolated_ctx).await; + let bucket = "metadata-cache-early-stop-fence"; + let object = "objects/repeated-key"; + let disks = set.disks.read().await.clone(); + for disk in disks.iter().flatten() { + disk.make_volume(bucket) + .await + .expect("test bucket should be created on each disk"); + } + set.put_object( + bucket, + object, + &mut PutObjReader::from_vec(vec![0x5a; 1024 * 1024]), + &ObjectOptions::default(), + ) + .await + .expect("test object should be written before enabling the slow-tail fault"); + + temp_env::async_with_vars( + [ + ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_TWO_PHASE_READ_PLAN_ENABLE", Some("true")), + ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("false")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS", Some("500")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DISKS", Some("3")), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_BUCKET", Some(bucket)), + ("RUSTFS_GET_METADATA_SLOWTAIL_FAULT_OBJECT_PREFIX", Some("objects/")), + ], + async { + let barrier = MetadataCachePublishBarrier::install(bucket, object); + let stale_generation = set + .get_object_metadata_cache_generation(bucket, object) + .expect("metadata cache generation should be active"); + let reader_set = Arc::clone(&set); + let read = tokio::spawn(async move { + reader_set + .get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, true) + .await + }); + barrier.wait_until_paused().await; + set.invalidate_get_object_metadata_cache(bucket, object).await; + barrier.release(); + + let snapshot = read + .await + .expect("early-stop metadata read should not panic") + .expect("read quorum should resolve the selected FileInfo"); + assert!(snapshot.owned.is_some()); + assert!( + set.get_object_metadata_cache + .get(&GetObjectMetadataCacheKey::new(bucket, object, stale_generation)) + .await + .is_none(), + "early-stop publication must not retain an entry under the retired generation" + ); + assert!( + set.cached_get_object_fileinfo(bucket, object).await.is_none(), + "invalidation must reject the quorum-selected early-stop publication" + ); + }, + ) + .await; + } + #[tokio::test] #[serial(metadata_cache_publish_barrier, body_cache_hook)] async fn conditional_inline_hedge_caches_only_terminal_metadata() { @@ -5085,12 +5418,15 @@ mod metadata_cache_tests { .get_object_metadata_cache_generation("bucket-a", "object-a") .expect("metadata cache generation should be active"); let first_key = GetObjectMetadataCacheKey::new("bucket-a", "object-a", generation); + let first_lookup = GetObjectMetadataCacheLookupKey::new("bucket-a", "object-a", generation); let second_key = GetObjectMetadataCacheKey { bucket: Arc::from("bucket-b"), object: Arc::from("object-b"), generation: generation.value, hash: generation.hash, }; + let second_lookup = GetObjectMetadataCacheLookupKey::new("bucket-b", "object-b", generation); + let unrelated_lookup = GetObjectMetadataCacheLookupKey::new("bucket-c", "object-c", generation); let first_fi = valid_test_fileinfo("object-a"); let second_fi = valid_test_fileinfo("object-b"); let entry = |fi: FileInfo| { @@ -5111,7 +5447,7 @@ mod metadata_cache_tests { assert_eq!( set.get_object_metadata_cache - .get(&first_key) + .get(&first_lookup) .await .expect("first colliding entry should remain addressable") .fi @@ -5120,13 +5456,14 @@ mod metadata_cache_tests { ); assert_eq!( set.get_object_metadata_cache - .get(&second_key) + .get(&second_lookup) .await .expect("second colliding entry should remain addressable") .fi .name, "object-b" ); + assert!(set.get_object_metadata_cache.get(&unrelated_lookup).await.is_none()); } #[tokio::test] @@ -5293,6 +5630,16 @@ mod tests { const CODEC_STREAMING_TEST_OBJECT: &str = "object"; static CAPTURED_READ_REPAIR_CALLS: AtomicUsize = AtomicUsize::new(0); + #[test] + fn metadata_quorum_outcome_uses_bounded_error_categories() { + assert_eq!(metadata_quorum_outcome(&StorageError::ErasureReadQuorum), "not_reached"); + assert_eq!(metadata_quorum_outcome(&StorageError::FileNotFound), "not_found"); + assert_eq!( + metadata_quorum_outcome(&StorageError::Io(std::io::Error::other("disk unavailable"))), + "unavailable" + ); + } + fn capture_read_repair_submitter( _request: rustfs_heal_contracts::heal_channel::HealChannelRequest, ) -> ReadRepairAdmissionFuture { diff --git a/crates/ecstore/src/storage_api_contracts/mod.rs b/crates/ecstore/src/storage_api_contracts/mod.rs index 414080eb7..00e507d52 100644 --- a/crates/ecstore/src/storage_api_contracts/mod.rs +++ b/crates/ecstore/src/storage_api_contracts/mod.rs @@ -63,6 +63,8 @@ pub(crate) mod namespace { pub(crate) mod object { use super::{Debug, Error, FileInfo, GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader}; use crate::storage_api_contracts::range::HTTPRangeSpec; + #[cfg(test)] + pub(crate) use rustfs_storage_api::object::DeleteObjectCondition; pub(crate) use rustfs_storage_api::{ DeleteAccounting, DeletedObject, HTTPPreconditions, ObjectIO, ObjectLockDeleteOptions, ObjectLockRetentionOptions, ObjectOperations, ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState, ObjectToDelete, diff --git a/crates/ecstore/src/store/bucket.rs b/crates/ecstore/src/store/bucket.rs index 57ccd3581..f7690f903 100644 --- a/crates/ecstore/src/store/bucket.rs +++ b/crates/ecstore/src/store/bucket.rs @@ -20,6 +20,7 @@ use crate::bucket::{ use crate::error::is_err_bucket_not_found; use crate::runtime::sources as runtime_sources; use crate::set_disk::{BucketInfoQuorum, get_lock_acquire_timeout}; +use crate::storage_api_contracts::admin::DiskSetSelector; use crate::storage_api_contracts::bucket::{BUCKET_LIFECYCLE_LOCK_OBJECT, SRBucketDeleteOp}; use crate::storage_api_contracts::namespace::NamespaceLocking as _; use futures::stream::{self, StreamExt}; @@ -1151,7 +1152,7 @@ impl ECStore { let mut scoped_results = futures::future::join_all(self.bucket_sets().map(|(pool_index, set_index, set)| async move { let result = match quorum { BucketInfoQuorum::Read => set.stat_bucket_with_quorum(bucket, quorum).await, - BucketInfoQuorum::Write => set.get_bucket_info(bucket, opts).await, + BucketInfoQuorum::Write | BucketInfoQuorum::HealCandidates => set.get_bucket_info(bucket, opts).await, }; (pool_index, set_index, result) })) @@ -1232,13 +1233,13 @@ impl ECStore { pub(super) async fn handle_list_bucket(&self, opts: &BucketOptions) -> Result> { // TODO(backlog): support cached bucket listing via opts.cached Ok(self - .list_bucket_from_sets(opts, usize::MAX, BucketInfoQuorum::Write) + .list_bucket_from_sets(opts, usize::MAX, BucketInfoQuorum::Write, None) .await? .buckets) } pub async fn list_bucket_for_scanner(&self, opts: &BucketOptions) -> Result { - self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY, BucketInfoQuorum::Write) + self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY, BucketInfoQuorum::Write, None) .await } @@ -1254,20 +1255,44 @@ impl ECStore { }, SCANNER_BUCKET_LIST_SET_CONCURRENCY, BucketInfoQuorum::Read, + None, ) .await? .buckets) } + /// Preserve namespace completeness for administrator traversal. A scoped + /// request must not depend on unrelated sets being available. + pub async fn list_buckets_for_admin_heal( + &self, + selector: Option, + ) -> Result { + self.list_bucket_from_sets( + &BucketOptions { + no_metadata: true, + ..Default::default() + }, + SCANNER_BUCKET_LIST_SET_CONCURRENCY, + BucketInfoQuorum::HealCandidates, + selector, + ) + .await + } + async fn list_bucket_from_sets( &self, opts: &BucketOptions, max_concurrency: usize, quorum: BucketInfoQuorum, + selector: Option, ) -> Result { - let set_count = self.pools.iter().map(|pool| pool.disk_set.len()).sum(); + let sets = self + .bucket_sets() + .filter(|(pool, set, _)| selector.is_none_or(|scope| scope.pool_idx == *pool && scope.set_idx == *set)) + .collect::>(); + let set_count = sets.len(); let concurrency = bucket_list_set_concurrency(set_count, max_concurrency); - let mut set_listings = stream::iter(self.bucket_sets().map(move |(pool_index, set_index, set)| async move { + let mut set_listings = stream::iter(sets.into_iter().map(move |(pool_index, set_index, set)| async move { set.list_bucket_with_quorum(quorum) .await .map(|(buckets, complete)| (pool_index, set_index, buckets, complete)) @@ -2236,6 +2261,64 @@ mod tests { } } + #[tokio::test] + async fn admin_bucket_discovery_preserves_completeness_and_scope() { + use crate::storage_api_contracts::admin::DiskSetSelector; + let (temp, store) = setup_bucket_quorum_test_env(&[2, 2], None).await; + let empty = store + .list_buckets_for_admin_heal(None) + .await + .expect("empty namespace enumeration"); + assert!(empty.buckets.is_empty()); + assert!(empty.topology_complete); + for disk in 0..2 { + tokio::fs::create_dir(temp.path().join(format!("pool0-disk{disk}/healthy-bucket"))) + .await + .expect("healthy bucket"); + } + tokio::fs::create_dir(temp.path().join("pool1-disk0/partial-bucket")) + .await + .expect("single surviving bucket directory"); + let global = store + .list_buckets_for_admin_heal(None) + .await + .expect("online disk enumeration"); + assert!(global.topology_complete, "every disk answered the candidate inventory"); + assert_eq!( + global.buckets.iter().map(|b| b.name.as_str()).collect::>(), + ["healthy-bucket", "partial-bucket"] + ); + let healthy = store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 0))) + .await + .expect("healthy scope"); + assert!(healthy.topology_complete); + assert_eq!(healthy.buckets.len(), 1); + assert_eq!(healthy.set_buckets.len(), 1); + let partial = store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(1, 0))) + .await + .expect("partial scope"); + assert!(partial.topology_complete); + assert_eq!(partial.buckets[0].name, "partial-bucket"); + let invalid = store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 1))) + .await + .expect("unconfigured scope"); + assert!(!invalid.topology_complete); + assert!(invalid.buckets.is_empty()); + let offline = take_set_disks_offline(&store, &store.pools[1].disk_set[0], &[0]).await; + assert!(store.list_buckets_for_admin_heal(None).await.is_err(), "quorum errors must remain errors"); + assert!( + store + .list_buckets_for_admin_heal(Some(DiskSetSelector::new(0, 0))) + .await + .expect("unrelated offline pool") + .topology_complete + ); + restore_set_disks(&store, &store.pools[1].disk_set[0], offline).await; + } + #[test] fn should_not_override_when_metadata_created_is_unix_epoch() { assert!(!should_override_created_from_metadata(OffsetDateTime::UNIX_EPOCH)); diff --git a/crates/ecstore/src/store/delete_recovery_runtime.rs b/crates/ecstore/src/store/delete_recovery_runtime.rs new file mode 100644 index 000000000..6256a71a9 --- /dev/null +++ b/crates/ecstore/src/store/delete_recovery_runtime.rs @@ -0,0 +1,1290 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded scheduling over the existing on-disk delete journals and receipts. +//! Hints are expendable: startup, returning drives and overflow retain a raw +//! discovery obligation. Only the canonical recovery path decides mutations. + +use super::ECStore; +use crate::bucket::utils::is_meta_bucketname; +use crate::disk::{DeleteRecoveryCursor, DiskAPI, DiskStore}; +use rustfs_heal_contracts::heal_channel::HealOpts; +use std::collections::{HashMap, HashSet, VecDeque}; +use std::sync::{ + Arc, Mutex, + atomic::{AtomicBool, Ordering}, +}; +use std::time::Duration; +use tokio::time::{Instant, MissedTickBehavior}; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +const HINT_CAPACITY: usize = 1_024; +const DISCOVERY_PAGE_SIZE: u32 = 128; +const PAGES_PER_PASS: usize = 8; +const MAX_ACTIVE_DISCOVERY_CURSORS: usize = 32; +const LOG_COMPONENT_ECSTORE: &str = "ecstore"; +const LOG_SUBSYSTEM_DELETE_RECOVERY: &str = "delete_recovery"; +const EVENT_DELETE_RECOVERY_STATUS: &str = "delete_recovery_status"; +const KEYS_PER_PASS: usize = 16; +const INTERVAL: Duration = Duration::from_secs(1); +const INVENTORY_INTERVAL: Duration = Duration::from_secs(30); +const DISCOVERY_BUDGET: Duration = Duration::from_secs(5); +const ANTI_ENTROPY_INTERVAL: Duration = Duration::from_secs(60 * 60); +const PASS_START_BUDGET: Duration = Duration::from_secs(30); + +#[derive(Clone, Debug, Eq, Hash, PartialEq)] +pub(crate) struct DeleteRecoveryKey { + pub(crate) pool: usize, + pub(crate) set: usize, + pub(crate) bucket: String, + pub(crate) object: String, + pub(crate) incarnation: Option, +} + +#[derive(Clone, Debug)] +pub(crate) struct DeleteRecoveryTicket { + key: Arc, + generation: Uuid, + admission: Option>, +} + +#[derive(Debug)] +struct AdmissionGuard { + state: Arc>, + key: Arc, + settled: AtomicBool, +} + +impl Drop for AdmissionGuard { + fn drop(&mut self) { + if !self.settled.load(Ordering::Acquire) { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if !state.pending.contains_key(&self.key) { + // A scan can finish before an evicted request persists its + // journal. Re-arm after that request finishes or is canceled. + state.rescan = Some(Uuid::new_v4()); + } + } + } +} + +#[derive(Debug)] +struct HintState { + pending: HashMap, (Uuid, Uuid, bool)>, + order: VecDeque, + rescan: Option, + started: HashSet, + #[cfg(test)] + compaction_visits: usize, +} + +impl HintState { + fn compact_order(&mut self, capacity: usize) { + // Twice the pending capacity guarantees at least capacity completed + // entries between compactions, even with a full retry backlog. This + // amortizes FIFO maintenance across successful delete requests. + // The queue generation preserves FIFO position across coalescing. + if self.order.len() < capacity.saturating_mul(2) { + return; + } + #[cfg(test)] + { + self.compaction_visits += self.order.len(); + } + let Self { pending, order, .. } = self; + order.retain(|ticket| pending.get(&ticket.key).is_some_and(|entry| entry.1 == ticket.generation)); + } +} + +pub(crate) struct DeleteRecoveryHints { + state: Arc>, + capacity: usize, +} + +impl Default for DeleteRecoveryHints { + fn default() -> Self { + Self::with_capacity(HINT_CAPACITY) + } +} + +impl DeleteRecoveryHints { + fn with_capacity(capacity: usize) -> Self { + Self { + state: Arc::new(Mutex::new(HintState { + pending: HashMap::new(), + order: VecDeque::new(), + rescan: Some(Uuid::new_v4()), + started: HashSet::new(), + #[cfg(test)] + compaction_visits: 0, + })), + capacity, + } + } + + // The queue mutex is held only for in-memory operations and is always + // released before disk, namespace, bucket lifecycle or movement locks. + pub(crate) fn register(&self, key: DeleteRecoveryKey) -> Option { + let key = Arc::new(key); + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let generation = Uuid::new_v4(); + if let Some((current, _, _)) = state.pending.get_mut(&key) { + *current = generation; + } else { + if state.pending.len() >= self.capacity { + // Durable evidence remains the authority. Rotate the oldest + // hint so permanently blocked keys cannot exclude later raw + // discoveries forever; the next scan revisits evicted keys. + while let Some(oldest) = state.order.pop_front() { + if state + .pending + .get(&oldest.key) + .is_some_and(|entry| entry.1 == oldest.generation) + { + state.pending.remove(&oldest.key); + break; + } + } + state.rescan = Some(Uuid::new_v4()); + } + state.compact_order(self.capacity); + state.pending.insert(key.clone(), (generation, generation, false)); + state.order.push_back(DeleteRecoveryTicket { + key: key.clone(), + generation, + admission: None, + }); + } + Some(DeleteRecoveryTicket { + admission: Some(Arc::new(AdmissionGuard { + state: self.state.clone(), + key: key.clone(), + settled: AtomicBool::new(false), + })), + key, + generation, + }) + } + + fn register_discovered(&self, key: DeleteRecoveryKey) -> bool { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.pending.contains_key(&key) { + return true; + } + if state.pending.len() >= self.capacity { + // Only previously attempted work may yield its slot. Unattempted + // keys apply backpressure so a fast scan cannot overwrite them. + let retried = state.order.iter().position(|ticket| { + state + .pending + .get(&ticket.key) + .is_some_and(|entry| entry.1 == ticket.generation && entry.2) + }); + let Some(index) = retried else { return false }; + if let Some(oldest) = state.order.remove(index) { + state.pending.remove(&oldest.key); + } + // This durable key has already been attempted. The hourly sweep + // revisits it; immediate rescans would loop on permanent failures. + } + state.compact_order(self.capacity); + let key = Arc::new(key); + let generation = Uuid::new_v4(); + state.pending.insert(key.clone(), (generation, generation, false)); + state.order.push_back(DeleteRecoveryTicket { + key, + generation, + admission: None, + }); + true + } + + pub(crate) fn ack(&self, ticket: &DeleteRecoveryTicket) { + if let Some(admission) = &ticket.admission { + admission.settled.store(true, Ordering::Release); + } + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state + .pending + .get(&ticket.key) + .is_some_and(|entry| entry.0 == ticket.generation) + { + state.pending.remove(&ticket.key); + } + } + + fn take(&self, limit: usize) -> Vec { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + let mut tickets = Vec::with_capacity(limit.min(state.pending.len())); + while tickets.len() < limit { + let Some(ticket) = state.order.pop_front() else { break }; + if let Some(&(generation, slot, _)) = state.pending.get(&ticket.key) + && slot == ticket.generation + { + state.pending.remove(&ticket.key); + tickets.push(DeleteRecoveryTicket { + key: ticket.key, + generation, + admission: None, + }); + } + } + tickets + } + + fn retry(&self, ticket: DeleteRecoveryTicket) { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.pending.contains_key(&ticket.key) { + return; + } + state.compact_order(self.capacity); + if state.pending.len() >= self.capacity { + // Previously attempted work yields to admitted keys until the + // periodic sweep. Foreground admission retains its own rescan guard. + return; + } + state + .pending + .insert(ticket.key.clone(), (ticket.generation, ticket.generation, true)); + state.order.push_back(ticket); + } + + fn pending_len(&self) -> usize { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .pending + .len() + } + + fn request_rescan(&self) { + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).rescan = Some(Uuid::new_v4()); + } + + fn rescan(&self) -> Option { + self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner).rescan + } + + fn ack_rescan(&self, generation: Uuid) { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.rescan == Some(generation) { + state.rescan = None; + } + } + + fn start(&self, store: Uuid) -> bool { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .started + .insert(store) + } +} + +#[derive(Clone)] +struct LocalSource { + pool: usize, + set: usize, + disk: DiskStore, +} + +struct BucketScan { + bucket: String, + incarnation: Option, + cursor: Option, + pending: VecDeque, +} + +struct DiskScan { + source: LocalSource, + buckets: Option>, +} + +fn admit_discovered_keys(hints: &DeleteRecoveryHints, source: &LocalSource, bucket: &mut BucketScan) { + while let Some(object) = bucket.pending.front() { + if !hints.register_discovered(DeleteRecoveryKey { + pool: source.pool, + set: source.set, + bucket: bucket.bucket.clone(), + object: object.clone(), + incarnation: bucket.incarnation, + }) { + break; + } + bucket.pending.pop_front(); + } +} + +#[derive(Default)] +struct RecoveryPass { + sources: Vec, + next_inventory: Option, + next_sweep: Option, + scans: VecDeque, + scan_generation: Option, + deferred: VecDeque<(Instant, DiskScan)>, +} + +impl RecoveryPass { + async fn refresh_inventory(&mut self, store: &Arc) { + if self.next_inventory.is_some_and(|next| Instant::now() < next) { + return; + } + let mut sources = Vec::new(); + for pool in &store.pools { + for set in &pool.disk_set { + for disk in set + .disk_inventory() + .await + .into_iter() + .flatten() + .filter(|disk| disk.is_local()) + { + sources.push(LocalSource { + pool: set.pool_index, + set: set.set_index, + disk, + }); + } + } + } + let current_source = |source: &LocalSource| { + sources.iter().any(|current| { + current.pool == source.pool && current.set == source.set && Arc::ptr_eq(¤t.disk, &source.disk) + }) + }; + if sources.len() != self.sources.len() || self.sources.iter().any(|source| !current_source(source)) { + store.ctx.delete_recovery_hints().request_rescan(); + self.scans.retain(|scan| current_source(&scan.source)); + self.deferred.retain(|(_, scan)| current_source(&scan.source)); + } + self.sources = sources; + self.next_inventory = Some(Instant::now() + INVENTORY_INTERVAL); + } + + fn defer(&mut self, mut scan: DiskScan) { + if let Some(buckets) = scan.buckets.as_mut() { + for bucket in buckets { + bucket.cursor = None; + bucket.incarnation = None; + } + } + let same_scope = |other: &DiskScan| { + scan.source.pool == other.source.pool + && scan.source.set == other.source.set + && scan.source.disk.to_string() == other.source.disk.to_string() + && scan + .buckets + .as_ref() + .and_then(|buckets| buckets.front()) + .map(|bucket| &bucket.bucket) + == other + .buckets + .as_ref() + .and_then(|buckets| buckets.front()) + .map(|bucket| &bucket.bucket) + }; + if !self.deferred.iter().any(|(_, pending)| same_scope(pending)) { + self.deferred.push_back((Instant::now() + INVENTORY_INTERVAL, scan)); + } + } + + async fn discover_page(&mut self, store: &Arc, deadline: Instant) { + let hints = store.ctx.delete_recovery_hints(); + let active_cursor = |scan: &DiskScan| { + scan.buckets + .as_ref() + .and_then(|buckets| buckets.front()) + .is_some_and(|bucket| bucket.cursor.is_some() || !bucket.pending.is_empty()) + }; + if self.scans.iter().filter(|scan| active_cursor(scan)).count() >= MAX_ACTIVE_DISCOVERY_CURSORS { + // Keep room for exact-key recovery and other discovery callers in + // the disk API's bounded session table. Finish existing cursors + // before starting another bucket; queued sources retain their place. + if let Some(index) = self.scans.iter().position(active_cursor) { + self.scans.rotate_left(index); + } + } + let Some(mut scan) = self.scans.pop_front() else { return }; + if scan.buckets.is_none() { + match tokio::time::timeout_at(deadline, scan.source.disk.list_volumes()).await { + Ok(Ok(volumes)) => { + scan.buckets = Some( + volumes + .into_iter() + .filter(|volume| !is_meta_bucketname(&volume.name)) + .map(|volume| BucketScan { + bucket: volume.name, + incarnation: None, + cursor: None, + pending: VecDeque::new(), + }) + .collect(), + ); + } + _ => { + self.defer(scan); + return; + } + } + } else if let Some(buckets) = scan.buckets.as_mut() + && let Some(mut bucket) = buckets.pop_front() + { + let had_pending = !bucket.pending.is_empty(); + admit_discovered_keys(hints, &scan.source, &mut bucket); + if !bucket.pending.is_empty() || (had_pending && bucket.cursor.is_none()) { + if !bucket.pending.is_empty() { + buckets.push_front(bucket); + } + if !buckets.is_empty() { + self.scans.push_back(scan); + } + return; + } + if bucket.incarnation.is_none() { + match tokio::time::timeout_at(deadline, store.bucket_incarnation_id_from_disk(&bucket.bucket)).await { + Ok(Ok(incarnation)) => bucket.incarnation = Some(incarnation), + _ => { + self.defer(DiskScan { + source: scan.source.clone(), + buckets: Some(VecDeque::from([bucket])), + }); + if !buckets.is_empty() { + self.scans.push_back(scan); + } + return; + } + } + } + match tokio::time::timeout_at( + deadline, + scan.source + .disk + .discover_delete_transactions(&bucket.bucket, None, bucket.cursor.take(), DISCOVERY_PAGE_SIZE), + ) + .await + { + Ok(Ok(page)) => { + bucket.pending = page.transactions.into_iter().map(|transaction| transaction.object).collect(); + bucket.cursor = page.next; + admit_discovered_keys(hints, &scan.source, &mut bucket); + if bucket.cursor.is_some() || !bucket.pending.is_empty() { + // Keep one bounded page until its keys are admitted; + // unopened buckets cannot displace its continuation. + buckets.push_front(bucket); + } + } + _ => { + // Retry only this source, never a full healthy tree + // because another disk is offline or a cursor expired. + self.defer(DiskScan { + source: scan.source.clone(), + buckets: Some(VecDeque::from([bucket])), + }); + } + } + } + if scan.buckets.as_ref().is_some_and(|buckets| !buckets.is_empty()) { + self.scans.push_back(scan); + } + } + + async fn discover(&mut self, store: &Arc, cancel: &CancellationToken) { + let hints = store.ctx.delete_recovery_hints(); + // Renew only this worker's bounded local cursors before any await. + // Backpressure and a slow canonical rollback must not erase progress. + for scan in &mut self.scans { + if let Some(bucket) = scan.buckets.as_mut().and_then(|buckets| buckets.front_mut()) + && let Some(cursor) = bucket.cursor.as_ref() + { + let renewed = match scan.source.disk.as_ref() { + crate::disk::Disk::Local(local) => local.get_disk().renew_delete_recovery_cursor(&bucket.bucket, cursor), + crate::disk::Disk::Remote(_) => false, + }; + if !renewed { + bucket.cursor = None; + hints.request_rescan(); + } + } + } + let discovery_deadline = Instant::now() + DISCOVERY_BUDGET; + tokio::select! { + _ = cancel.cancelled() => return, + result = tokio::time::timeout_at(discovery_deadline, self.refresh_inventory(store)) => { + if result.is_err() { hints.request_rescan(); } + } + } + if self.scan_generation.is_none() { + if self.next_sweep.is_some_and(|next| Instant::now() >= next) { + hints.request_rescan(); + self.next_sweep = None; + } + if let Some(generation) = hints.rescan() { + self.scan_generation = Some(generation); + self.scans = self + .sources + .iter() + .cloned() + .map(|source| DiskScan { source, buckets: None }) + .collect(); + } + } + for _ in 0..self.deferred.len() { + let Some((ready, scan)) = self.deferred.pop_front() else { break }; + if ready <= Instant::now() { + self.scans.push_back(scan); + } else { + self.deferred.push_back((ready, scan)); + } + } + for _ in 0..PAGES_PER_PASS { + if self.scans.is_empty() || Instant::now() >= discovery_deadline { + break; + } + tokio::select! { + _ = cancel.cancelled() => return, + _ = self.discover_page(store, discovery_deadline) => {} + } + } + if self.scans.is_empty() + && let Some(generation) = self.scan_generation.take() + { + hints.ack_rescan(generation); + self.next_sweep = Some(Instant::now() + ANTI_ENTROPY_INTERVAL); + } + } + + #[cfg(test)] + async fn run(&mut self, store: &Arc, cancel: &CancellationToken) { + self.discover(store, cancel).await; + recover_hints(store, cancel).await; + } +} + +#[derive(Default, PartialEq, Eq)] +struct RecoveryFailures { + pending: usize, + blocked: usize, + corrupt: usize, + failed: usize, +} + +async fn recover_hints(store: &Arc, cancel: &CancellationToken) -> RecoveryFailures { + let hints = store.ctx.delete_recovery_hints(); + let deadline = Instant::now() + PASS_START_BUDGET; + let mut failures = RecoveryFailures::default(); + // Remove only the selected generation before awaiting. A concurrent + // DELETE can enqueue the same key independently; neither successful + // recovery nor retry of this ticket acknowledges that newer hint. + let attempts = hints.pending_len().min(KEYS_PER_PASS); + for _ in 0..attempts { + if Instant::now() >= deadline || cancel.is_cancelled() { + break; + } + // Only the operation about to start leaves the queue. A slow first + // key must not turn untouched work into evictable retry entries. + let Some(ticket) = hints.take(1).pop() else { break }; + let key = &ticket.key; + let opts = HealOpts { + pool: Some(key.pool), + set: Some(key.set), + ..Default::default() + }; + // The pass deadline limits starting work, not an admitted rollback. + // Canonical disk/RPC recovery owns its configured large-shard timeout. + let outcome = tokio::select! { + _ = cancel.cancelled() => None, + result = store.recover_delete_transactions_at_incarnation( + &key.bucket, Some(&key.object), key.incarnation, &opts, + ) => Some(result), + }; + if !matches!(outcome, Some(Ok(_))) { + match outcome { + Some(Err(crate::error::StorageError::ErasureReadQuorum)) => failures.pending += 1, + Some(Err(crate::error::StorageError::SlowDown)) => failures.blocked += 1, + Some(Err(crate::error::StorageError::FileCorrupt)) => failures.corrupt += 1, + Some(Err(_)) => failures.failed += 1, + _ => {} + } + hints.retry(ticket); + } + } + failures +} + +pub(super) fn start_delete_recovery(store: Arc, cancel: CancellationToken) { + if !store.ctx.delete_recovery_hints().start(store.id) { + return; + } + tokio::spawn(async move { + let discovery = async { + let mut interval = tokio::time::interval(INTERVAL); + interval.set_missed_tick_behavior(MissedTickBehavior::Delay); + let mut pass = RecoveryPass::default(); + let mut previous_deferred = 0; + loop { + tokio::select! { + biased; + _ = cancel.cancelled() => return, + _ = interval.tick() => {}, + } + pass.discover(&store, &cancel).await; + let deferred = pass.deferred.len(); + if deferred != previous_deferred { + tracing::warn!( + event = EVENT_DELETE_RECOVERY_STATUS, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DELETE_RECOVERY, + state = "discovery_retry_changed", + deferred_sources = deferred, + pending_keys = store.ctx.delete_recovery_hints().pending_len(), + "Delete recovery discovery retry state changed" + ); + previous_deferred = deferred; + } + } + }; + let recovery = async { + let mut interval = tokio::time::interval(INTERVAL); + interval.set_missed_tick_behavior(MissedTickBehavior::Delay); + let mut previous = RecoveryFailures::default(); + loop { + tokio::select! { + biased; + _ = cancel.cancelled() => return, + _ = interval.tick() => {}, + } + let failures = recover_hints(&store, &cancel).await; + if failures != previous { + tracing::warn!( + event = EVENT_DELETE_RECOVERY_STATUS, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_DELETE_RECOVERY, + state = "recovery_retry_changed", + pending = failures.pending, + blocked = failures.blocked, + corrupt = failures.corrupt, + failed = failures.failed, + pending_keys = store.ctx.delete_recovery_hints().pending_len(), + "Delete recovery settlement retry state changed" + ); + previous = failures; + } + } + }; + // Long canonical verification uses its configured deadline while raw + // discovery continues to advance bounded, expiring disk cursors. + tokio::join!(discovery, recovery); + }); +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks}; + use crate::disk::{DeleteOptions, DeleteRecoveryAction, STORAGE_FORMAT_FILE}; + use crate::object_api::{ObjectOptions, PutObjReader}; + use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; + use crate::storage_api_contracts::object::{ObjectIO, ObjectOperations}; + use rustfs_filemeta::FileInfo; + + fn key(object: &str) -> DeleteRecoveryKey { + DeleteRecoveryKey { + pool: 0, + set: 0, + bucket: "bucket".to_owned(), + object: object.to_owned(), + incarnation: Some(Uuid::from_u128(1)), + } + } + + #[test] + fn delete_recovery_hints_coalesce_without_losing_fifo_or_new_generations() { + let hints = DeleteRecoveryHints::with_capacity(2); + let first = hints.register(key("a")).expect("first hint"); + hints.register(key("b")).expect("second hint"); + let newer = hints.register(key("a")).expect("coalesced hint"); + hints.ack(&first); + let taken = hints.take(1); + assert_eq!(taken.len(), 1); + assert_eq!(taken[0].key.object, "a", "coalescing must preserve fairness"); + assert_eq!(taken[0].generation, newer.generation); + let latest = hints.register(key("a")).expect("hint during recovery"); + hints.ack(&taken[0]); + hints.retry(taken.into_iter().next().expect("selected ticket")); + let pending = hints.take(2); + assert_eq!(pending[0].key.object, "b"); + assert_eq!(pending[1].generation, latest.generation, "old recovery cannot replace a newer hint"); + } + + #[test] + fn delete_recovery_overflow_rearms_after_preparation_and_preserves_new_scan_obligations() { + let hints = DeleteRecoveryHints::with_capacity(1); + let overflow = hints.register(key("occupied")).expect("retain request lifetime"); + hints.register(key("later")).expect("evict before preparation"); + let scan = hints.rescan().expect("overflow requests discovery"); + hints.ack_rescan(scan); + assert!(hints.rescan().is_none()); + drop(overflow); + let next = hints.rescan().expect("completion after discovery must rearm overflow"); + hints.request_rescan(); + hints.ack_rescan(next); + assert!(hints.rescan().is_some(), "old scan cannot clear a newer obligation"); + let completed = hints.register(key("completed")).expect("overflow ticket"); + hints.ack(&completed); + let final_scan = hints.rescan().expect("queued scan"); + hints.ack_rescan(final_scan); + drop(completed); + assert!(hints.rescan().is_none(), "synchronous settlement needs no new scan"); + } + + #[test] + fn delete_recovery_hints_bound_tombstones_and_retry_after_drain() { + let hints = DeleteRecoveryHints::with_capacity(2); + for _ in 0..100 { + let ticket = hints.register(key("same")).expect("register reusable key"); + hints.ack(&ticket); + } + let _a = hints.register(key("a")).expect("a"); + hints.register(key("b")).expect("b"); + assert!(hints.state.lock().expect("hint state").order.len() <= 2); + let mut tickets = hints.take(2); + assert_eq!(tickets.len(), 2); + hints.retry(tickets.remove(0)); + assert_eq!(hints.take(2).len(), 1, "failed work survives an empty queue"); + } + + #[test] + fn delete_recovery_hint_compaction_stays_amortized_near_capacity() { + let capacity = HINT_CAPACITY; + let completed = capacity * 4; + let hints = DeleteRecoveryHints::with_capacity(capacity); + let blocked: Vec<_> = (0..capacity - 1) + .map(|index| hints.register(key(&format!("blocked-{index}"))).expect("blocked hint")) + .collect(); + hints.ack_rescan(hints.rescan().expect("startup discovery")); + for _ in 0..completed { + let ticket = hints.register(key("completed")).expect("healthy delete"); + hints.ack(&ticket); + let state = hints.state.lock().expect("hint state"); + assert_eq!(state.pending.len(), capacity - 1, "healthy completion cannot evict blocked work"); + assert!(state.order.len() <= capacity * 2, "tombstones remain bounded"); + } + let visits = hints.state.lock().expect("hint state").compaction_visits; + eprintln!("compaction visited {visits} tickets for {completed} successful deletes near capacity"); + assert!( + visits <= (completed + capacity) * 2, + "queue maintenance must be amortized, not capacity per request" + ); + assert!(hints.rescan().is_none(), "successful deletes must not cause false overflow"); + let remaining = hints.take(capacity); + assert_eq!(remaining.len(), blocked.len()); + for (actual, expected) in remaining.iter().zip(&blocked) { + assert_eq!(actual.generation, expected.generation, "compaction preserves FIFO and generations"); + } + } + + #[test] + fn delete_recovery_saturated_blocked_hints_cannot_exclude_later_discovery() { + let hints = DeleteRecoveryHints::with_capacity(2); + hints.register(key("blocked-a")).expect("first blocked key"); + hints.register(key("blocked-b")).expect("second blocked key"); + for ticket in hints.take(2) { + hints.retry(ticket); + } + hints + .register(key("healthy-later")) + .expect("later page can replace oldest hint"); + let next = hints.take(2); + assert_eq!(next.len(), 2); + assert!(next.iter().any(|ticket| ticket.key.object == "healthy-later")); + assert!(hints.rescan().is_some(), "evicted durable work remains a discovery obligation"); + } + + async fn fixture(bucket: &str) -> (Vec, Arc, Vec) { + let (dirs, store) = isolated_store_over_temp_disks().await; + // These tests drive discovery and recovery explicitly. Stop the loops + // started by ECStore::init before adding work or advancing virtual time. + store.background_cancel_token().expect("fixture background token").cancel(); + tokio::task::yield_now().await; + init_bucket_metadata_sys(store.clone(), Vec::new()).await; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create recovery bucket"); + let mut reader = PutObjReader::from_vec(vec![19; 4096]); + store + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("write recovery source"); + let disks = store.pools[0].disk_set[0] + .disk_inventory() + .await + .into_iter() + .map(|disk| disk.expect("online disk")) + .collect(); + (dirs, store, disks) + } + + async fn prepare(store: &Arc, disks: &[DiskStore], bucket: &str) -> Uuid { + let transaction = Uuid::new_v4(); + let incarnation = store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("bucket incarnation"); + for disk in disks { + disk.delete_version_recoverable( + bucket, + "object", + FileInfo { + name: "object".to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("persist preparation"); + } + transaction + } + + async fn publish_one(disk: &DiskStore, bucket: &str, transaction: Uuid) { + let inspection = disk + .inspect_delete_transaction(bucket, "object", transaction) + .await + .expect("inspect preparation"); + disk.settle_delete_transaction( + bucket, + "object", + transaction, + inspection.fingerprint, + DeleteRecoveryAction::PublishCommit, + ) + .await + .expect("persist receipt with lost response"); + } + + fn hint_only_pass(store: &Arc) -> RecoveryPass { + let hints = store.ctx.delete_recovery_hints(); + if let Some(scan) = hints.rescan() { + hints.ack_rescan(scan); + } + RecoveryPass { + next_inventory: Some(Instant::now() + INVENTORY_INTERVAL), + ..Default::default() + } + } + + async fn register(store: &Arc, bucket: &str) { + store.ctx.delete_recovery_hints().register(DeleteRecoveryKey { + bucket: bucket.to_owned(), + object: "object".to_owned(), + pool: 0, + set: 0, + incarnation: Some( + store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("source incarnation"), + ), + }); + } + + #[tokio::test] + async fn delete_recovery_clean_startup_discovers_hidden_preparation() { + let bucket = "runtime-clean-start"; + let (_dirs, store, disks) = fixture(bucket).await; + prepare(&store, &disks, bucket).await; + assert!( + disks[0] + .read_all(bucket, &format!("object/{STORAGE_FORMAT_FILE}")) + .await + .is_err() + ); + let mut pass = RecoveryPass::default(); + for _ in 0..8 { + pass.run(&store, &CancellationToken::new()).await; + } + assert!( + disks[0] + .read_all(bucket, &format!("object/{STORAGE_FORMAT_FILE}")) + .await + .is_ok(), + "startup must restore an uncommitted hidden object without a client DELETE or dirty shutdown" + ); + assert!( + store.ctx.delete_recovery_hints().rescan().is_none(), + "a successful scan is not repeated each tick" + ); + } + + #[tokio::test] + async fn delete_recovery_partial_publication_converges_without_another_delete() { + let bucket = "runtime-lost-response"; + let (_dirs, store, disks) = fixture(bucket).await; + let mut pass = hint_only_pass(&store); + register(&store, bucket).await; + let transaction = prepare(&store, &disks, bucket).await; + publish_one(&disks[0], bucket, transaction).await; + pass.run(&store, &CancellationToken::new()).await; + for disk in &disks { + let page = disk + .discover_delete_transactions(bucket, Some("object"), None, 128) + .await + .expect("inspect cleanup"); + assert!( + page.transactions.is_empty(), + "partial publication must converge through canonical recovery" + ); + } + assert!(store.ctx.delete_recovery_hints().take(1).is_empty()); + } + + #[tokio::test] + async fn delete_recovery_offline_participant_retains_work_until_return() { + let bucket = "runtime-offline-return"; + let (_dirs, store, disks) = fixture(bucket).await; + let mut pass = hint_only_pass(&store); + register(&store, bucket).await; + let transaction = prepare(&store, &disks, bucket).await; + publish_one(&disks[0], bucket, transaction).await; + let set = &store.pools[0].disk_set[0]; + let offline = set.disks.write().await[3].take(); + pass.run(&store, &CancellationToken::new()).await; + assert_eq!( + store + .ctx + .delete_recovery_hints() + .state + .lock() + .expect("pending state") + .pending + .len(), + 1 + ); + set.disks.write().await[3] = offline; + pass.run(&store, &CancellationToken::new()).await; + assert!( + store.ctx.delete_recovery_hints().take(1).is_empty(), + "returning participant releases retained work" + ); + assert!( + disks[0] + .discover_delete_transactions(bucket, Some("object"), None, 128) + .await + .expect("reclaimed journals") + .transactions + .is_empty() + ); + } + + #[tokio::test] + async fn delete_recovery_raw_discovery_is_bounded_and_rotates_disks_and_buckets() { + let bucket = "runtime-page-budget"; + let (_dirs, store, _disks) = fixture(bucket).await; + for index in 0..8 { + store + .make_bucket(&format!("runtime-other-{index}"), &MakeBucketOptions::default()) + .await + .expect("create scan bucket"); + } + let mut pass = RecoveryPass::default(); + pass.run(&store, &CancellationToken::new()).await; + assert_eq!(pass.scans.len(), 4, "one pass cannot consume all bucket pages"); + let remaining: Vec<_> = pass + .scans + .iter() + .map(|scan| scan.buckets.as_ref().expect("initialized disk").len()) + .collect(); + assert!( + remaining.iter().all(|count| *count == 8), + "each disk receives one page fairly after volume discovery" + ); + for _ in 0..8 { + pass.run(&store, &CancellationToken::new()).await; + } + assert!(pass.scans.is_empty()); + assert!(store.ctx.delete_recovery_hints().rescan().is_none()); + } + #[tokio::test] + async fn delete_recovery_blocked_capacity_yields_until_periodic_sweep() { + let (_dirs, store, _disks) = fixture("runtime-blocked-sweep").await; + let hints = store.ctx.delete_recovery_hints(); + hints.request_rescan(); + let scanning = hints.rescan().expect("startup discovery obligation"); + for index in 0..HINT_CAPACITY { + assert!(hints.register_discovered(key(&format!("blocked-{index}")))); + } + for ticket in hints.take(KEYS_PER_PASS) { + hints.retry(ticket); + } + for index in 0..KEYS_PER_PASS { + assert!( + hints.register_discovered(key(&format!("later-page-{index}"))), + "attempted failures yield to later discovery" + ); + } + let in_flight = hints.take(1).pop().expect("one admitted recovery attempt"); + assert!(hints.register_discovered(key("arrived-during-recovery"))); + hints.retry(in_flight); + let mut pass = RecoveryPass { + next_inventory: Some(Instant::now() + ANTI_ENTROPY_INTERVAL), + scan_generation: Some(scanning), + ..Default::default() + }; + pass.discover(&store, &CancellationToken::new()).await; + assert!( + hints.rescan().is_none(), + "attempted failures cannot invalidate the completed scan generation" + ); + let scheduled = pass.next_sweep.expect("periodic discovery remains scheduled"); + assert!(scheduled >= Instant::now() + ANTI_ENTROPY_INTERVAL - Duration::from_secs(1)); + pass.discover(&store, &CancellationToken::new()).await; + assert!(pass.scan_generation.is_none()); + assert_eq!(pass.next_sweep, Some(scheduled), "the next tick must wait for the hourly sweep"); + hints + .register(key("foreground-new-request")) + .expect("foreground overflow remains tracked"); + let foreground = hints + .rescan() + .expect("foreground eviction still requests immediate discovery"); + assert_ne!(foreground, scanning); + hints.ack_rescan(scanning); + assert_eq!(hints.rescan(), Some(foreground), "an older scan cannot clear a foreground obligation"); + } + + #[tokio::test] + async fn delete_recovery_slow_first_key_preserves_unattempted_backpressure() { + let bucket = "runtime-slow-first"; + let (_dirs, store, _disks) = fixture(bucket).await; + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity"); + let hints = store.ctx.delete_recovery_hints(); + for index in 0..HINT_CAPACITY { + hints.register(DeleteRecoveryKey { + pool: 0, + set: 0, + bucket: bucket.to_owned(), + object: format!("waiting-{index}"), + incarnation: Some(incarnation), + }); + } + let gate = store.ctx.data_movement_operation_gate(); + let held = gate.write().await; + tokio::time::pause(); + let worker_store = store.clone(); + let worker = tokio::spawn(async move { recover_hints(&worker_store, &CancellationToken::new()).await }); + for _ in 0..100 { + if hints.pending_len() != HINT_CAPACITY { + break; + } + tokio::task::yield_now().await; + } + assert_eq!(hints.pending_len(), HINT_CAPACITY - 1, "only the admitted key leaves the queue"); + assert!(hints.register_discovered(key("newly-discovered")), "one in-flight key releases one slot"); + assert!(!hints.register_discovered(key("must-wait")), "untouched keys must apply backpressure"); + tokio::time::advance(PASS_START_BUDGET + Duration::from_secs(1)).await; + drop(held); + worker.await.expect("finish the admitted operation"); + tokio::time::resume(); + let state = hints.state.lock().expect("retained pending work"); + for index in 1..HINT_CAPACITY { + let queued = state + .pending + .iter() + .find(|(key, _)| key.object == format!("waiting-{index}")) + .expect("unattempted key remains queued") + .1; + assert!(!queued.2, "budget exhaustion must not grant discovery permission to evict untouched work"); + } + } + + #[test] + fn delete_recovery_backpressure_eventually_attempts_every_healthy_key() { + for total in [4_096, 16_384] { + let hints = DeleteRecoveryHints::with_capacity(HINT_CAPACITY); + let mut recovered = HashSet::new(); + let mut next = 0usize; + let healthy = total - total / 1_024 * KEYS_PER_PASS; + for _ in 0..(total / KEYS_PER_PASS * 4) { + for _ in 0..(PAGES_PER_PASS * usize::try_from(DISCOVERY_PAGE_SIZE).expect("page size")) { + if !recovered.contains(&next) && !hints.register_discovered(key(&next.to_string())) { + break; + } + next = (next + 1) % total; + } + for ticket in hints.take(KEYS_PER_PASS) { + let index: usize = ticket.key.object.parse().expect("numeric test key"); + if index % 1_024 < KEYS_PER_PASS { + hints.retry(ticket); + } else { + recovered.insert(index); + } + } + assert!(hints.pending_len() <= HINT_CAPACITY); + if recovered.len() == healthy { + break; + } + } + assert_eq!( + recovered.len(), + healthy, + "permanent failures at every page-group boundary cannot starve later keys" + ); + } + } + + #[tokio::test] + async fn delete_recovery_real_single_and_batch_delete_register_and_ack_cleanup() { + for batch in [false, true] { + for held_snapshot in [false, true] { + let bucket = "runtime-real-delete"; + let (_dirs, store, disks) = fixture(bucket).await; + let mut reader = PutObjReader::from_vec(vec![37; 1024 * 1024]); + store + .put_object( + bucket, + "object", + &mut reader, + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("external source for snapshot cleanup"); + let _pass = hint_only_pass(&store); + let mut leases = Vec::new(); + if held_snapshot { + for disk in &disks { + let info = disk + .read_version("", bucket, "object", "", &crate::disk::ReadOptions::default()) + .await + .expect("source data directory"); + let path = format!("object/{}", info.data_dir.expect("external payload")); + let lease = disk.acquire_snapshot_lease(bucket, &path).await.expect("protect cleanup"); + leases.push((disk.clone(), path, lease)); + } + } + if batch { + let (_, errors) = store + .delete_objects( + bucket, + vec![crate::storage_api_contracts::object::ObjectToDelete { + object_name: "object".to_owned(), + ..Default::default() + }], + ObjectOptions::default(), + ) + .await; + assert!(errors.iter().all(Option::is_none), "logical batch delete succeeds: {errors:?}"); + } else { + store + .delete_object(bucket, "object", ObjectOptions::default()) + .await + .expect("logical single delete succeeds"); + } + assert_eq!( + store.ctx.delete_recovery_hints().pending_len(), + usize::from(held_snapshot), + "the real preparation hook retains cleanup; synchronous settlement acknowledges its ticket" + ); + if held_snapshot { + for (disk, path, lease) in leases { + disk.release_snapshot_lease(bucket, &path, lease) + .await + .expect("release deferred payload"); + } + recover_hints(&store, &CancellationToken::new()).await; + assert_eq!( + store.ctx.delete_recovery_hints().pending_len(), + 0, + "maintenance finishes the real DELETE without manual registration" + ); + } + for disk in &disks { + assert!( + disk.discover_delete_transactions(bucket, Some("object"), None, 128) + .await + .expect("no residual delete journal") + .transactions + .is_empty() + ); + } + } + } + } + + #[tokio::test] + async fn delete_recovery_continues_current_bucket_before_opening_later_buckets() { + let bucket = "runtime-cursor-progress"; + let (dirs, store, disks) = fixture(bucket).await; + let other = "runtime-after-large"; + store + .make_bucket(other, &MakeBucketOptions::default()) + .await + .expect("next bucket"); + for index in 0..4_130 { + std::fs::write(dirs[0].path().join(bucket).join(format!("filler-{index}")), []).expect("raw discovery visit budget"); + } + let mut pass = RecoveryPass::default(); + pass.scans.push_back(DiskScan { + source: LocalSource { + pool: 0, + set: 0, + disk: disks[0].clone(), + }, + buckets: Some(VecDeque::from([bucket, other].map(|bucket| BucketScan { + bucket: bucket.to_owned(), + incarnation: None, + cursor: None, + pending: VecDeque::new(), + }))), + }); + pass.discover_page(&store, Instant::now() + Duration::from_secs(30)).await; + let first = pass.scans[0] + .buckets + .as_ref() + .expect("pending buckets") + .front() + .expect("current bucket"); + assert_eq!(first.bucket, bucket); + assert!(first.cursor.is_some(), "raw entry budget must leave a continuation"); + pass.discover_page(&store, Instant::now() + Duration::from_secs(30)).await; + assert_eq!( + pass.scans[0] + .buckets + .as_ref() + .expect("remaining bucket") + .front() + .expect("next bucket") + .bucket, + other, + "the second page completes instead of cycling through unopened buckets" + ); + } +} diff --git a/crates/ecstore/src/store/heal.rs b/crates/ecstore/src/store/heal.rs index 7a8ee37bb..719196833 100644 --- a/crates/ecstore/src/store/heal.rs +++ b/crates/ecstore/src/store/heal.rs @@ -117,6 +117,47 @@ enum HealFormatPoolSkip { Retryable, } +#[derive(Debug)] +enum HealFormatRebalance { + Missing, + Current(RebalanceMeta), + Historical, +} + +impl HealFormatRebalance { + fn current(&self) -> Option<&RebalanceMeta> { + match self { + Self::Current(meta) => Some(meta), + Self::Missing | Self::Historical => None, + } + } +} + +#[derive(Debug, thiserror::Error)] +#[error( + "format repair blocked: rebalance metadata describes {stored_pools} pools but the deployment has {current_pools}; the historical run is not confirmed terminal" +)] +struct HealFormatRebalanceTopologyMismatch { + stored_pools: usize, + current_pools: usize, +} + +fn classify_heal_format_rebalance(meta: Option, current_pools: usize) -> Result { + let Some(meta) = meta else { + return Ok(HealFormatRebalance::Missing); + }; + if meta.pool_stats.len() == current_pools { + return Ok(HealFormatRebalance::Current(meta)); + } + if meta.has_terminal_topology() { + return Ok(HealFormatRebalance::Historical); + } + Err(Error::other(HealFormatRebalanceTopologyMismatch { + stored_pools: meta.pool_stats.len(), + current_pools, + })) +} + fn classify_heal_format_pool( pool_idx: usize, pool_cmd_line: &str, @@ -365,13 +406,18 @@ impl ECStore { } } } - if summary.pending > 0 { + if object.is_some() && summary.pending > 0 { return Err(StorageError::ErasureReadQuorum); } if object.is_some() && !opts.dry_run && summary.blocked > 0 { return Err(StorageError::SlowDown); } - if summary.discovered == 0 { + let delete_recovery = rustfs_madmin::heal_commands::DeleteRecoveryStatus { + pending: summary.pending, + blocked: summary.blocked, + failed: summary.failed, + }; + if summary.discovered == 0 && !delete_recovery.has_unresolved() { return Ok(None); } Ok(Some(HealResultItem { @@ -384,14 +430,17 @@ impl ECStore { object: object.or(prefix).unwrap_or_default(), bucket, detail: format!( - "delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}", + "delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}, pending={}, failed={}", if opts.dry_run { " (dry run)" } else { "" }, summary.discovered, summary.committed, summary.rolled_back, summary.reclaimed, - summary.blocked + summary.blocked, + summary.pending, + summary.failed ), + delete_recovery: Some(delete_recovery), ..Default::default() })) }) @@ -404,7 +453,17 @@ impl ECStore { expected: uuid::Uuid, opts: &HealOpts, ) -> Result { - self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move { + self.run_bucket_heal_at_incarnation(bucket, expected, opts, move |store, bucket, mut opts| async move { + if !opts.recreate && !opts.remove { + let mut missing = false; + for (pool, set) in store.heal_erasure_set_scopes(&opts).await? { + missing |= store.pools[pool].get_disks(set).bucket_needs_reconstruction(&bucket).await?; + } + if missing { + crate::bucket::metadata_sys::validate_bucket_reconstruction_in(&store.ctx, &bucket, expected).await?; + opts.recreate = true; + } + } store.heal_bucket(&bucket, &opts).await }) .await @@ -469,7 +528,7 @@ impl ECStore { NamespaceLockGuard, NamespaceLockGuard, PoolMeta, - Option, + HealFormatRebalance, )> { let metadata_pool = self .pools @@ -525,12 +584,9 @@ impl ECStore { Err(err) => return Err(err), }; - if rebalance_meta - .as_ref() - .is_some_and(|meta| meta.pool_stats.len() != self.pools.len()) - { - return Err(heal_format_fence_lost_error()); - } + // Keep historical status on disk, but never use its obsolete indices + // to gate the current pools. The current pool identity remains fenced. + let rebalance_meta = classify_heal_format_rebalance(rebalance_meta, self.pools.len())?; write_state.ensure_write_safe("heal format fence failed")?; if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { @@ -733,7 +789,8 @@ impl ECStore { first_error.get_or_insert(heal_format_fence_lost_error()); break; } - if let Some(skip) = classify_heal_format_pool(pool_idx, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.as_ref()) + if let Some(skip) = + classify_heal_format_pool(pool_idx, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.current()) { if matches!(skip, HealFormatPoolSkip::Completed) { count_completed += 1; @@ -807,6 +864,74 @@ impl ECStore { Ok((r, None)) } + /// Read-only admission for one legacy pre-format recovery handoff. The + /// caller holds the target execution leases and validates their owners. + pub async fn replacement_preformat_retry_is_safe( + &self, + pool_index: usize, + set_index: usize, + targets: &[String], + ) -> Result { + let pool = self + .pools + .get(pool_index) + .ok_or_else(|| invalid_heal_pool_index(pool_index, self.pools.len()))?; + let Some(set) = pool.disk_set.get(set_index) else { + return Err(StorageError::PreconditionFailed); + }; + let (write_state, pool_guard, rebalance_guard, pool_meta, rebalance_meta) = self.acquire_heal_format_fence().await?; + if !matches!(rebalance_meta, HealFormatRebalance::Historical) + || classify_heal_format_pool(pool_index, &pool.endpoints.cmd_line, &pool_meta, None).is_some() + || targets.is_empty() + { + return Ok(false); + } + + let mut slots = BTreeSet::new(); + let mut target_disks = Vec::with_capacity(targets.len()); + for target in targets { + let Some((slot, endpoint)) = set + .set_endpoints + .iter() + .enumerate() + .find(|(_, endpoint)| endpoint.to_string() == *target) + else { + return Ok(false); + }; + if !endpoint.is_local || !slots.insert(slot) { + return Ok(false); + } + // Open the current mounted target; a cached disk handle can still + // refer to the previous disk at the same endpoint. + let target_disk = disk::new_disk( + endpoint, + &disk::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await?; + if !target_disk.is_blank_replacement_target().await? { + return Ok(false); + } + target_disks.push(target_disk); + } + + let (_, error) = set + .heal_replacement_format_with_fence(true, targets, || pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost()) + .await?; + write_state.ensure_write_safe("pre-format recovery admission fence failed")?; + if pool_guard.is_lock_lost() || rebalance_guard.is_lock_lost() { + return Err(heal_format_fence_lost_error()); + } + for target_disk in target_disks { + if !target_disk.is_blank_replacement_target().await? { + return Ok(false); + } + } + Ok(error.is_none()) + } + #[instrument(skip(self, targets), fields(pool_index, set_index, target_count = targets.len()))] pub async fn heal_replacement_format( &self, @@ -828,7 +953,8 @@ impl ECStore { })?; let (mut write_state, pool_guard, rebalance_guard, pool_meta, rebalance_meta) = self.acquire_heal_format_fence().await?; - if let Some(skip) = classify_heal_format_pool(pool_index, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.as_ref()) { + if let Some(skip) = classify_heal_format_pool(pool_index, &pool.endpoints.cmd_line, &pool_meta, rebalance_meta.current()) + { return Ok((HealResultItem::default(), Some(heal_format_pool_skip_error(skip)))); } @@ -1451,6 +1577,178 @@ mod tests { .expect("minimal pool should build") } + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_recreates_missing_directories_at_current_incarnation() { + let (root, store, shutdown) = multi_pool_heal_store().await; + let bucket = "admin-bucket-recovery"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create a committed bucket generation"); + let incarnation = store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("persisted generation"); + for disk in 1..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("remove only an empty fixture bucket directory"); + } + store + .heal_bucket_at_incarnation(bucket, incarnation, &HealOpts::default()) + .await + .expect("default administrator recovery should use committed generation authority"); + for disk in 0..4 { + assert!(root.path().join(format!("pool0-disk{disk}/{bucket}")).is_dir()); + } + assert_eq!( + store + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("recovered generation"), + incarnation + ); + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_rejects_missing_corrupt_uncommitted_and_retired_authority() { + let (root, store, shutdown) = multi_pool_heal_store().await; + for state in ["missing", "corrupt", "uncommitted", "mismatched", "retired"] { + let bucket = format!("recovery-authority-{state}"); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create fixture generation"); + let incarnation = store + .bucket_incarnation_id_from_disk(&bucket) + .await + .expect("fixture identity"); + let (mut metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket) + .await + .expect("committed metadata"); + assert!(persisted && metadata.bucket_creation_committed); + let path = metadata.save_file_path(); + match state { + "missing" => delete_config(store.clone(), &path).await.expect("remove metadata authority"), + "corrupt" => save_config(store.clone(), &path, b"corrupt bucket metadata".to_vec()) + .await + .expect("corrupt authority"), + "uncommitted" | "mismatched" => { + if state == "uncommitted" { + metadata.bucket_creation_committed = false; + } else { + metadata.bucket_incarnation_id = Uuid::new_v4(); + } + metadata + .save_with_store_committed(store.clone()) + .await + .expect("inject inconsistent metadata authority"); + } + "retired" => { + // A returning disk may retain a directory after retirement was committed. + crate::bucket::retirement::commit_retirement(store.clone(), &bucket, incarnation, &ObjectOptions::default()) + .await + .expect("persist retirement evidence"); + } + _ => unreachable!("fixed fixture states"), + } + for disk in 1..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("lose fixture directories"); + } + assert!( + store + .heal_bucket_at_incarnation(&bucket, incarnation, &HealOpts::default()) + .await + .is_err(), + "must reject {state}" + ); + for disk in 1..4 { + assert!( + !root.path().join(format!("pool0-disk{disk}/{bucket}")).exists(), + "{state} cannot authorize reconstruction" + ); + } + } + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_rejects_offline_disks_and_fully_missing_sets() { + for offline in [false, true] { + let (root, store, shutdown) = multi_pool_heal_store().await; + let bucket = "recovery-unproven-set"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("fixture identity"); + let first_missing = usize::from(offline); + for disk in first_missing..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("remove fixture directories"); + } + if offline { + let disk = store.pools[0].disk_set[0].disks.read().await[0] + .clone() + .expect("online fixture disk"); + disk.close().await.expect("stop disk monitoring"); + disk.force_offline_for_test(); + } + assert!( + store + .heal_bucket_at_incarnation(bucket, incarnation, &HealOpts::default()) + .await + .is_err() + ); + for disk in first_missing..4 { + assert!(!root.path().join(format!("pool0-disk{disk}/{bucket}")).exists()); + } + shutdown.cancel(); + } + } + + #[tokio::test] + #[serial_test::serial] + async fn admin_bucket_recovery_dry_run_preserves_missing_directories() { + let (root, store, shutdown) = multi_pool_heal_store().await; + let bucket = "recovery-dry-run"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("fixture identity"); + for disk in 1..4 { + tokio::fs::remove_dir(root.path().join(format!("pool0-disk{disk}/{bucket}"))) + .await + .expect("remove fixture directories"); + } + store + .heal_bucket_at_incarnation( + bucket, + incarnation, + &HealOpts { + dry_run: true, + ..Default::default() + }, + ) + .await + .expect("dry run can inspect eligible recovery"); + for disk in 1..4 { + assert!( + !root.path().join(format!("pool0-disk{disk}/{bucket}")).exists(), + "dry run must not reconstruct directories" + ); + } + shutdown.cancel(); + } + #[tokio::test] #[serial_test::serial] async fn bucket_incarnation_heal_preserves_successor_shards_and_allows_fresh_repair() { @@ -2151,9 +2449,13 @@ mod tests { } async fn multi_pool_heal_store() -> (tempfile::TempDir, Arc, CancellationToken) { + heal_store_with_pool_count(2).await + } + + async fn heal_store_with_pool_count(pool_count: usize) -> (tempfile::TempDir, Arc, CancellationToken) { let temp_dir = tempfile::tempdir().expect("multi-pool heal test directory should be created"); let mut pool_endpoints = Vec::new(); - for pool_index in 0..2 { + for pool_index in 0..pool_count { let mut endpoints = Vec::new(); for disk_index in 0..4 { let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}")); @@ -2470,6 +2772,224 @@ mod tests { ); } + fn completed_two_pool_rebalance() -> RebalanceMeta { + RebalanceMeta { + id: Uuid::new_v4().to_string(), + pool_stats: vec![ + RebalanceStats { + participating: true, + info: RebalanceInfo { + status: RebalStatus::Completed, + ..Default::default() + }, + ..Default::default() + }, + RebalanceStats::default(), + ], + ..Default::default() + } + } + + #[test] + fn historical_rebalance_topology_requires_every_pool_to_be_terminal() { + use crate::services::rebalance::{RebalanceStopPropagationRecord, encode_rebalance_stop_propagation_record}; + + assert!(matches!( + classify_heal_format_rebalance(Some(completed_two_pool_rebalance()), 1), + Ok(HealFormatRebalance::Historical) + )); + // Expansion has the same historical-index problem as removal. + assert!(matches!( + classify_heal_format_rebalance(Some(completed_two_pool_rebalance()), 3), + Ok(HealFormatRebalance::Historical) + )); + assert!(matches!(classify_heal_format_rebalance(None, 1), Ok(HealFormatRebalance::Missing))); + for status in [RebalStatus::Completed, RebalStatus::Stopped, RebalStatus::Failed] { + let mut meta = completed_two_pool_rebalance(); + meta.pool_stats[0].info.status = status; + assert!(meta.has_terminal_topology(), "terminal status {status:?}"); + } + + let mut invalid = Vec::new(); + for index in 0..2 { + let mut active = completed_two_pool_rebalance(); + active.stopped_at = Some(OffsetDateTime::UNIX_EPOCH); + active.pool_stats[index].info.status = RebalStatus::Started; + invalid.push(active); + let mut stopping = completed_two_pool_rebalance(); + stopping.pool_stats[index].info.stopping = true; + invalid.push(stopping); + } + let mut unknown_participant = completed_two_pool_rebalance(); + unknown_participant.pool_stats[0].info.status = RebalStatus::None; + invalid.push(unknown_participant); + let mut no_id = completed_two_pool_rebalance(); + no_id.id.clear(); + invalid.push(no_id); + invalid.push(RebalanceMeta::default()); + for record in [ + RebalanceStopPropagationRecord { + stop_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + }, + RebalanceStopPropagationRecord { + stop_failures: vec!["peer".to_string()], + ..Default::default() + }, + RebalanceStopPropagationRecord { + terminal_reload_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + terminal_reload_failures: vec!["peer".to_string()], + ..Default::default() + }, + ] { + let mut meta = completed_two_pool_rebalance(); + meta.pool_stats[1].info.last_error = Some(encode_rebalance_stop_propagation_record(&record)); + invalid.push(meta); + } + let mut malformed = completed_two_pool_rebalance(); + malformed.pool_stats[0].info.last_error = Some("rebalance stop propagation incomplete: invalid".to_string()); + invalid.push(malformed); + for meta in invalid { + let error = classify_heal_format_rebalance(Some(meta), 1).expect_err("uncertain history must block repair"); + assert!(error.to_string().contains("not confirmed terminal")); + assert!(!error.to_string().contains("request rate")); + } + let mut acknowledged = completed_two_pool_rebalance(); + acknowledged.pool_stats[0].info.last_error = + Some(encode_rebalance_stop_propagation_record(&RebalanceStopPropagationRecord { + stop_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + terminal_reload_attempt_at: Some(OffsetDateTime::UNIX_EPOCH), + ..Default::default() + })); + assert!(acknowledged.has_terminal_topology()); + } + + #[tokio::test] + #[serial_test::serial] + async fn replacement_format_after_historical_pool_removal() { + let (temp_dir, store, shutdown) = heal_store_with_pool_count(1).await; + let mut meta = completed_two_pool_rebalance(); + meta.save(store.pools[0].clone()) + .await + .expect("persist historical two-pool run"); + let (before, _) = read_config_no_lock_preserve_empty_with_metadata(store.pools[0].clone(), REBAL_META_NAME) + .await + .expect("read historical run before repair"); + let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await; + + // Even a worker in a removed pool must fence every current format write. + meta.pool_stats[1].participating = true; + meta.pool_stats[1].info.status = RebalStatus::Started; + meta.save(store.pools[0].clone()) + .await + .expect("persist active historical pool"); + let err = store + .heal_replacement_format(false, 0, 0, std::slice::from_ref(&target)) + .await + .expect_err("removed active source must block format repair"); + assert!(err.to_string().contains("2 pools but the deployment has 1")); + assert_heal_test_format_missing(&temp_dir, 0, 3).await; + meta.pool_stats[1] = RebalanceStats::default(); + meta.save(store.pools[0].clone()) + .await + .expect("persist terminal history again"); + + let (result, error) = store + .heal_replacement_format(false, 0, 0, std::slice::from_ref(&target)) + .await + .expect("terminal historical topology must allow target format repair"); + assert!(error.is_none(), "target repair: {error:?}"); + assert!(result.after.drives.iter().all(|drive| !drive.uuid.is_empty())); + let disk = store.pools[0].disk_set[0].disks.read().await[3] + .clone() + .expect("replacement disk"); + let repaired = load_format_erasure(&disk, true).await.expect("read repaired target format"); + assert!(!repaired.erasure.this.is_nil()); + + remove_heal_test_format(&temp_dir, &store, 0, 3).await; + let (_, error) = store + .handle_heal_format(false) + .await + .expect("full format heal uses the same historical fence"); + assert!(error.is_none(), "full format repair: {error:?}"); + assert!(heal_test_format_path(&temp_dir, 0, 3).exists()); + let (after, _) = read_config_no_lock_preserve_empty_with_metadata(store.pools[0].clone(), REBAL_META_NAME) + .await + .expect("historical status remains readable"); + assert_eq!(before, after, "format repair must not rewrite historical rebalance status"); + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial] + async fn preformat_retry_requires_historical_topology_and_a_blank_target() { + let (temp_dir, store, shutdown) = heal_store_with_pool_count(1).await; + completed_two_pool_rebalance() + .save(store.pools[0].clone()) + .await + .expect("persist historical two-pool run"); + let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await; + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, std::slice::from_ref(&target)) + .await + .expect("inspect target with old metadata"), + "missing format alone must not admit a disk containing old metadata" + ); + let target_path = temp_dir.path().join("pool0-disk3"); + for entry in std::fs::read_dir(&target_path).expect("list target fixture") { + let entry = entry.expect("target fixture entry"); + if entry.file_type().expect("fixture entry type").is_dir() { + std::fs::remove_dir_all(entry.path()).expect("clear replacement fixture directory"); + } else { + std::fs::remove_file(entry.path()).expect("clear replacement fixture file"); + } + } + let target_disk = store.pools[0].disk_set[0].disks.read().await[3] + .clone() + .expect("replacement disk"); + if !cfg!(target_os = "linux") { + assert!(target_disk.acquire_replacement_execution_lease().await.is_err()); + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, &[target]) + .await + .expect("platforms without pinned mount identity must reject compatibility recovery") + ); + shutdown.cancel(); + return; + } + let _lease = target_disk + .acquire_replacement_execution_lease() + .await + .expect("own replacement executor"); + assert!( + store + .replacement_preformat_retry_is_safe(0, 0, std::slice::from_ref(&target)) + .await + .expect("inspect blank target") + ); + assert_heal_test_format_missing(&temp_dir, 0, 3).await; + std::fs::create_dir(target_path.join("user-bucket")).expect("simulate residual user bucket"); + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, std::slice::from_ref(&target)) + .await + .expect("inspect nonblank target") + ); + std::fs::remove_dir(target_path.join("user-bucket")).expect("remove fixture bucket"); + delete_config(store.pools[0].clone(), REBAL_META_NAME) + .await + .expect("remove historical run"); + assert!( + !store + .replacement_preformat_retry_is_safe(0, 0, &[target]) + .await + .expect("ordinary replacement has no compatibility authority") + ); + shutdown.cancel(); + } + #[tokio::test] #[serial_test::serial] async fn replacement_format_heal_coexists_with_capacity_read_fence() { @@ -3762,7 +4282,6 @@ mod tests { let mut recoverable_temp_dirs = Vec::new(); let mut recoverable_endpoints = Vec::new(); let mut recoverable_disks = Vec::new(); - let mut unformatted_disk = None; for disk_index in 0..3 { let temp_dir = tempfile::tempdir().expect("temporary disk root should be created"); let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("temporary path should be UTF-8")) @@ -3779,15 +4298,11 @@ mod tests { ) .await .expect("temporary disk should open"); - if disk_index < 2 { - let mut disk_format = recoverable_format.clone(); - disk_format.erasure.this = recoverable_format.erasure.sets[0][disk_index]; - save_format_file(&Some(disk.clone()), &Some(disk_format)) - .await - .expect("recoverable format should be written"); - } else { - unformatted_disk = Some(disk.clone()); - } + let mut disk_format = recoverable_format.clone(); + disk_format.erasure.this = recoverable_format.erasure.sets[0][disk_index]; + save_format_file(&Some(disk.clone()), &Some(disk_format)) + .await + .expect("recoverable format should be written before pool metadata"); recoverable_temp_dirs.push(temp_dir); recoverable_endpoints.push(endpoint); recoverable_disks.push(Some(disk)); @@ -3836,6 +4351,36 @@ mod tests { .expect("pool metadata should be persisted before format heal"); *store.pool_meta_save_gate.lock().await = PoolMetaWriteState::default(); + let replacement_endpoint = store.pools[1].endpoints.endpoints.as_ref()[2].clone(); + tokio::fs::remove_file( + recoverable_temp_dirs[2] + .path() + .join(crate::disk::RUSTFS_META_BUCKET) + .join(FORMAT_CONFIG_FILE), + ) + .await + .expect("the target format must be removed only after durable pool metadata exists"); + let replacement_disk = new_disk( + &replacement_endpoint, + &DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("the unformatted replacement disk must reopen"); + assert!(matches!( + load_format_erasure(&replacement_disk, true).await, + Err(crate::disk::error::DiskError::UnformattedDisk) + )); + let replacement_set = &store.pools[1].disk_set[0]; + let replacement_slot = replacement_set + .set_endpoints + .iter() + .position(|endpoint| endpoint == &replacement_endpoint) + .expect("the replacement must remain in its original set"); + replacement_set.disks.write().await[replacement_slot] = Some(replacement_disk.clone()); + let (result, err) = store .handle_heal_format(false) .await @@ -3845,7 +4390,7 @@ mod tests { "foreign format majority must not be downgraded to a successful heal: {err:?}" ); assert_eq!(result.disk_count, 3, "the recoverable pool should still be inspected"); - let healed = load_format_erasure(&unformatted_disk.expect("the unformatted disk handle should be retained"), true) + let healed = load_format_erasure(&replacement_disk, true) .await .expect("the later pool should be healed despite the first pool error"); assert_eq!(healed.erasure.this, recoverable_format.erasure.sets[0][2]); diff --git a/crates/ecstore/src/store/init.rs b/crates/ecstore/src/store/init.rs index 86cf16be7..3786aa60d 100644 --- a/crates/ecstore/src/store/init.rs +++ b/crates/ecstore/src/store/init.rs @@ -831,6 +831,8 @@ impl ECStore { } }); + super::delete_recovery_runtime::start_delete_recovery(self.clone(), rx.clone()); + runtime_sources::init_bucket_monitor_for_current_endpoints(); crate::bucket::bucket_target_sys::BucketTargetSys::get().start_heartbeat(); @@ -875,6 +877,7 @@ mod tests { run_local_decommission_watchdog, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init, should_defer_rebalance_auto_start, should_retry_format_load, wait_for_local_decommission_resume_delay, }; + use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG; use crate::core::pools::PoolMetaBootstrapAuthority; #[cfg(feature = "test-util")] use crate::disk::DiskAPI; @@ -936,7 +939,7 @@ mod tests { }, validate_durable_ilm_record, }, - bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_VERSIONING_CONFIG}, + bucket::metadata::BUCKET_VERSIONING_CONFIG, config::com, core::pools::DecomBucketInfo, data_movement::SourceCleanupDeleteBarrier, @@ -1092,6 +1095,9 @@ mod tests { revision: AtomicUsize, pool_meta_write_attempts: AtomicUsize, fail_pool_meta_write_at: AtomicUsize, + identity_repair_failures: AtomicUsize, + identity_write_attempts: AtomicUsize, + identity_unconverged: AtomicBool, pool_meta_written_versions: Mutex>, objects: Mutex, String)>>, } @@ -1112,6 +1118,9 @@ mod tests { revision: AtomicUsize::new(0), pool_meta_write_attempts: AtomicUsize::new(0), fail_pool_meta_write_at: AtomicUsize::new(0), + identity_repair_failures: AtomicUsize::new(0), + identity_write_attempts: AtomicUsize::new(0), + identity_unconverged: AtomicBool::new(false), pool_meta_written_versions: Mutex::new(Vec::new()), objects: Mutex::new(objects), } @@ -1128,6 +1137,9 @@ mod tests { revision: AtomicUsize::new(0), pool_meta_write_attempts: AtomicUsize::new(0), fail_pool_meta_write_at: AtomicUsize::new(0), + identity_repair_failures: AtomicUsize::new(0), + identity_write_attempts: AtomicUsize::new(0), + identity_unconverged: AtomicBool::new(false), pool_meta_written_versions: Mutex::new(Vec::new()), objects: Mutex::new(HashMap::new()), } @@ -1242,6 +1254,17 @@ mod tests { .lock() .unwrap_or_else(std::sync::PoisonError::into_inner) .insert(object.to_string(), (payload, etag.clone())); + if object == POOL_META_IDENTITY_NAME { + self.identity_write_attempts.fetch_add(1, Ordering::SeqCst); + let partial = self + .identity_repair_failures + .try_update(Ordering::SeqCst, Ordering::SeqCst, |remaining| remaining.checked_sub(1)) + .is_ok(); + self.identity_unconverged.store(partial, Ordering::SeqCst); + if partial { + return Err(Error::other(crate::set_disk::SystemMetadataRepairRequired)); + } + } Ok(self.object_info(bucket, object, size, etag)) } } @@ -1748,6 +1771,220 @@ mod tests { assert!(startup_object(&pool0, POOL_META_NAME).is_none()); } + #[tokio::test] + async fn test_four_pool_expansion_repairs_missing_identity_disk_without_data_change() { + use crate::disk::{DiskAPI, RUSTFS_META_BUCKET, ReadOptions}; + use crate::set_disk::hermetic_set_disks_isolated; + + let mut dirs = Vec::new(); + let mut disks = Vec::new(); + let mut pools = Vec::new(); + for _ in 0..4 { + let (pool_dirs, pool_disks, pool) = hermetic_set_disks_isolated(4).await; + dirs.extend(pool_dirs); + disks.push(pool_disks); + pools.push(pool); + } + let deployment_id = Uuid::new_v4(); + let mut founder = PoolMetaWriteState::for_startup(deployment_id, true); + establish_pool_meta_bootstrap_identity_if_proven(pools[..3].to_vec(), &mut founder, true) + .await + .expect("seed three healthy pool identities"); + let (_, replicas) = load_pool_meta_for_startup(pools[..3].to_vec(), &mut founder) + .await + .expect("load the original deployment"); + persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(3), + pools[..3].to_vec(), + replicas, + &mut founder, + true, + true, + ) + .await + .expect("commit three healthy pools"); + let identity = crate::config::com::read_config_no_lock(pools[0].clone(), POOL_META_IDENTITY_NAME) + .await + .expect("read the initialized deployment identity"); + let bucket = "identity-repair-data"; + for disk in &disks[0] { + disk.make_volume(bucket).await.expect("create the data bucket"); + } + let original = vec![0x73; 256 * 1024]; + pools[0] + .put_object( + bucket, + "retained", + &mut PutObjReader::from_vec(original.clone()), + &ObjectOptions { + write_completion: crate::object_api::WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("seed user bytes before expansion"); + + // Model the startup snapshot containing a missing disk handle, while + // the other three disks can commit and serve the new pool identity. + let missing = pools[3].disks.write().await[0].take().expect("healthy fixture disk"); + let startup = tokio::spawn({ + let pools = pools.clone(); + async move { + let mut owner = PoolMetaWriteState::for_startup_with_pool_bootstrap_authorities( + deployment_id, + vec![PoolMetaBootstrapAuthority::None; 4], + true, + ); + let (_, replicas) = load_pool_meta_for_startup(pools.clone(), &mut owner) + .await + .expect("load the three-pool predecessor"); + let committed = persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(4), + pools, + replicas, + &mut owner, + true, + true, + ) + .await + .expect("startup must recover the partial identity and commit expansion"); + owner + .ensure_write_safe("expanded startup") + .expect("repaired startup remains writable"); + committed + } + }); + tokio::time::timeout(std::time::Duration::from_secs(30), async { + loop { + let mut committed = 0; + for disk in &disks[3][1..] { + if disk + .read_version("", RUSTFS_META_BUCKET, POOL_META_IDENTITY_NAME, "", &ReadOptions::default()) + .await + .is_ok() + { + committed += 1; + } + } + if committed == 3 { + break; + } + assert!(!startup.is_finished(), "startup must reach the partial-commit barrier"); + tokio::task::yield_now().await; + } + }) + .await + .expect("three disks must physically commit the identity"); + assert!( + disks[3][0] + .read_version("", RUSTFS_META_BUCKET, POOL_META_IDENTITY_NAME, "", &ReadOptions::default()) + .await + .is_err() + ); + assert_eq!( + crate::config::com::read_config_no_lock(pools[3].clone(), POOL_META_IDENTITY_NAME) + .await + .expect("quorum can already read the unhealed identity"), + identity + ); + assert!(!startup.is_finished(), "quorum readback must not acknowledge unrepaired identity"); + pools[3].disks.write().await[0] = Some(missing); + let committed = tokio::time::timeout(std::time::Duration::from_secs(30), startup) + .await + .expect("startup must finish after reconnect") + .expect("startup task must join"); + assert_eq!(committed.pools.len(), 4); + let metadata = crate::config::com::read_config_no_lock(pools[0].clone(), POOL_META_NAME) + .await + .expect("read committed expanded topology"); + for (pool, pool_disks) in pools.iter().zip(&disks) { + for (object, payload) in [(POOL_META_IDENTITY_NAME, &identity), (POOL_META_NAME, &metadata)] { + assert_eq!( + crate::config::com::read_config_no_lock(pool.clone(), object) + .await + .expect("read every pool"), + *payload + ); + let reference = pool_disks[0] + .read_version("", RUSTFS_META_BUCKET, object, "", &ReadOptions::default()) + .await + .expect("metadata exists on the first disk"); + for disk in &pool_disks[1..] { + let actual = disk + .read_version("", RUSTFS_META_BUCKET, object, "", &ReadOptions::default()) + .await + .expect("metadata must exist on every physical disk"); + assert_eq!(actual.metadata.get("etag"), reference.metadata.get("etag")); + assert_eq!(actual.mod_time, reference.mod_time); + } + } + } + let mut reader = pools[0] + .get_object_reader(bucket, "retained", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("user data remains readable after expansion"); + let mut actual = Vec::new(); + reader.stream.read_to_end(&mut actual).await.expect("read all retained bytes"); + assert_eq!(actual, original); + } + + #[tokio::test(start_paused = true)] + async fn test_four_pool_expansion_repairs_identity_before_completing_startup() { + let deployment_id = Uuid::new_v4(); + let mut pools = (0..3) + .map(|_| Arc::new(StartupPoolMetaStorage::new(Vec::new()))) + .collect::>(); + let mut founder = PoolMetaWriteState::for_startup(deployment_id, true); + establish_pool_meta_bootstrap_identity_if_proven(pools.clone(), &mut founder, true) + .await + .expect("seed the three-pool deployment identity"); + let (_, replicas) = load_pool_meta_for_startup(pools.clone(), &mut founder) + .await + .expect("load the fresh deployment"); + persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(3), + pools.clone(), + replicas, + &mut founder, + true, + true, + ) + .await + .expect("commit the three-pool deployment"); + let identity = startup_object(&pools[0], POOL_META_IDENTITY_NAME).expect("initialized identity"); + let new_pool = Arc::new(StartupPoolMetaStorage::new(Vec::new())); + new_pool.identity_repair_failures.store(1, Ordering::SeqCst); + pools.push(new_pool.clone()); + let mut owner = PoolMetaWriteState::for_startup_with_pool_bootstrap_authorities( + deployment_id, + vec![PoolMetaBootstrapAuthority::None; 4], + true, + ); + let (_, replicas) = load_pool_meta_for_startup(pools.clone(), &mut owner) + .await + .expect("read the committed predecessor during expansion"); + let committed = persist_pool_meta_for_startup_if_safe( + &init_test_pool_meta_with_pools(4), + pools.clone(), + replicas, + &mut owner, + true, + true, + ) + .await + .expect("the owning startup must repair its partial identity write before publishing expansion"); + assert_eq!(committed.pools.len(), 4); + assert_eq!(new_pool.identity_write_attempts.load(Ordering::SeqCst), 2); + assert!(!new_pool.identity_unconverged.load(Ordering::SeqCst)); + owner + .ensure_write_safe("expanded startup") + .expect("verified repair must not latch the write gate"); + for pool in &pools { + assert_eq!(startup_object(pool, POOL_META_IDENTITY_NAME).as_ref(), Some(&identity)); + assert_eq!(startup_object(pool, POOL_META_NAME), startup_object(&pools[0], POOL_META_NAME)); + } + } + #[tokio::test] async fn test_fresh_pool_joining_an_initialized_deployment_never_reopens_bootstrap() { let deployment_id = Uuid::new_v4(); @@ -2714,6 +2951,106 @@ mod tests { (instance_ctx, store, shutdown) } + /// CopyObject destinations and CreateMultipartUpload create keys, so they + /// must reject exactly the keys PutObject rejects, with the same error. + /// The copy path reaches the pools' put_object directly and used to admit + /// keys PutObject refused (e.g. over 1024 bytes, or Windows-unaddressable). + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn copy_destination_and_multipart_create_share_put_object_admission() { + let temp_dir = tempfile::tempdir().expect("create key-admission store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "key-admission", &[4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + + let bucket = format!("key-admission-{}", Uuid::new_v4()); + let source_object = "source.bin"; + let payload = b"key admission source".to_vec(); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("key-admission bucket should be created"); + store + .put_object( + &bucket, + source_object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions::default(), + ) + .await + .expect("source object should be written"); + let source_info = store + .get_object_info(&bucket, source_object, &ObjectOptions::default()) + .await + .expect("source metadata should be readable"); + + // Every segment fits NAME_MAX, but the key exceeds S3's 1024 bytes. + let too_long = vec!["s".repeat(210); 5].join("/"); + let mut rejected = vec![too_long]; + if cfg!(target_os = "windows") { + rejected.extend(["a:b", "a\\b", "NUL.txt", "tab\tname"].map(str::to_owned)); + } + for dst_object in &rejected { + let put_err = store + .put_object( + &bucket, + dst_object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions::default(), + ) + .await + .expect_err("PutObject must reject the key"); + let mut copy_source = source_info.clone(); + copy_source.put_object_reader = Some(PutObjReader::from_vec(payload.clone())); + let copy_err = store + .copy_object( + &bucket, + source_object, + &bucket, + dst_object, + &mut copy_source, + &ObjectOptions::default(), + &ObjectOptions::default(), + ) + .await + .expect_err("CopyObject must reject the destination key"); + assert_eq!(format!("{copy_err:?}"), format!("{put_err:?}"), "{dst_object:?}"); + let multipart_err = store + .new_multipart_upload(&bucket, dst_object, &ObjectOptions::default()) + .await + .expect_err("CreateMultipartUpload must reject the key"); + assert_eq!(format!("{multipart_err:?}"), format!("{put_err:?}"), "{dst_object:?}"); + } + + // A segment over NAME_MAX used to be admitted by CreateMultipartUpload + // and only fail at completion with ENAMETOOLONG (rustfs#5785). + let oversized_segment = "m".repeat(256); + assert!(matches!( + store + .new_multipart_upload(&bucket, &oversized_segment, &ObjectOptions::default()) + .await, + Err(StorageError::ObjectNameInvalid(_, _)) + )); + + // In-place copies rewrite an existing key and keep the copy-only check. + let mut in_place = source_info.clone(); + in_place.metadata_only = true; + store + .copy_object( + &bucket, + source_object, + &bucket, + source_object, + &mut in_place, + &ObjectOptions::default(), + &ObjectOptions::default(), + ) + .await + .expect("in-place copy of an existing key should succeed"); + + shutdown.cancel(); + } + #[cfg(feature = "test-util")] #[tokio::test] #[serial_test::serial(storage_class_env)] @@ -3560,7 +3897,7 @@ mod tests { #[cfg(feature = "test-util")] #[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[serial_test::serial(storage_class_env)] - async fn early_ack_put_tails_block_scanner_publication_until_all_renames_finish() { + async fn early_ack_put_tails_shade_buckets_without_fencing_scanner_publication() { use crate::storage_api_contracts::namespace::NamespaceLocking as _; let temp_dir = tempfile::tempdir().expect("create scanner PUT tail store dir"); @@ -3636,31 +3973,28 @@ mod tests { assert!(!active, "ordinary PUT tails are not decommission or rebalance work"); assert!(!blocked, "ordinary PUT tails must not block the movement-only scan baseline"); assert_eq!(observed_movement_generation, movement_generation); - assert!(store.scanner_data_usage_publication_blocked().await); + assert!( + !store.scanner_data_usage_publication_blocked().await, + "ordinary PUT tails shade their bucket instead of fencing the local usage publication" + ); assert!(store.scanner_data_usage_publication_admission_guard().await.is_some()); assert!(set.scanner_data_usage_publication_admission_guard().await.is_some()); - for error in [ - store - .acquire_scanner_publication_lease( - movement_generation, - crate::runtime::instance::SCANNER_PUBLICATION_LEASE_TTL, - ) - .await - .expect_err("a new remote publication lease must reject pending PUT tails"), - store - .validate_scanner_publication_lease(old_lease, movement_generation) - .await - .expect_err("an existing remote lease must not bypass pending PUT tails"), + let (tail_lease, tail_generation) = store + .acquire_scanner_publication_lease(movement_generation, crate::runtime::instance::SCANNER_PUBLICATION_LEASE_TTL) + .await + .expect("a remote publication lease is granted while PUT tails are pending"); + assert_eq!(tail_generation, movement_generation); + store + .validate_scanner_publication_lease(old_lease, movement_generation) + .await + .expect("an existing remote lease validates across pending PUT tails"); + drop( store .acquire_scanner_publication_lease_guard(old_lease) .await - .expect_err("target-side publication admission must reject pending PUT tails"), - ] { - assert!( - error.to_string().contains("blocked"), - "publication must fail because of active tails: {error}" - ); - } + .expect("target-side publication admission accepts pending PUT tails"), + ); + assert!(store.release_scanner_publication_lease(tail_lease).await); store.release_scanner_publication_lease(old_lease).await; for (index, barrier) in barriers.iter().enumerate() { @@ -3682,7 +4016,7 @@ mod tests { assert!(store.scanner_namespace_mutation_generation() > namespace_generation); let pending = index + 1 < barriers.len(); assert_eq!(ctx.namespace_commits_pending(), pending); - assert_eq!(store.scanner_data_usage_publication_blocked().await, pending); + assert!(!store.scanner_data_usage_publication_blocked().await); assert!(!store.scanner_data_movement_activity().await.1); assert!(store.scanner_data_usage_publication_admission_guard().await.is_some()); assert!(set.scanner_data_usage_publication_admission_guard().await.is_some()); @@ -3801,7 +4135,7 @@ mod tests { #[cfg(feature = "test-util")] #[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[serial_test::serial(storage_class_env)] - async fn cancelled_early_ack_put_keeps_scanner_publication_blocked_until_tail_finishes() { + async fn cancelled_early_ack_put_keeps_namespace_commit_pending_until_tail_finishes() { let temp_dir = tempfile::tempdir().expect("create cancelled scanner PUT tail store dir"); let (ctx, store, shutdown) = without_storage_class_env(build_isolated_test_store(temp_dir.path(), "scanner-cancelled-put-tail", &[4])).await; @@ -3852,7 +4186,7 @@ mod tests { assert!(tracker.running() >= 1); assert!(ctx.namespace_commits_pending()); assert!(!store.scanner_data_movement_activity().await.1); - assert!(store.scanner_data_usage_publication_blocked().await); + assert!(!store.scanner_data_usage_publication_blocked().await); assert!(store.scanner_data_usage_publication_admission_guard().await.is_some()); assert!( store.pools[0].disk_set[0] @@ -4163,6 +4497,178 @@ mod tests { OfflineTestDisks { disks } } + #[cfg(feature = "test-util")] + async fn durable_quota_writes_with_one_offline_disk(sharded: bool) { + use crate::bucket::quota::BucketQuota; + use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause}; + + let temp_dir = tempfile::tempdir().expect("create degraded quota store dir"); + let (ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "degraded-quota", &[4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = format!("degraded-quota-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create degraded quota bucket"); + let quota = if sharded { + BucketQuota::new_sharded(16_384) + } else { + BucketQuota::new(Some(16_384)) + }; + crate::bucket::metadata_sys::update_in( + &ctx, + &bucket, + crate::bucket::metadata::BUCKET_QUOTA_CONFIG_FILE, + serde_json::to_vec("a).expect("encode durable quota"), + ) + .await + .expect("configure durable quota before losing a disk"); + + let set = &store.pools[0].disk_set[0]; + assert_eq!(set.disks.read().await.len(), 4); + let offline_disk = force_set_disk_range_offline_for_test(set, 0..1).await; + let _without_proof = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_none()); + + let payload = vec![0x51; 4096]; + let mut reader = PutObjReader::from_vec(payload.clone()); + store + .put_object(&bucket, "put", &mut reader, &ObjectOptions::default()) + .await + .expect("three healthy disks must admit a quota PUT without fleet proof"); + + let upload = store + .new_multipart_upload(&bucket, "multipart", &ObjectOptions::default()) + .await + .expect("create multipart upload with one disk offline"); + let mut reader = PutObjReader::from_vec(payload.clone()); + let part = store + .put_object_part(&bucket, "multipart", &upload.upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + .expect("stage multipart part with one disk offline"); + Arc::clone(&store) + .complete_multipart_upload( + &bucket, + "multipart", + &upload.upload_id, + vec![crate::storage_api_contracts::multipart::CompletePart { + part_num: part.part_num, + etag: part.etag, + ..Default::default() + }], + &ObjectOptions::default(), + ) + .await + .expect("three healthy disks must complete a quota multipart upload without fleet proof"); + + let source = store + .get_object_reader(&bucket, "put", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("read degraded copy source"); + let mut copy_info = source.object_info.clone(); + let size = copy_info.get_actual_size().expect("copy source logical size"); + copy_info.put_object_reader = Some(PutObjReader::new( + rustfs_rio::HashReader::from_stream(source.stream, size, size, None, None, false) + .expect("construct copy source reader"), + )); + store + .copy_object( + &bucket, + "put", + &bucket, + "copy", + &mut copy_info, + &ObjectOptions::default(), + &ObjectOptions::default(), + ) + .await + .expect("three healthy disks must copy into a quota bucket without fleet proof"); + + // Hold the last available credit before publication so the competing + // writer must account for an outstanding reservation, not just objects. + let barrier = PutObjectCommitBarrier::install(&bucket, "reserved", PutObjectCommitPause::BeforeQuotaRename); + let first_store = Arc::clone(&store); + let first_bucket = bucket.clone(); + let first = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(vec![0x52; 4096]); + first_store + .put_object(&first_bucket, "reserved", &mut reader, &ObjectOptions::default()) + .await + }); + tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused()) + .await + .expect("the last within-limit writer must reach its commit barrier"); + let mut reader = PutObjReader::from_vec(vec![0x53; 4096]); + let denied = store + .put_object(&bucket, "over-limit", &mut reader, &ObjectOptions::default()) + .await + .expect_err("an outstanding reservation must prevent concurrent quota oversubscription"); + assert!(matches!(denied, StorageError::QuotaExceeded { limit: 16_384, .. })); + barrier.release(); + first + .await + .expect("reserved PUT task should not panic") + .expect("the reserved writer must commit while one disk is offline"); + + for object in ["put", "multipart", "copy", "reserved"] { + let mut reader = store + .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("degraded quota writes must be readable"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("read committed quota object"); + let expected = if object == "reserved" { + vec![0x52; 4096] + } else { + payload.clone() + }; + assert_eq!(body, expected, "committed body for {object}"); + } + store + .get_object_info(&bucket, "over-limit", &ObjectOptions::default()) + .await + .expect_err("the over-limit writer must not publish an object"); + + let second_offline_disk = force_set_disk_range_offline_for_test(set, 1..2).await; + let mut reader = PutObjReader::from_vec(vec![0x54; 4096]); + let denied = store + .put_object(&bucket, "put", &mut reader, &ObjectOptions::default()) + .await + .expect_err("a zero-growth overwrite must still fail when the actual write quorum is lost"); + assert!( + matches!( + denied, + StorageError::ErasureWriteQuorum + | StorageError::ErasureReadQuorum + | StorageError::InsufficientWriteQuorum(_, _) + | StorageError::InsufficientReadQuorum(_, _) + ), + "loss of actual disk quorum must return a storage quorum error, got {denied:?}" + ); + shutdown.cancel(); + drop(second_offline_disk); + drop(offline_disk); + } + + #[cfg(feature = "test-util")] + #[test] + #[serial_test::serial(storage_class_env)] + fn durable_quota_v1_writes_with_one_offline_disk_and_no_fleet_proof() { + run_large_stack_async_test("degraded-quota-v1", || durable_quota_writes_with_one_offline_disk(false)); + } + + #[cfg(feature = "test-util")] + #[test] + #[serial_test::serial(storage_class_env)] + fn durable_quota_v2_writes_with_one_offline_disk_and_no_fleet_proof() { + run_large_stack_async_test("degraded-quota-v2", || durable_quota_writes_with_one_offline_disk(true)); + } + fn active_rebalance_meta_for_pool(pool_count: usize, active_pool_idx: usize) -> RebalanceMeta { let now = OffsetDateTime::now_utc(); let mut pool_stats = vec![RebalanceStats::default(); pool_count]; @@ -5030,6 +5536,339 @@ mod tests { shutdown.cancel(); } + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn repeated_abort_recovers_exact_orphan_before_decommission_drains() { + use crate::storage_api_contracts::namespace::NamespaceLocking as _; + use sha2::Digest; + + let temp_dir = tempfile::tempdir().expect("create orphan recovery store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-multipart-drain", &[4, 4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("orphan-multipart-{}", uuid::Uuid::new_v4()); + let object = "exact-object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let incarnation = store.bucket_incarnation_id(&bucket).await.expect("bucket incarnation"); + let lifecycle_guard = store.acquire_bucket_lifecycle_read_lock(&bucket).await.expect("bucket guard"); + let mut upload_opts = ObjectOptions { + expected_bucket_incarnation_id: Some(incarnation), + ..Default::default() + }; + upload_opts.add_bucket_lifecycle_lock_guard(&lifecycle_guard); + let upload = store.pools[0] + .new_multipart_upload(&bucket, object, &upload_opts) + .await + .expect("create source upload"); + drop(lifecycle_guard); + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("initial abort"); + + let upload_sha = + hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower); + let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id); + let staging_dir = |pool_index: usize, disk_index: usize| { + temp_dir + .path() + .join(format!("pool{pool_index}/set0/disk{disk_index}")) + .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET) + .join(&upload_sha) + .join(&upload_uuid) + }; + for disk_index in 0..3 { + let path = staging_dir(0, disk_index); + tokio::fs::create_dir_all(&path).await.expect("create late part directory"); + tokio::fs::write(path.join("part.1"), b"late part after abort") + .await + .expect("write late part fixture"); + } + mark_test_pool_decommissioning(&store, 0).await; + assert!( + store + .list_multipart_uploads(&bucket, "", None, None, None, 100) + .await + .expect("logical upload listing") + .uploads + .is_empty() + ); + assert!(store.ensure_decommission_multipart_uploads_drained_for_test(0).await.is_err()); + assert!( + store + .abort_multipart_upload(&bucket, "different-key", &upload.upload_id, &ObjectOptions::default()) + .await + .is_err(), + "the same UploadID for a different key must not recover this path" + ); + + // A minority metadata copy in another pool blocks recovery everywhere. + let conflicting_metadata = staging_dir(1, 0).join(crate::disk::STORAGE_FORMAT_FILE); + tokio::fs::create_dir_all(conflicting_metadata.parent().expect("metadata parent")) + .await + .expect("create conflicting pool staging"); + tokio::fs::write(&conflicting_metadata, b"unreadable metadata") + .await + .expect("write conflicting metadata"); + assert!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .is_err(), + "no pool may be cleaned until every pool proves absence" + ); + for disk_index in 0..3 { + assert!(staging_dir(0, disk_index).join("part.1").exists()); + } + tokio::fs::remove_file(conflicting_metadata) + .await + .expect("remove conflicting fixture"); + let (paused, release) = crate::store::multipart::pause_orphan_multipart_cleanup_for_test(&bucket); + let recovery_store = store.clone(); + let recovery_bucket = bucket.clone(); + let recovery_upload = upload.upload_id.clone(); + let mut recovery = tokio::spawn(async move { + recovery_store + .abort_multipart_upload(&recovery_bucket, object, &recovery_upload, &ObjectOptions::default()) + .await + }); + tokio::time::timeout(Duration::from_secs(10), async { + tokio::select! { + paused = paused => paused.expect("cleanup owner announces arrival"), + result = &mut recovery => panic!("recovery returned before physical cleanup: {result:?}"), + } + }) + .await + .expect("recovery should reach physical cleanup"); + recovery.abort(); + assert!(recovery.await.expect_err("request waiter is cancellable").is_cancelled()); + let upload_lock = store.pools[0] + .get_disks_by_key(object) + .new_ns_lock(crate::disk::RUSTFS_META_MULTIPART_BUCKET, &format!("{upload_sha}/{upload_uuid}")) + .await + .expect("same upload namespace"); + let mut upload_writer = Box::pin(upload_lock.get_write_lock(Duration::from_secs(10))); + let mut bucket_writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket)); + assert!( + futures::poll!(&mut upload_writer).is_pending(), + "cancelled recovery must retain the upload lock" + ); + assert!( + futures::poll!(&mut bucket_writer).is_pending(), + "cancelled recovery must retain the real bucket guard" + ); + release.send(()).expect("allow physical cleanup to finish"); + let (upload_guard, bucket_guard) = + tokio::time::timeout(Duration::from_secs(10), async { tokio::join!(upload_writer, bucket_writer) }) + .await + .expect("both guards should release after physical cleanup"); + drop(upload_guard.expect("upload writer resumes")); + drop(bucket_guard.expect("bucket writer resumes")); + for pool_index in 0..2 { + for disk_index in 0..4 { + assert!(!staging_dir(pool_index, disk_index).exists(), "every checked staging path must be absent"); + } + } + store + .ensure_decommission_multipart_uploads_drained_for_test(0) + .await + .expect("physical decommission gate should open after recovery"); + assert!( + matches!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await, + Err(StorageError::InvalidUploadID(_, _, _)) + ), + "an absent upload must not claim to have recovered anything" + ); + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn repeated_abort_keeps_no_such_upload_while_a_disk_is_offline() { + use sha2::Digest; + + let temp_dir = tempfile::tempdir().expect("create offline-disk recovery store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-multipart-offline", &[4, 4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("orphan-multipart-{}", uuid::Uuid::new_v4()); + let object = "offline-object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let incarnation = store.bucket_incarnation_id(&bucket).await.expect("bucket incarnation"); + let lifecycle_guard = store.acquire_bucket_lifecycle_read_lock(&bucket).await.expect("bucket guard"); + let mut upload_opts = ObjectOptions { + expected_bucket_incarnation_id: Some(incarnation), + ..Default::default() + }; + upload_opts.add_bucket_lifecycle_lock_guard(&lifecycle_guard); + let upload = store.pools[0] + .new_multipart_upload(&bucket, object, &upload_opts) + .await + .expect("create source upload"); + drop(lifecycle_guard); + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("initial abort"); + + let upload_sha = + hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower); + let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id); + let staging_dir = |disk_index: usize| { + temp_dir + .path() + .join(format!("pool0/set0/disk{disk_index}")) + .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET) + .join(&upload_sha) + .join(&upload_uuid) + }; + for disk_index in 0..3 { + let path = staging_dir(disk_index); + tokio::fs::create_dir_all(&path).await.expect("create late part directory"); + tokio::fs::write(path.join("part.1"), b"late part after abort") + .await + .expect("write late part fixture"); + } + + // One offline disk means absence cannot be proven. The idempotent Abort + // must keep answering NoSuchUpload instead of surfacing a disk error, + // and must not delete anything. + let set = store.pools[0].get_disks_by_key(object); + let offline = set.disks.write().await[3].take(); + assert!(offline.is_some(), "fixture disk must have been online"); + assert!( + matches!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await, + Err(StorageError::InvalidUploadID(_, _, _)) + ), + "unprovable absence must keep NoSuchUpload while a disk is offline" + ); + for disk_index in 0..3 { + assert!( + staging_dir(disk_index).join("part.1").exists(), + "no staging may be deleted while a disk is offline" + ); + } + + set.disks.write().await[3] = offline; + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("recovery resumes once every disk is observable"); + for disk_index in 0..4 { + assert!(!staging_dir(disk_index).exists(), "recovered staging must be absent"); + } + shutdown.cancel(); + } + + #[tokio::test] + #[serial_test::serial(storage_class_env)] + async fn orphan_multipart_recovery_preserves_current_bucket_incarnation() { + use sha2::Digest; + + let temp_dir = tempfile::tempdir().expect("create bucket lifetime recovery store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "orphan-bucket-lifetime", &[4])).await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("orphan-lifetime-{}", uuid::Uuid::new_v4()); + let object = "object.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create original bucket"); + let original_incarnation = store.bucket_incarnation_id(&bucket).await.expect("original incarnation"); + let upload = store + .new_multipart_upload(&bucket, object, &ObjectOptions::default()) + .await + .expect("create original upload"); + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("abort original upload"); + store + .delete_bucket(&bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default()) + .await + .expect("delete original bucket"); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("recreate bucket"); + assert_ne!(store.bucket_incarnation_id(&bucket).await.expect("new incarnation"), original_incarnation); + let mut reader = PutObjReader::from_vec(b"current bucket object".to_vec()); + store + .put_object(&bucket, object, &mut reader, &ObjectOptions::default()) + .await + .expect("write current bucket object"); + let current_upload = store + .new_multipart_upload(&bucket, object, &ObjectOptions::default()) + .await + .expect("create current incarnation upload for the same key"); + + let upload_sha = + hex_simd::encode_to_string(sha2::Sha256::digest(format!("{bucket}/{object}").as_bytes()), hex_simd::AsciiCase::Lower); + let upload_uuid = crate::runtime::sources::upload_uuid_suffix(&upload.upload_id); + let leftover = temp_dir + .path() + .join("pool0/set0/disk0") + .join(crate::disk::RUSTFS_META_MULTIPART_BUCKET) + .join(upload_sha) + .join(upload_uuid) + .join("part.1"); + tokio::fs::create_dir_all(leftover.parent().expect("staging parent")) + .await + .expect("create old incarnation staging"); + tokio::fs::write(&leftover, b"previous bucket lifetime") + .await + .expect("write previous incarnation fixture"); + for opts in [ + ObjectOptions { + expected_bucket_incarnation_id: Some(original_incarnation), + ..Default::default() + }, + ObjectOptions { + expected_bucket_incarnation_id: Some(uuid::Uuid::nil()), + ..Default::default() + }, + ] { + assert!( + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &opts) + .await + .is_err(), + "a stale or nil current-bucket identity must fail closed" + ); + assert!(leftover.exists(), "failed bucket lifetime proof must preserve the path"); + } + store + .abort_multipart_upload(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .await + .expect("exact metadata-less staging from an old incarnation is unreferenced"); + assert!(!leftover.exists()); + store + .get_multipart_info(&bucket, object, ¤t_upload.upload_id, &ObjectOptions::default()) + .await + .expect("recovery must preserve the current incarnation's active upload"); + let mut reader = store + .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("current object must remain readable"); + let mut body = Vec::new(); + reader.stream.read_to_end(&mut body).await.expect("read current object"); + assert_eq!(body, b"current bucket object"); + shutdown.cancel(); + } + #[tokio::test] #[serial_test::serial(storage_class_env)] async fn active_multipart_upload_routes_before_faulted_suspended_source() { @@ -5181,6 +6020,179 @@ mod tests { shutdown.cancel(); } + #[test] + #[serial_test::serial(storage_class_env)] + fn decommission_listing_accepts_committed_inline_layout_change() { + // Keep the native migration scenario off the small debug test stack. + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .worker_threads(2) + .build() + .expect("build migration listing runtime"); + runtime.block_on(async move { + use crate::config::storageclass::{ + INLINE_BLOCK_ENV, OPTIMIZE_ENV, RRS_ENV, STANDARD_ENV, lookup_config_for_pools, + }; + + let temp_dir = tempfile::tempdir().expect("create migration listing store"); + let (_ctx, store, shutdown) = Box::pin(without_storage_class_env(build_isolated_test_store( + temp_dir.path(), + "decommission-inline-listing", + &[3, 9], + ))) + .await; + let _shutdown_guard = shutdown.drop_guard(); + crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = "decommission-inline-listing"; + let object = "object.bin"; + let payload = vec![0x5a; 64 * 1024]; + Box::pin(store.make_bucket(bucket, &MakeBucketOptions::default())) + .await + .expect("create migration listing bucket"); + + // A source written with inline storage disabled remains valid after + // the destination starts using the default inline policy. + let external_config = temp_env::with_vars( + [ + (STANDARD_ENV, None), + (RRS_ENV, None), + (OPTIMIZE_ENV, None), + (INLINE_BLOCK_ENV, Some("0")), + ], + || lookup_config_for_pools(&KVS::new(), &[3, 9]), + ) + .expect("external storage policy"); + for set in &store.pools[1].disk_set { + set.set_test_storage_class_config(external_config.clone()); + } + let source = Box::pin(store.pools[1].put_object( + bucket, + object, + &mut PutObjReader::from_vec(payload.clone()), + &ObjectOptions::default(), + )) + .await + .expect("write external source"); + assert!(!source.inlined, "fixture must use external source shards"); + store.pool_meta.write().await.pools[1].decommission = Some(PoolDecommissionInfo { + start_time: Some(OffsetDateTime::now_utc()), + ..Default::default() + }); + + let reader = Box::pin(store.pools[1].get_object_reader( + bucket, + object, + None, + HeaderMap::new(), + &ObjectOptions { + no_lock: true, + data_movement: true, + raw_data_movement_read: true, + ..Default::default() + }, + )) + .await + .expect("read migration source"); + let incarnation = store.bucket_incarnation_id(bucket).await.expect("bucket incarnation"); + Box::pin(crate::data_movement::migrate_decommission_object( + store.clone(), + 1, + bucket.to_string(), + reader, + Some(incarnation), + "test_decommission_inline_listing", + None, + )) + .await + .expect("commit migrated target while retaining source"); + + for pool_index in [0, 1] { + let mut reader = Box::pin(store.pools[pool_index].get_object_reader( + bucket, + object, + None, + HeaderMap::new(), + &ObjectOptions::default(), + )) + .await + .expect("both physical copies must remain readable"); + assert_eq!(reader.object_info.inlined, pool_index == 0); + assert_eq!(reader.object_info.etag, source.etag); + assert_eq!(reader.object_info.mod_time, source.mod_time); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("read complete object body"); + assert_eq!(actual, payload); + } + + Box::pin(store.pools[0].put_object( + bucket, + "z-sentinel", + &mut PutObjReader::from_vec(b"sentinel".to_vec()), + &ObjectOptions::default(), + )) + .await + .expect("write pagination sentinel"); + let listing = Box::pin( + store + .clone() + .inner_list_objects_v2(bucket, "", None, None, 1, false, None, false), + ) + .await + .expect("listing must merge committed source and target copies"); + assert!(listing.is_truncated); + assert_eq!(listing.objects.len(), 1); + assert_eq!(listing.objects[0].name, object); + assert_eq!(listing.objects[0].etag, source.etag); + assert_eq!(listing.objects[0].size, i64::try_from(payload.len()).expect("payload size")); + let next = Box::pin(store.clone().inner_list_objects_v2( + bucket, + "", + listing.next_continuation_token, + None, + 1, + false, + None, + false, + )) + .await + .expect("continue after the merged object"); + assert!(!next.is_truncated); + assert_eq!(next.objects.len(), 1); + assert_eq!(next.objects[0].name, "z-sentinel"); + + Box::pin(store.pools[1].delete_object(bucket, object, ObjectOptions::default())) + .await + .expect("remove the migrated source copy"); + let after_cleanup = Box::pin( + store + .clone() + .inner_list_objects_v2(bucket, "", None, None, 100, false, None, false), + ) + .await + .expect("listing must remain complete after source cleanup"); + assert!(!after_cleanup.is_truncated); + assert_eq!( + after_cleanup + .objects + .iter() + .map(|info| info.name.as_str()) + .collect::>(), + [object, "z-sentinel"] + ); + }); + }) + .expect("spawn migration listing test") + .join() + .expect("migration listing test must complete"); + } + #[test] #[serial_test::serial(storage_class_env)] fn data_movement_conflicts_and_multipart_retries_converge_safely() { @@ -16495,11 +17507,10 @@ mod tests { .delete_object_with_tier_delete_journal(bucket, object, ObjectOptions::default()) .await .expect_err("one post-commit error plus one offline disk must miss delete quorum"); - assert!(matches!(err, StorageError::Io(_)), "unexpected local delete failure: {err:?}"); - assert!( - err.to_string() - .contains("Storage resources are insufficient for the write operation"), - "the caller must observe the wrapped quorum failure: {err}" + assert_eq!( + err, + StorageError::InsufficientWriteQuorum(bucket.to_owned(), object.to_owned()), + "the caller must observe the typed quorum failure with its object identity" ); set.disks.write().await[1] = Some(saved_disk); @@ -17879,6 +18890,15 @@ mod tests { .put_object(&bucket, object, &mut reader, &ObjectOptions::default()) .await .expect("put expired object"); + let lifecycle_xml = br#"expireEnabled0"#; + crate::bucket::metadata_sys::update_in(&store.ctx, &bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle_xml.to_vec()) + .await + .expect("install lifecycle policy"); + let expiry_policy = crate::bucket::lifecycle::get_expiry_configs(&store, &bucket) + .await + .unwrap() + .lifecycle + .map(Into::into); let bucket_incarnation_id = store.bucket_incarnation_id(&bucket).await.expect("read bucket incarnation"); let barrier = crate::store::object::DeleteAfterObjectLockSnapshotBarrier::install(&bucket); @@ -17889,6 +18909,7 @@ mod tests { &object_info, &crate::bucket::lifecycle::lifecycle::Event { action: crate::bucket::lifecycle::lifecycle::IlmAction::DeleteAction, + expiry_policy, ..Default::default() }, &crate::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc::Scanner, @@ -18569,11 +19590,21 @@ mod tests { .await .expect("put replacement object"); + let lifecycle_xml = br#"expireEnabled0"#; + crate::bucket::metadata_sys::update_in(&store.ctx, &bucket, BUCKET_LIFECYCLE_CONFIG, lifecycle_xml.to_vec()) + .await + .expect("install lifecycle policy"); + let expiry_policy = crate::bucket::lifecycle::get_expiry_configs(&store, &bucket) + .await + .unwrap() + .lifecycle + .map(Into::into); let applied = crate::bucket::lifecycle::bucket_lifecycle_ops::apply_expiry_on_non_transitioned_objects( store.clone(), &old_object, &crate::bucket::lifecycle::lifecycle::Event { action: crate::bucket::lifecycle::lifecycle::IlmAction::DeleteAction, + expiry_policy, ..Default::default() }, &crate::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc::Scanner, @@ -18591,6 +19622,70 @@ mod tests { assert_eq!(body, b"new-generation"); } + #[tokio::test(flavor = "multi_thread")] + #[serial_test::serial] + #[serial_test::serial(storage_class_env)] + async fn cancelled_ecstore_put_part_keeps_bucket_lifecycle_lock_through_commit() { + for pause in [ + crate::set_disk::MultipartCommitPause::PutPartAfterCapacityAdmission, + crate::set_disk::MultipartCommitPause::PutPartAfterRename, + ] { + let temp_dir = tempfile::tempdir().expect("create cancelled multipart lifecycle store dir"); + let (_ctx, store, shutdown) = + without_storage_class_env(build_isolated_test_store(temp_dir.path(), "cancelled-multipart-lifecycle", &[4, 4])) + .await; + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = format!("cancelled-part-lifecycle-{}", Uuid::new_v4()); + let object = "part.bin"; + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket for multipart lifecycle cancellation"); + let upload = store + .new_multipart_upload(&bucket, object, &ObjectOptions::default()) + .await + .expect("create upload through the owning ECStore"); + let barrier = crate::set_disk::MultipartCommitBarrier::install(&bucket, object, pause); + let put_store = Arc::clone(&store); + let put_bucket = bucket.clone(); + let put_upload_id = upload.upload_id.clone(); + let put = tokio::spawn(async move { + let mut reader = PutObjReader::from_vec(vec![0x45; 4096]); + put_store + .put_object_part(&put_bucket, object, &put_upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + }); + barrier.wait_until_paused().await; + put.abort(); + assert!( + put.await + .expect_err("the ECStore request waiter must remain cancellable") + .is_cancelled() + ); + + let mut writer = Box::pin(store.acquire_bucket_lifecycle_write_lock(&bucket)); + assert!( + tokio::time::timeout(Duration::from_millis(100), &mut writer).await.is_err(), + "canceling UploadPart at {pause:?} must not release its bucket lifecycle owner before the commit drains" + ); + barrier.release(); + let lifecycle_guard = tokio::time::timeout(Duration::from_secs(10), writer) + .await + .expect("the bucket lifecycle writer must resume after the part commit drains") + .expect("acquire the bucket lifecycle write lock after cancellation"); + drop(lifecycle_guard); + + let parts = store + .list_object_parts(&bucket, object, &upload.upload_id, None, 100, &ObjectOptions::default()) + .await + .expect("the cancelled request's owned part commit must finish successfully"); + assert_eq!(parts.parts.len(), 1, "the commit must publish exactly the uploaded part at {pause:?}"); + assert_eq!(parts.parts[0].part_num, 1); + assert_eq!(parts.parts[0].size, 4096); + shutdown.cancel(); + } + } + #[tokio::test] #[serial_test::serial(storage_class_env)] async fn pre_upgrade_multipart_upload_can_complete_and_abort_in_the_same_bucket_lifetime() { @@ -20147,6 +21242,11 @@ mod tests { let lifecycle_event = crate::bucket::lifecycle::lifecycle::Event { action: rustfs_scanner_metrics::metrics::IlmAction::DeleteAllVersionsAction, rule_id: "delete-all-versions".to_string(), + expiry_policy: crate::bucket::lifecycle::get_expiry_configs(&store, bucket) + .await + .unwrap() + .lifecycle + .map(Into::into), ..Default::default() }; let legacy_applied = @@ -22143,7 +23243,6 @@ mod tests { .make_bucket(bucket, &MakeBucketOptions::default()) .await .expect("kill-point bucket should be created"); - let _compaction_proof = install_transition_transaction_compaction_fleet_proof_for_test("object-transaction-fencing-test"); for (index, point, expected_state, expected_revision, expected_committed, expected_recovered) in [ ( @@ -22187,10 +23286,14 @@ mod tests { .await .expect("kill-point source should be written"); let remote_before = backend.object_count().await; + // Earlier cancellation/recovery cases must not consume this + // transaction's 30-second homogeneous-fleet proof window. + let _compaction_proof = + install_transition_transaction_compaction_fleet_proof_for_test("object-transaction-fencing-test"); let barrier = TransitionTransactionKillPointBarrier::install(bucket, &object, point); let transition_store = store.clone(); let transition_object = object.clone(); - let transition = tokio::spawn(async move { + let mut transition = tokio::spawn(async move { transition_store .transition_object( bucket, @@ -22209,7 +23312,12 @@ mod tests { ) .await }); - barrier.wait_until_paused().await; + tokio::select! { + _ = barrier.wait_until_paused() => {} + result = &mut transition => { + panic!("transition exited before reaching {point:?}: {result:?}"); + } + } transition.abort(); assert!( transition diff --git a/crates/ecstore/src/store/list_objects.rs b/crates/ecstore/src/store/list_objects.rs index ce7eb0006..9c737318b 100644 --- a/crates/ecstore/src/store/list_objects.rs +++ b/crates/ecstore/src/store/list_objects.rs @@ -24,7 +24,7 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::free_version_remote_tuple_ma use crate::bucket::metadata_sys::{ get_versioning_config, has_authoritative_never_versioned_state, has_authoritative_never_versioned_state_in, }; -use crate::bucket::utils::check_list_objs_args; +use crate::bucket::utils::{check_list_objs_args, has_bad_path_component}; use crate::bucket::versioning::VersioningApi; use crate::cache_value::metacache_set::{ FallbackClaimTracker, ListPathRawOptions, list_path_raw_with_claim_tracker, list_path_raw_with_partial_result, @@ -56,7 +56,7 @@ use rustfs_filemeta::{ }; use rustfs_io_metrics::{ LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, LIST_OBJECTS_SOURCE_WALKER, - ListObjectsGatherObservation, ListObjectsIndexPageObservation, + ListObjectsGatherCaller, ListObjectsGatherObservation, ListObjectsIndexPageObservation, }; use rustfs_utils::path::{self, SLASH_SEPARATOR, base_dir_from_prefix}; use std::cmp::Reverse; @@ -109,6 +109,7 @@ fn recursive_walk_list_path_options(bucket: String, prefix: &str, opts: &WalkOpt incl_deleted: true, recursive: true, versioned: true, + caller: ListObjectsGatherCaller::S3ListVersions, walkdir_timeout: opts.walkdir_timeout, walkdir_stall_timeout: opts.walkdir_stall_timeout, ..Default::default() @@ -274,6 +275,7 @@ fn log_list_path_finished( #[derive(Debug, Default, Clone)] pub struct ListPathOptions { + pub caller: ListObjectsGatherCaller, pub id: Option, // Bucket of the listing. @@ -4112,10 +4114,17 @@ impl ECStore { max_keys: i32, incl_deleted: bool, ) -> Result { + // Stored keys cannot contain repeated separators. Keep the query literal + // and avoid walking or cleaning up an aliased filesystem path. + if prefix.contains("//") && !has_bad_path_component(prefix) && !prefix.contains('\0') { + check_list_objs_args(bucket, "", &marker)?; + return Ok(ListObjectsInfo::default()); + } let max_keys = normalize_max_keys(max_keys); let effective_max_keys = if max_keys <= 0 { 0 } else { max_keys_plus_one(max_keys, true) }; let mut opts = ListPathOptions { bucket: bucket.to_owned(), + caller: ListObjectsGatherCaller::S3List, prefix: prefix.to_owned(), separator: delimiter.clone(), // Always request max_keys + 1 to detect if there are more results @@ -4261,6 +4270,79 @@ impl ECStore { .await } + /// Capture one logical key's version group for write-triggered expiry. + pub(crate) async fn get_object_versions_for_lifecycle( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + bucket_incarnation_id: Uuid, + ) -> Result> { + crate::bucket::utils::check_object_args(bucket, object)?; + let mut opts = opts.clone(); + // Lock order matches writes: bucket lifecycle, metadata validation, + // then object namespace. Borrow fences supplied by an outer operation. + let bucket_guard = if opts.bucket_lifecycle_lock_fence.is_none() { + let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?; + opts.add_bucket_lifecycle_lock_guard(&guard); + Some(guard) + } else { + None + }; + let incarnation = crate::bucket::metadata_sys::get_bucket_incarnation_id_for_options_in(&self.ctx, bucket, &opts).await?; + if incarnation != bucket_incarnation_id { + return Err(Error::BucketNotFound(bucket.to_owned())); + } + let metadata_sys = crate::bucket::metadata_sys::bucket_metadata_sys_of(&self.ctx)?; + let metadata_sys = metadata_sys.read().await.clone(); + let (versioning, _) = metadata_sys.get_versioning_config(bucket).await?; + let disk_object = path::encode_dir_object(object); + let object_guard = self + .acquire_object_read_lock_if_needed("lifecycle_versions", bucket, &disk_object, &mut opts) + .await?; + + // As with version listings, include every physical set: migration can + // leave a key's history in more than its current hash destination. + let reads = self + .pools + .iter() + .flat_map(|pool| &pool.disk_set) + .map(|set| set.read_object_versions_for_lifecycle(bucket, object)); + let mut entries = Vec::new(); + for result in join_all(reads).await { + if let Some(entry) = result.map_err(|error| to_object_err(error.into(), vec![bucket, object]))? { + entries.push(entry); + } + } + if self.pools.is_empty() + || bucket_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + || opts + .bucket_lifecycle_lock_fence + .as_ref() + .is_some_and(|fence| fence.is_lock_lost()) + || opts.namespace_lock_fence.as_ref().is_some_and(|fence| fence.is_lock_lost()) + || object_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()) + { + return Err(Error::PreconditionFailed); + } + let mut entries = entries.into_iter(); + let Some(mut entry) = entries.next() else { + return Ok(Vec::new()); + }; + if entries.len() > 0 { + merge_object_entry_versions(&mut entry, entries)?; + } + // This projection excludes tier free-version bookkeeping but retains + // pending version purges for the evaluator's replication safeguards. + let versions = entry.file_info_versions(bucket)?; + let versioned = versioning.versioned(object); + Ok(versions + .versions + .iter() + .map(|version| ObjectInfo::from_file_info(version, bucket, object, versioned)) + .collect()) + } + async fn inner_list_object_versions_with_projection( self: Arc, input: ListObjectVersionsInput<'_>, @@ -4289,6 +4371,11 @@ impl ECStore { let effective_max_keys = list_versions_scan_limit(max_keys, has_version_marker); let mut opts = ListPathOptions { bucket: bucket.to_owned(), + caller: if include_version_purge { + ListObjectsGatherCaller::Lifecycle + } else { + ListObjectsGatherCaller::S3ListVersions + }, prefix: prefix.to_owned(), separator: delimiter.clone(), limit: effective_max_keys, @@ -4993,11 +5080,45 @@ where let _cancel_guard = cancel.clone().drop_guard(); let (result_tx, mut result_rx) = mpsc::channel(1); let mut workers = JoinSet::new(); - workers.spawn(async move { producer.await.map(|()| None) }.instrument(tracing::Span::current())); - let gather_cancel = cancel.clone(); - workers.spawn( - async move { gather_results(gather_cancel, opts, recv, result_tx).await.map(Some) }.instrument(tracing::Span::current()), + #[cfg(feature = "diagnostic-read-version-caller")] + let mut diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::ListObjects, ); + #[cfg(feature = "diagnostic-read-version-caller")] + let request_caller = diagnostic_context.caller.or_else(|| { + use rustfs_io_metrics::ListObjectsGatherCaller; + use rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller; + + match opts.caller { + ListObjectsGatherCaller::S3List | ListObjectsGatherCaller::S3ListVersions => { + Some(ReadVersionRequestCaller::ListObjects) + } + ListObjectsGatherCaller::Lifecycle => Some(ReadVersionRequestCaller::Lifecycle), + ListObjectsGatherCaller::Other => Some(ReadVersionRequestCaller::Other), + ListObjectsGatherCaller::Unattributed => None, + } + }); + #[cfg(feature = "diagnostic-read-version-caller")] + { + diagnostic_context.caller = request_caller; + } + let producer = async move { producer.await.map(|()| None) }; + #[cfg(feature = "diagnostic-read-version-caller")] + let producer = crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ListProducer, + producer, + ); + workers.spawn(producer.instrument(tracing::Span::current())); + let gather_cancel = cancel.clone(); + let gather = async move { gather_results(gather_cancel, opts, recv, result_tx).await.map(Some) }; + #[cfg(feature = "diagnostic-read-version-caller")] + let gather = crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context( + diagnostic_context, + rustfs_io_metrics::internode_metrics::ReadVersionContextBoundary::ListGather, + gather, + ); + workers.spawn(gather.instrument(tracing::Span::current())); let mut failure = None; let mut stopping = false; @@ -5133,6 +5254,7 @@ async fn gather_results( let duration_ms = started.elapsed().as_secs_f64() * 1000.0; rustfs_io_metrics::record_list_objects_gather(ListObjectsGatherObservation { source: LIST_OBJECTS_SOURCE_WALKER, + caller: opts.caller, outcome: LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, limit: opts.limit, scanned_entries, @@ -5195,6 +5317,7 @@ async fn gather_results( let duration_ms = started.elapsed().as_secs_f64() * 1000.0; rustfs_io_metrics::record_list_objects_gather(ListObjectsGatherObservation { source: LIST_OBJECTS_SOURCE_WALKER, + caller: opts.caller, outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, limit: opts.limit, scanned_entries, @@ -5326,6 +5449,23 @@ async fn send_or_cancel(rx: &CancellationToken, out_channel: &Sender, validation: &'static str) -> Error { + const EVENT_LISTING_MERGE_REJECTED: &str = "listing_merge_rejected"; + const LOG_COMPONENT_ECSTORE: &str = "ecstore"; + const LOG_SUBSYSTEM_LISTING: &str = "listing"; + error!( + event = EVENT_LISTING_MERGE_REJECTED, + component = LOG_COMPONENT_ECSTORE, + subsystem = LOG_SUBSYSTEM_LISTING, + result = "rejected", + object, + version_id = ?version_id, + validation, + "Listing version merge rejected metadata" + ); + Error::FileCorrupt +} + /// Each input has already been resolved inside its own erasure set. This is a /// union of version histories, never a quorum vote between unrelated pools. fn merge_object_entry_versions(first: &mut MetaCacheEntry, others: impl Iterator) -> Result<()> { @@ -5334,23 +5474,26 @@ fn merge_object_entry_versions(first: &mut MetaCacheEntry, others: impl Iterator for mut entry in std::iter::once(std::mem::take(first)).chain(others) { let meta = match entry.cached.take() { Some(meta) => meta, - None => FileMeta::load(&entry.metadata).map_err(|_| Error::FileCorrupt)?, + None => FileMeta::load(&entry.metadata).map_err(|_| listing_merge_corruption(&name, None, "metadata_decode"))?, }; if meta.versions.is_empty() { - return Err(Error::FileCorrupt); + return Err(listing_merge_corruption(&name, None, "empty_version_stack")); } for version in meta.versions { - let parsed = version.parse_version_meta().map_err(|_| Error::FileCorrupt)?; + let corrupt = |validation| listing_merge_corruption(&name, version.header.version_id, validation); + let parsed = version.parse_version_meta().map_err(|_| corrupt("version_decode"))?; if !parsed.valid() || parsed.version_type != version.header.version_type { - return Err(Error::FileCorrupt); + return Err(corrupt("version_type_or_validity")); } - let fi = parsed.into_fileinfo("", &name, true).map_err(|_| Error::FileCorrupt)?; + let fi = parsed + .into_fileinfo("", &name, true) + .map_err(|_| corrupt("file_info_decode"))?; let version_id = fi.version_id.filter(|id| !id.is_nil()); if version_id != version.header.version_id.filter(|id| !id.is_nil()) || fi.mod_time != version.header.mod_time || fi.tier_free_version() != version.header.free_version() { - return Err(Error::FileCorrupt); + return Err(corrupt("version_header_mismatch")); } let info = ObjectInfo::from_file_info(&fi, "", &name, true); let identity = (version_id, version.header.free_version()); @@ -5374,7 +5517,7 @@ fn merge_object_entry_versions(first: &mut MetaCacheEntry, others: impl Iterator crate::data_movement::is_equivalent_data_movement_object_identity(&info, previous_info, true, true) }; if !equivalent { - return Err(Error::FileCorrupt); + return Err(corrupt("version_identity_conflict")); } // Equivalent migrated copies can have different coding or // data directories. Choose a stable representation without @@ -7229,6 +7372,9 @@ fn calc_common_counter(infos: &[DiskInfo], read_quorum: usize) -> u64 { #[cfg(test)] mod differential_tests; +#[cfg(test)] +mod lifecycle_exact_tests; + #[cfg(test)] mod test { use super::{ @@ -7458,7 +7604,7 @@ mod test { } } - fn test_pending_version_purge_meta_entry(bucket: &str, name: &str) -> (MetaCacheEntry, Uuid) { + pub(super) fn test_pending_version_purge_meta_entry(bucket: &str, name: &str) -> (MetaCacheEntry, Uuid) { let version_id = Uuid::from_u128(0x1234_5678_9abc_def0_1234_5678_9abc_def0); let data_dir = Uuid::from_u128(0xabcd_ef12_3456_7890_abcd_ef12_3456_7890); let mod_time = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"); @@ -12117,6 +12263,82 @@ mod test { assert_eq!(versions.versions[0].metadata.get("etag").map(String::as_str), Some("same-etag")); } + #[tokio::test] + async fn merge_entry_channels_data_movement_accepts_inline_layout_changes() { + let time = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"); + let external = test_object_meta_entry_with_erasure_versions("key", &[(time, "same-etag", 2, 1)]); + let inline = rewrite_test_version( + test_object_meta_entry_with_erasure_versions("key", &[(time, "same-etag", 5, 4)]), + |version| { + rustfs_utils::http::insert_bytes( + &mut version.object.as_mut().expect("object version").meta_sys, + rustfs_utils::http::SUFFIX_INLINE_DATA, + b"true".to_vec(), + ); + }, + ); + let legacy_inline = rewrite_test_version(inline.clone(), |version| { + version + .object + .as_mut() + .expect("object version") + .meta_sys + .remove(&rustfs_utils::http::internal_key_rustfs(rustfs_utils::http::SUFFIX_INLINE_DATA)); + }); + let copies = [external, legacy_inline, inline]; + let mut expected = None; + for order in [[0, 1, 2], [0, 2, 1], [1, 0, 2], [1, 2, 0], [2, 0, 1], [2, 1, 0]] { + let merged = merge_test_object_entries(order.map(|index| copies[index].clone()).to_vec()) + .await + .expect("physical inline placement and legacy aliases must not change object identity"); + assert_eq!( + merged + .file_info_versions("bucket") + .expect("decode merged history") + .versions + .len(), + 1 + ); + if let Some(expected) = &expected { + assert_eq!(&merged.metadata, expected, "representation must not depend on pool order"); + } else { + expected = Some(merged.metadata); + } + } + } + + #[test] + fn data_movement_listing_conflict_logs_identity_without_metadata_values() { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _guard = tracing::subscriber::set_default(subscriber); + let time = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp"); + let mut original = test_object_meta_entry_with_erasure_versions("failing-key", &[(time, "same-etag", 2, 1)]); + let conflicting = rewrite_test_version(original.clone(), |version| { + version + .object + .as_mut() + .expect("object version") + .meta_user + .insert("x-amz-meta-private".to_string(), "sensitive-value-must-not-be-logged".to_string()); + }); + assert_eq!( + super::merge_object_entry_versions(&mut original, std::iter::once(conflicting)) + .expect_err("conflicting user metadata must still fail closed"), + StorageError::FileCorrupt + ); + let output = logs.contents(); + assert!(output.contains("listing_merge_rejected")); + assert!(output.contains("failing-key")); + assert!(output.contains("00000000-0000-0000-0000-000000000001")); + assert!(output.contains("version_identity_conflict")); + assert!(!output.contains("sensitive-value-must-not-be-logged")); + } + #[tokio::test] async fn merge_entry_channels_defers_free_version_while_same_remote_source_is_live() { let live = test_transitioned_meta_entry("key", "remote/shared", false); @@ -12294,6 +12516,44 @@ mod test { ); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn list_path_producer_worker_preserves_s3_and_caller_context() { + let mut request = rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard::new("GET"); + let api_operation = request + .in_scope(|| { + rustfs_io_metrics::record_s3_op(rustfs_s3_ops::S3Operation::ListObjectsV2); + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_context() + }) + .expect("dispatched list operation should be captured"); + request.response(200); + + let (sender, recv) = mpsc::channel(1); + let opts = ListPathOptions { + caller: rustfs_io_metrics::ListObjectsGatherCaller::S3List, + ..Default::default() + }; + let result = rustfs_io_metrics::s3_http_metrics::with_s3_http_operation_context(Some(api_operation), async move { + collect_list_path_results(CancellationToken::new(), opts, recv, async move { + assert_eq!( + rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), + Some("s3:ListObjectsV2") + ); + assert_eq!( + crate::cluster::rpc::runtime_sources::current_read_version_request_caller(), + Some(rustfs_io_metrics::internode_metrics::ReadVersionRequestCaller::ListObjects) + ); + drop(sender); + Ok(()) + }) + .await + }) + .await; + + assert!(result.err.is_some(), "the empty page channel remains an error"); + assert_eq!(rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(), None); + } + #[tokio::test] async fn list_path_workers_preserve_error_after_input_closes() { let (sender, recv) = mpsc::channel(1); diff --git a/crates/ecstore/src/store/list_objects/lifecycle_exact_tests.rs b/crates/ecstore/src/store/list_objects/lifecycle_exact_tests.rs new file mode 100644 index 000000000..91cf0c0a5 --- /dev/null +++ b/crates/ecstore/src/store/list_objects/lifecycle_exact_tests.rs @@ -0,0 +1,498 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::bucket::lifecycle::core::IlmAction; +use crate::bucket::lifecycle::evaluator::Evaluator; +use crate::bucket::metadata::{BUCKET_LIFECYCLE_CONFIG, BUCKET_VERSIONING_CONFIG}; +use crate::bucket::metadata_sys; +use crate::disk::lifecycle_test_io::Observer; +use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints}; +use crate::object_api::PutObjReader; +use crate::runtime::instance::InstanceContext; +use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions}; +use crate::storage_api_contracts::multipart::{CompletePart, MultipartOperations}; +use crate::storage_api_contracts::object::ObjectIO; +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use tempfile::TempDir; +use time::OffsetDateTime; + +async fn fixture(pool_count: usize) -> (Vec, Arc, String, Uuid) { + let mut dirs = Vec::new(); + let mut pools = Vec::new(); + for pool_idx in 0..pool_count { + let mut endpoints = Vec::new(); + for disk_idx in 0..4 { + let dir = tempfile::tempdir().expect("disk directory"); + let mut endpoint = + crate::disk::endpoint::Endpoint::try_from(dir.path().to_str().expect("disk path")).expect("disk endpoint"); + endpoint.set_pool_index(pool_idx); + endpoint.set_set_index(0); + endpoint.set_disk_index(disk_idx); + dirs.push(dir); + endpoints.push(endpoint); + } + pools.push(PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 4, + endpoints: Endpoints::from(endpoints), + cmd_line: format!("lifecycle-exact-pool-{pool_idx}"), + platform: "test".into(), + }); + } + let endpoints = EndpointServerPools(pools); + let ctx = Arc::new(InstanceContext::new()); + crate::store::init_local_disks_with_instance_ctx(&ctx, endpoints.clone()) + .await + .expect("initialize disks"); + let store = + ECStore::new_with_instance_ctx("127.0.0.1:0".parse().expect("test address"), endpoints, CancellationToken::new(), ctx) + .await + .expect("initialize store"); + metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await; + let bucket = format!("lifecycle-exact-{}", Uuid::new_v4().simple()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let incarnation = metadata_sys::get_bucket_incarnation_id_in(&store.ctx, &bucket) + .await + .expect("bucket incarnation"); + (dirs, store, bucket, incarnation) +} + +fn version(object: &str, id: Option, timestamp: i64) -> FileInfo { + let mut fi = FileInfo::new(object, 2, 2); + fi.name = object.to_owned(); + fi.version_id = id; + fi.versioned = id.is_some(); + fi.data_dir = Some(Uuid::from_u128(0x1234)); + fi.mod_time = Some(OffsetDateTime::from_unix_timestamp(timestamp).expect("version timestamp")); + fi.size = 1; + fi.parts = vec![ObjectPartInfo { + number: 1, + size: 1, + actual_size: 1, + ..Default::default() + }]; + fi.metadata.insert("etag".into(), timestamp.to_string()); + fi +} + +async fn seed(dirs: &[TempDir], bucket: &str, object: &str, versions: &[FileInfo]) { + for (index, dir) in dirs.iter().enumerate() { + let mut meta = FileMeta::new(); + for version in versions { + let mut fi = version.clone(); + fi.volume = bucket.to_owned(); + if !fi.is_canonical_delete_marker() { + fi.erasure.index = index % 4 + 1; + } + fi.validate_for_metadata_read().expect("valid fixture version"); + meta.add_version(fi).expect("encode version"); + } + let path = dir.path().join(bucket).join(path::encode_dir_object(object)); + tokio::fs::create_dir_all(&path).await.expect("object directory"); + tokio::fs::write(path.join(crate::disk::STORAGE_FORMAT_FILE), meta.marshal_msg().expect("encode metadata")) + .await + .expect("persist fixture metadata"); + } +} + +async fn configure_versioning(store: &ECStore, bucket: &str, state: &str) { + if state == "Unversioned" { + return; + } + metadata_sys::update_in( + &store.ctx, + bucket, + BUCKET_VERSIONING_CONFIG, + format!("{state}").into_bytes(), + ) + .await + .expect("persist versioning configuration"); +} + +#[tokio::test] +async fn lifecycle_exact_matches_listing_for_complete_version_groups() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + for state in ["Unversioned", "Enabled", "Suspended"] { + configure_versioning(&store, &bucket, state).await; + for object in ["abc", "nested/abc", "directory/"] { + let mut versions = vec![version(object, None, 1_700_000_000)]; + if state != "Unversioned" { + versions.push(version(object, Some(Uuid::from_u128(1)), 1_700_000_001)); + versions.push(FileInfo { + name: object.into(), + version_id: Some(Uuid::from_u128(2)), + deleted: true, + mod_time: Some(OffsetDateTime::from_unix_timestamp(1_700_000_002).expect("marker timestamp")), + ..Default::default() + }); + } + seed(&dirs, &bucket, object, &versions).await; + let sibling = format!("{object}-sibling"); + seed(&dirs, &bucket, &sibling, &[version(&sibling, None, 1_700_000_003)]).await; + let observed = Observer::new(&bucket); + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .unwrap_or_else(|error| panic!("read {state} versions for {object}: {error}")); + let counts = observed.snapshot(); + assert_eq!(counts.walks, 0, "exact metadata reads must never enumerate a directory"); + assert_eq!(counts.reads.len(), 4, "one exact metadata read per physical drive"); + assert!(counts.reads.iter().all(|key| key == &path::encode_dir_object(object))); + let mut listed = store + .clone() + .list_object_versions_for_lifecycle(&bucket, object, None, None, None, 1000) + .await + .expect("reference lifecycle listing") + .objects + .into_iter() + .filter(|info| info.name == object) + .collect::>(); + // The legacy list projection uses ambient bucket versioning. This + // fixture has an isolated store; restore its known null-version + // representation before comparing the version-group semantics. + if state != "Unversioned" { + for info in &mut listed { + if info.version_id.is_none() { + info.version_id = Some(Uuid::nil()); + } + } + assert!(exact.iter().any(|info| info.version_id == Some(Uuid::nil()))); + } + assert!(observed.snapshot().walks > 0, "the old prefix-listing path must exercise the observer"); + assert_eq!(exact.len(), versions.len(), "{state}: {object}"); + assert_eq!(exact.len(), listed.len()); + for (exact, listed) in exact.iter().zip(&listed) { + assert_eq!( + ( + exact.version_id, + exact.is_latest, + exact.delete_marker, + exact.num_versions, + exact.successor_mod_time + ), + ( + listed.version_id, + listed.is_latest, + listed.delete_marker, + listed.num_versions, + listed.successor_mod_time + ) + ); + assert_eq!(exact.user_defined, listed.user_defined); + assert_eq!(exact.version_purge_status, listed.version_purge_status); + } + let policy = crate::bucket::utils::deserialize( + br#"historyEnabled3011"#, + ).expect("reference policy"); + let evaluator = Evaluator::new(Arc::new(policy)); + let to_opts = |objects: &[ObjectInfo]| { + objects + .iter() + .map(crate::bucket::lifecycle::core::object_opts_from_object_info) + .collect::>() + }; + let exact_events = evaluator.eval(&to_opts(&exact)).await.expect("evaluate exact group"); + let listed_events = evaluator.eval(&to_opts(&listed)).await.expect("evaluate listed group"); + for (exact, listed) in exact_events.iter().zip(listed_events) { + assert_eq!((exact.action, exact.due, &exact.rule_id), (listed.action, listed.due, &listed.rule_id)); + } + } + } +} + +#[tokio::test] +async fn lifecycle_exact_merges_pools_and_more_than_one_page_of_versions() { + let (dirs, store, bucket, incarnation) = fixture(2).await; + configure_versioning(&store, &bucket, "Enabled").await; + let object = "history"; + let versions = (0..1002) + .map(|index| { + version( + object, + Some(Uuid::from_u128(index + 1)), + 1_700_000_000 + i64::try_from(index).expect("timestamp index"), + ) + }) + .collect::>(); + seed(&dirs[..4], &bucket, object, &versions[..601]).await; + seed(&dirs[4..], &bucket, object, &versions[600..]).await; + let observed = Observer::new(&bucket); + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .expect("merge independently authoritative pools"); + assert_eq!(exact.len(), 1002, "overlapping migration copies must be deduplicated without pagination"); + assert_eq!(exact[0].version_id, Some(Uuid::from_u128(1002))); + assert!(exact[0].is_latest); + assert!(exact.iter().all(|info| info.num_versions == 1002)); + for pair in exact.windows(2) { + assert!(!pair[1].is_latest); + assert_eq!(pair[1].successor_mod_time, pair[0].mod_time); + } + assert_eq!(observed.snapshot().walks, 0); + assert_eq!(observed.snapshot().reads.len(), 8, "read count depends on drives, not version pages"); + let mut conflict = versions[600].clone(); + conflict.metadata.insert("etag".into(), "conflicting-content".into()); + seed(&dirs[4..], &bucket, object, &[conflict]).await; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .is_err(), + "conflicting cross-pool copies cannot authorize expiry" + ); +} + +#[tokio::test] +async fn lifecycle_exact_rejects_uncertain_metadata_and_stale_bucket_identity() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + let object = "probe"; + let opts = ObjectOptions::default(); + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .expect("confirmed absence") + .is_empty() + ); + seed(&dirs, &bucket, object, &[version(object, None, 1_700_000_000)]).await; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, Uuid::new_v4()) + .await + .is_err() + ); + for dir in &dirs { + tokio::fs::write( + dir.path() + .join(bucket.as_str()) + .join(object) + .join(crate::disk::STORAGE_FORMAT_FILE), + b"corrupt", + ) + .await + .expect("corrupt fixture metadata"); + } + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .is_err(), + "corruption is not absence" + ); + seed(&dirs, &bucket, object, &[version(object, None, 1_700_000_000)]).await; + let set = &store.pools[0].disk_set[0]; + let saved = set.disks.read().await.clone(); + *set.disks.write().await = vec![saved[0].clone(), saved[1].clone(), None, None]; + assert_eq!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .expect("exactly the object's read quorum") + .len(), + 1 + ); + *set.disks.write().await = vec![saved[0].clone(), None, None, None]; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &opts, incarnation) + .await + .is_err(), + "a minority cannot authorize expiry" + ); + assert!( + store + .get_object_versions_for_lifecycle(&bucket, "missing", &opts, incarnation) + .await + .is_err(), + "missing on one drive does not prove absence" + ); + *set.disks.write().await = saved; + let lost = ObjectOptions { + no_lock: true, + namespace_lock_fence: Some(crate::object_api::NamespaceLockFence::lost_for_test()), + ..Default::default() + }; + assert!( + store + .get_object_versions_for_lifecycle(&bucket, object, &lost, incarnation) + .await + .is_err(), + "an inherited lost object fence must reject the snapshot" + ); +} + +#[tokio::test] +async fn lifecycle_exact_write_hooks_never_walk_parent_directories() { + let (dirs, store, bucket, _) = fixture(1).await; + metadata_sys::update_in(&store.ctx, &bucket, BUCKET_LIFECYCLE_CONFIG, + br#"expireEnabled30"#.to_vec()) + .await.expect("configure expiration"); + for index in 0..64 { + let key = format!("probe-{index}"); + seed(&dirs, &bucket, &key, &[version(&key, None, 1_700_000_000)]).await; + } + for state in ["Unversioned", "Enabled", "Suspended"] { + configure_versioning(&store, &bucket, state).await; + let opts = ObjectOptions { + versioned: state == "Enabled", + version_suspended: state == "Suspended", + ..Default::default() + }; + let observer = Observer::new(&bucket); + let mut source = store + .put_object(&bucket, "probe", &mut PutObjReader::from_vec(b"payload".to_vec()), &opts) + .await + .expect("PUT with lifecycle"); + source.put_object_reader = Some(PutObjReader::from_vec(b"payload".to_vec())); + let copied = store + .copy_object(&bucket, "probe", &bucket, "copy", &mut source, &ObjectOptions::default(), &opts) + .await + .expect("COPY with lifecycle"); + assert_eq!(copied.size, source.size); + let upload = store + .new_multipart_upload(&bucket, "multipart", &opts) + .await + .expect("init multipart"); + let part = store + .put_object_part( + &bucket, + "multipart", + &upload.upload_id, + 1, + &mut PutObjReader::from_vec(b"part".to_vec()), + &opts, + ) + .await + .expect("upload part"); + let complete = store + .clone() + .complete_multipart_upload( + &bucket, + "multipart", + &upload.upload_id, + vec![CompletePart { + part_num: 1, + etag: part.etag, + checksum_crc32: None, + checksum_crc32c: None, + checksum_sha1: None, + checksum_sha256: None, + checksum_crc64nvme: None, + }], + &opts, + ) + .await + .expect("complete multipart with lifecycle"); + assert_eq!(complete.size, 4); + let counts = observer.snapshot(); + assert_eq!(counts.walks, 0, "{state}: successful writes must not run the version walker"); + for key in ["probe", "copy", "multipart"] { + assert!( + counts.reads.iter().filter(|read| read.as_str() == key).count() >= 4, + "{state}: {key} must run exact expiry evaluation" + ); + } + assert!( + !counts.reads.iter().any(|key| key.starts_with("probe-")), + "prefix siblings must never enter evaluation" + ); + } +} + +#[tokio::test] +async fn lifecycle_exact_keeps_pending_purges_visible_to_expiry_guards() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + configure_versioning(&store, &bucket, "Enabled").await; + let object = "pending"; + let (entry, _) = super::test::test_pending_version_purge_meta_entry(&bucket, object); + for dir in &dirs { + let path = dir.path().join(&bucket).join(object); + tokio::fs::create_dir_all(&path).await.expect("object directory"); + tokio::fs::write(path.join(crate::disk::STORAGE_FORMAT_FILE), &entry.metadata) + .await + .expect("persist pending purge"); + } + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .expect("read pending purge"); + assert_eq!(exact.len(), 1); + assert_eq!(exact[0].num_versions, 1); + assert_eq!(exact[0].version_purge_status, crate::bucket::replication::VersionPurgeStatusType::Pending); + let policy = crate::bucket::utils::deserialize( + br#"expireEnabled1"#, + ).expect("expiration policy"); + let opts = crate::bucket::lifecycle::core::object_opts_from_object_info(&exact[0]); + let events = Evaluator::new(Arc::new(policy)) + .eval(&[opts]) + .await + .expect("evaluate pending purge"); + assert_eq!(events[0].action, IlmAction::NoneAction); +} + +#[tokio::test] +async fn lifecycle_exact_io_is_independent_of_parent_directory_width() { + let (dirs, store, bucket, incarnation) = fixture(1).await; + let object = "probe"; + seed(&dirs, &bucket, object, &[version(object, None, 1_700_000_000)]).await; + let mut previous = 0; + for width in [64, 1024, 8192] { + let roots = dirs.iter().map(|dir| dir.path().join(&bucket)).collect::>(); + tokio::task::spawn_blocking(move || { + for root in roots { + for index in previous..width { + std::fs::create_dir(root.join(format!("sibling-{index:05}"))).expect("parent-directory entry"); + } + } + }) + .await + .expect("create wide directory"); + previous = width; + let mut exact_times = Vec::new(); + let mut listing_times = Vec::new(); + for _ in 0..5 { + let observer = Observer::new(&bucket); + let started = std::time::Instant::now(); + let exact = store + .get_object_versions_for_lifecycle(&bucket, object, &ObjectOptions::default(), incarnation) + .await + .expect("exact lifecycle read"); + exact_times.push(started.elapsed().as_micros()); + assert_eq!(exact.len(), 1); + assert_eq!(observer.snapshot().walks, 0); + assert_eq!(observer.snapshot().reads.len(), 4, "directory width {width}"); + let started = std::time::Instant::now(); + let listed = store + .clone() + .list_object_versions_for_lifecycle(&bucket, object, None, None, None, 1000) + .await + .expect("reference prefix scan"); + listing_times.push(started.elapsed().as_micros()); + assert_eq!(listed.objects.len(), 1); + assert!(observer.snapshot().walks > 0); + } + exact_times.sort_unstable(); + listing_times.sort_unstable(); + eprintln!( + "lifecycle directory width={width}: exact median={}us max={}us; prefix median={}us max={}us", + exact_times[2], exact_times[4], listing_times[2], listing_times[4] + ); + } +} diff --git a/crates/ecstore/src/store/mod.rs b/crates/ecstore/src/store/mod.rs index 008d5f01b..f28479a13 100644 --- a/crates/ecstore/src/store/mod.rs +++ b/crates/ecstore/src/store/mod.rs @@ -27,6 +27,7 @@ use crate::bucket::utils::check_get_obj_args; use crate::bucket::utils::check_list_multipart_args; use crate::bucket::utils::check_list_parts_args; use crate::bucket::utils::check_new_multipart_args; +use crate::bucket::utils::check_new_object_args; use crate::bucket::utils::check_object_args; use crate::bucket::utils::check_put_object_args; use crate::bucket::utils::check_put_object_part_args; @@ -428,6 +429,7 @@ mod bucket; mod bucket_fence; pub(crate) use bucket::await_bucket_namespace_operation; pub use bucket_fence::BucketIncarnationFenceGuard; +pub(crate) mod delete_recovery_runtime; mod heal; pub use heal::{HealObjectAbsenceProof, HealObjectStorageResult}; pub(crate) use heal::{bucket_heal_scope, bucket_heal_scope_for_object}; @@ -438,7 +440,7 @@ pub(crate) mod init_format; pub(crate) mod list_objects; mod multipart; mod object; -#[cfg(feature = "test-util")] +#[cfg(any(test, feature = "test-util"))] pub use object::DeleteAfterObjectLockSnapshotBarrier; pub(crate) use object::{ DecommissionFixedReadAnchor, ObjectLockDiagGuard, RemoteTuplePublicationCommitGuard, RemoteTuplePublicationFence, @@ -926,12 +928,27 @@ impl ECStore { /// Returns whether scanner metadata may still be hidden by a local /// data-movement state. Terminal failed/canceled decommission entries /// remain suspended until an operator clears or retries them, so they are - /// a publication barrier even after the worker has stopped. Active PUT - /// rename fanouts also defer publication, including post-ACK tails. + /// a publication barrier even after the worker has stopped. + /// + /// An ordinary PUT whose rename fanout is still pending is not a barrier. + /// Its bucket is shaded through the dirty-usage generation and certified + /// per bucket by the scanner, so a sustained write load cannot hold the + /// authoritative usage snapshot back (rustfs/rustfs#8494). The scanner + /// publication leases below are movement-only for the same reason + /// (rustfs/rustfs#8305). pub async fn scanner_data_usage_publication_blocked(&self) -> bool { let operation_gate = self.ctx.data_movement_operation_gate(); let _operation_guard = operation_gate.read_owned().await; - self.scanner_data_usage_publication_snapshot_blocked().await || self.ctx.namespace_commits_pending() + self.scanner_data_usage_publication_snapshot_blocked().await + } + + /// Whether an ordinary namespace commit (a PUT rename fanout, including a + /// tail that outlives a cancelled handler) still holds its physical owner. + /// Usage publication no longer consults it; tests use it to observe owner + /// retention directly. + #[cfg(any(test, feature = "test-util"))] + pub fn namespace_commits_pending_for_test(&self) -> bool { + self.ctx.namespace_commits_pending() } pub async fn scanner_data_movement_pause_status(&self) -> ScannerDataMovementPauseStatus { @@ -1118,9 +1135,13 @@ impl ECStore { } else { ScannerPublicationLeasePurpose::Authoritative }; - // Strict leases bind the namespace generation before their async - // admission checks. An observation only needs the movement fence, but - // still fails closed if generation tracking is exhausted. + // Both purposes fence data movement only. An ordinary namespace commit + // on this node is not a reason to withhold the lease: the bucket it + // writes is shaded for the coordinator's next cycle, and a commit + // pending at every lease attempt would otherwise keep a continuously + // written peer from ever granting one (rustfs/rustfs#8305). The + // namespace generation is still recorded on an authoritative lease + // for diagnostics, and generation exhaustion still fails closed. let observed_namespace_generation = self.scanner_namespace_mutation_generation(); let operation_gate = self.ctx.data_movement_operation_gate(); let operation_guard = operation_gate.read_owned().await; @@ -1135,17 +1156,7 @@ impl ECStore { return Err(Error::other("scanner publication lease is blocked by data movement")); } - let namespace_generation = if observational_only { - None - } else { - if self.ctx.namespace_commits_pending() { - return Err(Error::other("scanner publication lease is blocked by pending namespace commit")); - } - if self.scanner_namespace_mutation_generation() != observed_namespace_generation { - return Err(Error::other("scanner publication lease namespace changed during acquisition")); - } - Some(observed_namespace_generation) - }; + let namespace_generation = (!observational_only).then_some(observed_namespace_generation); let token = Uuid::new_v4(); let expires_at = tokio::time::Instant::now() + ttl; @@ -1209,20 +1220,12 @@ impl ECStore { if self.scanner_data_movement_snapshot_locked().await.1 { return Err(Error::other("scanner publication lease is blocked by data movement")); } - let Some((lease_generation, lease_namespace_generation, purpose)) = - self.ctx.scanner_publication_lease_generations(token).await - else { + let Some((lease_generation, _, _)) = self.ctx.scanner_publication_lease_generations(token).await else { return Err(Error::other("scanner publication lease is unknown or expired")); }; - if purpose == ScannerPublicationLeasePurpose::Authoritative && self.ctx.namespace_commits_pending() { - return Err(Error::other("scanner publication lease is blocked by pending namespace commit")); - } - let namespace_generation = self.scanner_namespace_mutation_generation(); - if lease_generation != self.ctx.data_movement_generation() - || namespace_generation == u64::MAX - || (purpose == ScannerPublicationLeasePurpose::Authoritative - && lease_namespace_generation != Some(namespace_generation)) - { + // Ordinary namespace commits neither block nor stale a lease; only the + // movement generation and generation exhaustion do. + if lease_generation != self.ctx.data_movement_generation() || self.scanner_namespace_mutation_generation() == u64::MAX { return Err(Error::other("scanner publication lease generation is stale")); } Ok(()) @@ -1274,9 +1277,7 @@ impl ECStore { if self.scanner_data_movement_snapshot_locked().await.1 { return Err(Error::other("scanner publication lease is blocked by data movement")); } - let Some((lease_generation, lease_namespace_generation, purpose)) = - self.ctx.scanner_publication_lease_generations(token).await - else { + let Some((lease_generation, _, purpose)) = self.ctx.scanner_publication_lease_generations(token).await else { return Err(Error::other("scanner publication lease is unknown or expired")); }; if expected_purpose.is_some_and(|expected| expected != purpose) { @@ -1285,15 +1286,11 @@ impl ECStore { if purpose == ScannerPublicationLeasePurpose::Observational && observed_target.is_none() && !validation_only { return Err(Error::other("observational lease is limited to observed usage publication")); } - if purpose == ScannerPublicationLeasePurpose::Authoritative && self.ctx.namespace_commits_pending() { - return Err(Error::other("scanner publication lease is blocked by pending namespace commit")); - } - let namespace_generation = self.scanner_namespace_mutation_generation(); - if lease_generation != self.ctx.data_movement_generation() - || namespace_generation == u64::MAX - || (purpose == ScannerPublicationLeasePurpose::Authoritative - && lease_namespace_generation != Some(namespace_generation)) - { + // An ordinary namespace commit pending or completed on this node + // neither blocks nor stales the target admission: the bucket it + // writes is shaded for the next cycle, and the aggregate rename below + // touches only scanner-owned metadata. + if lease_generation != self.ctx.data_movement_generation() || self.scanner_namespace_mutation_generation() == u64::MAX { return Err(Error::other("scanner publication lease generation is stale")); } Ok(operation_guard) @@ -1363,7 +1360,13 @@ impl crate::storage_api_contracts::object::ObjectIO for ECStore { h: HeaderMap, opts: &ObjectOptions, ) -> Result { - self.handle_get_object_reader(bucket, object, range, h, opts).await + let read = self.handle_get_object_reader(bucket, object, range, h, opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::cluster::rpc::runtime_sources::with_read_version_entry( + rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::StoreObjectReader, + read, + ); + read.await } #[instrument(level = "debug", skip(self, data))] async fn put_object(&self, bucket: &str, object: &str, data: &mut PutObjReader, opts: &ObjectOptions) -> Result { @@ -2672,11 +2675,22 @@ mod tests { let ctx = Arc::new(InstanceContext::new()); let store = build_store_with_ctx(ctx.clone()); let namespace_commit = ctx.begin_namespace_commit(); - let strict_error = store + let (strict_token, strict_generation) = store .acquire_scanner_publication_lease(0, crate::runtime::instance::SCANNER_PUBLICATION_LEASE_TTL) .await - .expect_err("authoritative lease must wait for namespace commits"); - assert!(strict_error.to_string().contains("pending namespace commit")); + .expect("an authoritative lease is granted while an ordinary namespace commit is pending"); + assert_eq!(strict_generation, 0); + store + .validate_scanner_publication_lease(strict_token, 0) + .await + .expect("the pending commit does not invalidate the authoritative lease"); + drop( + store + .acquire_scanner_publication_lease_guard(strict_token) + .await + .expect("target-side admission accepts the pending commit"), + ); + assert!(store.release_scanner_publication_lease(strict_token).await); let (token, movement_generation) = store .acquire_scanner_publication_lease_with_purpose(0, crate::runtime::instance::SCANNER_PUBLICATION_LEASE_TTL, true) @@ -2747,7 +2761,7 @@ mod tests { } #[tokio::test] - async fn scanner_publication_lease_rejects_namespace_change_during_admission() { + async fn scanner_publication_lease_survives_namespace_change_during_admission() { let ctx = Arc::new(InstanceContext::new()); let store = build_store_with_ctx(ctx.clone()); let snapshot_blocker = store.rebalance_meta.write().await; @@ -2759,19 +2773,21 @@ mod tests { assert_eq!(ctx.namespace_commit_generation(), 2); assert!(!ctx.namespace_commits_pending()); drop(snapshot_blocker); - let error = tokio::time::timeout(Duration::from_secs(1), acquire) + let (token, generation) = tokio::time::timeout(Duration::from_secs(1), acquire) .await .expect("snapshot admission must finish") - .expect_err("a namespace change during admission must not install a lease"); - assert_eq!( - error.to_string(), - "Io error: scanner publication lease namespace changed during acquisition" + .expect("an ordinary namespace commit during admission shades its bucket; it does not reject the lease"); + assert_eq!(generation, 0); + assert!( + ctx.data_movement_operation_gate().try_write_owned().is_err(), + "the installed lease holds the movement permit" ); - assert!(ctx.data_movement_operation_gate().try_write_owned().is_ok(), "no lease was installed"); + assert!(store.release_scanner_publication_lease(token).await); + assert!(ctx.data_movement_operation_gate().try_write_owned().is_ok()); } #[tokio::test] - async fn scanner_publication_lease_rechecks_namespace_after_validate_snapshot() { + async fn scanner_publication_lease_validates_across_a_completed_namespace_commit() { let ctx = Arc::new(InstanceContext::new()); let store = build_store_with_ctx(ctx.clone()); let (token, generation) = store @@ -2792,14 +2808,13 @@ mod tests { assert_eq!(ctx.namespace_commit_generation(), 2); assert!(!ctx.namespace_commits_pending()); drop(next_writer); - let error = tokio::time::timeout(Duration::from_secs(1), validate) + tokio::time::timeout(Duration::from_secs(1), validate) .await .expect("validation must finish without nesting movement read locks") - .expect_err("the final snapshot must reject a completed namespace commit"); - assert_eq!(error.to_string(), "Io error: scanner publication lease generation is stale"); + .expect("a completed ordinary namespace commit does not stale the lease"); assert!( ctx.data_movement_operation_gate().try_write_owned().is_err(), - "stale lookup retains the lease permit" + "validation retains the lease permit" ); assert!(store.release_scanner_publication_lease(token).await); assert!(ctx.data_movement_operation_gate().try_write_owned().is_ok()); diff --git a/crates/ecstore/src/store/multipart.rs b/crates/ecstore/src/store/multipart.rs index a2b1f8896..b4e817be6 100644 --- a/crates/ecstore/src/store/multipart.rs +++ b/crates/ecstore/src/store/multipart.rs @@ -32,6 +32,42 @@ fn data_movement_multipart_discovery_counts() -> &'static std::sync::Mutex, tokio::sync::oneshot::Receiver<()>); + +#[cfg(test)] +static ORPHAN_MULTIPART_CLEANUP_PAUSES: std::sync::OnceLock< + std::sync::Mutex>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +pub(crate) fn pause_orphan_multipart_cleanup_for_test( + bucket: &str, +) -> (tokio::sync::oneshot::Receiver<()>, tokio::sync::oneshot::Sender<()>) { + let (arrived_tx, arrived_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = tokio::sync::oneshot::channel(); + let previous = ORPHAN_MULTIPART_CLEANUP_PAUSES + .get_or_init(Default::default) + .lock() + .expect("orphan cleanup pause mutex should not poison") + .insert(bucket.to_owned(), (arrived_tx, release_rx)); + assert!(previous.is_none(), "each bucket has at most one orphan cleanup pause"); + (arrived_rx, release_tx) +} + +#[cfg(test)] +async fn pause_orphan_multipart_cleanup(bucket: &str) { + let pause = ORPHAN_MULTIPART_CLEANUP_PAUSES + .get_or_init(Default::default) + .lock() + .expect("orphan cleanup pause mutex should not poison") + .remove(bucket); + if let Some((arrived, release)) = pause { + let _ = arrived.send(()); + let _ = release.await; + } +} + fn decommission_multipart_target_clear_pending(opts: &ObjectOptions, target: Option<&ObjectInfo>) -> Result { let expected_mod_time = opts.mod_time.ok_or_else(|| Error::DecommissionCapacityBlocked { message: "multipart cleanup cannot prove exact target absence without a modification time".to_string(), @@ -520,7 +556,7 @@ impl ECStore { opts: &ObjectOptions, mutation_fence: Option<&ObjectLockDiagGuard>, ) -> Result<(MultipartUploadResult, usize, Option)> { - check_new_multipart_args(bucket, object)?; + check_new_object_args(bucket, object, opts.data_movement)?; let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; if self.single_pool() { @@ -529,6 +565,12 @@ impl ECStore { let result = self.pools[0].new_multipart_upload(bucket, object, &opts).await?; return Ok((result, 0, opts.expected_bucket_incarnation_id)); } + if opts.http_preconditions.is_some() + && !opts.data_movement + && let Some(err) = self.check_write_precondition(bucket, object, &opts).await + { + return Err(err); + } let capacity_owner = DecommissionCapacityOwner::from_options(&opts); if opts.data_movement && (opts.version_id.is_some() || capacity_owner.is_some()) { @@ -654,7 +696,10 @@ impl ECStore { opts: &ObjectOptions, ) -> Result { check_put_object_part_args(bucket, object, upload_id)?; - let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; + let (mut opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; + if let Some(guard) = bucket_lifecycle_guard { + opts.bucket_lifecycle_lock_owner = Some(Arc::new(guard)); + } if !self.single_pool() { opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await; } @@ -750,19 +795,12 @@ impl ECStore { opts: &ObjectOptions, ) -> Result<()> { check_abort_multipart_args(bucket, object, upload_id)?; - let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; - let opts = &opts; - - // TODO(backlog): defer DeleteUploadID to background for faster abort response - - if self.single_pool() { - return self.pools[0].abort_multipart_upload(bucket, object, upload_id, opts).await; - } + let (opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?; for pool_idx in self.existing_multipart_pool_order().await { let pool = &self.pools[pool_idx]; - let err = match pool.abort_multipart_upload(bucket, object, upload_id, opts).await { + let err = match pool.abort_multipart_upload(bucket, object, upload_id, &opts).await { Ok(_) => return Ok(()), Err(err) => { // @@ -775,7 +813,92 @@ impl ECStore { } } - Err(StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned())) + self.recover_orphan_multipart_upload(bucket, object, upload_id, opts, bucket_lifecycle_guard) + .await + } + + async fn recover_orphan_multipart_upload( + &self, + bucket: &str, + object: &str, + upload_id: &str, + opts: ObjectOptions, + bucket_lifecycle_guard: Option, + ) -> Result<()> { + let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()); + if self.ctx.lock_manager().is_disabled() { + return Err(invalid_upload()); + } + let Some(bucket_lifecycle_guard) = bucket_lifecycle_guard else { + // A signal borrowed from an enclosing operation cannot keep its lock + // alive when the request waiter disappears during physical cleanup. + return Err(invalid_upload()); + }; + let Some(expected_incarnation) = opts.expected_bucket_incarnation_id.filter(|id| !id.is_nil()) else { + return Err(invalid_upload()); + }; + // Recovery is a side path of an idempotent Abort. Until the first + // deletion is issued, anything that stops staging absence from being + // proven (an offline disk, lock contention, partially overlapping lock + // domains, unreadable bucket metadata) keeps the upload reported as + // gone, exactly as before recovery existed; only failures inside the + // deletion task are request failures, because a retry can finish them. + let (metadata, persisted) = + match crate::bucket::metadata_sys::get_config_from_disk_with_presence_in(&self.ctx, bucket).await { + Ok(loaded) => loaded, + Err(error) => { + debug!(bucket, object, upload_id, %error, "orphan multipart cleanup skipped: bucket metadata unavailable"); + return Err(invalid_upload()); + } + }; + if !persisted + || !metadata.bucket_creation_committed + || !metadata.bucket_incarnation_sidecar + || metadata.bucket_incarnation_id != expected_incarnation + { + return Err(invalid_upload()); + } + + // Lock order: bucket lifecycle, then upload locks in pool-index order. + // Prove metadata absence across every pool before deleting any staging. + let mut cleanup_guards = Vec::with_capacity(self.pools.len()); + for pool in &self.pools { + let guard = match pool + .get_disks_by_key(object) + .lock_orphan_multipart_cleanup(bucket, object, upload_id, &opts, &cleanup_guards) + .await + { + Ok(guard) => guard, + Err(error) => { + if !is_err_invalid_upload_id(&error) { + debug!(bucket, object, upload_id, %error, "orphan multipart cleanup skipped: staging absence not proven"); + } + return Err(invalid_upload()); + } + }; + cleanup_guards.push(guard); + } + let bucket = bucket.to_owned(); + let object = object.to_owned(); + let upload_id = upload_id.to_owned(); + tokio::spawn(async move { + // Own both the real bucket guard and all upload guards until every + // disk deletion settles, including after cancellation of the waiter. + let _bucket_lifecycle_guard = bucket_lifecycle_guard; + #[cfg(test)] + pause_orphan_multipart_cleanup(&bucket).await; + let mut recovered = false; + for guard in &cleanup_guards { + recovered |= guard.delete(&bucket, &object, &opts).await?; + } + if recovered { + Ok(()) + } else { + Err(StorageError::InvalidUploadID(bucket, object, upload_id)) + } + }) + .await + .map_err(|error| Error::other_with_context("orphan multipart cleanup task failed", error))? } pub(crate) async fn abort_multipart_upload_for_data_movement( @@ -970,6 +1093,15 @@ impl ECStore { Ok(()) } + async fn check_write_precondition(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Option { + let mut lookup_opts = opts.clone(); + let http_preconditions = lookup_opts.http_preconditions?; + lookup_opts.http_preconditions = None; + lookup_opts.no_lock = true; + let oi = self.get_object_info_for_delete(bucket, object, &lookup_opts).await; + crate::set_disk::SetDisks::evaluate_write_precondition(oi, &http_preconditions, bucket, object, opts) + } + #[instrument(skip(self))] pub(super) async fn handle_complete_multipart_upload( self: Arc, @@ -992,6 +1124,12 @@ impl ECStore { .complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts) .await; } + if opts.http_preconditions.is_some() + && !opts.data_movement + && let Some(err) = self.check_write_precondition(bucket, object, opts).await + { + return Err(err); + } match self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await { Ok(pool_idx) => { diff --git a/crates/ecstore/src/store/object.rs b/crates/ecstore/src/store/object.rs index 9e16d5b90..ff37c53b0 100644 --- a/crates/ecstore/src/store/object.rs +++ b/crates/ecstore/src/store/object.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use super::rebalance::support::{LatestObjectInfoCandidate, resolve_latest_object_info_candidates_with_pool_state}; use super::*; use crate::bucket::lifecycle::{ bucket_lifecycle_ops::{enqueue_committed_free_versions, eval_action_from_lifecycle}, @@ -30,7 +31,7 @@ use crate::bucket::lifecycle::{ }; use crate::bucket::metadata_sys::{ acquire_bucket_metadata_transaction_read_lock_in, get_bucket_incarnation_id_for_options_in, - get_cached_bucket_incarnation_id_in, get_object_lock_config_and_incarnation_from_disk_in, + get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in, get_object_lock_config_and_incarnation_from_disk_in, }; use crate::bucket::object_lock::objectlock_sys::{ check_object_lock_for_deletion_with_state, ensure_recursive_force_delete_allowed_for_state, @@ -39,7 +40,7 @@ use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObj use crate::bucket::versioning::VersioningApi; use crate::core::pools::{DecommissionCapacityAdmission, DecommissionCapacityOwner, ensure_decommission_capacity_mutation_id}; use crate::diagnostics::object_lock::ObjectLockAttempt; -use crate::disk::OldCurrentSize; +use crate::disk::{CurrentDeleteGeneration, OldCurrentSize}; use crate::object_api::{ NamespaceLockFence, ObjectLockConfigSnapshot, ScannerPublicationCommitScopeGuard, ScannerPublicationCommitState, TierFreeVersionReceiptSink, @@ -58,13 +59,14 @@ use crate::storage_api_contracts::{ namespace::NamespaceLocking as _, object::{DeleteAccounting, ObjectIO as _, ObjectOperations as _}, }; -use futures::StreamExt as _; +use futures::{StreamExt as _, TryStreamExt as _}; use parking_lot::Mutex as ParkingMutex; use rustfs_filemeta::ObjectPartInfo; use rustfs_io_metrics::{ record_object_lock_diag_acquire_duration, record_object_lock_diag_hold_duration, record_object_lock_diag_slow_acquire, record_object_lock_diag_slow_hold, }; +use std::collections::HashSet; use std::{ fmt, pin::Pin, @@ -1993,6 +1995,10 @@ fn batch_delete_creates_latest_marker(object: &ObjectToDelete, delete_config_sna versioned || version_suspended } +fn batch_delete_action_key(object: &ObjectToDelete) -> (&str, Option) { + (object.object_name.as_str(), object.version_id) +} + fn batch_delete_targets_pool(creates_latest_marker: bool, marker_target_pool_idx: Option, pool_idx: usize) -> bool { !creates_latest_marker || marker_target_pool_idx == Some(pool_idx) } @@ -2484,6 +2490,250 @@ fn sorted_unique_delete_object_names(objects: &[ObjectToDelete]) -> Vec<&str> { object_names } +#[cfg(test)] +static CONDITIONAL_DELETE_POST_ADMISSION_REBALANCE: std::sync::OnceLock>> = + std::sync::OnceLock::new(); + +#[cfg(test)] +struct ConditionalDeletePostAdmissionOverwrite { + pool_idx: usize, + object: String, + payload: Vec, + versioned: bool, +} + +#[cfg(test)] +static CONDITIONAL_DELETE_POST_ADMISSION_OVERWRITE: std::sync::OnceLock< + std::sync::Mutex>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +async fn inject_conditional_delete_post_admission_mutations(store: &ECStore, bucket: &str) { + let pool_idx = CONDITIONAL_DELETE_POST_ADMISSION_REBALANCE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional rebalance injection must not poison") + .remove(bucket); + if let Some(pool_idx) = pool_idx { + let mut pool_stats = vec![crate::services::rebalance::RebalanceStats::default(); store.pools.len()]; + pool_stats[pool_idx].participating = true; + pool_stats[pool_idx].info.status = crate::services::rebalance::RebalStatus::Started; + *store.rebalance_meta.write().await = Some(crate::services::rebalance::RebalanceMeta { + pool_stats, + ..Default::default() + }); + } + let overwrite = CONDITIONAL_DELETE_POST_ADMISSION_OVERWRITE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional overwrite injection must not poison") + .remove(bucket); + if let Some(overwrite) = overwrite { + // Model a writer in a different lock domain after admission. The + // destructive path must still compare metadata under the disk lease. + store.pools[overwrite.pool_idx] + .put_object( + bucket, + &overwrite.object, + &mut PutObjReader::from_vec(overwrite.payload), + &ObjectOptions { + no_lock: true, + versioned: overwrite.versioned, + ..Default::default() + }, + ) + .await + .expect("post-admission generation overwrite"); + } +} + +const CONDITIONAL_DELETE_SNAPSHOT_CONCURRENCY: usize = 16; +const CONDITIONAL_DELETE_CAPABILITY_SET_CONCURRENCY: usize = 16; + +#[cfg(test)] +struct ConditionalCapabilitySetProbeGateState { + started: std::sync::atomic::AtomicUsize, + active: std::sync::atomic::AtomicUsize, + max_active: std::sync::atomic::AtomicUsize, + reject_first: AtomicBool, + changed: tokio::sync::Notify, + release: tokio::sync::Semaphore, +} + +#[cfg(test)] +struct ConditionalCapabilitySetProbeGate { + bucket: String, + state: Arc, + abort_task: Option, +} + +#[cfg(test)] +static CONDITIONAL_CAPABILITY_SET_PROBE_GATES: std::sync::OnceLock< + std::sync::Mutex>>, +> = std::sync::OnceLock::new(); + +#[cfg(test)] +impl ConditionalCapabilitySetProbeGate { + fn install(bucket: &str, reject_first: bool) -> Self { + let state = Arc::new(ConditionalCapabilitySetProbeGateState { + started: std::sync::atomic::AtomicUsize::new(0), + active: std::sync::atomic::AtomicUsize::new(0), + max_active: std::sync::atomic::AtomicUsize::new(0), + reject_first: AtomicBool::new(reject_first), + changed: tokio::sync::Notify::new(), + release: tokio::sync::Semaphore::new(0), + }); + let previous = CONDITIONAL_CAPABILITY_SET_PROBE_GATES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional capability probe gates must not poison") + .insert(bucket.to_owned(), Arc::clone(&state)); + assert!(previous.is_none(), "one probe gate per test bucket"); + Self { + bucket: bucket.to_owned(), + state, + abort_task: None, + } + } + + async fn wait_started(&self, count: usize) { + tokio::time::timeout(Duration::from_secs(30), async { + loop { + let changed = self.state.changed.notified(); + if self.state.started.load(Ordering::Acquire) >= count { + break; + } + changed.await; + } + }) + .await + .expect("bounded set probes must reach the requested admission count"); + } +} + +#[cfg(test)] +impl Drop for ConditionalCapabilitySetProbeGate { + fn drop(&mut self) { + if let Some(task) = &self.abort_task { + task.abort(); + } + self.state.release.close(); + let mut gates = CONDITIONAL_CAPABILITY_SET_PROBE_GATES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional capability probe gates must not poison"); + if gates.get(&self.bucket).is_some_and(|state| Arc::ptr_eq(state, &self.state)) { + gates.remove(&self.bucket); + } + } +} + +#[cfg(test)] +struct ConditionalCapabilitySetProbeActivity(Arc); + +#[cfg(test)] +impl Drop for ConditionalCapabilitySetProbeActivity { + fn drop(&mut self) { + self.0.active.fetch_sub(1, Ordering::AcqRel); + } +} + +#[cfg(test)] +async fn await_conditional_capability_set_probe_gate(bucket: &str) -> Result<()> { + let state = CONDITIONAL_CAPABILITY_SET_PROBE_GATES + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("conditional capability probe gates must not poison") + .get(bucket) + .cloned(); + let Some(state) = state else { + return Ok(()); + }; + let active = state.active.fetch_add(1, Ordering::AcqRel) + 1; + let _activity = ConditionalCapabilitySetProbeActivity(Arc::clone(&state)); + state.max_active.fetch_max(active, Ordering::AcqRel); + state.started.fetch_add(1, Ordering::AcqRel); + state.changed.notify_one(); + state.release.acquire().await.map_err(|_| Error::OperationCanceled)?.forget(); + if state.reject_first.swap(false, Ordering::AcqRel) { + return Err(StorageError::GuardedDeleteUnsupported); + } + Ok(()) +} + +struct BatchDeletePhasePlan<'a> { + indices: &'a [usize], + pool_generations: HashMap>, + marker_targets: &'a HashMap, +} + +struct ConditionalDeleteSnapshot { + current: ObjectInfo, + current_pool_idx: usize, + pool_generations: HashMap, +} + +fn resolve_conditional_delete_snapshot( + bucket: &str, + object: &str, + results: Vec<(usize, Result<(ObjectInfo, CurrentDeleteGeneration)>)>, + suspended_pools: &[bool], + opts: &ObjectOptions, +) -> Result { + let mut candidates = Vec::with_capacity(results.len()); + let mut pool_generations = HashMap::with_capacity(results.len()); + for (idx, result) in results { + match result { + Ok((info, generation)) => { + pool_generations.insert(idx, generation); + candidates.push(LatestObjectInfoCandidate { + info: Some(info), + idx, + err: None, + }); + } + Err(error) if is_err_object_not_found(&error) || is_err_version_not_found(&error) => { + pool_generations.insert(idx, CurrentDeleteGeneration::Absent); + candidates.push(LatestObjectInfoCandidate { + info: None, + idx, + err: Some(error), + }); + } + // A readable old copy cannot establish the condition while a + // different pool might hold a newer, unreadable generation. + Err(error) => return Err(error), + } + } + let (current, current_pool_idx) = + resolve_latest_object_info_candidates_with_pool_state(candidates, suspended_pools, bucket, object, opts)?; + if current.delete_marker { + return Err(StorageError::ObjectNotFound(bucket.to_owned(), object.to_owned())); + } + Ok(ConditionalDeleteSnapshot { + current, + current_pool_idx, + pool_generations, + }) +} + +fn check_batch_delete_condition(object: &ObjectToDelete, snapshot: &ConditionalDeleteSnapshot, bucket: &str) -> Result<()> { + let Some(condition) = object.delete_condition.as_ref() else { + return Ok(()); + }; + if object.version_id.is_some() || condition.validate().is_err() { + return Err(StorageError::InvalidArgument( + bucket.to_owned(), + object.object_name.clone(), + "invalid current-object delete condition".to_owned(), + )); + } + if !condition.matches(snapshot.current.etag.as_deref().unwrap_or_default()) { + return Err(StorageError::PreconditionFailed); + } + Ok(()) +} + /// A CopyObject branch that writes a fresh local data stream owns new bytes and /// must not inherit the source version's remote-tier ownership record. FileMeta /// exposes internal transition fields through `ObjectInfo::user_defined` for @@ -2794,6 +3044,40 @@ impl RemoteTuplePublicationFence { } impl ECStore { + async fn conditional_delete_snapshot( + &self, + bucket: &str, + object: &str, + opts: &ObjectOptions, + ) -> Result { + let mut lookup_opts = opts.clone(); + lookup_opts.no_lock = true; + lookup_opts.metadata_cache_safe = false; + lookup_opts.suppress_read_repair = true; + lookup_opts.http_preconditions = None; + lookup_opts.version_id = None; + lookup_opts.conditional_delete_generations = None; + let results = join_all(self.pools.iter().map(|pool| { + let lookup_opts = &lookup_opts; + async move { + ( + pool.pool_idx, + pool.get_disks_by_key(object) + .conditional_delete_current_snapshot(bucket, object, lookup_opts) + .await, + ) + } + })) + .await; + let suspended_pools = { + let pool_meta = self.pool_meta.read().await; + (0..self.pools.len()) + .map(|idx| pool_meta.is_suspended(idx)) + .collect::>() + }; + resolve_conditional_delete_snapshot(bucket, object, results, &suspended_pools, &lookup_opts) + } + /// Acquire the pre-copy half of a data-movement publication fence. Legacy /// remote objects without a destination identity pin the named tier; an /// ordinary local source takes the zero-tier-manager-work `None` lease @@ -3173,7 +3457,7 @@ impl ECStore { Ok(guards) } - async fn acquire_object_read_lock_if_needed( + pub(super) async fn acquire_object_read_lock_if_needed( &self, op: &'static str, bucket: &str, @@ -4278,7 +4562,7 @@ impl ECStore { } async fn prepare_put_object(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<(String, ObjectOptions)> { - check_put_object_args(bucket, object)?; + check_new_object_args(bucket, object, opts.data_movement)?; let object = encode_dir_object(object); let mut opts = opts.clone(); @@ -4498,7 +4782,17 @@ impl ECStore { dst_opts: &ObjectOptions, ) -> Result { check_copy_obj_args(src_bucket, src_object)?; - check_copy_obj_args(dst_bucket, dst_object)?; + if src_bucket == dst_bucket && src_object == dst_object { + // In-place copies (metadata replace, restore) rewrite a key that + // already exists, so they must keep working for keys that new + // writes would no longer accept. + check_copy_obj_args(dst_bucket, dst_object)?; + } else { + // A copy onto another key creates that key: it must be admitted + // exactly like PutObject, since the copy reaches the pools' own + // put_object without passing through prepare_put_object. + check_new_object_args(dst_bucket, dst_object, dst_opts.data_movement)?; + } let src_object = encode_dir_object(src_object); let dst_object = encode_dir_object(dst_object); @@ -5003,7 +5297,7 @@ impl ECStore { } let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard.as_ref() { let (state, incarnation_id, config_revision) = - get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?; + get_delete_config_from_disk_in(&self.ctx, bucket, None, opts.lifecycle_expiry_policy.as_ref()).await?; opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence( self.id, bucket, @@ -5437,137 +5731,53 @@ impl ECStore { .await } - async fn handle_delete_objects_with_journal_and_accounting_inner( + /// Each phase owns separate disk transactions, receipts, and rollback. + /// The caller retains the batch namespace locks across both phases. + async fn run_batch_delete_phase( &self, bucket: &str, - objects: Vec, - opts: ObjectOptions, - tier_journal_api: Option>, + objects: &[ObjectToDelete], + opts: &ObjectOptions, + plan: BatchDeletePhasePlan<'_>, ) -> (Vec, Vec>, Vec>) { - // encode object name - let objects: Vec = objects - .iter() - .map(|v| { - let mut v = v.clone(); - v.object_name = encode_dir_object(v.object_name.as_str()); - v - }) - .collect(); - - // Default return value + let BatchDeletePhasePlan { + indices, + mut pool_generations, + marker_targets: conditional_marker_targets, + } = plan; + let mut active = vec![false; objects.len()]; + for idx in indices { + active[*idx] = true; + } let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let mut del_errs = vec![None; objects.len()]; let mut accounting = vec![None; objects.len()]; - - let mut del_errs = Vec::with_capacity(objects.len()); - for _ in 0..objects.len() { - del_errs.push(None) - } - - let mut opts = opts; - opts.tier_delete_journal_api = tier_journal_api; - let _bucket_lifecycle_guard = if is_meta_bucketname(bucket) { - None - } else { - match self.acquire_bucket_lifecycle_read_lock(bucket).await { - Ok(guard) => Some(Arc::new(guard)), - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - } - }; - if let Some(guard) = _bucket_lifecycle_guard.as_ref() { - opts.add_bucket_lifecycle_lock_guard(guard); - opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone()))); - } - if opts.delete_replication_config_snapshot.is_none() { - match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await { - Ok(snapshot) => opts.delete_replication_config_snapshot = Some(Arc::new(snapshot)), - Err(err) => { - let message = err.to_string(); - let errors = (0..objects.len()).map(|_| Some(Error::other(message.clone()))).collect(); - return (del_objects, errors, accounting); - } - } - } - if !is_meta_bucketname(bucket) - && let Err(err) = get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await - { - return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err); - } - let _object_lock_metadata_guard = if is_meta_bucketname(bucket) { - None - } else { - Some(Arc::new( - match acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await { - Ok(guard) => guard, - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - }, - )) - }; - if let Some(guard) = _object_lock_metadata_guard.as_ref() { - opts.add_namespace_lock_guard(guard); - } - let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard.as_ref() { - let (state, incarnation_id, config_revision) = - match get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await { - Ok(snapshot) => snapshot, - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - }; - opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence( - self.id, - bucket, - incarnation_id, - config_revision, - state, - opts.bucket_lifecycle_lock_fence - .clone() - .unwrap_or_else(NamespaceLockFence::new), - Arc::clone(metadata_guard), - ))); - Some(incarnation_id) - } else { - None - }; - if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id) - && expected != current - { - return return_batch_delete_lock_error_with_accounting( - objects.as_slice(), - StorageError::BucketNotFound(bucket.to_string()), - ); - } - if opts.expected_bucket_incarnation_id.is_none() { - opts.expected_bucket_incarnation_id = current_bucket_incarnation_id; - } - #[cfg(any(test, feature = "test-util"))] - if current_bucket_incarnation_id.is_some() { - pause_delete_after_object_lock_snapshot(bucket).await; - } - let _object_lock_guards = Arc::new(match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await { - Ok(guards) => guards, - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), - }); - opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), _object_lock_guards.clone()))); - #[cfg(any(test, feature = "test-util"))] - if !_object_lock_guards.is_empty() { - notify_delete_namespace_acquired(bucket); - } - let delete_config_snapshot = opts .delete_replication_config_snapshot .as_deref() .expect("batch delete replication config snapshot should be loaded"); let latest_marker_objects = objects .iter() - .map(|object| batch_delete_creates_latest_marker(object, delete_config_snapshot)) + .enumerate() + .map(|(idx, object)| { + active[idx] && del_errs[idx].is_none() && batch_delete_creates_latest_marker(object, delete_config_snapshot) + }) .collect::>(); - let marker_target_results = join_all(objects.iter().zip(&latest_marker_objects).map( - |(object, creates_marker)| async move { + let marker_target_results = join_all(objects.iter().zip(&latest_marker_objects).map(|(object, creates_marker)| { + let admitted_target = conditional_marker_targets.get(&object.object_name).copied(); + async move { if *creates_marker { - Some(self.get_pool_idx_no_lock(bucket, &object.object_name, 0).await) + // Routing is part of condition admission. A rebalance + // start cannot replace the validated current owner. + Some(match admitted_target { + Some(pool_idx) => Ok(pool_idx), + None => self.get_pool_idx_no_lock(bucket, &object.object_name, 0).await, + }) } else { None } - }, - )) + } + })) .await; let mut marker_target_pool_indices = Vec::with_capacity(objects.len()); for (idx, target_result) in marker_target_results.into_iter().enumerate() { @@ -5593,7 +5803,13 @@ impl ECStore { .await { Ok(fence) => Some(fence), - Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + Err(err) => { + let (mut deleted, errors, receipts) = return_batch_delete_lock_error_with_accounting(objects, err); + for (result, object) in deleted.iter_mut().zip(objects) { + result.object_name.clone_from(&object.object_name); + } + return (deleted, errors, receipts); + } } } else { None @@ -5601,15 +5817,22 @@ impl ECStore { let mut futures = Vec::with_capacity(self.pools.len()); for pool in self.pools.iter() { - if self.is_pool_rebalancing(pool.pool_idx).await { + let rebalancing = self.is_pool_rebalancing(pool.pool_idx).await; + let guarded_generations = pool_generations.get(&pool.pool_idx); + if rebalancing && guarded_generations.is_none() { continue; } let (object_indices, pool_objects): (Vec<_>, Vec<_>) = objects .iter() .enumerate() - .filter(|(idx, _)| { - batch_delete_targets_pool(latest_marker_objects[*idx], marker_target_pool_indices[*idx], pool.pool_idx) + .filter(|(idx, object)| { + active[*idx] && del_errs[*idx].is_none() + // A newly rebalancing pool still owes every admitted + // conditional key its guarded mutation. Ordinary keys + // retain the existing rebalance skip behavior. + && (!rebalancing || guarded_generations.is_some_and(|generations| generations.contains_key(&object.object_name))) + && batch_delete_targets_pool(latest_marker_objects[*idx], marker_target_pool_indices[*idx], pool.pool_idx) }) .map(|(idx, object)| (idx, object.clone())) .unzip(); @@ -5617,7 +5840,8 @@ impl ECStore { continue; } - let pool_opts = opts.clone(); + let mut pool_opts = (*opts).clone(); + pool_opts.conditional_delete_generations = pool_generations.remove(&pool.pool_idx).map(Arc::new); futures.push(async move { #[cfg(test)] let pool_object_names = pool_objects @@ -5665,6 +5889,348 @@ impl ECStore { } } + (del_objects, del_errs, accounting) + } + + async fn handle_delete_objects_with_journal_and_accounting_inner( + &self, + bucket: &str, + objects: Vec, + opts: ObjectOptions, + tier_journal_api: Option>, + ) -> (Vec, Vec>, Vec>) { + // encode object name + let mut objects: Vec = objects + .iter() + .map(|v| { + let mut v = v.clone(); + v.object_name = encode_dir_object(v.object_name.as_str()); + v + }) + .collect(); + + // Default return value + let mut del_objects = vec![DeletedObject::default(); objects.len()]; + let mut accounting = vec![None; objects.len()]; + + let mut del_errs = Vec::with_capacity(objects.len()); + for _ in 0..objects.len() { + del_errs.push(None) + } + + let mut opts = opts; + opts.conditional_delete_generations = None; + opts.tier_delete_journal_api = tier_journal_api; + let _bucket_lifecycle_guard = if is_meta_bucketname(bucket) { + None + } else { + match self.acquire_bucket_lifecycle_read_lock(bucket).await { + Ok(guard) => Some(Arc::new(guard)), + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + } + }; + if let Some(guard) = _bucket_lifecycle_guard.as_ref() { + opts.add_bucket_lifecycle_lock_guard(guard); + opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone()))); + } + if opts.delete_replication_config_snapshot.is_none() { + match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await { + Ok(snapshot) => opts.delete_replication_config_snapshot = Some(Arc::new(snapshot)), + Err(err) => { + let message = err.to_string(); + let errors = (0..objects.len()).map(|_| Some(Error::other(message.clone()))).collect(); + return (del_objects, errors, accounting); + } + } + } + if !is_meta_bucketname(bucket) + && let Err(err) = get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await + { + return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err); + } + let _object_lock_metadata_guard = if is_meta_bucketname(bucket) { + None + } else { + Some(Arc::new( + match acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await { + Ok(guard) => guard, + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + }, + )) + }; + if let Some(guard) = _object_lock_metadata_guard.as_ref() { + opts.add_namespace_lock_guard(guard); + } + let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard.as_ref() { + let (state, incarnation_id, config_revision) = + match get_delete_config_from_disk_in(&self.ctx, bucket, None, opts.lifecycle_expiry_policy.as_ref()).await { + Ok(snapshot) => snapshot, + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + }; + opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence( + self.id, + bucket, + incarnation_id, + config_revision, + state, + opts.bucket_lifecycle_lock_fence + .clone() + .unwrap_or_else(NamespaceLockFence::new), + Arc::clone(metadata_guard), + ))); + Some(incarnation_id) + } else { + None + }; + if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id) + && expected != current + { + return return_batch_delete_lock_error_with_accounting( + objects.as_slice(), + StorageError::BucketNotFound(bucket.to_string()), + ); + } + if opts.expected_bucket_incarnation_id.is_none() { + opts.expected_bucket_incarnation_id = current_bucket_incarnation_id; + } + #[cfg(any(test, feature = "test-util"))] + if current_bucket_incarnation_id.is_some() { + pause_delete_after_object_lock_snapshot(bucket).await; + } + let _object_lock_guards = Arc::new(match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await { + Ok(guards) => guards, + Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err), + }); + opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), _object_lock_guards.clone()))); + #[cfg(any(test, feature = "test-util"))] + if !_object_lock_guards.is_empty() { + notify_delete_namespace_acquired(bucket); + } + + let requested_conditions = objects + .iter() + .map(|object| object.delete_condition.is_some()) + .collect::>(); + let has_conditions = requested_conditions.iter().any(|condition| *condition); + let mut pool_generations: HashMap> = HashMap::new(); + let mut conditional_marker_targets = HashMap::new(); + let mut representatives: Vec = (0..objects.len()).collect(); + let mut admitted_conditions = vec![false; objects.len()]; + if has_conditions { + let mut rebalancing = false; + for pool in &self.pools { + rebalancing |= self.is_pool_rebalancing(pool.pool_idx).await; + } + let mut conditional_names = Vec::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if let Some(condition) = object.delete_condition.as_ref() { + if self.ctx.lock_manager().is_disabled() || object.version_id.is_some() || condition.validate().is_err() { + del_errs[idx] = Some(StorageError::InvalidArgument( + bucket.to_owned(), + object.object_name.clone(), + "conditional batch delete requires a valid current-object condition and namespace locking".to_owned(), + )); + } else if rebalancing { + del_errs[idx] = Some(StorageError::SlowDown); + } else { + conditional_names.push(object.object_name.clone()); + } + } + } + conditional_names.sort_unstable(); + conditional_names.dedup(); + let snapshots = futures::stream::iter(conditional_names.into_iter().map(|object| { + let opts = &opts; + async move { + let snapshot = self.conditional_delete_snapshot(bucket, &object, opts).await; + (object, snapshot) + } + })) + .buffer_unordered(CONDITIONAL_DELETE_SNAPSHOT_CONCURRENCY) + .collect::>() + .await; + + for (idx, object) in objects.iter_mut().enumerate() { + if object.delete_condition.is_none() || del_errs[idx].is_some() { + continue; + } + let snapshot = snapshots + .get(&object.object_name) + .expect("each valid conditional key was included in the bounded snapshot pass"); + let result = match snapshot { + Ok(snapshot) => check_batch_delete_condition(object, snapshot, bucket), + Err(error) => Err(error.clone()), + }; + if let Err(error) = result { + del_errs[idx] = Some(error); + continue; + } + let snapshot = snapshot.as_ref().expect("condition admission requires a resolved snapshot"); + if !conditional_marker_targets.contains_key(&object.object_name) { + conditional_marker_targets.insert(object.object_name.clone(), snapshot.current_pool_idx); + } + for (pool_idx, generation) in &snapshot.pool_generations { + let generations = pool_generations.entry(*pool_idx).or_default(); + if !generations.contains_key(&object.object_name) { + generations.insert(object.object_name.clone(), generation.clone()); + } + } + // The condition has been consumed once, against the global + // current object; disks receive only their local generation. + admitted_conditions[idx] = true; + object.delete_condition = None; + } + + // All relevant peers are checked before marker selection, capacity + // admission, or any destructive work, even in a mixed batch. + let mut probe_sets = Vec::new(); + for (pool_idx, generations) in &pool_generations { + let pool = &self.pools[*pool_idx]; + let mut checked_sets = HashSet::new(); + for object in generations.keys() { + let set = pool.get_disks_by_key(object); + if checked_sets.insert(set.set_index) { + probe_sets.push(set); + } + } + } + let preflight = futures::stream::iter(probe_sets.into_iter().map(|set| async move { + #[cfg(test)] + await_conditional_capability_set_probe_gate(bucket).await?; + set.preflight_guarded_delete_versions().await + })) + .buffer_unordered(CONDITIONAL_DELETE_CAPABILITY_SET_CONCURRENCY) + .try_collect::>() + .await; + // Every selected set must succeed before conditional mutation. + // These probes are read-only: the first error can safely cancel + // pending work instead of holding namespace locks for its timeout. + if let Err(error) = preflight { + for (idx, admitted) in admitted_conditions.iter().enumerate() { + if *admitted { + del_errs[idx] = Some(error.clone()); + } + } + pool_generations.clear(); + conditional_marker_targets.clear(); + } + } + + #[cfg(test)] + inject_conditional_delete_post_admission_mutations(self, bucket).await; + + if !has_conditions { + let indices = (0..objects.len()).collect::>(); + (del_objects, del_errs, accounting) = self + .run_batch_delete_phase( + bucket, + &objects, + &opts, + BatchDeletePhasePlan { + indices: &indices, + pool_generations, + marker_targets: &conditional_marker_targets, + }, + ) + .await; + } else { + let mut conditional_actions = HashMap::with_capacity(objects.len()); + let mut conditional_indices = Vec::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if admitted_conditions[idx] && del_errs[idx].is_none() { + let representative = *conditional_actions.entry(batch_delete_action_key(object)).or_insert(idx); + representatives[idx] = representative; + if representative == idx { + conditional_indices.push(idx); + } + } + } + if !conditional_indices.is_empty() { + let (deleted, mut errors, mut receipts) = self + .run_batch_delete_phase( + bucket, + &objects, + &opts, + BatchDeletePhasePlan { + indices: &conditional_indices, + pool_generations, + marker_targets: &conditional_marker_targets, + }, + ) + .await; + for idx in &conditional_indices { + del_objects[*idx] = deleted[*idx].clone(); + del_errs[*idx] = errors[*idx].take(); + accounting[*idx] = receipts[*idx].take(); + } + } + let successful_conditional_actions = conditional_actions + .into_iter() + .filter(|(_, idx)| del_errs[*idx].is_none()) + .collect::>(); + let mut ordinary_actions = HashMap::with_capacity(objects.len()); + let mut ordinary_indices = Vec::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if requested_conditions[idx] || del_errs[idx].is_some() { + continue; + } + let action = batch_delete_action_key(object); + if object.expected_identity.is_none() + && let Some(representative) = successful_conditional_actions.get(&action) + { + // Only a committed conditional action can satisfy its + // unconditional alias. A CAS failure grants no deletion. + representatives[idx] = *representative; + continue; + } + let representative = *ordinary_actions.entry(action).or_insert(idx); + representatives[idx] = representative; + if representative == idx { + ordinary_indices.push(idx); + } + } + if !ordinary_indices.is_empty() { + // The guarded phase has finished all rollback/receipt work. + // These actions receive neither its CAS nor its transaction. + let (deleted, mut errors, mut receipts) = self + .run_batch_delete_phase( + bucket, + &objects, + &opts, + BatchDeletePhasePlan { + indices: &ordinary_indices, + pool_generations: HashMap::new(), + marker_targets: &HashMap::new(), + }, + ) + .await; + for idx in ordinary_indices { + del_objects[idx] = deleted[idx].clone(); + del_errs[idx] = errors[idx].take(); + accounting[idx] = receipts[idx].take(); + } + } + } + + if has_conditions { + for (idx, representative) in representatives.into_iter().enumerate() { + if representative != idx { + del_objects[idx] = del_objects[representative].clone(); + del_errs[idx] = del_errs[representative].clone(); + accounting[idx] = None; + } + } + let mut first_successes = HashMap::with_capacity(objects.len()); + for (idx, object) in objects.iter().enumerate() { + if del_errs[idx].is_none() { + let first = *first_successes.entry(batch_delete_action_key(object)).or_insert(idx); + if first != idx && accounting[idx].is_some() { + accounting[first] = accounting[idx].take(); + } + } + } + } + #[cfg(test)] for (idx, object) in objects.iter().enumerate() { if del_errs[idx].is_none() && del_objects[idx].delete_marker { @@ -8784,6 +9350,99 @@ mod tests { assert_eq!(sorted_unique_delete_object_names(&objects), vec!["alpha", "beta"]); } + fn conditional_pool_fixture(etag: &str, timestamp: i64) -> (ObjectInfo, CurrentDeleteGeneration) { + let mod_time = OffsetDateTime::from_unix_timestamp(timestamp).expect("fixture timestamp"); + let data_dir = Some(Uuid::from_u128(u128::try_from(timestamp).expect("positive fixture timestamp"))); + let info = ObjectInfo { + etag: Some(etag.to_owned()), + mod_time: Some(mod_time), + data_dir, + ..Default::default() + }; + let generation = CurrentDeleteGeneration::Present { + version_id: None, + data_dir, + mod_time, + delete_marker: false, + etag: etag.to_owned(), + transition: None, + }; + (info, generation) + } + + #[test] + fn conditional_delete_snapshot_rejects_unknown_pool_even_with_readable_old_copy() { + let error = resolve_conditional_delete_snapshot( + "bucket", + "key", + vec![ + (0, Ok(conditional_pool_fixture("old", 1))), + (1, Err(StorageError::ErasureReadQuorum)), + ], + &[false, false], + &ObjectOptions::default(), + ) + .err() + .expect("unknown pool must veto admission"); + assert_eq!(error, StorageError::ErasureReadQuorum); + } + + #[test] + fn conditional_delete_snapshot_keeps_each_pool_generation_and_explicit_absence() { + let old = conditional_pool_fixture("old", 1); + let new = conditional_pool_fixture("new", 2); + let old_generation = old.1.clone(); + let new_generation = new.1.clone(); + let snapshot = resolve_conditional_delete_snapshot( + "bucket", + "key", + vec![ + (0, Ok(old)), + (1, Ok(new)), + (2, Err(StorageError::ObjectNotFound("bucket".to_owned(), "key".to_owned()))), + ], + &[true, false, false], + &ObjectOptions::default(), + ) + .expect("all pools are known"); + assert_eq!(snapshot.current.etag.as_deref(), Some("new")); + assert_eq!(snapshot.current_pool_idx, 1); + assert_eq!(snapshot.pool_generations.get(&0), Some(&old_generation)); + assert_eq!(snapshot.pool_generations.get(&1), Some(&new_generation)); + assert_eq!(snapshot.pool_generations.get(&2), Some(&CurrentDeleteGeneration::Absent)); + let conditions = ["old", "new", "*", "new"]; + let results = conditions.map(|etag| { + check_batch_delete_condition( + &ObjectToDelete { + object_name: "key".to_owned(), + delete_condition: Some( + crate::storage_api_contracts::object::DeleteObjectCondition::parse(etag).expect("valid condition"), + ), + ..Default::default() + }, + &snapshot, + "bucket", + ) + }); + assert_eq!(results[0], Err(StorageError::PreconditionFailed)); + assert!(results[1..].iter().all(Result::is_ok), "duplicate conditions use one current snapshot"); + } + + #[test] + fn conditional_delete_snapshot_rejects_current_marker_and_missing_object() { + let (mut marker, generation) = conditional_pool_fixture("marker", 2); + marker.delete_marker = true; + for results in [ + vec![(0, Ok((marker, generation)))], + vec![(0, Err(StorageError::ObjectNotFound("bucket".to_owned(), "key".to_owned())))], + ] { + let error = resolve_conditional_delete_snapshot("bucket", "key", results, &[false], &ObjectOptions::default()) + .err() + .expect("missing live object must fail"); + assert!(is_err_object_not_found(&error)); + } + } + async fn new_read_lock_test_store() -> ECStore { let format = FormatV3::new(1, 2); let endpoints = vec![ @@ -9152,6 +9811,10 @@ mod tests { } async fn multipool_version_test_store(bucket: &str) -> (Vec, Arc) { + multipool_delete_test_store(bucket, true).await + } + + async fn multipool_delete_test_store(bucket: &str, versioned: bool) -> (Vec, Arc) { let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; let (second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; @@ -9162,15 +9825,587 @@ mod tests { .handle_make_bucket( bucket, &MakeBucketOptions { - versioning_enabled: true, + versioning_enabled: versioned, ..Default::default() }, ) .await - .expect("create the versioned bucket in both pools"); + .expect("create the bucket in both pools"); (dirs, store) } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_set_probes_are_bounded_before_mutation() { + assert_conditional_set_probe_admission(false).await; + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_set_probe_error_cancels_pending_probes() { + assert_conditional_set_probe_admission(true).await; + } + + async fn assert_conditional_set_probe_admission(reject_first: bool) { + use crate::storage_api_contracts::object::DeleteObjectCondition; + let bucket = if reject_first { + "conditional-capability-set-rejected" + } else { + "conditional-capability-set-supported" + }; + let protected = "protected.bin"; + let ordinary = "ordinary.bin"; + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let mut dirs = Vec::new(); + let mut sets = Vec::new(); + for _ in 0..17 { + let (set_dirs, set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + dirs.extend(set_dirs); + sets.push(set); + } + let store = Arc::new(new_prepared_reader_test_store_with_ctx(&sets, ctx).await); + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + store + .handle_make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create all pool buckets"); + let protected_before = store.pools[0] + .put_object( + bucket, + protected, + &mut PutObjReader::from_vec(b"protected data".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("write conditional target"); + let ordinary_before = store.pools[0] + .put_object( + bucket, + ordinary, + &mut PutObjReader::from_vec(b"ordinary data".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("write ordinary control"); + let mut gate = ConditionalCapabilitySetProbeGate::install(bucket, reject_first); + let delete_store = Arc::clone(&store); + let condition_etag = protected_before.etag.clone().expect("conditional target ETag"); + let delete = tokio::spawn(async move { + delete_store + .handle_delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: protected.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(condition_etag)), + ..Default::default() + }, + ObjectToDelete { + object_name: ordinary.to_owned(), + ..Default::default() + }, + ], + ObjectOptions::default(), + ) + .await + }); + gate.abort_task = Some(delete.abort_handle()); + gate.wait_started(16).await; + assert_eq!(gate.state.started.load(Ordering::Acquire), 16, "the seventeenth probe must remain queued"); + assert_eq!(gate.state.active.load(Ordering::Acquire), 16); + let read_options = ObjectOptions { + no_lock: true, + metadata_cache_safe: false, + suppress_read_repair: true, + ..Default::default() + }; + for (key, before) in [(protected, &protected_before), (ordinary, &ordinary_before)] { + let current = store.pools[0] + .get_object_info(bucket, key, &read_options) + .await + .expect("read disk metadata under the retained outer write lock"); + assert_eq!(current.data_dir, before.data_dir, "pending probes cannot mutate {key}"); + assert_eq!(current.etag, before.etag); + } + gate.state.release.add_permits(1); + if !reject_first { + gate.wait_started(17).await; + assert!(!delete.is_finished(), "one successful set cannot bypass still-running capability probes"); + for (key, before) in [(protected, &protected_before), (ordinary, &ordinary_before)] { + let current = store.pools[0] + .get_object_info(bucket, key, &read_options) + .await + .expect("read metadata while the final set remains paused"); + assert_eq!(current.data_dir, before.data_dir, "all sets must approve before mutating {key}"); + assert_eq!(current.etag, before.etag); + } + gate.state.release.add_permits(16); + } + let (_, errors) = tokio::time::timeout(Duration::from_secs(30), delete) + .await + .expect("capability admission must finish without releasing failed peers") + .expect("delete task joins"); + assert_eq!(gate.state.max_active.load(Ordering::Acquire), 16, "capability fanout must be bounded"); + assert_eq!( + gate.state.active.load(Ordering::Acquire), + 0, + "completed or canceled probes must release their activity" + ); + assert_eq!( + gate.state.started.load(Ordering::Acquire), + if reject_first { 16 } else { 17 }, + "a rejected capability must cancel queued set probes" + ); + if reject_first { + assert_eq!(errors[0], Some(StorageError::GuardedDeleteUnsupported)); + let after = store + .get_object_info(bucket, protected, &ObjectOptions::default()) + .await + .expect("rejected condition preserves data"); + assert_eq!(after.data_dir, protected_before.data_dir); + assert_eq!(after.etag, protected_before.etag); + } else { + assert!(errors[0].is_none(), "all supported sets authorize conditional deletion: {errors:?}"); + assert!( + store + .get_object_info(bucket, protected, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + assert!( + errors[1].is_none(), + "ordinary deletion remains independent of capability admission: {errors:?}" + ); + assert!( + store + .get_object_info(bucket, ordinary, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_conflict_preserves_unconditional_same_key_actions() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + for delete_current in [false, true] { + let bucket = if delete_current { + "conditional-conflict-current" + } else { + "conditional-conflict-history" + }; + let object = "phase-conflict.bin"; + let (_dirs, store) = multipool_version_test_store(bucket).await; + let options = ObjectOptions { + versioned: true, + ..Default::default() + }; + let history = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"history".to_vec()), &options) + .await + .expect("write unconditional historical target"); + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"admitted".to_vec()), &options) + .await + .expect("write admitted current generation"); + CONDITIONAL_DELETE_POST_ADMISSION_OVERWRITE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("overwrite injection lock") + .insert( + bucket.to_owned(), + ConditionalDeletePostAdmissionOverwrite { + pool_idx: 1, + object: object.to_owned(), + payload: b"replacement".to_vec(), + versioned: true, + }, + ); + let conditional = ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current ETag"))), + ..Default::default() + }; + let (deleted, errors) = store + .handle_delete_objects( + bucket, + vec![ + conditional.clone(), + ObjectToDelete { + object_name: object.to_owned(), + version_id: if delete_current { None } else { history.version_id }, + ..Default::default() + }, + conditional, + ], + options, + ) + .await; + assert_eq!(errors[0], Some(StorageError::ConditionalDeleteConflict)); + assert_eq!(errors[2], Some(StorageError::ConditionalDeleteConflict)); + assert!( + errors[1].is_none(), + "unconditional action must not inherit the current-object CAS: {errors:?}" + ); + let versions = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list independent transaction results"); + if delete_current { + let marker = deleted[1] + .delete_marker_version_id + .expect("ordinary current delete creates its own marker"); + assert_eq!( + versions.objects.iter().filter(|entry| entry.delete_marker).count(), + 1, + "a rejected guarded transaction cannot leave or roll back the ordinary marker" + ); + assert!( + versions + .objects + .iter() + .any(|entry| entry.delete_marker && entry.version_id == Some(marker)) + ); + } else { + assert_eq!(deleted[1].version_id, history.version_id); + assert!(!versions.objects.iter().any(|entry| entry.version_id == history.version_id)); + assert!(versions.objects.iter().all(|entry| !entry.delete_marker)); + let mut reader = store + .handle_get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("replacement survives the failed guarded action"); + let mut body = Vec::new(); + reader + .stream + .read_to_end(&mut body) + .await + .expect("read surviving replacement"); + assert_eq!(body, b"replacement"); + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_success_reuses_unconditional_current_alias_once() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + for versioned in [false, true] { + let bucket = if versioned { + "conditional-alias-marker" + } else { + "conditional-alias-plain" + }; + let object = "phase-alias.bin"; + let (_dirs, store) = multipool_delete_test_store(bucket, versioned).await; + let options = ObjectOptions { + versioned, + ..Default::default() + }; + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"one action".to_vec()), &options) + .await + .expect("write current alias target"); + let conditional = ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current ETag"))), + ..Default::default() + }; + let (deleted, errors, accounting) = store + .handle_delete_objects_with_journal_and_accounting( + bucket, + vec![ + ObjectToDelete { + object_name: object.to_owned(), + ..Default::default() + }, + conditional.clone(), + conditional, + ], + options, + None, + ) + .await; + assert!(errors.iter().all(Option::is_none), "all aliases succeed: {errors:?}"); + if versioned { + let marker = deleted[0].delete_marker_version_id.expect("one physical marker"); + assert!(deleted.iter().all(|entry| entry.delete_marker_version_id == Some(marker))); + let versions = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list reused marker"); + assert_eq!(versions.objects.iter().filter(|entry| entry.delete_marker).count(), 1); + } else { + assert!(accounting[0].is_some(), "the first successful input owns the physical accounting"); + assert!(accounting[1..].iter().all(Option::is_none)); + assert!( + store + .get_object_info(bucket, object, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)) + ); + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_directory_marker_keeps_synthetic_null_delete_distinct() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + let bucket = "conditional-directory-targets"; + let object = "dir/"; + let (_dirs, store) = multipool_version_test_store(bucket).await; + // Directory PUTs use the null data version even in an enabled bucket. + let current = store + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(b"directory payload".to_vec()), + &ObjectOptions::default(), + ) + .await + .expect("write the null directory data version"); + let before = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list the null data fixture"); + assert_eq!(before.objects.len(), 1); + assert!(before.objects[0].version_id.is_none_or(|id| id.is_nil()) && !before.objects[0].delete_marker); + let (deleted, errors) = store + .handle_delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.expect("directory ETag"))), + ..Default::default() + }, + ObjectToDelete { + object_name: object.to_owned(), + version_id: Some(Uuid::nil()), + synthetic_version_id: true, + ..Default::default() + }, + ], + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await; + assert!(errors.iter().all(Option::is_none), "both distinct targets must succeed: {errors:?}"); + let marker = deleted[0] + .delete_marker_version_id + .expect("conditional current delete creates a marker"); + assert!(!marker.is_nil()); + assert!(deleted[0].delete_marker); + assert!( + !deleted[1].delete_marker, + "the synthetic request deletes null data, not the created marker" + ); + assert_eq!(deleted[1].version_id, Some(Uuid::nil())); + let after = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list both action results"); + assert_eq!(after.objects.len(), 1, "one UUID marker remains and null data must be removed"); + assert!(after.objects[0].delete_marker); + assert_eq!(after.objects[0].version_id, Some(marker)); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_freezes_routes_when_rebalance_starts_after_admission() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + for versioned in [false, true] { + let bucket = if versioned { + "conditional-route-versioned" + } else { + "conditional-route-plain" + }; + let object = "conditional-routed.bin"; + let ordinary = "ordinary-routed.bin"; + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + let (second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + dirs.extend(second_dirs); + let store = Arc::new(new_prepared_reader_test_store_with_ctx(&[first_set, second_set], ctx).await); + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + store + .handle_make_bucket( + bucket, + &MakeBucketOptions { + versioning_enabled: versioned, + ..Default::default() + }, + ) + .await + .expect("create bucket in both pools"); + let options = ObjectOptions { + versioned, + ..Default::default() + }; + store.pools[0] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"old".to_vec()), &options) + .await + .expect("write an older copy to the other pool"); + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"current".to_vec()), &options) + .await + .expect("write the current copy to the future rebalance pool"); + let ordinary_before = store.pools[1] + .put_object(bucket, ordinary, &mut PutObjReader::from_vec(b"ordinary".to_vec()), &options) + .await + .expect("write the unconditional control key to the same pool"); + CONDITIONAL_DELETE_POST_ADMISSION_REBALANCE + .get_or_init(|| std::sync::Mutex::new(HashMap::new())) + .lock() + .expect("rebalance injection lock") + .insert(bucket.to_owned(), 1); + let (deleted, errors) = store + .handle_delete_objects( + bucket, + vec![ + ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current ETag"))), + ..Default::default() + }, + ObjectToDelete { + object_name: ordinary.to_owned(), + version_id: ordinary_before.version_id, + ..Default::default() + }, + ], + options, + ) + .await; + assert!( + store.is_pool_rebalancing(1).await, + "rebalance must have started in the admission-to-dispatch window" + ); + assert!(errors[0].is_none(), "the frozen conditional action must commit: {errors:?}"); + *store.rebalance_meta.write().await = None; + let ordinary_after = store.pools[1] + .get_object_info(bucket, ordinary, &ObjectOptions::default()) + .await + .expect("unconditional key must retain the existing rebalance skip behavior"); + assert_eq!(ordinary_after.data_dir, ordinary_before.data_dir); + if versioned { + let marker = deleted[0].delete_marker_version_id.expect("conditional marker result"); + for pool in &store.pools { + let versions = pool + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list the frozen marker destination"); + assert_eq!( + versions + .objects + .iter() + .any(|entry| entry.delete_marker && entry.version_id == Some(marker)), + pool.pool_idx == 1, + "the marker must stay with the admitted current owner" + ); + } + } else { + for pool in &store.pools { + assert!( + pool.get_object_info(bucket, object, &ObjectOptions::default()) + .await + .err() + .is_some_and(|error| is_err_object_not_found(&error)), + "the current copy and stale copy must both be deleted" + ); + } + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn conditional_batch_delete_mixed_duplicates_preserve_historical_actions() { + use crate::storage_api_contracts::object::DeleteObjectCondition; + let bucket = "conditional-batch-duplicates"; + let object = "conditional-history.bin"; + let (_dirs, store) = multipool_version_test_store(bucket).await; + let options = ObjectOptions { + versioned: true, + ..Default::default() + }; + let old = store.pools[0] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"old".to_vec()), &options) + .await + .expect("write older pool-local copy"); + let current = store.pools[1] + .put_object(bucket, object, &mut PutObjReader::from_vec(b"new".to_vec()), &options) + .await + .expect("write current pool-local copy"); + let mut objects = vec![ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(old.etag.clone().expect("old etag"))), + ..Default::default() + }]; + objects.extend((0..997).map(|_| ObjectToDelete { + object_name: object.to_owned(), + delete_condition: Some(DeleteObjectCondition::MatchEtag(current.etag.clone().expect("current etag"))), + ..Default::default() + })); + objects.push(ObjectToDelete { + object_name: object.to_owned(), + version_id: old.version_id, + ..Default::default() + }); + objects.push(ObjectToDelete { + object_name: "missing".to_owned(), + delete_condition: Some(DeleteObjectCondition::Exists), + ..Default::default() + }); + let calls = disk_call_counters::observe(object); + let (deleted, errors, accounting) = store + .handle_delete_objects_with_journal_and_accounting(bucket, objects, options, None) + .await; + assert!( + calls.total(disk_call_counters::KIND_READ_VERSION) <= 64, + "997 identical conditions must share one metadata snapshot and mutation plan" + ); + assert_eq!(errors[0], Some(StorageError::PreconditionFailed)); + assert!( + errors[1..=998].iter().all(Option::is_none), + "accepted conditions and historical delete: {errors:?}" + ); + assert!(errors[999].as_ref().is_some_and(is_err_object_not_found)); + let marker_id = deleted[1].delete_marker_version_id.expect("one created marker"); + assert!( + deleted[1..=997] + .iter() + .all(|entry| entry.delete_marker && entry.delete_marker_version_id == Some(marker_id)) + ); + assert!( + accounting[2..=997].iter().all(Option::is_none), + "duplicate actions cannot duplicate accounting" + ); + let versions = store + .clone() + .inner_list_object_versions(bucket, object, None, None, None, 10) + .await + .expect("list remaining versions"); + assert_eq!(versions.objects.len(), 2, "one marker and the current data version must remain"); + assert!( + versions + .objects + .iter() + .any(|entry| entry.version_id == current.version_id && !entry.delete_marker) + ); + assert!( + versions + .objects + .iter() + .any(|entry| entry.version_id == Some(marker_id) && entry.delete_marker) + ); + assert!(!versions.objects.iter().any(|entry| entry.version_id == old.version_id)); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn batch_delete_rejects_dot_segments_and_keeps_valid_result_positions() { let bucket = "batch-delete-key-validation"; @@ -9315,8 +10550,9 @@ mod tests { disk.read_all(bucket, &format!("{object}/xl.meta")).await.unwrap(), )); } - for slot in 12..16 { - set.disks.write().await[slot] = None; + for disk in before.iter().skip(12).flatten() { + disk.close().await.expect("stop stale member health monitoring"); + disk.force_offline_for_test(); } for version in historical { store @@ -9357,7 +10593,9 @@ mod tests { .await .unwrap() ); - *set.disks.write().await = before.clone(); + for disk in before.iter().skip(12).flatten() { + disk.reset_health_for_store_init_retry(); + } store .recover_delete_transactions_at_incarnation( bucket, @@ -9376,8 +10614,9 @@ mod tests { .await .expect("durable retirement") ); - for slot in 12..16 { - set.disks.write().await[slot] = None; + for disk in before.iter().skip(12).flatten() { + disk.close().await.expect("stop returning member health monitoring"); + disk.force_offline_for_test(); } store .handle_make_bucket(bucket, &create) @@ -9411,7 +10650,9 @@ mod tests { tokio::fs::write(root.join("xl.meta"), bytes).await.unwrap(); } } - *set.disks.write().await = before.clone(); + for disk in before.iter().skip(12).flatten() { + disk.reset_health_for_store_init_retry(); + } let heal = rustfs_heal_contracts::heal_channel::HealOpts { remove: true, scan_mode: rustfs_heal_contracts::heal_channel::HealScanMode::Deep, @@ -9495,14 +10736,16 @@ mod tests { .unwrap(); } } - set.disks.write().await[15] = None; + let offline = before[15].as_ref().expect("cleanup target must exist"); + offline.close().await.expect("stop cleanup target health monitoring"); + offline.force_offline_for_test(); let incomplete = store .heal_object_with_proof(bucket, object, &marker.to_string(), &heal) .await .unwrap(); assert!(incomplete.error.as_ref().is_some_and(Error::is_retired_marker_deferred)); assert!(incomplete.absence.is_none(), "offline targets cannot be omitted from a cleanup receipt"); - set.disks.write().await[15] = before[15].clone(); + offline.reset_health_for_store_init_retry(); let failure = crate::set_disk::DanglingDeleteFailure::install(bucket, object, 15, crate::disk::error::DiskError::FaultyDisk); let partial = store @@ -9621,9 +10864,13 @@ mod tests { .expect("bucket incarnation"); let set = &store.pools[0].disk_set[0]; let before = set.disks.read().await.clone(); - for slot in 12..16 { - set.disks.write().await[slot] = None; + // Keep the handles attached so reconnect probes inherit the forced + // outage instead of recreating healthy handles for these local paths. + for disk in before.iter().skip(12).flatten() { + disk.close().await.expect("stop stale member health monitoring"); + disk.force_offline_for_test(); } + set.connect_disks().await; store .delete_object( bucket, @@ -9636,8 +10883,6 @@ mod tests { ) .await .expect("acknowledged delete should remove the marker from the online quorum"); - *set.disks.write().await = before.clone(); - for disk in before.iter().take(12).flatten() { assert!(matches!( disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default()) @@ -9646,6 +10891,8 @@ mod tests { )); } for disk in before.iter().skip(12).flatten() { + assert_eq!(disk.runtime_state(), crate::disk::health_state::RuntimeDriveHealthState::Offline); + disk.reset_health_for_store_init_retry(); assert!( disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default()) .await diff --git a/crates/ecstore/src/store/peer.rs b/crates/ecstore/src/store/peer.rs index 655492c66..be4560216 100644 --- a/crates/ecstore/src/store/peer.rs +++ b/crates/ecstore/src/store/peer.rs @@ -18,6 +18,11 @@ use crate::disk::error::{DiskError, Result as DiskResult}; use crate::disk::{DeleteOptions, Disk, RenameDataGuards, RenameDataResp}; use crate::runtime::instance::{InstanceContext, NamespaceCommitGuard}; use crate::runtime::sources as runtime_sources; +use rustfs_io_metrics::{ + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT, PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP, PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL, +}; use tracing::{debug, error}; const LOG_COMPONENT_ECSTORE: &str = "ecstore"; @@ -80,6 +85,13 @@ fn validate_bootstrap_volume(volume: &str) -> DiskResult<()> { } impl ECStore { + /// Resolve a quota lease against this request's captured storage instance. + pub async fn quota_mutation_fence_disk(&self, disk_ref: &str) -> DiskResult { + let (disk, id) = local_disk_candidate(&self.ctx, disk_ref).await?; + let _owner = admit_local_disk(&self.ctx, &disk, id, false).await?; + Ok(disk) + } + pub async fn write_local_metadata_at_incarnation( self: &Arc, disk_ref: &str, @@ -331,15 +343,38 @@ async fn rename_local_data_with_ctx( destination: (&str, &str), mut guards: RenameDataGuards, ) -> DiskResult { + let handler_started = rustfs_io_metrics::put_stage_timer(); + let lookup_started = rustfs_io_metrics::put_stage_timer(); let (disk, disk_id) = local_disk_candidate(ctx, disk_ref).await?; + let lookup_ms = lookup_started.map(|started| started.elapsed().as_secs_f64() * 1000.0); + if let Some(duration_ms) = lookup_ms { + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP, duration_ms); + } let mutates_namespace = !is_meta_bucketname(source.0) || !is_meta_bucketname(destination.0); + let admit_started = rustfs_io_metrics::put_stage_timer(); let owner = admit_local_disk(ctx, &disk, disk_id, mutates_namespace).await?; + let admit_ms = admit_started.map(|started| started.elapsed().as_secs_f64() * 1000.0); + if let Some(duration_ms) = admit_ms { + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT, duration_ms); + } guards.namespace_owner = owner.as_ref().map(|owner| owner.clone() as Arc); + let local_rename_started = rustfs_io_metrics::put_stage_timer(); let result = disk .rename_data_borrowed_with_fence_observed(source.0, source.1, fi, destination.0, destination.1, guards) .await .result; + let local_rename_ms = local_rename_started.map(|started| started.elapsed().as_secs_f64() * 1000.0); + if let Some(duration_ms) = local_rename_ms { + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME, duration_ms); + } drop(owner); + if let Some(handler_started) = handler_started { + let handler_ms = handler_started.elapsed().as_secs_f64() * 1000.0; + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL, handler_ms); + let child_ms = lookup_ms.unwrap_or(0.0) + admit_ms.unwrap_or(0.0) + local_rename_ms.unwrap_or(0.0); + let overhead_ms = (handler_ms - child_ms).max(0.0); + rustfs_io_metrics::record_put_object_stage_duration(PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD, overhead_ms); + } result } @@ -1090,15 +1125,20 @@ mod tests { .await .expect("bounded physical entry"); drop(rename); - assert!(store.scanner_data_usage_publication_blocked().await); + assert!( + !store.scanner_data_usage_publication_blocked().await, + "a pending namespace commit is not a local publication barrier" + ); assert!(ctx.namespace_commits_pending()); assert!(!sibling.namespace_commits_pending()); - assert!( + // The pending physical rename owns the namespace commit, which is + // ordinary namespace activity and no longer fences another scanner + // admission on the same token. + drop( store - .rename_local_data(&disk_ref, ("target-bucket", "staged"), &fi, ("target-bucket", "another"), Some(token)) + .acquire_scanner_publication_lease_guard(token) .await - .is_err(), - "real pending rename blocks another scanner publication" + .expect("a pending ordinary rename does not fence another scanner admission"), ); assert!(store.release_scanner_publication_lease(token).await, "remove registered token"); let gate = ctx.data_movement_operation_gate(); @@ -1255,7 +1295,7 @@ mod tests { } #[tokio::test] - async fn completed_namespace_commit_rejects_stale_scanner_target_admission() { + async fn completed_namespace_commit_keeps_scanner_target_admission() { use futures::FutureExt; use std::time::{Duration, Instant}; use tokio::time::timeout; @@ -1466,21 +1506,21 @@ mod tests { assert_eq!(final_state, (false, namespace_completed, movement_before, operation_epoch_before)); assert_eq!(user_latest.data, user_new.data); assert_eq!(user_latest.version_id, user_new.version_id); + // An ordinary commit that completed after the lease was granted is + // namespace activity the scanner certifies per bucket; it is not a + // reason to withhold the scanner-owned metadata rename. assert!( - stale_validate.is_err() && target_rename.is_err(), - "completed namespace commit must reject both stale admissions: validate={stale_validate:?}, target={:?}, metadata_changed={metadata_changed}", + stale_validate.is_ok() && target_rename.is_ok(), + "a completed ordinary commit must keep both admissions: validate={stale_validate:?}, target={:?}, metadata_changed={metadata_changed}", target_rename.as_ref().map(|_| ()), ); - assert_eq!(metadata_after, metadata_before, "stale target must not rewrite metadata"); - assert_eq!(metadata_latest.data, metadata_old.data, "latest metadata body must remain the old one"); - assert_eq!( - metadata_latest.version_id, metadata_old.version_id, - "latest metadata version must remain the old one" - ); + assert!(metadata_changed, "the admitted target publishes the staged metadata"); + assert!(latest_is_staged, "latest metadata must be the published one"); + assert_ne!(metadata_latest.data, metadata_old.data); } #[tokio::test] - async fn renewed_namespace_lease_allows_real_metadata_publication() { + async fn scanner_lease_allows_real_metadata_publication_after_an_ordinary_write() { use futures::FutureExt; use std::time::Duration; use tokio::time::timeout; @@ -1546,12 +1586,10 @@ mod tests { .expect("latest ordinary committed object"); assert_eq!(user_latest.version_id, user_new.version_id); assert_eq!(user_latest.data, user_new.data); - assert!( - store - .validate_scanner_publication_lease(old_token, movement_generation) - .await - .is_err() - ); + store + .validate_scanner_publication_lease(old_token, movement_generation) + .await + .expect("an ordinary write after the grant does not stale the lease"); assert_eq!( ctx.scanner_publication_lease_generations(old_token).await, Some(( @@ -1559,7 +1597,7 @@ mod tests { Some(namespace_generation), crate::runtime::instance::ScannerPublicationLeasePurpose::Authoritative, )), - "rejection must not refresh or discard the old lease" + "validation must not refresh or discard the lease" ); let (fresh_token, fresh_generation) = store .acquire_scanner_publication_lease(movement_generation, ttl) @@ -1669,7 +1707,9 @@ mod tests { #[serial_test::serial] async fn target_ordinary_timeout_keeps_its_physical_namespace_owner() { use crate::disk::os::prepared_publication_test_hooks as hooks; - temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("1"))], async { + // Allow real fixture I/O to finish under load. Expire the operation + // with virtual time only after its physical publication is paused. + temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("60"))], async { let ctx = Arc::new(InstanceContext::new()); let store = super::super::tests::build_store_with_ctx(ctx.clone()); let root = tempfile::tempdir().expect("root"); @@ -1703,7 +1743,7 @@ mod tests { .await .expect("bounded entry"); tokio::time::pause(); - tokio::time::advance(std::time::Duration::from_secs(2)).await; + tokio::time::advance(std::time::Duration::from_secs(61)).await; tokio::time::resume(); let result = tokio::time::timeout(std::time::Duration::from_secs(5), &mut rename) .await diff --git a/crates/ecstore/src/store/rebalance.rs b/crates/ecstore/src/store/rebalance.rs index e55cc5fdc..6a0b69ebb 100644 --- a/crates/ecstore/src/store/rebalance.rs +++ b/crates/ecstore/src/store/rebalance.rs @@ -125,9 +125,8 @@ async fn pause_prepared_pool_read_before_refetch(object: &str) { #[cfg(test)] use support::resolve_latest_object_info_candidates; use support::{ - LatestObjectInfoCandidate, PoolErr, PoolObjInfo, RebalanceDeletePoolResult, pool_lookup_not_found_error, - rebalance_disk_set_lookup_error, resolve_latest_object_info_candidates_with_pool_state, - resolve_rebalance_delete_from_all_pools_result, resolve_rebalance_delete_from_all_pools_results, + LatestObjectInfoCandidate, PoolErr, PoolObjInfo, pool_lookup_not_found_error, rebalance_disk_set_lookup_error, + resolve_latest_object_info_candidates_with_pool_state, resolve_rebalance_delete_from_all_pools_results, resolve_store_rebalance_pool_meta_reload_result, validate_prepared_pool_refetch_identity, }; @@ -996,42 +995,27 @@ impl ECStore { let mut results = Vec::with_capacity(errs.len()); - for pe in errs.iter() { - if let Some(err) = &pe.err - && err == &StorageError::ErasureWriteQuorum - { - if let Some(idx) = pe.index { - results.push(RebalanceDeletePoolResult { - pool_idx: idx, - result: Err(StorageError::ErasureWriteQuorum), - }); - } + for pe in &errs { + let Some(idx) = pe.index else { continue }; + if pe.err.as_ref() == Some(&StorageError::ErasureWriteQuorum) { + results.push(Err(StorageError::ErasureWriteQuorum)); continue; } - - if let Some(idx) = pe.index { - let pool = self.pools[idx].clone(); - results.push(RebalanceDeletePoolResult { - pool_idx: idx, - result: self - .run_external_decommission_capacity_object_delete( - idx, - bucket, - object, - object, - opts.clone(), - |opts| async move { pool.delete_object(bucket, object, opts).await }, - ) - .await, - }); - } + let pool = self.pools[idx].clone(); + results.push( + self.run_external_decommission_capacity_object_delete( + idx, + bucket, + object, + object, + opts.clone(), + |opts| async move { pool.delete_object(bucket, object, opts).await }, + ) + .await, + ); } - resolve_rebalance_delete_from_all_pools_result( - resolve_rebalance_delete_from_all_pools_results(results, bucket, object), - bucket, - object, - ) + resolve_rebalance_delete_from_all_pools_results(results, bucket, object) } /// Peer reload entry: refreshes in-memory pool metadata from the shared @@ -2460,56 +2444,22 @@ mod tests { assert!(err_message.contains(&Error::SlowDown.to_string())); } - #[test] - fn resolve_rebalance_delete_from_all_pools_result_passthrough_ok() { - let info = ObjectInfo { - bucket: "bucket".to_string(), - name: "object".to_string(), - ..Default::default() - }; - - let resolved = resolve_rebalance_delete_from_all_pools_result(Ok(info.clone()), "bucket", "object") - .expect("successful rebalance delete should pass through"); - - assert_eq!(resolved.bucket, info.bucket); - assert_eq!(resolved.name, info.name); - } - - #[test] - fn resolve_rebalance_delete_from_all_pools_result_wraps_object_context() { - let err = resolve_rebalance_delete_from_all_pools_result(Err(Error::SlowDown), "bucket", "object") - .expect_err("failed rebalance delete should be wrapped"); - let rendered = err.to_string(); - - assert!(rendered.contains("failed to delete rebalance source object bucket/object"), "{rendered}"); - assert!(rendered.contains(&Error::SlowDown.to_string()), "{rendered}"); - } - #[test] fn resolve_rebalance_delete_from_all_pools_results_fails_on_later_pool_error() { let err = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Ok(ObjectInfo { - bucket: "bucket".to_string(), - name: "object".to_string(), - ..Default::default() - }), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::SlowDown), - }, + Ok(ObjectInfo { + bucket: "bucket".to_string(), + name: "object".to_string(), + ..Default::default() + }), + Err(Error::SlowDown), ], "bucket", "object", ) .expect_err("non-ignorable errors from later pools must not be hidden"); - let rendered = err.to_string(); - - assert!(rendered.contains("pool 1 delete failed for bucket/object"), "{rendered}"); - assert!(rendered.contains(&Error::SlowDown.to_string()), "{rendered}"); + assert_eq!(err, Error::SlowDown); } #[test] @@ -2522,14 +2472,8 @@ mod tests { let resolved = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Ok(info.clone()), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), - }, + Ok(info.clone()), + Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), ], "bucket", "object", @@ -2550,14 +2494,8 @@ mod tests { let resolved = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Ok(info.clone()), - }, + Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), + Ok(info.clone()), ], "bucket", "object", @@ -2572,50 +2510,35 @@ mod tests { fn resolve_rebalance_delete_from_all_pools_results_fails_when_all_results_are_ignored_errors() { let err = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::VersionNotFound("bucket".to_string(), "object".to_string(), "vid-1".to_string())), - }, + Err(Error::ObjectNotFound("bucket".to_string(), "object".to_string())), + Err(Error::VersionNotFound("bucket".to_string(), "object".to_string(), "vid-1".to_string())), ], "bucket", "object", ) .expect_err("all ignored errors without any successful delete should still fail"); - let rendered = err.to_string(); - - assert!(rendered.contains("pool 1 delete failed for bucket/object"), "{rendered}"); - assert!(rendered.contains("Version not found"), "{rendered}"); + assert_eq!( + err, + Error::VersionNotFound("bucket".to_string(), "object".to_string(), "vid-1".to_string()) + ); } #[test] fn resolve_rebalance_delete_from_all_pools_results_fails_on_write_quorum_even_with_success() { let err = resolve_rebalance_delete_from_all_pools_results( vec![ - RebalanceDeletePoolResult { - pool_idx: 0, - result: Ok(ObjectInfo { - bucket: "bucket".to_string(), - name: "object".to_string(), - ..Default::default() - }), - }, - RebalanceDeletePoolResult { - pool_idx: 1, - result: Err(Error::ErasureWriteQuorum), - }, + Ok(ObjectInfo { + bucket: "bucket".to_string(), + name: "object".to_string(), + ..Default::default() + }), + Err(Error::ErasureWriteQuorum), ], "bucket", "object", ) .expect_err("write quorum failures must fail the aggregate delete"); - let rendered = err.to_string(); - - assert!(rendered.contains("pool 1 delete failed for bucket/object"), "{rendered}"); - assert!(rendered.contains(&Error::ErasureWriteQuorum.to_string()), "{rendered}"); + assert_eq!(err, Error::ErasureWriteQuorum); } #[test] diff --git a/crates/ecstore/src/store/rebalance/support.rs b/crates/ecstore/src/store/rebalance/support.rs index 376ac05b9..13be3844c 100644 --- a/crates/ecstore/src/store/rebalance/support.rs +++ b/crates/ecstore/src/store/rebalance/support.rs @@ -46,15 +46,10 @@ impl Clone for PoolObjInfo { } } -pub(super) struct LatestObjectInfoCandidate { - pub(super) info: Option, - pub(super) idx: usize, - pub(super) err: Option, -} - -pub(super) struct RebalanceDeletePoolResult { - pub(super) pool_idx: usize, - pub(super) result: Result, +pub(in crate::store) struct LatestObjectInfoCandidate { + pub(in crate::store) info: Option, + pub(in crate::store) idx: usize, + pub(in crate::store) err: Option, } pub(super) fn pool_lookup_not_found_error(bucket: &str, object: &str, opts: &ObjectOptions) -> Error { @@ -71,53 +66,30 @@ pub(super) fn resolve_store_rebalance_pool_meta_reload_result(result: Result< result.map_err(|err| Error::other(format!("store rebalance pool meta reload failed during {stage}: {err}"))) } -pub(super) fn resolve_rebalance_delete_from_all_pools_result( - result: Result, - bucket: &str, - object: &str, -) -> Result { - result.map_err(|err| { - if matches!(&err, Error::PreconditionFailed | Error::PrefixAccessDenied(_, _)) { - err - } else { - Error::other(format!("failed to delete rebalance source object {bucket}/{object}: {err}")) - } - }) -} - fn is_ignorable_rebalance_delete_error(err: &Error) -> bool { is_err_object_not_found(err) || is_err_version_not_found(err) } -fn rebalance_delete_pool_error(pool_idx: usize, bucket: &str, object: &str, err: Error) -> Error { - if matches!(&err, Error::PreconditionFailed | Error::PrefixAccessDenied(_, _)) { - err - } else { - Error::other(format!("pool {pool_idx} delete failed for {bucket}/{object}: {err}")) - } -} - pub(super) fn resolve_rebalance_delete_from_all_pools_results( - results: Vec, + results: Vec>, bucket: &str, object: &str, ) -> Result { let mut deleted = None; let mut ignored_error = None; - for pool_result in results { - let pool_idx = pool_result.pool_idx; - match pool_result.result { + for result in results { + match result { Ok(info) => { if deleted.is_none() { deleted = Some(info); } } Err(err) if is_ignorable_rebalance_delete_error(&err) => { - ignored_error = Some((pool_idx, err)); + ignored_error = Some(err); } Err(err) => { - return Err(rebalance_delete_pool_error(pool_idx, bucket, object, err)); + return Err(err); } } } @@ -126,8 +98,8 @@ pub(super) fn resolve_rebalance_delete_from_all_pools_results( return Ok(info); } - if let Some((pool_idx, err)) = ignored_error { - return Err(rebalance_delete_pool_error(pool_idx, bucket, object, err)); + if let Some(err) = ignored_error { + return Err(err); } Err(Error::other(format!( @@ -295,7 +267,7 @@ pub(super) fn resolve_latest_object_info_candidates( resolve_latest_object_info_candidates_with_pool_state(candidates, &[], bucket, object, opts) } -pub(super) fn resolve_latest_object_info_candidates_with_pool_state( +pub(in crate::store) fn resolve_latest_object_info_candidates_with_pool_state( candidates: Vec, suspended_pools: &[bool], bucket: &str, @@ -377,52 +349,56 @@ mod tests { } #[test] - fn rebalance_delete_result_preserves_precondition_failed() { - let err = resolve_rebalance_delete_from_all_pools_result(Err(Error::PreconditionFailed), "bucket", "object") - .expect_err("precondition failure should remain structured"); - - assert_eq!(err, Error::PreconditionFailed); + fn rebalance_delete_aggregation_preserves_typed_failures() { + for expected in [ + Error::UnsupportedDisk, + Error::GuardedDeleteUnsupported, + Error::ErasureReadQuorum, + Error::ErasureWriteQuorum, + Error::SlowDown, + Error::PreconditionFailed, + Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned()), + Error::FileCorrupt, + ] { + for results in [ + vec![Err(expected.clone())], + vec![Ok(ObjectInfo::default()), Err(expected.clone())], + vec![Err(expected.clone()), Ok(ObjectInfo::default())], + vec![ + Err(Error::ObjectNotFound("bucket".to_owned(), "object".to_owned())), + Err(expected.clone()), + ], + ] { + let actual = resolve_rebalance_delete_from_all_pools_results(results, "bucket", "object") + .expect_err("a pool failure must retain its S3 error classification"); + assert_eq!(actual, expected); + } + } } #[test] - fn rebalance_delete_result_preserves_prefix_access_denied() { - let err = resolve_rebalance_delete_from_all_pools_result( - Err(Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())), - "bucket", - "object", - ) - .expect_err("prefix access denial should remain structured"); - - assert_eq!(err, Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())); + fn rebalance_delete_aggregation_preserves_missing_identity() { + for expected in [ + Error::ObjectNotFound("bucket".to_owned(), "object".to_owned()), + Error::VersionNotFound("bucket".to_owned(), "object".to_owned(), "version".to_owned()), + ] { + let actual = resolve_rebalance_delete_from_all_pools_results(vec![Err(expected.clone())], "bucket", "object") + .expect_err("all-missing deletes must preserve their typed object or version identity"); + assert_eq!(actual, expected); + } } #[test] - fn rebalance_delete_pool_result_preserves_precondition_failed() { - let err = resolve_rebalance_delete_from_all_pools_results( - vec![RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::PreconditionFailed), - }], - "bucket", - "object", - ) - .expect_err("precondition failure should remain structured"); - - assert_eq!(err, Error::PreconditionFailed); - } - - #[test] - fn rebalance_delete_pool_result_preserves_prefix_access_denied() { - let err = resolve_rebalance_delete_from_all_pools_results( - vec![RebalanceDeletePoolResult { - pool_idx: 0, - result: Err(Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())), - }], - "bucket", - "object", - ) - .expect_err("prefix access denial should remain structured"); - - assert_eq!(err, Error::PrefixAccessDenied("bucket".to_owned(), "object".to_owned())); + fn rebalance_delete_aggregation_rejects_no_attempted_pools() { + let error = resolve_rebalance_delete_from_all_pools_results(Vec::new(), "bucket", "object") + .expect_err("an empty attempt set cannot report a successful delete"); + let Error::Io(error) = error else { + panic!("an empty attempt set must remain an internal error"); + }; + assert_eq!(error.kind(), std::io::ErrorKind::Other); + assert_eq!( + error.to_string(), + "failed to delete rebalance source object bucket/object: no pools were attempted" + ); } } diff --git a/crates/ecstore/tests/cleanup_isolation_test.rs b/crates/ecstore/tests/cleanup_isolation_test.rs new file mode 100644 index 000000000..663681445 --- /dev/null +++ b/crates/ecstore/tests/cleanup_isolation_test.rs @@ -0,0 +1,174 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(target_os = "linux")] + +use bytes::Bytes; +mod storage_api; + +use std::time::Duration; +use storage_api::cleanup_isolation::{DeleteOptions, DiskAPI, DiskError, DiskOption, Endpoint, new_disk}; + +#[tokio::test] +async fn cleanup_disk_and_periodic_gc_use_affinity_without_changing_foreground() { + const CHILD: &str = "RUSTFS_CLEANUP_LINUX_TEST_CHILD"; + let foreground = rustix::thread::sched_getaffinity(None).unwrap(); + let cpu = (0..rustix::thread::CpuSet::MAX_CPU) + .find(|cpu| foreground.is_set(*cpu)) + .expect("the test process must have an allowed CPU"); + if std::env::var_os(CHILD).is_none() { + for mode in ["custom", "default", "none", "restricted"] { + if mode == "default" && (!foreground.is_set(1) || !foreground.is_set(2)) { + continue; + } + let output = tokio::task::spawn_blocking(move || { + let mut command = std::process::Command::new(std::env::current_exe().unwrap()); + command + .args([ + "--exact", + "cleanup_disk_and_periodic_gc_use_affinity_without_changing_foreground", + "--nocapture", + ]) + .env(CHILD, mode) + .env("RUSTFS_CLEANUP_ISOLATE_ENABLE", "true") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING", "4") + .env("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS", "1") + .env("RUSTFS_CLEANUP_DISK_MAX_PENDING", "4") + .env("RUSTFS_CLEANUP_DISK_WORKERS", "1") + .env("RUSTFS_CLEANUP_GC_WORKERS", "1") + .env("RUSTFS_CLEANUP_BLOCKING_THREADS", "1") + .env("RUSTFS_CLEANUP_ASYNC_THREADS", "1") + .env_remove("RUSTFS_CLEANUP_CPUS") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER") + .env_remove("RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP"); + match mode { + "custom" => { + command.env("RUSTFS_CLEANUP_CPUS", cpu.to_string()); + } + "none" => { + command.env("RUSTFS_CLEANUP_CPUS", "none"); + } + _ => {} + } + command.output().unwrap() + }) + .await + .unwrap(); + assert!( + output.status.success(), + "{}\n{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8_lossy(&output.stdout).contains("1 passed")); + } + return; + } + + let mode = std::env::var(CHILD).unwrap(); + if mode == "restricted" { + let mut restricted = rustix::thread::CpuSet::new(); + restricted.set(cpu); + rustix::thread::sched_setaffinity(None, &restricted).expect("restrict initialization to one allowed CPU"); + } + let dir = tempfile::tempdir().unwrap(); + let disk = new_disk(&Endpoint::try_from(dir.path().to_str().unwrap()).unwrap(), &DiskOption::default()) + .await + .unwrap(); + let bucket = "cleanup-affinity"; + disk.make_volume(bucket).await.unwrap(); + disk.write_all(bucket, "old/part.1", Bytes::from_static(b"garbage")) + .await + .unwrap(); + let receipt = disk + .write_all(bucket, "old/.rustfs-old-data-cleanup-receipt.json", Bytes::from_static(b"receipt")) + .await; + if mode == "restricted" { + assert!( + receipt + .expect_err("unavailable default CPUs must reject cleanup") + .to_string() + .contains("allowed CPU set") + ); + assert_eq!(disk.read_all(bucket, "old/part.1").await.unwrap(), Bytes::from_static(b"garbage")); + return; + } + receipt.unwrap(); + let expected_mask = match mode.as_str() { + "default" => "1-2".to_owned(), + "none" => std::fs::read_to_string("/proc/thread-self/status") + .unwrap() + .lines() + .find_map(|line| line.strip_prefix("Cpus_allowed_list:")) + .unwrap() + .trim() + .to_owned(), + _ => cpu.to_string(), + }; + disk.write_all(bucket, "live/part.1", Bytes::from_static(b"keep")) + .await + .unwrap(); + disk.delete( + bucket, + "old", + DeleteOptions { + recursive: true, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!(matches!(disk.read_all(bucket, "old/part.1").await, Err(DiskError::FileNotFound))); + + // One async worker and one lazily started blocking worker. Inspect the + // real process threads, including the threads executing filesystem work. + let mut cleanup_threads = 0; + for entry in std::fs::read_dir("/proc/self/task").unwrap() { + let task = entry.unwrap().path(); + let Ok(name) = std::fs::read_to_string(task.join("comm")) else { + continue; + }; + if name.trim() != "rustfs-cleanup" { + continue; + } + cleanup_threads += 1; + let status = std::fs::read_to_string(task.join("status")).unwrap(); + let mask = status + .lines() + .find_map(|line| line.strip_prefix("Cpus_allowed_list:")) + .unwrap() + .trim(); + assert_eq!(mask, expected_mask); + } + assert_eq!(cleanup_threads, 2, "async and blocking workers must both use the configured runtime"); + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), foreground); + + let trash = dir.path().join(".rustfs.sys/tmp/.trash"); + assert!(std::fs::read_dir(&trash).unwrap().next().is_some()); + // Advance the foreground timer that dispatches the five-minute periodic + // scan; the dedicated runtime then performs the actual physical removal. + tokio::time::pause(); + tokio::time::advance(Duration::from_secs(301)).await; + tokio::time::resume(); + tokio::time::timeout(Duration::from_secs(20), async { + while std::fs::read_dir(&trash).unwrap().next().is_some() { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .expect("periodic GC must physically remove trash"); + assert_eq!(disk.read_all(bucket, "live/part.1").await.unwrap(), Bytes::from_static(b"keep")); + assert_eq!(rustix::thread::sched_getaffinity(None).unwrap(), foreground); +} diff --git a/crates/ecstore/tests/fd_relative_io_test.rs b/crates/ecstore/tests/fd_relative_io_test.rs new file mode 100644 index 000000000..fa19deb52 --- /dev/null +++ b/crates/ecstore/tests/fd_relative_io_test.rs @@ -0,0 +1,283 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(all(target_os = "linux", feature = "test-util"))] + +mod storage_api; + +use bytes::Bytes; +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use std::{path::Path, time::Duration}; +use storage_api::fd_relative::{ + DiskAPI, DiskError, DiskOption, DiskStore, Endpoint, LocalPublicationPause, LocalPublicationStage, ReadOptions, new_disk, +}; +use time::OffsetDateTime; +use uuid::Uuid; + +const RUSTFS_META_TMP_BUCKET: &str = ".rustfs.sys/tmp"; + +async fn disk_at(path: &Path) -> DiskStore { + new_disk( + &Endpoint::try_from(path.to_str().expect("UTF-8 test path")).expect("endpoint"), + &DiskOption::default(), + ) + .await + .expect("local disk") +} + +fn metadata(data: Bytes, inline: bool, directory: Uuid) -> FileInfo { + let mut fi = FileInfo::new("object", 1, 0); + fi.erasure.index = 1; + fi.version_id = Some(Uuid::nil()); + fi.data_dir = Some(directory); + fi.size = i64::try_from(data.len()).expect("test data size"); + fi.parts = vec![ObjectPartInfo { + number: 1, + size: data.len(), + actual_size: fi.size, + ..Default::default() + }]; + fi.mod_time = Some(OffsetDateTime::now_utc()); + fi.data = inline.then_some(data); + if inline { + fi.set_inline_data(); + } + fi +} + +async fn put(disk: &DiskStore, staging: &str, data: Bytes, inline: bool) -> Uuid { + let directory = Uuid::new_v4(); + if !inline { + disk.write_all(RUSTFS_META_TMP_BUCKET, &format!("{staging}/{directory}/part.1"), data.clone()) + .await + .expect("stage shard"); + } + disk.rename_data(RUSTFS_META_TMP_BUCKET, staging, metadata(data, inline, directory), "bucket", "object") + .await + .expect("commit object"); + directory +} + +#[tokio::test] +async fn pinned_root_reads_and_writes_original_disk_after_path_replacement() { + let temp = tempfile::tempdir().expect("test directory"); + let root = temp.path().join("disk"); + std::fs::create_dir(&root).expect("disk root"); + let disk = disk_at(&root).await; + disk.make_volume("bucket").await.expect("bucket"); + disk.write_all("bucket", "metadata", Bytes::from_static(b"original")) + .await + .expect("initial write"); + let moved = temp.path().join("original"); + std::fs::rename(&root, &moved).expect("move mount path"); + std::fs::create_dir_all(root.join("bucket")).expect("replacement disk"); + std::fs::write(root.join("bucket/metadata"), b"replacement").expect("replacement sentinel"); + assert_eq!( + disk.read_all("bucket", "metadata").await.expect("pinned read"), + Bytes::from_static(b"original") + ); + disk.write_all("bucket", "metadata", Bytes::from_static(b"updated")) + .await + .expect("pinned write"); + assert_eq!(std::fs::read(moved.join("bucket/metadata")).expect("original disk"), b"updated"); + assert_eq!(std::fs::read(root.join("bucket/metadata")).expect("replacement disk"), b"replacement"); +} + +#[tokio::test] +async fn relative_reads_and_writes_reject_symlink_escape() { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + let outside = tempfile::tempdir().expect("outside directory"); + std::fs::write(outside.path().join("sentinel"), b"preserved").expect("sentinel"); + std::os::unix::fs::symlink(outside.path(), temp.path().join("bucket/link")).expect("directory link"); + std::os::unix::fs::symlink(outside.path().join("sentinel"), temp.path().join("bucket/file-link")).expect("file link"); + for path in ["link/sentinel", "file-link"] { + assert!(matches!(disk.read_all("bucket", path).await, Err(DiskError::InvalidPath))); + assert!(matches!( + disk.write_all("bucket", path, Bytes::from_static(b"unsafe")).await, + Err(DiskError::InvalidPath) + )); + } + assert_eq!(std::fs::read(outside.path().join("sentinel")).expect("outside bytes"), b"preserved"); +} + +#[tokio::test] +async fn inline_and_non_inline_overwrites_remain_readable_after_reopen() { + for inline in [true, false] { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "first", Bytes::from_static(b"old bytes"), inline).await; + let directory = put(&disk, "second", Bytes::from_static(b"new bytes"), inline).await; + drop(disk); + let disk = disk_at(temp.path()).await; + let fi = disk + .read_version( + "bucket", + "bucket", + "object", + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("committed metadata after reopen"); + assert_eq!(fi.data_dir, Some(directory)); + if inline { + assert_eq!(fi.data, Some(Bytes::from_static(b"new bytes"))); + } else { + assert_eq!( + disk.read_all("bucket", &format!("object/{directory}/part.1")) + .await + .expect("shard bytes"), + Bytes::from_static(b"new bytes") + ); + } + } +} + +#[tokio::test] +async fn rename_rejects_symlinked_destination_metadata() { + for inline in [true, false] { + let temp = tempfile::tempdir().expect("disk directory"); + let outside = tempfile::tempdir().expect("outside directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "seed", Bytes::from_static(b"old object"), inline).await; + let destination = temp.path().join("bucket/object/xl.meta"); + let original = std::fs::read(&destination).expect("valid existing metadata"); + let external_metadata = outside.path().join("xl.meta"); + std::fs::write(&external_metadata, &original).expect("external valid metadata"); + std::fs::remove_file(&destination).expect("replace destination metadata"); + std::os::unix::fs::symlink(&external_metadata, &destination).expect("metadata escape link"); + + let directory = Uuid::new_v4(); + let data = Bytes::from_static(b"replacement"); + if !inline { + disk.write_all(RUSTFS_META_TMP_BUCKET, &format!("replacement/{directory}/part.1"), data.clone()) + .await + .expect("stage non-inline data"); + } + let result = disk + .rename_data( + RUSTFS_META_TMP_BUCKET, + "replacement", + metadata(data, inline, directory), + "bucket", + "object", + ) + .await; + assert!( + matches!(result, Err(DiskError::InvalidPath)), + "symlinked metadata must be rejected: {result:?}" + ); + assert_eq!(std::fs::read_link(&destination).expect("link must remain"), external_metadata); + assert_eq!(std::fs::read(&external_metadata).expect("external metadata must survive"), original); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn prepared_publication_rejects_source_or_directory_replacement() { + for replace_directory in [false, true] { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "first", Bytes::from_static(b"old bytes"), true).await; + let old = std::fs::read(temp.path().join("bucket/object/xl.meta")).expect("old metadata"); + let mut pause = LocalPublicationPause::install(&disk, "bucket", "object/xl.meta", LocalPublicationStage::PreparedRename) + .expect("publication pause"); + let writer = disk.clone(); + let task = tokio::spawn(async move { + writer + .rename_data( + RUSTFS_META_TMP_BUCKET, + "second", + metadata(Bytes::from_static(b"new bytes"), true, Uuid::new_v4()), + "bucket", + "object", + ) + .await + }); + tokio::time::timeout(Duration::from_secs(30), pause.entered()) + .await + .expect("publication reached pause") + .expect("pause entered"); + if replace_directory { + std::fs::rename(temp.path().join("bucket/object"), temp.path().join("detached")).expect("detach object"); + std::fs::create_dir(temp.path().join("bucket/object")).expect("replacement directory"); + std::fs::write(temp.path().join("bucket/object/xl.meta"), &old).expect("replacement metadata"); + } else { + let src = temp.path().join(RUSTFS_META_TMP_BUCKET).join("second/xl.meta"); + std::fs::rename(&src, src.with_extension("detached")).expect("detach staged metadata"); + std::fs::write(src, b"unrelated").expect("replacement staged file"); + } + drop(pause); + assert!(task.await.expect("writer task").is_err()); + assert_eq!( + std::fs::read(temp.path().join("bucket/object/xl.meta")).expect("old metadata survives"), + old + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cancelled_waiter_keeps_publication_serialized_until_physical_completion() { + let temp = tempfile::tempdir().expect("test directory"); + let disk = disk_at(temp.path()).await; + disk.make_volume("bucket").await.expect("bucket"); + put(&disk, "first", Bytes::from_static(b"old bytes"), true).await; + let mut pause = LocalPublicationPause::install(&disk, "bucket", "object/xl.meta", LocalPublicationStage::PreparedRename) + .expect("publication pause"); + let writer = disk.clone(); + let pending = tokio::spawn(async move { put(&writer, "shared-stage", Bytes::from_static(b"cancelled waiter"), true).await }); + tokio::time::timeout(Duration::from_secs(30), pause.entered()) + .await + .expect("publication reached pause") + .expect("pause entered"); + pending.abort(); + assert!(pending.await.expect_err("waiter aborted").is_cancelled()); + let writer = disk.clone(); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let mut retry = tokio::spawn(async move { + let _ = started_tx.send(()); + put(&writer, "shared-stage", Bytes::from_static(b"latest bytes"), true).await + }); + started_rx.await.expect("retry started"); + assert!( + tokio::time::timeout(Duration::from_millis(200), &mut retry).await.is_err(), + "a retry must not publish while the cancelled writer still owns physical IO" + ); + drop(pause); + tokio::time::timeout(Duration::from_secs(30), retry) + .await + .expect("retry completed") + .expect("retry task"); + let fi = disk + .read_version( + "bucket", + "bucket", + "object", + "", + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("latest metadata"); + assert_eq!(fi.data, Some(Bytes::from_static(b"latest bytes"))); +} diff --git a/crates/ecstore/tests/inline_heal_staging_test.rs b/crates/ecstore/tests/inline_heal_staging_test.rs new file mode 100644 index 000000000..95867d93e --- /dev/null +++ b/crates/ecstore/tests/inline_heal_staging_test.rs @@ -0,0 +1,226 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(all(feature = "test-util", not(windows)))] + +mod storage_api; + +use bytes::Bytes; +use rustfs_filemeta::{FileInfo, ObjectPartInfo}; +use std::{process::Command, time::Duration}; +use storage_api::inline_heal_staging::{ + DeleteOptions, DiskAPI, DiskError, DiskOption, Endpoint, LocalPublicationPause, LocalPublicationStage, ReadOptions, new_disk, +}; +use time::OffsetDateTime; +use uuid::Uuid; + +const TMP_BUCKET: &str = ".rustfs.sys/tmp"; +const CHILD: &str = "RUSTFS_INLINE_HEAL_STAGING_CHILD"; + +fn metadata(object: &str, version: Uuid, data_dir: Uuid, data: Bytes, inline: bool) -> FileInfo { + let mut info = FileInfo::new(object, 1, 0); + info.erasure.index = 1; + info.version_id = Some(version); + info.data_dir = Some(data_dir); + info.size = i64::try_from(data.len()).expect("fixture size"); + info.parts = vec![ObjectPartInfo { + number: 1, + size: data.len(), + actual_size: info.size, + ..Default::default() + }]; + info.mod_time = Some(OffsetDateTime::now_utc()); + if inline { + info.data = Some(data); + info.set_inline_data(); + } + info +} + +#[test] +fn inline_heal_stages_backup_in_production_durability_modes() { + if std::env::var_os(CHILD).is_none() { + // Durability is cached process-wide; use fresh production-library processes. + for mode in ["none", "relaxed"] { + let result = Command::new(std::env::current_exe().expect("integration test binary")) + .args([ + "--exact", + "inline_heal_stages_backup_in_production_durability_modes", + "--nocapture", + ]) + .env(CHILD, "1") + .env("RUSTFS_DURABILITY_MODE", mode) + .status() + .expect("run production-library durability fixture"); + assert!(result.success(), "inline staging failed in {mode} mode"); + } + return; + } + tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .expect("fixture runtime") + .block_on(async { + for version in [Uuid::nil(), Uuid::new_v4()] { + for old_inline in [true, false] { + let root = tempfile::tempdir().expect("fixture directory"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = new_disk(&endpoint, &DiskOption::default()).await.expect("local disk"); + disk.make_volume("bucket").await.expect("bucket"); + match disk.make_volume(TMP_BUCKET).await { + Ok(()) | Err(DiskError::VolumeExists) => {} + Err(error) => panic!("temporary bucket: {error:?}"), + } + let old_dir = Uuid::new_v4(); + let old_body = Bytes::from_static(b"old bytes must remain recoverable"); + if !old_inline { + disk.write_all(TMP_BUCKET, &format!("seed/{old_dir}/part.1"), old_body.clone()) + .await + .expect("stage external source"); + } + disk.rename_data( + TMP_BUCKET, + "seed", + metadata("object", version, old_dir, old_body, old_inline), + "bucket", + "object", + ) + .await + .expect("seed old version"); + let object_dir = root.path().join("bucket/object"); + let old_metadata = tokio::fs::read(object_dir.join("xl.meta")).await.expect("old metadata"); + let new_body = Bytes::from_static(b"repaired inline bytes"); + let mut repaired = metadata("object", version, Uuid::new_v4(), new_body.clone(), true); + repaired.set_healing(); + let mut pause = + LocalPublicationPause::install(&disk, "bucket", "object/xl.meta", LocalPublicationStage::PreparedRename) + .expect("pause physical metadata publication"); + let task_disk = disk.clone(); + let task = + tokio::spawn( + async move { task_disk.rename_data(TMP_BUCKET, "heal", repaired, "bucket", "object").await }, + ); + tokio::time::timeout(Duration::from_secs(10), pause.entered()) + .await + .expect("publication must reach its physical boundary") + .expect("publication pause"); + let backup = root.path().join(TMP_BUCKET).join("heal/xl.meta.bkp"); + assert_eq!( + tokio::fs::read(&backup) + .await + .expect("production heal must stage rollback bytes"), + old_metadata + ); + assert_eq!( + tokio::fs::read(object_dir.join("xl.meta")) + .await + .expect("unpublished metadata"), + old_metadata + ); + drop(pause); + let response = tokio::time::timeout(Duration::from_secs(10), task) + .await + .expect("publication must finish") + .expect("publication task") + .expect("healing commit"); + assert!(response.rollback_data_dir.is_none(), "inline heal must not publish a live backup"); + assert_eq!(response.cleanup_data_dir.is_some(), !old_inline); + assert!(!backup.exists(), "committed staging backup must be retired"); + for entry in std::fs::read_dir(&object_dir).expect("object directory") { + let entry = entry.expect("object entry"); + if entry.file_type().expect("entry type").is_dir() { + assert!(!entry.path().join("xl.meta.bkp").exists(), "no live rollback backup may be published"); + } + } + let current = disk + .read_version( + "", + "bucket", + "object", + &version.to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read healed physical target"); + assert_eq!(current.data, Some(new_body)); + assert_eq!(current.version_id.unwrap_or_default(), version); + } + } + }); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn recursive_tmp_cleanup_prunes_nested_stage_without_relocking() { + for key in ["stage", "stage/child", "stage/deep/child"] { + for layout in ["absent", "empty-parent", "present"] { + for has_sibling in [false, true] { + let root = tempfile::tempdir().expect("nested cleanup fixture"); + let endpoint = Endpoint::try_from(root.path().to_str().expect("UTF-8 fixture path")).expect("endpoint"); + let disk = new_disk(&endpoint, &DiskOption::default()).await.expect("local disk"); + let temporary = root.path().join(TMP_BUCKET); + let target = temporary.join(key); + tokio::fs::create_dir_all(&temporary).await.expect("TMP bucket"); + if layout != "absent" { + tokio::fs::create_dir_all(target.parent().expect("target parent")) + .await + .expect("prepare cleanup ancestors"); + } + if layout == "present" { + disk.write_all(TMP_BUCKET, &format!("{key}/payload"), Bytes::from_static(b"removed child")) + .await + .expect("seed cleanup target"); + } + let sibling = if key == "stage" { "other/payload" } else { "stage/sibling" }; + if has_sibling { + disk.write_all(TMP_BUCKET, sibling, Bytes::from_static(b"preserved sibling")) + .await + .expect("seed sibling"); + } + for _ in 0..2 { + tokio::time::timeout( + Duration::from_secs(10), + disk.delete( + TMP_BUCKET, + key, + DeleteOptions { + recursive: true, + ..Default::default() + }, + ), + ) + .await + .expect("nested TMP cleanup must not reacquire its staging lease") + .expect("recursive TMP cleanup"); + } + assert!(!target.exists(), "cleanup target must be gone: {key}"); + assert!(temporary.is_dir(), "TMP bucket must remain"); + if has_sibling { + assert_eq!( + tokio::fs::read(temporary.join(sibling)).await.expect("sibling survives"), + b"preserved sibling" + ); + } + assert_eq!( + temporary.join("stage").exists(), + has_sibling && key != "stage", + "empty staging ancestors must be pruned: {key}" + ); + } + } + } +} diff --git a/crates/ecstore/tests/list_prefix_test.rs b/crates/ecstore/tests/list_prefix_test.rs new file mode 100644 index 000000000..214f12eb3 --- /dev/null +++ b/crates/ecstore/tests/list_prefix_test.rs @@ -0,0 +1,82 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(feature = "test-util")] + +mod storage_api; + +use std::sync::Arc; +use storage_api::contract_compat::StorageListOperations; +use storage_api::metadata_lock::{ + BucketOperations, Error, MakeBucketOptions, ObjectIO, ObjectOptions, PutObjReader, init_bucket_metadata_sys, + isolated_store_over_temp_disks, +}; + +#[tokio::test] +async fn double_slash_list_prefix_is_literal_and_empty() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "list-double-slash-prefix"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + let key = "foo/foo/base/backup"; + let mut data = PutObjReader::from_vec(b"backup".to_vec()); + store + .put_object(bucket, key, &mut data, &ObjectOptions::default()) + .await + .expect("write control object"); + + for prefix in ["foo//foo/base/", "zzz//yyy", "foo///", "//foo/", "/foo/"] { + for delimiter in [None, Some("/".to_owned())] { + for max in [0, 1, 100] { + for marker in [None, Some(key.to_owned())] { + let page = store + .clone() + .list_objects_v2(bucket, prefix, marker.clone(), delimiter.clone(), max, false, None, false) + .await + .expect("a literal prefix with no stored matches is a valid list request"); + assert!(page.objects.is_empty(), "{prefix} must not alias {key}"); + assert!(page.prefixes.is_empty(), "{prefix} must not publish normalized prefixes"); + assert!(!page.is_truncated); + assert!(page.next_continuation_token.is_none()); + assert_eq!(page.continuation_token, marker); + } + } + } + } + let oversized = format!("foo//{}", "x".repeat(32 << 10)); + for prefix in ["foo//../secret", "foo//./secret", "foo//bad\0name", &oversized] { + let err = store + .clone() + .list_objects_v2(bucket, prefix, None, None, 100, false, None, false) + .await + .expect_err("invalid prefixes must still be rejected"); + assert!(matches!(err, Error::ObjectNameInvalid(_, _)), "{err:?}"); + } + let err = store + .clone() + .list_objects_v2("invalid_bucket", "foo//bar", None, None, 100, false, None, false) + .await + .expect_err("invalid buckets must still be rejected"); + assert!(matches!(err, Error::BucketNameInvalid(_)), "{err:?}"); + + let control = store + .clone() + .list_objects_v2(bucket, "foo/", None, None, 100, false, None, false) + .await + .expect("normal prefix still lists the control object"); + assert_eq!(control.objects.iter().map(|object| object.name.as_str()).collect::>(), [key]); +} diff --git a/crates/ecstore/tests/metadata_read_lock_reuse_test.rs b/crates/ecstore/tests/metadata_read_lock_reuse_test.rs index 6433bf0cd..c37949477 100644 --- a/crates/ecstore/tests/metadata_read_lock_reuse_test.rs +++ b/crates/ecstore/tests/metadata_read_lock_reuse_test.rs @@ -18,6 +18,7 @@ mod storage_api; use std::sync::Arc; use std::time::Duration; +use storage_api::contract_compat::ECStore; use storage_api::metadata_lock::{ BucketOperations, CompletePart, Error, MakeBucketOptions, MultipartOperations, NamespaceLocking, ObjectIO, ObjectOperations, ObjectOptions, PutObjReader, PutObjectCommitBarrier, PutObjectCommitPause, init_bucket_metadata_sys, @@ -170,6 +171,100 @@ async fn reused_metadata_lock_still_enforces_quota() { .expect("acquire writer"); } +#[tokio::test] +async fn authoritative_quota_read_does_not_ignore_later_quota() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "authoritative-quota-quota-set"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + + let mut first = PutObjReader::from_vec(b"cache no quota".to_vec()); + store + .put_object(bucket, "warm", &mut first, &ObjectOptions::default()) + .await + .expect("warm no-quota metadata cache"); + store + .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec()) + .await + .expect("set quota after no-quota cache was observed"); + + let mut opts = quota_snapshot_options(&store, bucket).await; + assert!(opts.set_quota_admission(0, 1)); + let mut data = PutObjReader::from_vec(b"too large".to_vec()); + let error = store + .put_object(bucket, "object", &mut data, &opts) + .await + .expect_err("quota set after a no-quota cache hit must still reject growth"); + assert!(matches!(error, Error::QuotaExceeded { limit: 1, .. }), "{error:?}"); +} + +#[tokio::test] +async fn authoritative_quota_read_allows_write_after_quota_is_cleared() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "authoritative-quota-quota-cleared"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + store + .update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec()) + .await + .expect("set quota"); + store + .update_bucket_metadata_config(bucket, "quota.json", Vec::new()) + .await + .expect("clear quota"); + + let mut data = PutObjReader::from_vec(b"allowed after clear".to_vec()); + store + .put_object(bucket, "object", &mut data, &ObjectOptions::default()) + .await + .expect("authoritatively cleared quota should allow writes"); + + let mut reader = store + .get_object_reader(bucket, "object", None, Default::default(), &ObjectOptions::default()) + .await + .expect("read object written after quota clear"); + let mut bytes = Vec::new(); + reader.stream.read_to_end(&mut bytes).await.expect("read body"); + assert_eq!(bytes, b"allowed after clear"); +} + +#[tokio::test] +async fn authoritative_quota_read_fails_closed_on_invalid_quota_json() { + let (_dirs, store) = isolated_store_over_temp_disks().await; + init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "authoritative-quota-invalid-quota"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + store + .update_bucket_metadata_config(bucket, "quota.json", b"{not-json".to_vec()) + .await + .expect("persist invalid quota bytes for fail-closed read"); + + let mut data = PutObjReader::from_vec(b"must fail".to_vec()); + store + .put_object(bucket, "object", &mut data, &ObjectOptions::default()) + .await + .expect_err("invalid quota JSON must not degrade to no-quota"); +} + +async fn quota_snapshot_options(store: &Arc, bucket: &str) -> ObjectOptions { + ObjectOptions { + versioned: true, + version_id: Some(Uuid::new_v4().to_string()), + expected_bucket_incarnation_id: Some(store.bucket_incarnation_id(bucket).await.expect("load incarnation")), + object_lock_config_snapshot: Some(store.object_lock_config_snapshot(bucket).await.expect("capture snapshot")), + ..Default::default() + } +} + #[tokio::test] async fn cancelled_put_releases_shared_metadata_guard() { let (_dirs, store) = isolated_store_over_temp_disks().await; diff --git a/crates/ecstore/tests/put_object_cancellation_test.rs b/crates/ecstore/tests/put_object_cancellation_test.rs new file mode 100644 index 000000000..bee170ca4 --- /dev/null +++ b/crates/ecstore/tests/put_object_cancellation_test.rs @@ -0,0 +1,133 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![cfg(feature = "test-util")] + +mod storage_api; + +use std::{path::PathBuf, time::Duration}; +use storage_api::metadata_lock::{ + ObjectIO, ObjectOptions, PutObjReader, PutObjectCommitBarrier, PutObjectCommitPause, isolated_store_over_temp_disks, +}; +use tempfile::TempDir; +use tokio::{ + fs, + io::AsyncReadExt, + time::{Instant, timeout}, +}; + +async fn staged_workspaces(dirs: &[TempDir]) -> Vec { + let mut staged = Vec::new(); + for dir in dirs { + let mut entries = fs::read_dir(dir.path().join(".rustfs.sys/tmp")) + .await + .expect("read temporary namespace"); + while let Some(entry) = entries.next_entry().await.expect("read temporary entry") { + if entry.file_name() != ".trash" { + staged.push(entry.path()); + } + } + } + staged +} + +async fn wait_for_staging_cleanup(dirs: &[TempDir]) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + let staged = staged_workspaces(dirs).await; + if staged.is_empty() { + return; + } + assert!(Instant::now() < deadline, "cancelled PUT left staging: {staged:?}"); + tokio::time::sleep(Duration::from_millis(25)).await; + } +} + +async fn checkpoint_timeout_at(pause: PutObjectCommitPause) { + let (dirs, store) = isolated_store_over_temp_disks().await; + let bucket = ".rustfs.sys"; + let object = "buckets/scanner-timeout/.usage-cache.bin.bkp"; + let old_body = vec![0x31; 1 << 20]; + let opts = ObjectOptions { + max_parity: true, + ..Default::default() + }; + store + .put_object(bucket, object, &mut PutObjReader::from_vec(old_body.clone()), &opts) + .await + .expect("seed previous checkpoint"); + wait_for_staging_cleanup(&dirs).await; + + let barrier = PutObjectCommitBarrier::install(bucket, object, pause); + let mut reader = PutObjReader::from_vec(vec![0x32; 1 << 20]); + let mut put = Box::pin(store.put_object(bucket, object, &mut reader, &opts)); + tokio::select! { + result = &mut put => panic!("replacement completed before the barrier: {result:?}"), + paused = timeout(Duration::from_secs(10), barrier.wait_until_paused()) => { + paused.expect("replacement must reach its commit barrier"); + } + } + let staged = staged_workspaces(&dirs).await; + assert_eq!(staged.len(), 4, "replacement must stage on every disk before cancellation"); + for workspace in staged { + let mut entries = fs::read_dir(workspace).await.expect("read staged workspace"); + let data_dir = entries + .next_entry() + .await + .expect("read staged directory") + .expect("a real shard"); + assert!( + fs::metadata(data_dir.path().join("part.1")) + .await + .expect("real staged payload") + .len() + > 0 + ); + } + // Hold the deterministic barrier so the deadline, not disk speed, drops + // the save future. No detached owner may finish and mask missing cleanup. + timeout(Duration::from_secs(1), put) + .await + .expect_err("held replacement must time out"); + // delete_all hands staging to .trash for ordinary background GC; it does + // not promise immediate space reclamation when the request is cancelled. + wait_for_staging_cleanup(&dirs).await; + drop(barrier); + let mut reader = store + .get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default()) + .await + .expect("previous checkpoint remains readable"); + let mut actual = Vec::new(); + reader + .stream + .read_to_end(&mut actual) + .await + .expect("read previous checkpoint"); + assert_eq!(actual, old_body, "timeout must neither publish replacement nor delete old data"); +} + +#[tokio::test] +async fn checkpoint_timeout_before_namespace_cleans_staging() { + checkpoint_timeout_at(PutObjectCommitPause::BeforeNamespace).await; +} + +#[tokio::test] +async fn checkpoint_timeout_before_commit_owner_cleans_staging() { + checkpoint_timeout_at(PutObjectCommitPause::AfterNamespace).await; +} + +#[tokio::test] +async fn checkpoint_timeout_after_commit_owner_cleans_staging() { + checkpoint_timeout_at(PutObjectCommitPause::AfterQuotaReservation).await; +} diff --git a/crates/ecstore/tests/storage_api.rs b/crates/ecstore/tests/storage_api.rs index acfd5c85e..c14769b3f 100644 --- a/crates/ecstore/tests/storage_api.rs +++ b/crates/ecstore/tests/storage_api.rs @@ -61,6 +61,25 @@ pub(crate) mod minio_generated_read { }; } +pub(crate) mod cleanup_isolation { + pub(crate) use super::{DiskAPI, DiskOption, Endpoint, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{DeleteOptions, DiskError}; +} + +#[cfg(all(feature = "test-util", not(windows)))] +pub(crate) mod inline_heal_staging { + pub(crate) use super::{DiskAPI, DiskOption, Endpoint, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{ + DeleteOptions, DiskError, LocalPublicationPause, LocalPublicationStage, ReadOptions, + }; +} + +#[cfg(all(feature = "test-util", target_os = "linux"))] +pub(crate) mod fd_relative { + pub(crate) use super::{DiskAPI, DiskOption, DiskStore, Endpoint, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{DiskError, LocalPublicationPause, LocalPublicationStage, ReadOptions}; +} + pub(crate) mod pinned_erasure_fixtures { pub(crate) use super::storage_contracts::{BucketOperations, HTTPRangeSpec, MakeBucketOptions, ObjectIO}; pub(crate) use super::{ECStore, Endpoint, Erasure, Error, ObjectOptions}; diff --git a/crates/filemeta/benches/xl_meta_bench.rs b/crates/filemeta/benches/xl_meta_bench.rs index ee46865a4..dc093e21d 100644 --- a/crates/filemeta/benches/xl_meta_bench.rs +++ b/crates/filemeta/benches/xl_meta_bench.rs @@ -12,9 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use criterion::{Criterion, Throughput, criterion_group, criterion_main}; -use rustfs_filemeta::{FileMeta, MetaCacheEntry, test_data::*}; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; +use rustfs_filemeta::{FileInfo, FileMeta, MetaCacheEntry, ObjectPartInfo, test_data::*}; use std::hint::black_box; +use std::time::Duration; +use time::OffsetDateTime; +use uuid::Uuid; fn bench_create_real_xlmeta(c: &mut Criterion) { c.bench_function("create_real_xlmeta", |b| b.iter(|| black_box(create_real_xlmeta().unwrap()))); @@ -134,6 +137,73 @@ fn bench_list_page_to_fileinfo(c: &mut Criterion) { group.finish(); } +fn bench_delete_inline_versions(c: &mut Criterion) { + let mut group = c.benchmark_group("delete_inline_versions"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(500)); + group.measurement_time(Duration::from_secs(2)); + for count in [1000_u32, 2000, 10000] { + let mut meta = FileMeta::new(); + let mut inline = vec![1]; + rmp::encode::write_map_len(&mut inline, count).expect("inline map header"); + let payload = vec![42; 9 * 1024]; + let mut requests = Vec::new(); + for id in 1..=count { + let version = Uuid::from_u128(u128::from(id)); + let mut fi = FileInfo::new("object", 1, 0); + fi.version_id = Some(version); + fi.data_dir = Some(Uuid::from_u128(u128::from(count) + u128::from(id))); + fi.mod_time = Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1_700_000_000 + i64::from(id))); + fi.erasure.index = 1; + fi.size = i64::try_from(payload.len()).expect("payload length"); + fi.parts.push(ObjectPartInfo { + number: 1, + size: payload.len(), + actual_size: fi.size, + ..Default::default() + }); + fi.set_inline_data(); + meta.add_version(fi).expect("seed version metadata"); + rmp::encode::write_str(&mut inline, &version.to_string()).expect("inline key"); + rmp::encode::write_bin(&mut inline, &payload).expect("inline body"); + if id <= 1000 { + requests.push(FileInfo { + version_id: Some(version), + ..Default::default() + }); + } + } + meta.data.update(&inline); + group.throughput(Throughput::Elements(u64::try_from(requests.len()).expect("batch size"))); + group.bench_with_input(BenchmarkId::new("batch", count), &count, |b, _| { + b.iter_batched( + || meta.clone(), + |mut meta| black_box(meta.delete_versions(black_box(&requests)).expect("batch delete")), + BatchSize::LargeInput, + ); + }); + if count <= 2000 { + group.bench_with_input(BenchmarkId::new("sequential", count), &count, |b, _| { + b.iter_batched( + || meta.clone(), + |mut meta| { + for request in black_box(&requests) { + if let Some(dir) = meta.delete_version(request).expect("sequential delete") { + meta.data + .remove(vec![request.version_id.expect("explicit version"), dir]) + .expect("remove inline body"); + } + } + black_box(meta); + }, + BatchSize::LargeInput, + ); + }); + } + } + group.finish(); +} + criterion_group!( benches, bench_create_real_xlmeta, @@ -147,7 +217,8 @@ criterion_group!( bench_version_stats, bench_validate_integrity, bench_into_fileinfo_realistic, - bench_list_page_to_fileinfo + bench_list_page_to_fileinfo, + bench_delete_inline_versions ); criterion_main!(benches); diff --git a/crates/filemeta/src/filemeta.rs b/crates/filemeta/src/filemeta.rs index 3d721ef41..e4cf323f2 100644 --- a/crates/filemeta/src/filemeta.rs +++ b/crates/filemeta/src/filemeta.rs @@ -347,21 +347,25 @@ impl FileMeta { self.versions.sort_by(cmp_shallow_versions_for_order); } - fn find_inline_data_for_version(&self, version_id: Option) -> Result>> { + fn find_inline_data_in_slice(inline_data: &[u8], version_id: Option) -> Result>> { let key = data_key_for_version(version_id); - if let Some(data) = self.data.find(key.as_str())? { + if let Some(data) = InlineData::find_in_slice(inline_data, key.as_str())? { return Ok(Some(data)); } if let Some(legacy_key) = legacy_data_key_for_version(version_id) && legacy_key != key { - return self.data.find(legacy_key.as_str()); + return InlineData::find_in_slice(inline_data, legacy_key.as_str()); } Ok(None) } + fn find_inline_data_for_version(&self, version_id: Option) -> Result>> { + Self::find_inline_data_in_slice(self.data.as_slice(), version_id) + } + // Find version pub fn find_version(&self, vid: Option) -> Result<(usize, FileMetaVersion)> { let vid = vid.unwrap_or_default(); @@ -696,9 +700,55 @@ impl FileMeta { // Ok(()) } + /// Apply a disk deletion batch while compacting inline payloads only once. + /// Missing non-marker requests retain the disk layer's idempotent semantics. + pub fn delete_versions(&mut self, requests: &[FileInfo]) -> Result>> { + if requests.is_empty() { + return Ok(Vec::new()); + } + let mut inline_keys = self.data.key_set()?; + let mut removed_keys = std::collections::HashSet::new(); + let mut data_dirs = Vec::with_capacity(requests.len()); + let mut failure = None; + for request in requests { + // Later requests must observe earlier inline removals even though + // the payload buffer is not rebuilt until the batch is complete. + let has_inline_data = self.data.version_ok() + && (inline_keys.contains(data_key_for_version(request.version_id).as_bytes()) + || legacy_data_key_for_version(request.version_id).is_some_and(|key| inline_keys.contains(key.as_bytes()))); + match self.delete_version_inner(request, Some(has_inline_data)) { + Ok(data_dir) => { + if let Some(dir) = data_dir { + for key in [request.version_id.unwrap_or_default(), dir] { + let key = key.to_string().into_bytes(); + if inline_keys.remove(&key) { + removed_keys.insert(key); + } + } + } + data_dirs.push(data_dir); + } + Err(Error::FileNotFound | Error::FileVersionNotFound) if !request.deleted => data_dirs.push(None), + Err(err) => { + failure = Some(err); + break; + } + } + } + self.data.remove_keys(&removed_keys)?; + if let Some(err) = failure { + return Err(err); + } + Ok(data_dirs) + } + // delete_version deletes version, returns data_dir #[tracing::instrument(level = "debug", skip(self))] pub fn delete_version(&mut self, fi: &FileInfo) -> Result> { + self.delete_version_inner(fi, None) + } + + fn delete_version_inner(&mut self, fi: &FileInfo, has_inline_data: Option) -> Result> { fi.validate_for_delete_operation()?; let vid = Some(fi.version_id.unwrap_or(Uuid::nil())); @@ -930,7 +980,7 @@ impl FileMeta { return Err(e); } - if self.shared_data_dir_count(obj_version_id, obj_data_dir) > 0 { + if self.shared_data_dir_count_with_inline_data(obj_version_id, obj_data_dir, has_inline_data) > 0 { return Ok(None); } @@ -987,6 +1037,74 @@ impl FileMeta { version_id: &str, opts: FileInfoDecodeOptions, ) -> Result { + Self::select_and_decode_fileinfo( + self.versions.len(), + self.versions.iter().map(|ver| Ok((ver.header.clone(), ver.meta.as_slice()))), + self.data.as_slice(), + volume, + path, + version_id, + opts, + ) + } + + fn decode_fileinfo_from_buf( + buf: &[u8], + volume: &str, + path: &str, + version_id: &str, + opts: FileInfoDecodeOptions, + ) -> Result { + let vid = if version_id.is_empty() { + None + } else { + Some(Uuid::parse_str(version_id)?) + }; + + let (meta, inline_buf) = Self::parse_validated_xl_envelope(buf)?; + let (versions_len, header_ver, meta) = if meta.is_empty() { + (0, 0, meta) + } else { + let (versions_len, header_ver, _meta_ver, meta) = Self::decode_validated_xl_headers(meta)?; + (versions_len, header_ver, meta) + }; + + if versions_len == 0 { + return Ok(FileInfo { + volume: volume.to_owned(), + name: path.to_owned(), + version_id: vid, + is_latest: true, + deleted: true, + mod_time: Some(OffsetDateTime::from_unix_timestamp(1)?), + ..Default::default() + }); + } + + let mut cur = Cursor::new(meta); + Self::select_and_decode_fileinfo( + versions_len, + (0..versions_len).map(|_| Self::read_next_shallow_slice(meta, &mut cur, header_ver)), + inline_buf, + volume, + path, + version_id, + opts, + ) + } + + fn select_and_decode_fileinfo<'a, I>( + versions_len: usize, + versions: I, + inline_data: &[u8], + volume: &str, + path: &str, + version_id: &str, + opts: FileInfoDecodeOptions, + ) -> Result + where + I: IntoIterator>, + { let vid = { if !version_id.is_empty() { Uuid::parse_str(version_id)? @@ -1006,20 +1124,20 @@ impl FileMeta { let mut is_latest = true; let mut succ_mod_time = None; - let mut non_free_versions = self.versions.len(); + let mut non_free_versions = versions_len; let mut found = false; let mut found_free_version = None; - let mut found_fi = None; + let mut found_fi: Option> = None; - for ver in self.versions.iter() { - let header = &ver.header; + for item in versions { + let (header, ver_meta) = item?; if header.free_version() { non_free_versions -= 1; if opts.include_free_versions && found_free_version.is_none() - && let Ok(found_free_fi) = ver.parse_version_meta() + && let Ok(found_free_fi) = FileMetaVersion::try_from(ver_meta) && found_free_fi.version_type != VersionType::Invalid { // Graceful degradation: the free-version replication-accounting record @@ -1028,8 +1146,12 @@ impl FileMeta { // Known side effect: if a disk holds only free versions and they are // corrupt, `into_fileinfo` falls through to `FileNotFound` (not // `FileCorrupt`), so that disk is not enqueued for heal. - match found_free_fi.to_fileinfo_with_part_checksums(volume, path, opts.all_parts, opts.include_part_checksums) - { + match found_free_fi.into_fileinfo_with_part_checksums( + volume, + path, + opts.all_parts, + opts.include_part_checksums, + ) { Ok(mut free_fi) => { free_fi.is_latest = true; found_free_version = Some(free_fi); @@ -1057,18 +1179,23 @@ impl FileMeta { found = true; - let mut fi = ver.to_fileinfo_with_part_checksums(volume, path, opts.all_parts, opts.include_part_checksums)?; - fi.is_latest = is_latest; + found_fi = Some( + FileMetaVersion::try_from(ver_meta) + .and_then(|v| v.into_fileinfo_with_part_checksums(volume, path, opts.all_parts, opts.include_part_checksums)) + .and_then(|mut fi| { + fi.is_latest = is_latest; - if let Some(_d) = succ_mod_time { - fi.successor_mod_time = succ_mod_time; - } + if let Some(_d) = succ_mod_time { + fi.successor_mod_time = succ_mod_time; + } - if opts.read_data && fi.inline_data() { - fi.data = self.find_inline_data_for_version(fi.version_id)?.map(bytes::Bytes::from); - } + if opts.read_data && fi.inline_data() { + fi.data = Self::find_inline_data_in_slice(inline_data, fi.version_id)?.map(bytes::Bytes::from); + } - found_fi = Some(fi); + Ok(fi) + }), + ); } if !found { @@ -1085,7 +1212,8 @@ impl FileMeta { } } - if let Some(mut fi) = found_fi { + if let Some(fi_res) = found_fi { + let mut fi = fi_res?; fi.num_versions = non_free_versions; return Ok(fi); diff --git a/crates/filemeta/src/filemeta/codec.rs b/crates/filemeta/src/filemeta/codec.rs index d1feb19ac..095b2d016 100644 --- a/crates/filemeta/src/filemeta/codec.rs +++ b/crates/filemeta/src/filemeta/codec.rs @@ -130,10 +130,8 @@ impl FileMeta { Ok((bin_len, remaining)) } - #[hotpath::measure(impl_type = "FileMeta")] - pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result { - let i = buf.len() as u64; - + #[inline] + pub(super) fn parse_validated_xl_envelope(buf: &[u8]) -> Result<(&[u8], &[u8])> { // check version, buf = buf[8..] let (buf, _, _) = Self::check_xl2_v1(buf)?; @@ -188,12 +186,11 @@ impl FileMeta { } if !buf.is_empty() { - self.data.update(buf); - self.data.validate().map_err(|e| { + InlineData::validate_slice(buf).map_err(|e| { error!("data validation failed: {}", e); // Validation only examines bytes already read. Classify damage // as repairable corruption only for formats we understand. - if self.data.version_ok() && Self::decode_xl_headers(meta).is_ok() { + if InlineData::version_ok_slice(buf) && Self::decode_xl_headers(meta).is_ok() { Error::FileCorrupt } else { e @@ -201,82 +198,110 @@ impl FileMeta { })?; } + Ok((meta, buf)) + } + + #[inline] + pub(super) fn read_next_shallow_slice<'a>( + meta: &'a [u8], + cur: &mut Cursor<&'a [u8]>, + header_ver: u8, + ) -> Result<(FileMetaVersionHeader, &'a [u8])> { + let bin_len = rmp::decode::read_bin_len(cur).map_err(|e| { + error!("failed to read binary length for version header: {}", e); + Error::other(format!("failed to read binary length for version header: {e}")) + })? as usize; + + let start = cur.position() as usize; + let remaining = meta.len().saturating_sub(start); + if bin_len > remaining { + error!("corrupt xl.meta: version header length {} exceeds remaining {} bytes", bin_len, remaining); + return Err(Error::other("corrupt xl.meta: version header length exceeds metadata size")); + } + let end = start + bin_len; + + let mut header = FileMetaVersionHeader::default(); + header.unmarshal_v(header_ver, &meta[start..end]).map_err(|e| { + error!("failed to unmarshal version header: {}", e); + e + })?; + cur.set_position(end as u64); + + let bin_len = rmp::decode::read_bin_len(cur).map_err(|e| { + error!("failed to read binary length for version metadata: {}", e); + Error::other(format!("failed to read binary length for version metadata: {e}")) + })? as usize; + + let start = cur.position() as usize; + let remaining = meta.len().saturating_sub(start); + if bin_len > remaining { + error!( + "corrupt xl.meta: version metadata length {} exceeds remaining {} bytes", + bin_len, remaining + ); + return Err(Error::other("corrupt xl.meta: version metadata length exceeds metadata size")); + } + let end = start + bin_len; + cur.set_position(end as u64); + + Ok((header, &meta[start..end])) + } + + #[hotpath::measure(impl_type = "FileMeta")] + pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result { + let i = buf.len() as u64; + let (meta, inline_buf) = Self::parse_validated_xl_envelope(buf)?; + + if !inline_buf.is_empty() { + self.data.update(inline_buf); + } + // Parse meta if !meta.is_empty() { - let (versions_len, header_ver, meta_ver, meta) = Self::decode_xl_headers(meta).map_err(|e| { - error!("failed to decode XL headers: {}", e); - e - })?; - - // let (_, meta) = meta.split_at(read_size as usize); + let (versions_len, header_ver, meta_ver, meta) = Self::decode_validated_xl_headers(meta)?; self.meta_ver = meta_ver; - - // `versions_len` is decoded from a potentially corrupted buffer. - // Every version contributes at least two msgpack bin headers to - // `meta`, so a count larger than the remaining bytes is corrupt; - // reject it before sizing any allocation from it (rustfs/rustfs#2715). - if versions_len > meta.len() { - error!( - "corrupt xl.meta: version count {} exceeds remaining metadata size {}", - versions_len, - meta.len() - ); - return Err(Error::other(format!( - "corrupt xl.meta: version count {versions_len} exceeds metadata size {}", - meta.len() - ))); - } - self.versions = Vec::with_capacity(versions_len); let mut cur: Cursor<&[u8]> = Cursor::new(meta); for _ in 0..versions_len { - let bin_len = rmp::decode::read_bin_len(&mut cur).map_err(|e| { - error!("failed to read binary length for version header: {}", e); - Error::other(format!("failed to read binary length for version header: {e}")) - })? as usize; - - let remaining = meta.len().saturating_sub(cur.position() as usize); - if bin_len > remaining { - error!("corrupt xl.meta: version header length {} exceeds remaining {} bytes", bin_len, remaining); - return Err(Error::other("corrupt xl.meta: version header length exceeds metadata size")); - } - let mut header_buf = vec![0u8; bin_len]; - - cur.read_exact(&mut header_buf)?; - - let mut ver = FileMetaShallowVersion::default(); - ver.header.unmarshal_v(header_ver, &header_buf).map_err(|e| { - error!("failed to unmarshal version header: {}", e); - e - })?; - - let bin_len = rmp::decode::read_bin_len(&mut cur).map_err(|e| { - error!("failed to read binary length for version metadata: {}", e); - Error::other(format!("failed to read binary length for version metadata: {e}")) - })? as usize; - - let remaining = meta.len().saturating_sub(cur.position() as usize); - if bin_len > remaining { - error!( - "corrupt xl.meta: version metadata length {} exceeds remaining {} bytes", - bin_len, remaining - ); - return Err(Error::other("corrupt xl.meta: version metadata length exceeds metadata size")); - } - let mut ver_meta_buf = vec![0u8; bin_len]; - cur.read_exact(&mut ver_meta_buf)?; - - ver.meta.extend_from_slice(&ver_meta_buf); - - self.versions.push(ver); + let (header, ver_meta) = Self::read_next_shallow_slice(meta, &mut cur, header_ver)?; + self.versions.push(FileMetaShallowVersion { + header, + meta: ver_meta.to_vec(), + }); } } Ok(i) } + #[inline] + pub(super) fn decode_validated_xl_headers(meta: &[u8]) -> Result<(usize, u8, u8, &[u8])> { + let (versions_len, header_ver, meta_ver, meta) = Self::decode_xl_headers(meta).map_err(|e| { + error!("failed to decode XL headers: {}", e); + e + })?; + + // `versions_len` is decoded from a potentially corrupted buffer. + // Every version contributes at least two msgpack bin headers to + // `meta`, so a count larger than the remaining bytes is corrupt; + // reject it before sizing any allocation from it (rustfs/rustfs#2715). + if versions_len > meta.len() { + error!( + "corrupt xl.meta: version count {} exceeds remaining metadata size {}", + versions_len, + meta.len() + ); + return Err(Error::other(format!( + "corrupt xl.meta: version count {versions_len} exceeds metadata size {}", + meta.len() + ))); + } + + Ok((versions_len, header_ver, meta_ver, meta)) + } + // decode_xl_headers parses meta header, returns (versions count, xl_header_version, xl_meta_version, read data length) fn decode_xl_headers(buf: &[u8]) -> Result<(usize, u8, u8, &[u8])> { let mut cur = Cursor::new(buf); @@ -359,7 +384,8 @@ impl FileMeta { #[hotpath::measure(impl_type = "FileMeta")] pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = + Vec::with_capacity(24 + self.versions.iter().map(|v| 48 + v.meta.len()).sum::() + self.data.as_slice().len()); // header wr.write_all(XL_FILE_HEADER.as_slice())?; @@ -382,9 +408,12 @@ impl FileMeta { rmp::encode::write_sint(&mut wr, self.versions.len() as i64)?; + let mut hbuf = [0u8; 48]; for ver in self.versions.iter() { - let hmsg = ver.header.marshal_msg()?; - rmp::encode::write_bin(&mut wr, &hmsg)?; + let mut hcur = Cursor::new(&mut hbuf[..]); + ver.header.encode_to(&mut hcur)?; + let hlen = hcur.position() as usize; + rmp::encode::write_bin(&mut wr, &hbuf[..hlen])?; rmp::encode::write_bin(&mut wr, &ver.meta)?; } diff --git a/crates/filemeta/src/filemeta/inline_data.rs b/crates/filemeta/src/filemeta/inline_data.rs index e1154e5c3..286f49775 100644 --- a/crates/filemeta/src/filemeta/inline_data.rs +++ b/crates/filemeta/src/filemeta/inline_data.rs @@ -114,14 +114,24 @@ impl FileMeta { /// Count shared data directories pub fn shared_data_dir_count(&self, version_id: Option, data_dir: Option) -> usize { + self.shared_data_dir_count_with_inline_data(version_id, data_dir, None) + } + + pub(super) fn shared_data_dir_count_with_inline_data( + &self, + version_id: Option, + data_dir: Option, + has_inline_data: Option, + ) -> usize { let vid = version_id.unwrap_or_default(); let Some(data_dir) = data_dir else { return 0; }; - if self.data.entries().unwrap_or_default() > 0 - && self.find_inline_data_for_version(version_id).unwrap_or_default().is_some() - { + if has_inline_data.unwrap_or_else(|| { + self.data.entries().unwrap_or_default() > 0 + && self.find_inline_data_for_version(version_id).unwrap_or_default().is_some() + }) { return 0; } @@ -297,4 +307,191 @@ mod tests { let got = meta.find_unshared_data_dir_for_version(Some(target_version)); assert_eq!(got, None); } + + fn sequential_delete(meta: &mut FileMeta, requests: &[FileInfo]) -> Result>> { + let mut dirs = Vec::new(); + for request in requests { + match meta.delete_version(request) { + Ok(dir) => { + if let Some(dir) = dir { + meta.data.remove(vec![request.version_id.unwrap_or_default(), dir])?; + } + dirs.push(dir); + } + Err(Error::FileNotFound | Error::FileVersionNotFound) if !request.deleted => dirs.push(None), + Err(err) => return Err(err), + } + } + Ok(dirs) + } + + fn assert_batch_matches_sequential(original: &FileMeta, requests: &[FileInfo]) { + let mut expected = original.clone(); + let expected_dirs = sequential_delete(&mut expected, requests); + let mut actual = original.clone(); + assert_eq!(actual.delete_versions(requests), expected_dirs); + let decode = |meta: &FileMeta| { + meta.versions + .iter() + .map(|version| version.parse_version_meta().expect("decode retained version")) + .collect::>() + }; + assert_eq!(decode(&actual), decode(&expected)); + assert_eq!(actual.data, expected.data, "retained inline bytes and ordering must be identical"); + } + + #[test] + fn delete_versions_matches_sequential_mixed_version_subsets() { + let mut original = FileMeta::new(); + let mut requests = Vec::new(); + for id in 0_u128..8 { + let version = Uuid::from_u128(id); + // Versions 2 and 3 share physical data. Version 4's data-dir key + // aliases version 5's inline key, so later decisions must see its removal. + let dir = Uuid::from_u128(match id { + 2 | 3 | 5 => 100, + 4 => 5, + _ => 200 + id, + }); + let mut fi = make_plain_file_info(version, dir); + fi.mod_time = + Some(OffsetDateTime::UNIX_EPOCH + Duration::seconds(1_700_000_000 + i64::try_from(id).expect("small id"))); + if !matches!(id, 2 | 3) { + fi.data = Some(Bytes::from(vec![u8::try_from(id).expect("small id"); 97])); + fi.set_inline_data(); + } + original.add_version(fi).expect("seed mixed version chain"); + requests.push(FileInfo { + version_id: Some(version), + ..Default::default() + }); + } + for mask in 0_u32..256 { + let selected: Vec<_> = requests + .iter() + .enumerate() + .filter(|(i, _)| mask & (1 << i) != 0) + .map(|(_, fi)| fi.clone()) + .collect(); + assert_batch_matches_sequential(&original, &selected); + let reversed: Vec<_> = selected.into_iter().rev().collect(); + assert_batch_matches_sequential(&original, &reversed); + } + let mut duplicate_and_missing = requests.clone(); + duplicate_and_missing.extend(requests.iter().rev().cloned()); + duplicate_and_missing.push(FileInfo { + version_id: Some(Uuid::from_u128(999)), + ..Default::default() + }); + assert_batch_matches_sequential(&original, &duplicate_and_missing); + } + + #[test] + fn delete_versions_preserves_pending_purge_and_stops_at_invalid_request() { + let mut original = FileMeta::new(); + let mut fi = make_plain_file_info(Uuid::from_u128(1), Uuid::from_u128(101)); + fi.data = Some(Bytes::from_static(b"retained until replication completes")); + fi.set_inline_data(); + original.add_version(fi).expect("seed inline version"); + let pending = FileInfo { + version_id: Some(Uuid::from_u128(1)), + mark_deleted: true, + replication_state_internal: Some(ReplicationState { + version_purge_status_internal: Some("target=PENDING;".into()), + purge_targets: version_purge_statuses_map("target=PENDING;"), + ..Default::default() + }), + ..Default::default() + }; + assert_batch_matches_sequential(&original, std::slice::from_ref(&pending)); + let completed = FileInfo { + version_id: pending.version_id, + replication_state_internal: Some(ReplicationState { + version_purge_status_internal: Some("target=COMPLETE;".into()), + purge_targets: version_purge_statuses_map("target=COMPLETE;"), + ..Default::default() + }), + ..Default::default() + }; + assert_batch_matches_sequential(&original, &[pending, completed]); + let valid = FileInfo { + version_id: Some(Uuid::from_u128(1)), + ..Default::default() + }; + let invalid = FileInfo { + parts: vec![ + ObjectPartInfo { + number: 1, + ..Default::default() + }; + 2 + ], + ..Default::default() + }; + assert!(invalid.validate_for_delete_operation().is_err()); + assert_batch_matches_sequential(&original, &[valid, invalid]); + } + + #[test] + fn delete_versions_rejects_corrupt_inline_before_mutating_versions() { + let mut meta = FileMeta::new(); + meta.add_version(make_plain_file_info(Uuid::from_u128(1), Uuid::from_u128(101))) + .expect("seed version"); + for bytes in [ + vec![1, 0x81, 0xa1, b'x', 0xc4, 255], + vec![2, 0x80], + vec![0, 0x80], + vec![1, 0x81, 0xa1, 0xff, 0xc4, 0], + vec![1, 0x81, 0xdb, 1, 0, 0, 1], + vec![1, 0x81, 0xa1, b'x', 0xc6, 0xff, 0xff, 0xff, 0xff], + ] { + meta.data.update(&bytes); + let before = meta.clone(); + assert!( + meta.delete_versions(&[]) + .expect("empty batch does not inspect metadata") + .is_empty() + ); + assert!( + meta.delete_versions(&[FileInfo { + version_id: Some(Uuid::from_u128(1)), + ..Default::default() + }]) + .is_err() + ); + assert_eq!(meta, before); + } + } + + #[test] + fn delete_versions_preserves_both_null_inline_key_encodings() { + for legacy in [false, true] { + let mut original = FileMeta::new(); + let mut fi = make_plain_null_version_file_info(Uuid::from_u128(100)); + fi.data = Some(Bytes::from_static(b"null version body")); + fi.set_inline_data(); + original.add_version(fi).expect("seed null version"); + if legacy { + let value = original + .data + .find(NULL_VERSION_ID) + .expect("find null body") + .expect("null body"); + original.data.remove_key(NULL_VERSION_ID).expect("remove modern null key"); + original + .data + .replace(&Uuid::nil().to_string(), value) + .expect("write legacy null key"); + } + for version_id in [None, Some(Uuid::nil())] { + assert_batch_matches_sequential( + &original, + &[FileInfo { + version_id, + ..Default::default() + }], + ); + } + } + } } diff --git a/crates/filemeta/src/filemeta/msgp_decode.rs b/crates/filemeta/src/filemeta/msgp_decode.rs index b99e6a7ed..1ebb58755 100644 --- a/crates/filemeta/src/filemeta/msgp_decode.rs +++ b/crates/filemeta/src/filemeta/msgp_decode.rs @@ -17,45 +17,14 @@ use rmp::Marker; use std::io::Read; /// Maximum accepted length for a single length-prefixed msgpack element -/// (map key, string, binary blob, ext payload, or a whole serialized -/// xl.meta record inside a metacache stream). +/// (map key, string, binary blob, or ext payload) inside xl.meta. /// -/// Legitimate elements are far smaller: object keys are at most a few KiB -/// and even an xl.meta record fully populated up to the 10,000-version -/// object cap serializes to a few MiB (metacache streams carry xl.meta -/// without inline data), so 16 MiB leaves ample headroom. A larger value -/// means the length prefix itself is corrupt; decoding must fail with an -/// error instead of attempting a huge allocation that aborts the whole -/// process (see rustfs/rustfs#2715). +/// This is not a limit on the aggregate xl.meta carried by a metacache +/// stream: many versions and their inline data can exceed it. Bound each +/// individual value before allocation so corrupted prefixes return an error +/// instead of attempting a huge allocation (see rustfs/rustfs#2715). pub(crate) const MAX_MSGP_ELEMENT_SIZE: usize = 16 << 20; -/// Reads exactly `len` bytes into a fresh buffer, treating `len` as -/// untrusted input: lengths above [`MAX_MSGP_ELEMENT_SIZE`] and allocation -/// failures surface as decode errors instead of aborting the process. -pub(crate) fn read_exact_vec(rd: &mut R, len: usize) -> Result> { - if len > MAX_MSGP_ELEMENT_SIZE { - return Err(Error::other(format!( - "corrupt msgpack element: length {len} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" - ))); - } - let mut buf = Vec::new(); - buf.try_reserve_exact(len) - .map_err(|e| Error::other(format!("msgpack element allocation of {len} bytes failed: {e}")))?; - buf.resize(len, 0); - rd.read_exact(&mut buf).map_err(Error::from)?; - Ok(buf) -} - -/// Bounds a decoded collection count when it is used only as a -/// pre-allocation hint. The decode loop still consumes exactly the decoded -/// number of elements (a corrupt count fails with an EOF decode error once -/// the input runs out); this merely keeps the speculative reservation from -/// aborting the process on an absurd count. -pub(crate) fn prealloc_hint(len: usize) -> usize { - const MAX_PREALLOC_ITEMS: usize = 4096; - len.min(MAX_PREALLOC_ITEMS) -} - /// Reader that prepends a single byte to the stream. Used when we've read the marker /// and need to pass it to a decoder that expects to read the marker itself. pub(crate) struct PrependByteReader<'a, R> { @@ -77,7 +46,72 @@ impl Read for PrependByteReader<'_, R> { } } +/// Reads a nil marker (`0xc0`) or a MessagePack map length using `rmp::decode::read_map_len` +/// semantics so error variants match `read_map_len` while leaving `rd` unwrapped afterward. +#[inline] +pub(crate) fn read_nil_or_submap_len(rd: &mut R) -> Result> { + let mut first = [0u8; 1]; + rd.read_exact(&mut first).map_err(Error::from)?; + match first[0] { + 0xc0 => Ok(None), + 0x80..=0x8f => Ok(Some((first[0] & 0x0f) as usize)), + b => { + let mut prepend = PrependByteReader { + byte: Some(b), + inner: rd, + }; + let len = rmp::decode::read_map_len(&mut prepend)?; + Ok(Some(len as usize)) + } + } +} + +/// Reads a MessagePack string key into a 16-byte stack buffer when it fits, +/// falling back to `heap` only for longer keys. +#[inline] +pub(crate) fn read_small_str<'a, R: Read>(rd: &mut R, inline: &'a mut [u8; 16], heap: &'a mut Vec) -> Result<&'a str> { + let len = usize::try_from(rmp::decode::read_str_len(rd)?).map_err(Error::other)?; + let bytes = if len <= inline.len() { + rd.read_exact(&mut inline[..len]).map_err(Error::from)?; + &inline[..len] + } else { + *heap = read_exact_vec(rd, len)?; + heap.as_slice() + }; + std::str::from_utf8(bytes).map_err(|e| Error::FromUtf8(e.to_string())) +} + +/// Reads exactly `len` bytes into a fresh buffer, treating `len` as +/// untrusted input: lengths above [`MAX_MSGP_ELEMENT_SIZE`] and allocation +/// failures surface as decode errors instead of aborting the process. +#[inline] +pub(crate) fn read_exact_vec(rd: &mut R, len: usize) -> Result> { + if len > MAX_MSGP_ELEMENT_SIZE { + return Err(Error::other(format!( + "corrupt msgpack element: length {len} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" + ))); + } + let mut buf = Vec::new(); + buf.try_reserve_exact(len) + .map_err(|e| Error::other(format!("msgpack element allocation of {len} bytes failed: {e}")))?; + buf.resize(len, 0); + rd.read_exact(&mut buf).map_err(Error::from)?; + Ok(buf) +} + +/// Bounds a decoded collection count when it is used only as a +/// pre-allocation hint. The decode loop still consumes exactly the decoded +/// number of elements (a corrupt count fails with an EOF decode error once +/// the input runs out); this merely keeps the speculative reservation from +/// aborting the process on an absurd count. +#[inline] +pub(crate) fn prealloc_hint(len: usize) -> usize { + const MAX_PREALLOC_ITEMS: usize = 4096; + len.min(MAX_PREALLOC_ITEMS) +} + /// Read MessagePack nil or array length. Returns None for nil, Some(len) for array. +#[inline] pub(crate) fn read_nil_or_array_len(rd: &mut R) -> Result> { let mut buf = [0u8; 1]; rd.read_exact(&mut buf).map_err(Error::from)?; @@ -100,6 +134,7 @@ pub(crate) fn read_nil_or_array_len(rd: &mut R) -> Result } /// Read MessagePack nil or map length. Returns None for nil, Some(len) for map. +#[inline] pub(crate) fn read_nil_or_map_len(rd: &mut R) -> Result> { let mut buf = [0u8; 1]; rd.read_exact(&mut buf).map_err(Error::from)?; diff --git a/crates/filemeta/src/filemeta/version.rs b/crates/filemeta/src/filemeta/version.rs index 87beb3d80..e5384d5cd 100644 --- a/crates/filemeta/src/filemeta/version.rs +++ b/crates/filemeta/src/filemeta/version.rs @@ -23,7 +23,8 @@ //! legacy (rmp_serde) format. `try_from` falls back to rmp_serde when hand-written decode fails. use super::msgp_decode::{ - PrependByteReader, prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, skip_msgp_value, + prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, read_nil_or_submap_len, read_small_str, + skip_msgp_value, }; use super::*; use crate::{ChecksumInfo, TransitionVersionState}; @@ -85,6 +86,21 @@ fn fold_signature(crc: u64) -> [u8; 4] { ((crc ^ (crc >> 32)) as u32).to_le_bytes() } +struct Xxh64Writer(xxhash_rust::xxh64::Xxh64); + +impl std::io::Write for Xxh64Writer { + #[inline] + fn write(&mut self, buf: &[u8]) -> std::io::Result { + self.0.update(buf); + Ok(buf.len()) + } + + #[inline] + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + fn read_msgp_string(rd: &mut R) -> Result { let len = rmp::decode::read_str_len(rd)? as usize; let buf = read_exact_vec(rd, len)?; @@ -588,18 +604,8 @@ impl FileMetaShallowVersion { } pub fn into_fileinfo(&self, volume: &str, path: &str, all_parts: bool) -> Result { - self.parse_version_meta()?.into_fileinfo(volume, path, all_parts) - } - - pub(super) fn to_fileinfo_with_part_checksums( - &self, - volume: &str, - path: &str, - all_parts: bool, - include_part_checksums: bool, - ) -> Result { self.parse_version_meta()? - .to_fileinfo_with_part_checksums(volume, path, all_parts, include_part_checksums) + .into_fileinfo_with_part_checksums(volume, path, all_parts, true) } } @@ -652,15 +658,15 @@ impl FileMetaVersion { let mut cur = std::io::Cursor::new(buf); let mut fields = rmp::decode::read_map_len(&mut cur)?; let mut version_type = VersionType::Invalid; + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(&mut cur)? as usize; - let key_buf = read_exact_vec(&mut cur, key_len)?; - let key = String::from_utf8(key_buf)?; + let key = read_small_str(&mut cur, &mut inline_key, &mut heap_key)?; - match key.as_str() { + match key { "Type" => { let v: i64 = rmp::decode::read_int(&mut cur)?; version_type = VersionType::from_u8(v as u8); @@ -671,39 +677,29 @@ impl FileMetaVersion { continue; } - let mut first = [0u8; 1]; - cur.read_exact(&mut first)?; - if first[0] == 0xc0 { + let Some(mut obj_fields) = read_nil_or_submap_len(&mut cur)? else { return Ok(None); - } - - let mut prepend = PrependByteReader { - byte: Some(first[0]), - inner: &mut cur, }; - let mut obj_fields = rmp::decode::read_map_len(&mut prepend)?; let mut data_dir: Option = None; while obj_fields > 0 { obj_fields -= 1; - let obj_key_len = rmp::decode::read_str_len(&mut prepend)? as usize; - let obj_key_buf = read_exact_vec(&mut prepend, obj_key_len)?; - let obj_key = String::from_utf8(obj_key_buf)?; + let obj_key = read_small_str(&mut cur, &mut inline_key, &mut heap_key)?; if obj_key == "DDir" { - let bin_len = rmp::decode::read_bin_len(&mut prepend)? as usize; + let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; if bin_len != 16 { return Err(Error::other(format!("DDir must be 16 bytes, got {bin_len}"))); } let mut raw = [0u8; 16]; - prepend.read_exact(&mut raw)?; + cur.read_exact(&mut raw)?; let id = Uuid::from_bytes(raw); data_dir = if id.is_nil() { None } else { Some(id) }; break; } - skip_msgp_value(&mut prepend)?; + skip_msgp_value(&mut cur)?; } return Ok(data_dir); @@ -781,62 +777,44 @@ impl FileMetaVersion { pub fn decode_from(&mut self, rd: &mut R) -> Result<()> { let mut fields = rmp::decode::read_map_len(rd)?; *self = FileMetaVersion::default(); + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(rd)?; - let key_buf = read_exact_vec(rd, key_len as usize)?; - let key = String::from_utf8(key_buf)?; + let key = read_small_str(rd, &mut inline_key, &mut heap_key)?; - match key.as_str() { + match key { "Type" => { let v: i64 = rmp::decode::read_int(rd)?; self.version_type = VersionType::from_u8(v as u8); } "V1Obj" => { - let mut buf = [0u8; 1]; - rd.read_exact(&mut buf).map_err(Error::from)?; - if buf[0] == 0xc0 { - self.legacy_object = None; - } else { - let mut prepend = PrependByteReader { - byte: Some(buf[0]), - inner: rd, - }; + if let Some(obj_fields) = read_nil_or_submap_len(rd)? { let mut obj = MetaObjectV1::default(); - obj.decode_from(&mut prepend)?; + obj.decode_fields(rd, obj_fields)?; self.legacy_object = Some(obj); + } else { + self.legacy_object = None; } } "V2Obj" => { - let mut buf = [0u8; 1]; - rd.read_exact(&mut buf).map_err(Error::from)?; - if buf[0] == 0xc0 { - self.object = None; - } else { - let mut prepend = PrependByteReader { - byte: Some(buf[0]), - inner: rd, - }; + if let Some(obj_fields) = read_nil_or_submap_len(rd)? { let mut obj = MetaObject::default(); - obj.decode_from(&mut prepend)?; + obj.decode_fields(rd, obj_fields)?; self.object = Some(obj); + } else { + self.object = None; } } "DelObj" => { - let mut buf = [0u8; 1]; - rd.read_exact(&mut buf).map_err(Error::from)?; - if buf[0] == 0xc0 { - self.delete_marker = None; - } else { - let mut prepend = PrependByteReader { - byte: Some(buf[0]), - inner: rd, - }; + if let Some(dm_fields) = read_nil_or_submap_len(rd)? { let mut dm = MetaDeleteMarker::default(); - dm.decode_from(&mut prepend)?; + dm.decode_fields(rd, dm_fields)?; self.delete_marker = Some(dm); + } else { + self.delete_marker = None; } } "v" => { @@ -908,7 +886,7 @@ impl FileMetaVersion { } pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = Vec::with_capacity(256); self.encode_to(&mut wr)?; Ok(wr) } @@ -918,13 +896,33 @@ impl FileMetaVersion { } pub fn header(&self) -> FileMetaVersionHeader { - FileMetaVersionHeader::from(self.clone()) + FileMetaVersionHeader::from(self) } pub fn into_fileinfo(&self, volume: &str, path: &str, all_parts: bool) -> Result { self.to_fileinfo_with_part_checksums(volume, path, all_parts, true) } + pub(super) fn into_fileinfo_with_part_checksums( + self, + volume: &str, + path: &str, + all_parts: bool, + include_part_checksums: bool, + ) -> Result { + if self.version_type == VersionType::Object { + let mut fi = self.object.unwrap_or_default().into_fileinfo_with_part_checksums( + volume, + path, + all_parts, + include_part_checksums, + )?; + fi.uses_legacy_checksum = self.uses_legacy_checksum; + return Ok(fi); + } + self.to_fileinfo_with_part_checksums(volume, path, all_parts, include_part_checksums) + } + pub(super) fn to_fileinfo_with_part_checksums( &self, volume: &str, @@ -1215,27 +1213,31 @@ impl FileMetaVersionHeader { } pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = Vec::with_capacity(42); + self.encode_to(&mut wr)?; + Ok(wr) + } + pub fn encode_to(&self, wr: &mut W) -> Result<()> { // array len 7 - rmp::encode::write_array_len(&mut wr, 7)?; + rmp::encode::write_array_len(wr, 7)?; // version_id - rmp::encode::write_bin(&mut wr, self.version_id.unwrap_or_default().as_bytes())?; + rmp::encode::write_bin(wr, self.version_id.unwrap_or_default().as_bytes())?; // mod_time - rmp::encode::write_i64(&mut wr, self.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH).unix_timestamp_nanos() as i64)?; + rmp::encode::write_i64(wr, self.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH).unix_timestamp_nanos() as i64)?; // signature - rmp::encode::write_bin(&mut wr, self.signature.as_slice())?; + rmp::encode::write_bin(wr, self.signature.as_slice())?; // version_type - rmp::encode::write_uint8(&mut wr, self.version_type.to_u8())?; + rmp::encode::write_uint8(wr, self.version_type.to_u8())?; // flags - rmp::encode::write_uint8(&mut wr, self.flags)?; + rmp::encode::write_uint8(wr, self.flags)?; // ec_n - rmp::encode::write_uint8(&mut wr, self.ec_n)?; + rmp::encode::write_uint8(wr, self.ec_n)?; // ec_m - rmp::encode::write_uint8(&mut wr, self.ec_m)?; + rmp::encode::write_uint8(wr, self.ec_m)?; - Ok(wr) + Ok(()) } pub fn unmarshal_v(&mut self, version: u8, buf: &[u8]) -> Result { @@ -1396,8 +1398,8 @@ impl Ord for FileMetaVersionHeader { } } -impl From for FileMetaVersionHeader { - fn from(value: FileMetaVersion) -> Self { +impl From<&FileMetaVersion> for FileMetaVersionHeader { + fn from(value: &FileMetaVersion) -> Self { let flags = { let mut f: u8 = 0; if value.free_version() { @@ -1424,7 +1426,7 @@ impl From for FileMetaVersionHeader { version_id: value.get_version_id(), mod_time: value.get_mod_time(), signature: value.get_signature(), - version_type: value.version_type, + version_type: value.version_type.clone(), flags, ec_n, ec_m, @@ -1432,6 +1434,12 @@ impl From for FileMetaVersionHeader { } } +impl From for FileMetaVersionHeader { + fn from(value: FileMetaVersion) -> Self { + Self::from(&value) + } +} + #[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq)] // Because of custom message_pack, field order must be guaranteed pub struct MetaObject { @@ -1605,8 +1613,7 @@ impl MetaObjectV1 { && crate::fileinfo::is_valid_distribution(&self.erasure.distribution, data_blocks + parity_blocks) } - fn decode_from(&mut self, rd: &mut R) -> Result<()> { - let mut fields = rmp::decode::read_map_len(rd)?; + fn decode_fields(&mut self, rd: &mut R, mut fields: usize) -> Result<()> { *self = Self::default(); while fields > 0 { @@ -2049,7 +2056,7 @@ impl MetaObject { // marshal_msg custom messagepack naming consistent with go pub fn marshal_msg(&self) -> Result> { - let mut wr = Vec::new(); + let mut wr = Vec::with_capacity(256); self.encode_to(&mut wr)?; Ok(wr) } @@ -2058,32 +2065,30 @@ impl MetaObject { /// `xlMetaV2Object.Signature`. See [`FileMetaVersion::get_signature`] for /// why divergence detection requires covering every body field. pub fn get_signature(&self) -> [u8; 4] { - let mut c = self.clone(); - - // Zero fields that legitimately vary per disk within an erasure set. - c.erasure_index = 0; - - // Treat an all-empty PartETags vector the same as an absent one, so two - // disks that encode `[]` vs `["", ""]` do not falsely diverge. - if c.part_etags.iter().all(String::is_empty) { - c.part_etags.clear(); - } - // Fold maps in with an order-independent hash: msgpack map order is not // stable across disks, so they must not be part of the marshaled body. - let mut crc = hash_deterministic_string(&c.meta_user); - crc ^= hash_deterministic_bytes(&c.meta_sys); - c.meta_sys.clear(); - c.meta_user.clear(); + let mut crc = hash_deterministic_string(&self.meta_user); + crc ^= hash_deterministic_bytes(&self.meta_sys); - if let Ok(bytes) = c.marshal_msg() { - crc ^= xxhash_rust::xxh64::xxh64(&bytes, XXHASH_SEED); + // Zero fields that legitimately vary per disk within an erasure set, and + // treat an all-empty PartETags vector the same as an absent one so two + // disks that encode `[]` vs `["", ""]` do not falsely diverge. + let mut hasher = Xxh64Writer(xxhash_rust::xxh64::Xxh64::new(XXHASH_SEED)); + let include_part_etags = !self.part_etags.iter().all(String::is_empty); + if self.encode_to_with_options(&mut hasher, 0, include_part_etags, false).is_ok() { + crc ^= hasher.0.finish(); } fold_signature(crc) } - pub fn encode_to(&self, wr: &mut W) -> Result<()> { + fn encode_to_with_options( + &self, + wr: &mut W, + erasure_index: usize, + include_part_etags: bool, + include_maps: bool, + ) -> Result<()> { // Variable map size: omit PartIdx when empty let mut map_len = 18u32; if self.part_indices.is_empty() { @@ -2117,7 +2122,7 @@ impl MetaObject { // EcIndex rmp::encode::write_str(wr, "EcIndex")?; - rmp::encode::write_sint(wr, self.erasure_index as i64)?; + rmp::encode::write_sint(wr, erasure_index as i64)?; // EcDist rmp::encode::write_str(wr, "EcDist")?; @@ -2139,7 +2144,7 @@ impl MetaObject { // PartETags (write nil when empty) rmp::encode::write_str(wr, "PartETags")?; - if self.part_etags.is_empty() { + if !include_part_etags || self.part_etags.is_empty() { rmp::encode::write_nil(wr)?; } else { rmp::encode::write_array_len(wr, self.part_etags.len() as u32)?; @@ -2186,7 +2191,7 @@ impl MetaObject { // MetaSys (write nil when empty) rmp::encode::write_str(wr, "MetaSys")?; - if self.meta_sys.is_empty() { + if !include_maps || self.meta_sys.is_empty() { rmp::encode::write_nil(wr)?; } else { rmp::encode::write_map_len(wr, self.meta_sys.len() as u32)?; @@ -2198,7 +2203,7 @@ impl MetaObject { // MetaUsr (write nil when empty) rmp::encode::write_str(wr, "MetaUsr")?; - if self.meta_user.is_empty() { + if !include_maps || self.meta_user.is_empty() { rmp::encode::write_nil(wr)?; } else { rmp::encode::write_map_len(wr, self.meta_user.len() as u32)?; @@ -2211,44 +2216,28 @@ impl MetaObject { Ok(()) } + pub fn encode_to(&self, wr: &mut W) -> Result<()> { + self.encode_to_with_options(wr, self.erasure_index, !self.part_etags.is_empty(), true) + } + pub fn decode_from(&mut self, rd: &mut R) -> Result<()> { - let mut fields = rmp::decode::read_map_len(rd).map_err(|e| { + let fields = rmp::decode::read_map_len(rd).map_err(|e| { tracing::error!(error = %e, "decode_from: read_map_len failed"); e - })?; + })? as usize; + self.decode_fields(rd, fields) + } + + fn decode_fields(&mut self, rd: &mut R, mut fields: usize) -> Result<()> { *self = MetaObject::default(); + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(rd).map_err(|e| { - tracing::error!(error = %e, "decode_from: read_str_len key failed"); - e - })?; - let key_len = usize::try_from(key_len).map_err(|e| { - tracing::error!(error = %e, "decode_from: key length conversion failed"); - Error::other(e) - })?; - - let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; - let heap_key; - let key_buf = if key_len <= MSGP_OBJECT_KEY_STACK_CAP { - rd.read_exact(&mut inline_key[..key_len]).map_err(|e| { - tracing::error!(error = %e, "decode_from: read key_buf failed"); - Error::from(e) - })?; - &inline_key[..key_len] - } else { - heap_key = read_exact_vec(rd, key_len).map_err(|e| { - tracing::error!(error = %e, "decode_from: read key_buf failed"); - e - })?; - heap_key.as_slice() - }; - - let key = std::str::from_utf8(key_buf).map_err(|e| { - tracing::error!(error = %e, "decode_from: from_utf8 key failed"); - Error::FromUtf8(e.to_string()) + let key = read_small_str(rd, &mut inline_key, &mut heap_key).inspect_err(|e| { + tracing::error!(error = %e, "decode_from: read key failed"); })?; match key { @@ -2506,7 +2495,7 @@ impl MetaObject { Some(n) => n, }; self.meta_user.clear(); - self.meta_user.reserve(prealloc_hint(len)); + self.meta_user.reserve(prealloc_hint(len.saturating_add(self.meta_sys.len()))); for _ in 0..len { let k_len = rmp::decode::read_str_len(rd).map_err(|e| { tracing::error!(error = %e, "decode_from: read_str_len MetaUsr key failed"); @@ -2561,9 +2550,40 @@ impl MetaObject { path: &str, all_parts: bool, include_part_checksums: bool, + ) -> Result { + Self { + version_id: self.version_id, + data_dir: self.data_dir, + erasure_algorithm: self.erasure_algorithm.clone(), + erasure_m: self.erasure_m, + erasure_n: self.erasure_n, + erasure_block_size: self.erasure_block_size, + erasure_index: self.erasure_index, + erasure_dist: self.erasure_dist.clone(), + bitrot_checksum_algo: self.bitrot_checksum_algo.clone(), + part_numbers: self.part_numbers.clone(), + part_etags: if all_parts { self.part_etags.clone() } else { Vec::new() }, + part_sizes: self.part_sizes.clone(), + part_actual_sizes: self.part_actual_sizes.clone(), + part_indices: if all_parts { self.part_indices.clone() } else { Vec::new() }, + size: self.size, + mod_time: self.mod_time, + meta_sys: self.meta_sys.clone(), + meta_user: self.meta_user.clone(), + } + .into_fileinfo_with_part_checksums(volume, path, all_parts, include_part_checksums) + } + + fn into_fileinfo_with_part_checksums( + mut self, + volume: &str, + path: &str, + all_parts: bool, + include_part_checksums: bool, ) -> Result { let version_id = self.version_id.filter(|&vid| !vid.is_nil()); - let derived_metadata = DerivedInternalMetadata::from_meta_sys(&self.meta_sys)?; + let meta_sys = std::mem::take(&mut self.meta_sys); + let derived_metadata = DerivedInternalMetadata::from_meta_sys(&meta_sys)?; let parts = if all_parts { let n = self.part_numbers.len(); @@ -2578,74 +2598,39 @@ impl MetaObject { if n != 0 && (self.part_sizes.len() != n || self.part_actual_sizes.len() != n) { return Err(Error::FileCorrupt); } - let mut parts = vec![ObjectPartInfo::default(); n]; - - for (i, part) in parts.iter_mut().enumerate() { - part.number = self.part_numbers[i]; - part.size = self.part_sizes[i]; - part.actual_size = self.part_actual_sizes[i]; + let mut etags = (self.part_etags.len() == n).then(|| self.part_etags.into_iter()); + let mut indices = (self.part_indices.len() == n).then(|| self.part_indices.into_iter()); + let mut parts = Vec::with_capacity(n); + for i in 0..n { // etag/index stay soft-guarded: they are recomputable / optional, an empty // PartETags is legitimate MinIO interop, and a length mismatch there does // not corrupt returned data. - if self.part_etags.len() == n { - part.etag = self.part_etags[i].clone(); - } - - if self.part_indices.len() == n { - part.index = if self.part_indices[i].is_empty() { - None - } else { - Some(self.part_indices[i].clone()) - }; - } + parts.push(ObjectPartInfo { + number: self.part_numbers[i], + size: self.part_sizes[i], + actual_size: self.part_actual_sizes[i], + etag: etags.as_mut().and_then(Iterator::next).unwrap_or_default(), + index: indices.as_mut().and_then(Iterator::next).filter(|idx| !idx.is_empty()), + ..Default::default() + }); } parts } else { Vec::new() }; - let mut metadata = HashMap::with_capacity(self.meta_user.len() + self.meta_sys.len()); - for (k, v) in &self.meta_user { - if k == AMZ_META_UNENCRYPTED_CONTENT_LENGTH || k == AMZ_META_UNENCRYPTED_CONTENT_MD5 { - continue; - } - - if k == metadata_keys::STORAGE_CLASS && v == "STANDARD" { - continue; - } - - metadata.insert(k.to_owned(), v.to_owned()); - } - - for (k, v) in &self.meta_sys { - if has_internal_suffix(k, SUFFIX_TIER_FV_ID) || has_internal_suffix(k, SUFFIX_TIER_FV_MARKER) { - continue; - } - - if k.eq_ignore_ascii_case(metadata_keys::STORAGE_CLASS) && v == b"STANDARD" { - continue; - } - - if is_internal_key(k) { - metadata.insert(k.to_owned(), String::from_utf8(v.to_owned()).unwrap_or_default()); - } - } - - let replication_state_internal = get_internal_replication_state(&metadata); - - let mut deleted = false; - - if let Some(v) = replication_state_internal.as_ref() { - if !v.composite_version_purge_status().is_empty() { - deleted = true; - } - - let st = v.composite_replication_status(); - if !st.is_empty() { - metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), st.to_string()); - } + let mut metadata = self.meta_user; + metadata.remove(AMZ_META_UNENCRYPTED_CONTENT_LENGTH); + metadata.remove(AMZ_META_UNENCRYPTED_CONTENT_MD5); + if metadata.get(metadata_keys::STORAGE_CLASS).is_some_and(|v| v == "STANDARD") { + metadata.remove(metadata_keys::STORAGE_CLASS); } + let user_has_internal = metadata.keys().any(|k| is_internal_key(k)); + let mut has_replication_candidate = user_has_internal; + let mut has_part_checksums = user_has_internal || derived_metadata.part_checksums.is_some(); + let mut has_upload_integrity = user_has_internal; + let mut has_shard_or_inline_integrity = user_has_internal; let checksum = derived_metadata .checksum @@ -2658,7 +2643,7 @@ impl MetaObject { parity_blocks: self.erasure_n, block_size: self.erasure_block_size, index: self.erasure_index, - distribution: self.erasure_dist.iter().map(|&v| v as usize).collect(), + distribution: self.erasure_dist.into_iter().map(usize::from).collect(), ..Default::default() }; @@ -2677,11 +2662,6 @@ impl MetaObject { if derived_metadata.transitioned_version_state.is_some() { validate_transition_version_state(transition_version_state, transition_version.as_deref())?; } - for (key, value) in &self.meta_sys { - if is_transition_version_metadata_key(key) { - metadata.insert(key.to_owned(), transition_version_metadata_value(value, transition_version.as_deref())); - } - } let transition_version_id = transition_version.as_deref().and_then(|value| Uuid::parse_str(value).ok()); let transition_tier = derived_metadata .transition_tier @@ -2689,6 +2669,68 @@ impl MetaObject { .map(|v| String::from_utf8_lossy(v).to_string()) .unwrap_or_default(); + metadata.reserve(meta_sys.len()); + for (k, v) in meta_sys { + if has_internal_suffix(&k, SUFFIX_TIER_FV_ID) || has_internal_suffix(&k, SUFFIX_TIER_FV_MARKER) { + continue; + } + let Some(suffix) = strip_internal_prefix_preserving_case(&k) else { + continue; + }; + if suffix.eq_ignore_ascii_case(SUFFIX_TRANSITIONED_VERSION_ID) { + let val = transition_version_metadata_value(&v, transition_version.as_deref()); + metadata.insert(k, val); + continue; + } + if !k.is_ascii() { + has_replication_candidate = true; + has_part_checksums = true; + has_upload_integrity = true; + has_shard_or_inline_integrity = true; + } else { + match suffix.as_bytes().first().map(u8::to_ascii_lowercase) { + Some(b'p') => { + if suffix.eq_ignore_ascii_case(SUFFIX_PURGESTATUS) { + has_replication_candidate = true; + } else if suffix.eq_ignore_ascii_case(SUFFIX_PART_CHECKSUMS) { + has_part_checksums = true; + } + } + Some(b'r') => has_replication_candidate = true, + Some(b's') => { + if suffix.eq_ignore_ascii_case(crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY) { + has_upload_integrity = true; + } else if suffix.eq_ignore_ascii_case(crate::shard_integrity::SUFFIX_SHARD_INTEGRITY) + || suffix.eq_ignore_ascii_case(crate::shard_integrity::SUFFIX_INLINE_INTEGRITY) + { + has_shard_or_inline_integrity = true; + } + } + _ => {} + } + } + metadata.insert(k, String::from_utf8(v).unwrap_or_default()); + } + + let replication_state_internal = if has_replication_candidate { + get_internal_replication_state(&metadata) + } else { + None + }; + + let mut deleted = false; + + if let Some(v) = replication_state_internal.as_ref() { + if !v.composite_version_purge_status().is_empty() { + deleted = true; + } + + let st = v.composite_replication_status(); + if !st.is_empty() { + metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), st.to_string()); + } + } + let mut file_info = FileInfo { version_id, erasure, @@ -2710,15 +2752,18 @@ impl MetaObject { transition_tier, ..Default::default() }; - if all_parts && include_part_checksums { + if all_parts && include_part_checksums && has_part_checksums { file_info.hydrate_data_movement_part_checksums()?; } if !self.part_numbers.is_empty() + && has_upload_integrity && rustfs_utils::http::contains_key_str(&file_info.metadata, crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY) { return Err(Error::FileCorrupt); } - if let Some(commitments) = crate::shard_integrity::descriptor_from_metadata(&file_info.metadata)? { + if has_shard_or_inline_integrity + && let Some(commitments) = crate::shard_integrity::descriptor_from_metadata(&file_info.metadata)? + { let layout = crate::shard_integrity::IntegrityLayout::new( self.erasure_m, self.erasure_n, @@ -2938,6 +2983,7 @@ impl From for MetaObject { pub fn get_internal_replication_state(metadata: &HashMap) -> Option { let mut rs = ReplicationState::default(); let mut has = false; + let mut has_delete_marker_version_keys = false; for (k, v) in metadata.iter() { if has_internal_suffix(k, SUFFIX_PURGESTATUS) { @@ -2961,6 +3007,11 @@ pub fn get_internal_replication_state(metadata: &HashMap) -> Opt has = true; rs.replication_status_internal = Some(v.clone()); rs.targets = replication_statuses_map(v.as_str()); + } else if sub_key + .get(..SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX.len()) + .is_some_and(|prefix| prefix.eq_ignore_ascii_case(SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)) + { + has_delete_marker_version_keys = true; } else if let Some(arn) = rustfs_utils::http::internal_key_strip_suffix_prefix(k, SUFFIX_REPLICATION_RESET_ARN_PREFIX) { has = true; @@ -2981,9 +3032,12 @@ pub fn get_internal_replication_state(metadata: &HashMap) -> Opt // authority. `corrupt` means the dual internal prefixes disagreed: surface it // rather than guessing, so callers fail closed instead of purging the wrong // target version. - (rs.target_delete_marker_version_ids, rs.target_delete_marker_version_ids_corrupt) = target_delete_marker_versions(metadata); - has |= !rs.target_delete_marker_version_ids.is_empty(); - has |= rs.target_delete_marker_version_ids_corrupt; + if has_delete_marker_version_keys { + (rs.target_delete_marker_version_ids, rs.target_delete_marker_version_ids_corrupt) = + target_delete_marker_versions(metadata); + has |= !rs.target_delete_marker_version_ids.is_empty(); + has |= rs.target_delete_marker_version_ids_corrupt; + } if has { Some(rs) } else { None } } @@ -3101,6 +3155,10 @@ impl MetaDeleteMarker { } pub fn encode_to(&self, wr: &mut W) -> Result<()> { + self.encode_to_with_meta(wr, true) + } + + fn encode_to_with_meta(&self, wr: &mut W, include_meta_sys: bool) -> Result<()> { rmp::encode::write_map_len(wr, 3)?; // ID @@ -3114,27 +3172,35 @@ impl MetaDeleteMarker { // MetaSys rmp::encode::write_str(wr, "MetaSys")?; - rmp::encode::write_map_len(wr, self.meta_sys.len() as u32)?; - for (k, v) in &self.meta_sys { - rmp::encode::write_str(wr, k)?; - rmp::encode::write_bin(wr, v)?; + if !include_meta_sys || self.meta_sys.is_empty() { + rmp::encode::write_map_len(wr, 0)?; + } else { + rmp::encode::write_map_len(wr, self.meta_sys.len() as u32)?; + for (k, v) in &self.meta_sys { + rmp::encode::write_str(wr, k)?; + rmp::encode::write_bin(wr, v)?; + } } Ok(()) } pub fn decode_from(&mut self, rd: &mut R) -> Result<()> { - let mut fields = rmp::decode::read_map_len(rd)?; + let fields = rmp::decode::read_map_len(rd)? as usize; + self.decode_fields(rd, fields) + } + + fn decode_fields(&mut self, rd: &mut R, mut fields: usize) -> Result<()> { *self = MetaDeleteMarker::default(); + let mut inline_key = [0u8; MSGP_OBJECT_KEY_STACK_CAP]; + let mut heap_key = Vec::new(); while fields > 0 { fields -= 1; - let key_len = rmp::decode::read_str_len(rd)?; - let key_buf = read_exact_vec(rd, key_len as usize)?; - let key = String::from_utf8(key_buf)?; + let key = read_small_str(rd, &mut inline_key, &mut heap_key)?; - match key.as_str() { + match key { "ID" => { let _ = rmp::decode::read_bin_len(rd)?; let mut buf = [0u8; 16]; @@ -3193,14 +3259,11 @@ impl MetaDeleteMarker { /// Compute this delete marker's header signature, mirroring MinIO's /// `xlMetaV2DeleteMarker.Signature`. See [`FileMetaVersion::get_signature`]. pub fn get_signature(&self) -> [u8; 4] { - let mut c = self.clone(); - // MetaSys is order-unstable across disks; fold it in separately. - let mut crc = hash_deterministic_bytes(&c.meta_sys); - c.meta_sys.clear(); - - if let Ok(bytes) = c.marshal_msg() { - crc ^= xxhash_rust::xxh64::xxh64(&bytes, XXHASH_SEED); + let mut crc = hash_deterministic_bytes(&self.meta_sys); + let mut hasher = Xxh64Writer(xxhash_rust::xxh64::Xxh64::new(XXHASH_SEED)); + if self.encode_to_with_meta(&mut hasher, false).is_ok() { + crc ^= hasher.0.finish(); } fold_signature(crc) @@ -3578,33 +3641,18 @@ pub struct FileInfoOpts { } pub fn get_file_info(buf: &[u8], volume: &str, path: &str, version_id: &str, opts: FileInfoOpts) -> Result { - let vid = { - if version_id.is_empty() { - None - } else { - Some(Uuid::parse_str(version_id)?) - } - }; - - let meta = FileMeta::load(buf)?; - if meta.versions.is_empty() { - return Ok(FileInfo { - volume: volume.to_owned(), - name: path.to_owned(), - version_id: vid, - is_latest: true, - deleted: true, - mod_time: Some(OffsetDateTime::from_unix_timestamp(1)?), - ..Default::default() - }); - } - - let fi = if opts.include_part_checksums { - meta.into_fileinfo(volume, path, version_id, opts.data, opts.include_free_versions, true)? - } else { - meta.into_fileinfo_without_part_checksums(volume, path, version_id, opts.data, opts.include_free_versions)? - }; - Ok(fi) + FileMeta::decode_fileinfo_from_buf( + buf, + volume, + path, + version_id, + FileInfoDecodeOptions { + read_data: opts.data, + include_free_versions: opts.include_free_versions, + all_parts: true, + include_part_checksums: opts.include_part_checksums, + }, + ) } async fn read_more( @@ -5465,6 +5513,135 @@ mod tests { ); } + #[test] + fn into_fileinfo_preserves_upload_integrity_and_non_ascii_checksum_guards() { + for key in [ + format!("{RUSTFS_INTERNAL_PREFIX}{}", crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY), + format!( + "{}{}", + rustfs_utils::http::MINIO_INTERNAL_PREFIX, + crate::shard_integrity::SUFFIX_UPLOAD_INTEGRITY.to_ascii_uppercase() + ), + ] { + for all_parts in [false, true] { + let mut obj = MetaObject { + part_numbers: vec![1], + part_sizes: vec![1024], + part_actual_sizes: vec![1024], + ..Default::default() + }; + obj.meta_sys.insert(key.clone(), b"1".to_vec()); + assert!( + matches!(obj.into_fileinfo("bucket", "object", all_parts), Err(Error::FileCorrupt)), + "upload-integrity marker {key} with parts must fail closed (all_parts={all_parts})" + ); + } + } + + for prefix in [RUSTFS_INTERNAL_PREFIX, rustfs_utils::http::MINIO_INTERNAL_PREFIX] { + let mut obj = MetaObject { + part_numbers: vec![1], + part_sizes: vec![1024], + part_actual_sizes: vec![1024], + ..Default::default() + }; + let non_ascii_suffix = SUFFIX_PART_CHECKSUMS.replacen('k', "\u{212A}", 1); + obj.meta_sys + .insert(format!("{prefix}{non_ascii_suffix}"), br#"[[1,[["CRC32","AAAAAA=="]]]]"#.to_vec()); + assert!( + matches!(obj.into_fileinfo("bucket", "object", true), Err(Error::FileCorrupt)), + "non-ASCII part-checksums alias under {prefix} must fail closed" + ); + } + } + + #[test] + fn get_file_info_zero_copy_matches_loaded_filemeta_and_rejects_trailing_corruption() { + for raw in [ + crate::test_data::create_real_xlmeta().expect("real xl.meta"), + crate::test_data::create_complex_xlmeta().expect("complex xl.meta"), + crate::test_data::create_realistic_object_xlmeta().expect("realistic xl.meta"), + ] { + let loaded = FileMeta::load(&raw).expect("load fixture"); + for include_part_checksums in [false, true] { + let direct = get_file_info( + &raw, + "bucket", + "object", + "", + FileInfoOpts { + data: true, + include_free_versions: false, + include_part_checksums, + }, + ) + .expect("direct get_file_info"); + let via_loaded = if include_part_checksums { + loaded + .into_fileinfo("bucket", "object", "", true, false, true) + .expect("loaded into_fileinfo") + } else { + loaded + .into_fileinfo_without_part_checksums("bucket", "object", "", true, false) + .expect("loaded into_fileinfo_without_part_checksums") + }; + assert_eq!(direct, via_loaded); + } + } + + let mut fm = FileMeta::load(&crate::test_data::create_complex_xlmeta().expect("complex xl.meta")).expect("load"); + // Make version[0]'s body decode to FileCorrupt (mismatched part_sizes) while + // truncating the trailing version entry: the trailing framing error must still + // take precedence over version[0]'s body error, matching FileMeta::load. + let mut v0 = fm.versions[0].parse_version_meta().expect("parse v0"); + v0.object.as_mut().expect("v0 object").part_sizes.clear(); + fm.versions[0].meta = v0.marshal_msg().expect("marshal v0"); + fm.versions[1].header = FileMetaVersionHeader::default(); + let mut bad_buf = fm.marshal_msg().expect("marshal"); + let (_, remaining) = FileMeta::read_bytes_header(&bad_buf[8..]).expect("read bytes header"); + let meta_len = remaining.len() - 5; + let mut truncated_meta = remaining[..meta_len - 10].to_vec(); + let new_crc = xxhash_rust::xxh64::xxh64(&truncated_meta, XXHASH_SEED) as u32; + bad_buf.truncate(8); + bad_buf.push(0xc6); + bad_buf.extend_from_slice(&(truncated_meta.len() as u32).to_be_bytes()); + bad_buf.append(&mut truncated_meta); + bad_buf.push(0xce); + bad_buf.extend_from_slice(&new_crc.to_be_bytes()); + + let direct_err = get_file_info( + &bad_buf, + "bucket", + "object", + "", + FileInfoOpts { + data: false, + include_free_versions: false, + include_part_checksums: false, + }, + ) + .expect_err("truncated trailing version entry must fail even when version[0] matches"); + let load_err = FileMeta::load(&bad_buf).expect_err("FileMeta::load must fail on truncated trailing version"); + assert_eq!(direct_err, load_err, "get_file_info must match FileMeta::load error precedence"); + + // Submap non-nil error variants (`{"V2Obj": 0}` and truncated map16 `{"V2Obj": 0xde}`) + // must remain RmpDecodeValueRead. + for bad_ver in [&b"\x81\xa5V2Obj\x00"[..], &b"\x81\xa5V2Obj\xde"[..]] { + let mut ver = FileMetaVersion::default(); + assert!( + matches!(ver.unmarshal_msg(bad_ver), Err(Error::RmpDecodeValueRead(_))), + "invalid V2Obj marker must return RmpDecodeValueRead" + ); + } + + // InlineData validation must reject an empty key before reading a missing value. + let err = InlineData::validate_slice(&[0x01, 0x81, 0xa0]).expect_err("empty inline key must fail"); + assert!( + matches!(&err, Error::Io(io_err) if io_err.to_string().contains("InlineData key empty")), + "expected InlineData key empty error, got {err:?}" + ); + } + #[test] fn get_internal_replication_state_accepts_rfc3339_and_rustfs_display_timestamps() { let replica_timestamp = OffsetDateTime::UNIX_EPOCH + time::Duration::SECOND; diff --git a/crates/filemeta/src/filemeta_inline.rs b/crates/filemeta/src/filemeta_inline.rs index 658433e1e..26426b972 100644 --- a/crates/filemeta/src/filemeta_inline.rs +++ b/crates/filemeta/src/filemeta_inline.rs @@ -13,8 +13,9 @@ // limitations under the License. use crate::error::{Error, Result}; -use crate::filemeta::msgp_decode::{prealloc_hint, read_exact_vec}; +use crate::filemeta::msgp_decode::{MAX_MSGP_ELEMENT_SIZE, prealloc_hint, read_exact_vec}; use serde::{Deserialize, Serialize}; +use std::collections::HashSet; use std::io::Cursor; use uuid::Uuid; @@ -63,110 +64,98 @@ impl InlineData { Ok(false) } + fn borrowed_entries(&self) -> Result> { + let buf = self.after_version(); + if buf.is_empty() { + return Ok(Vec::new()); + } + let mut cur = Cursor::new(buf); + let count = usize::try_from(rmp::decode::read_map_len(&mut cur)?).map_err(Error::other)?; + let mut entries = Vec::with_capacity(prealloc_hint(count)); + for _ in 0..count { + let key_len = usize::try_from(rmp::decode::read_str_len(&mut cur)?).map_err(Error::other)?; + if key_len > MAX_MSGP_ELEMENT_SIZE { + return Err(Error::other("InlineData key exceeds the msgpack element limit")); + } + let start = usize::try_from(cur.position()).map_err(Error::other)?; + let end = start + .checked_add(key_len) + .ok_or_else(|| Error::other("InlineData key out of range"))?; + let key = buf + .get(start..end) + .ok_or_else(|| Error::other("InlineData key out of range"))?; + let key = std::str::from_utf8(key).map_err(|err| Error::FromUtf8(err.to_string()))?; + cur.set_position(u64::try_from(end).map_err(Error::other)?); + + let value_len = usize::try_from(rmp::decode::read_bin_len(&mut cur)?).map_err(Error::other)?; + let start = usize::try_from(cur.position()).map_err(Error::other)?; + let end = start + .checked_add(value_len) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + let value = buf + .get(start..end) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + cur.set_position(u64::try_from(end).map_err(Error::other)?); + entries.push((key, value)); + } + Ok(entries) + } + + pub(crate) fn key_set(&self) -> Result>> { + if !self.version_ok() { + return Err(Error::FileCorrupt); + } + Ok(self + .borrowed_entries()? + .into_iter() + .map(|(key, _)| key.as_bytes().to_vec()) + .collect()) + } + + pub(crate) fn remove_keys(&mut self, keys: &HashSet>) -> Result { + if keys.is_empty() { + return Ok(false); + } + self.remove_keys_by(|key| keys.contains(key)) + } + fn remove_keys_by(&mut self, mut should_remove: F) -> Result where F: FnMut(&[u8]) -> bool, { - let buf = self.after_version(); - if buf.is_empty() { + let mut entries = self.borrowed_entries()?; + let original_count = entries.len(); + entries.retain(|(key, _)| !should_remove(key.as_bytes())); + if entries.len() == original_count { return Ok(false); } - - let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize; - let mut keys = Vec::with_capacity(prealloc_hint(fields_len)); - let mut values = Vec::with_capacity(prealloc_hint(fields_len)); - let mut found = false; - - while fields_len > 0 { - fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)? as usize; - let field_buf = read_exact_vec(&mut cur, str_len)?; - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - let value = buf - .get(start..end) - .ok_or_else(|| Error::other("InlineData value out of range"))?; - cur.set_position(end as u64); - - if should_remove(field_buf.as_slice()) { - found = true; - continue; - } - - keys.push(String::from_utf8(field_buf)?); - values.push(value.to_vec()); - } - - if !found { - return Ok(false); - } - - if keys.is_empty() { - self.0 = Vec::new(); + if entries.is_empty() { + self.0.clear(); return Ok(true); } - self.serialize(keys, values)?; + // Borrow payloads until the final encoding so every retained body is + // copied only once, regardless of the number of removed versions. + let capacity = entries.iter().try_fold(6_usize, |size, (key, value)| { + // A string and binary value each need at most five header bytes. + size.checked_add(key.len()) + .and_then(|size| size.checked_add(value.len())) + .and_then(|size| size.checked_add(10)) + .ok_or_else(|| Error::other("InlineData encoded size overflow")) + })?; + let mut encoded = Vec::with_capacity(capacity); + encoded.push(INLINE_DATA_VER); + rmp::encode::write_map_len(&mut encoded, u32::try_from(entries.len()).map_err(Error::other)?)?; + for (key, value) in entries { + rmp::encode::write_str(&mut encoded, key)?; + rmp::encode::write_bin(&mut encoded, value)?; + } + self.0 = encoded; Ok(true) } fn remove_two_keys_by_bytes(&mut self, first_key: &[u8], second_key: &[u8]) -> Result { - let buf = self.after_version(); - if buf.is_empty() { - return Ok(false); - } - - let same = first_key == second_key; - let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize; - let mut keys = Vec::with_capacity(prealloc_hint(fields_len) + 1); - let mut values = Vec::with_capacity(prealloc_hint(fields_len) + 1); - let mut found = false; - - while fields_len > 0 { - fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)? as usize; - let field_buf = read_exact_vec(&mut cur, str_len)?; - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - let value = buf - .get(start..end) - .ok_or_else(|| Error::other("InlineData value out of range"))?; - cur.set_position(end as u64); - - let should_remove = if same { - field_buf.as_slice() == first_key - } else { - field_buf.as_slice() == first_key || field_buf.as_slice() == second_key - }; - - if should_remove { - found = true; - continue; - } - - keys.push(String::from_utf8(field_buf)?); - values.push(value.to_vec()); - } - - if !found { - return Ok(false); - } - - if keys.is_empty() { - self.0 = Vec::new(); - return Ok(true); - } - - self.serialize(keys, values)?; - Ok(true) + self.remove_keys_by(|key| key == first_key || key == second_key) } pub fn new() -> Self { @@ -179,11 +168,11 @@ impl InlineData { self.0.as_slice() } pub fn version_ok(&self) -> bool { - if self.0.is_empty() { - return true; - } + Self::version_ok_slice(&self.0) + } - self.0[0] > 0 && self.0[0] <= INLINE_DATA_VER + pub(crate) fn version_ok_slice(raw: &[u8]) -> bool { + raw.first().is_none_or(|version| *version > 0 && *version <= INLINE_DATA_VER) } pub fn after_version(&self) -> &[u8] { @@ -204,35 +193,56 @@ impl InlineData { Ok(fields_len as usize) } + #[inline] + fn read_entry_slice<'a>(buf: &'a [u8], cur: &mut Cursor<&'a [u8]>, reject_empty_key: bool) -> Result<(&'a str, &'a [u8])> { + let str_len = rmp::decode::read_str_len(cur)? as usize; + if str_len > MAX_MSGP_ELEMENT_SIZE { + return Err(Error::other(format!( + "corrupt msgpack element: length {str_len} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" + ))); + } + let key_start = cur.position() as usize; + let key_end = key_start + .checked_add(str_len) + .ok_or_else(|| Error::other("InlineData key out of range"))?; + let key_bytes = buf + .get(key_start..key_end) + .ok_or_else(|| Error::from(std::io::Error::new(std::io::ErrorKind::UnexpectedEof, "failed to fill whole buffer")))?; + let field = std::str::from_utf8(key_bytes).map_err(|e| Error::FromUtf8(e.to_string()))?; + if reject_empty_key && field.is_empty() { + return Err(Error::other("InlineData key empty")); + } + cur.set_position(key_end as u64); + + let bin_len = rmp::decode::read_bin_len(cur)? as usize; + let start = cur.position() as usize; + let end = start + .checked_add(bin_len) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + let value = buf + .get(start..end) + .ok_or_else(|| Error::other("InlineData value out of range"))?; + cur.set_position(end as u64); + Ok((field, value)) + } + pub fn find(&self, key: &str) -> Result>> { - if self.0.is_empty() || !self.version_ok() { + Self::find_in_slice(&self.0, key) + } + + pub(crate) fn find_in_slice(raw: &[u8], key: &str) -> Result>> { + if raw.is_empty() || raw[0] == 0 || raw[0] > INLINE_DATA_VER { return Ok(None); } - let buf = self.after_version(); - + let buf = &raw[1..]; let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)?; while fields_len > 0 { fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)?; - - let field_buff = read_exact_vec(&mut cur, str_len as usize)?; - - let field = String::from_utf8(field_buff)?; - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - let value = buf - .get(start..end) - .ok_or_else(|| Error::other("InlineData value out of range"))?; - cur.set_position(end as u64); - - if field.as_str() == key { + let (field, value) = Self::read_entry_slice(buf, &mut cur, false)?; + if field == key { return Ok(Some(value.to_vec())); } } @@ -241,34 +251,21 @@ impl InlineData { } pub fn validate(&self) -> Result<()> { - if self.0.is_empty() { + Self::validate_slice(&self.0) + } + + pub(crate) fn validate_slice(raw: &[u8]) -> Result<()> { + if raw.is_empty() { return Ok(()); } - let buf = self.after_version(); + let buf = &raw[1..]; let mut cur = Cursor::new(buf); - let mut fields_len = rmp::decode::read_map_len(&mut cur)?; while fields_len > 0 { fields_len -= 1; - - let str_len = rmp::decode::read_str_len(&mut cur)?; - - let field_buff = read_exact_vec(&mut cur, str_len as usize)?; - - let field = String::from_utf8(field_buff)?; - if field.is_empty() { - return Err(Error::other("InlineData key empty")); - } - - let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize; - let start = cur.position() as usize; - let end = start + bin_len; - if end > buf.len() { - return Err(Error::other("InlineData value out of range")); - } - cur.set_position(end as u64); + let _ = Self::read_entry_slice(buf, &mut cur, true)?; } Ok(()) @@ -420,4 +417,38 @@ mod tests { Some(b"keep".to_vec()) ); } + + #[test] + fn remove_keys_preserves_wire_bytes_and_rejects_truncated_payloads() { + let mut original = InlineData::new(); + for id in 0_u128..8 { + original + .replace(&Uuid::from_u128(id).to_string(), vec![u8::try_from(id).expect("small id"); 128]) + .expect("seed inline map"); + } + for mask in 0_u32..256 { + let keys: HashSet<_> = (0_u128..8) + .filter(|id| mask & (1 << id) != 0) + .map(|id| Uuid::from_u128(id).to_string().into_bytes()) + .collect(); + let mut actual = original.clone(); + actual.remove_keys(&keys).expect("batch removal"); + let retained: Vec<_> = (0_u128..8).filter(|id| mask & (1 << id) == 0).collect(); + let mut expected = InlineData::new(); + for id in retained { + expected + .replace(&Uuid::from_u128(id).to_string(), vec![u8::try_from(id).expect("small id"); 128]) + .expect("encode survivor"); + } + assert_eq!(actual, expected, "mask={mask}"); + } + let keys = HashSet::from([Uuid::nil().to_string().into_bytes()]); + for end in 2..original.as_slice().len() { + let mut truncated = InlineData::new(); + truncated.update(&original.as_slice()[..end]); + let before = truncated.clone(); + assert!(truncated.remove_keys(&keys).is_err(), "end={end}"); + assert_eq!(truncated, before); + } + } } diff --git a/crates/filemeta/src/metacache.rs b/crates/filemeta/src/metacache.rs index 61a2e4d8e..3109fd07c 100644 --- a/crates/filemeta/src/metacache.rs +++ b/crates/filemeta/src/metacache.rs @@ -12,7 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::filemeta::msgp_decode::MAX_MSGP_ELEMENT_SIZE; use crate::{ Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType, get_file_info, merge_file_meta_versions, merge_file_meta_versions_with_write_quorum, @@ -840,6 +839,7 @@ impl MetaCacheEntriesSorted { } const METACACHE_STREAM_VERSION: u8 = 2; +const METACACHE_READ_CHUNK_SIZE: usize = 64 << 10; #[derive(Debug)] pub struct MetacacheWriter { @@ -912,10 +912,10 @@ impl MetacacheWriter { pub struct MetacacheReader { rd: R, init: bool, + in_entry: bool, err: Option, buf: Vec, - offset: usize, - current: Option, + has_current: bool, } impl MetacacheReader { @@ -923,49 +923,66 @@ impl MetacacheReader { Self { rd, init: false, + in_entry: false, err: None, buf: Vec::new(), - offset: 0, - current: None, + has_current: false, } } pub async fn read_more(&mut self, read_size: usize) -> Result<&[u8]> { - // `read_size` is usually a length decoded from the stream itself, so - // it is untrusted: a corrupted length prefix must yield a decode - // error instead of a huge allocation that aborts the process - // (see rustfs/rustfs#2715). - if read_size > MAX_MSGP_ELEMENT_SIZE { - let err = Error::other(format!( - "metacache stream corrupt: element length {read_size} exceeds the {MAX_MSGP_ELEMENT_SIZE} byte limit" - )); - self.err = Some(err.clone()); - return Err(err); + if let Some(err) = &self.err { + return Err(err.clone()); } - let pref = self.offset; - let ext_size = pref + read_size; + let pref = self.buf.len(); + let Some(ext_size) = pref.checked_add(read_size) else { + let err = Error::other("metacache stream: element length overflow"); + self.err = Some(err.clone()); + return Err(err); + }; - if self.buf.len() < ext_size { - let extra = ext_size - self.buf.len(); + // A whole xl.meta can legitimately exceed the limit on its individual + // msgpack values. Grow only as payload arrives, so a corrupt length + // prefix cannot cause a speculative allocation of the advertised size. + while self.buf.len() < ext_size { + let start = self.buf.len(); + let extra = (ext_size - start).min(METACACHE_READ_CHUNK_SIZE); if let Err(e) = self.buf.try_reserve(extra) { let err = Error::other(format!("metacache stream: buffer allocation of {extra} bytes failed: {e}")); self.err = Some(err.clone()); return Err(err); } - self.buf.resize(ext_size, 0); + self.buf.resize(start + extra, 0); + if let Err(e) = self.rd.read_exact(&mut self.buf[start..]).await { + // Listings treat Unexpected/UnexpectedEof as clean exhaustion. + // An incomplete entry must not hide remaining keys. Preserve + // the existing empty/between-entry EOF handling. + let err = match e.kind() { + std::io::ErrorKind::UnexpectedEof if self.in_entry => Error::FileCorrupt, + _ => Error::from(e), + }; + self.err = Some(err.clone()); + return Err(err); + } } - self.rd.read_exact(&mut self.buf[pref..ext_size]).await?; - - self.offset += read_size; - Ok(&self.buf[pref..ext_size]) } + async fn discard(&mut self, mut size: usize) -> Result<()> { + self.reset(); + while size > 0 { + let chunk = size.min(METACACHE_READ_CHUNK_SIZE); + self.read_more(chunk).await?; + size -= chunk; + self.reset(); + } + Ok(()) + } + fn reset(&mut self) { self.buf.clear(); - self.offset = 0; } async fn check_init(&mut self) -> Result<()> { @@ -1048,11 +1065,15 @@ impl MetacacheReader { return Err(err.clone()); } + if size == 0 { + return Ok(()); + } + let mut n = size; - if self.current.is_some() { + if self.has_current { n -= 1; - self.current = None; + self.has_current = false; } while n > 0 { @@ -1069,10 +1090,12 @@ impl MetacacheReader { } }; + self.in_entry = true; let l = self.read_str_len().await?; - let _ = self.read_more(l as usize).await?; + self.discard(usize::try_from(l).map_err(Error::other)?).await?; let l = self.read_bin_len().await?; - let _ = self.read_more(l as usize).await?; + self.discard(usize::try_from(l).map_err(Error::other)?).await?; + self.in_entry = false; n -= 1; } @@ -1100,11 +1123,11 @@ impl MetacacheReader { } }; + self.in_entry = true; let l = self.read_str_len().await?; let buf = self.read_more(l as usize).await?; - let name_buf = buf.to_vec(); - let name = match from_utf8(&name_buf) { + let name = match from_utf8(buf) { Ok(decoded) => decoded.to_owned(), Err(err) => { self.err = Some(Error::other(err.to_string())); @@ -1118,17 +1141,16 @@ impl MetacacheReader { let metadata = buf.to_vec(); + self.in_entry = false; self.reset(); - let entry = Some(MetaCacheEntry { + self.has_current = true; + Ok(Some(MetaCacheEntry { name, metadata, cached: None, reusable: false, - }); - self.current = entry.clone(); - - Ok(entry) + })) } pub async fn read_all(&mut self) -> Result> { @@ -1294,6 +1316,41 @@ mod tests { assert_eq!(objs, nobjs); } + #[tokio::test] + async fn test_reader_consecutive_peeks_and_mixed_skips_preserve_entries() { + let entries: Vec<_> = [vec![], vec![1, 2, 3], vec![], vec![4, 5], vec![]] + .into_iter() + .enumerate() + .map(|(i, metadata)| MetaCacheEntry { + name: format!("object-{i}"), + metadata, + ..Default::default() + }) + .collect(); + let mut output = Cursor::new(Vec::new()); + let mut writer = MetacacheWriter::new(&mut output); + writer.write(&entries).await.expect("mixed metadata entries must encode"); + writer.close().await.expect("stream must close"); + let mut reader = MetacacheReader::new(Cursor::new(output.into_inner())); + assert_eq!(reader.peek().await.expect("first entry must decode"), Some(entries[0].clone())); + assert_eq!(reader.peek().await.expect("peek must advance without skip"), Some(entries[1].clone())); + reader.skip(0).await.expect("zero skip must preserve the current entry"); + reader + .skip(2) + .await + .expect("skip must consume the current and following entry"); + assert_eq!(reader.read_all().await.expect("remaining entries must decode"), entries[3..]); + } + + #[tokio::test] + async fn test_reader_invalid_utf8_name_remains_a_sticky_error() { + let stream = vec![0xcc, 2, 0xc3, 0xa1, 0xff, 0xc4, 0, 0xc2]; + let mut reader = MetacacheReader::new(Cursor::new(stream)); + let err = reader.peek().await.expect_err("invalid UTF-8 name must be rejected"); + assert!(err.to_string().contains("utf-8")); + assert_eq!(reader.peek().await.expect_err("invalid name must leave a sticky error"), err); + } + #[tokio::test] async fn empty_writer_emits_a_valid_stream() { let mut output = Cursor::new(Vec::new()); @@ -1304,6 +1361,45 @@ mod tests { assert!(reader.read_all().await.expect("empty stream should decode").is_empty()); } + #[tokio::test] + async fn test_reader_preserves_empty_and_between_entry_eof() { + let mut empty = MetacacheReader::new(Cursor::new(Vec::::new())); + assert_eq!(empty.peek().await.expect_err("legacy empty producer must remain EOF"), Error::Unexpected); + + // A complete entry without a final false marker is a supported legacy EOF. + let stream = vec![0xcc, 2, 0xc3, 0xa1, b'x', 0xc4, 1, 0x42]; + let mut reader = MetacacheReader::new(Cursor::new(stream.clone())); + assert!(reader.peek().await.expect("complete entry must decode").is_some()); + reader.skip(1).await.expect("current entry must advance"); + assert_eq!(reader.peek().await.expect_err("between-entry EOF must remain EOF"), Error::Unexpected); + + let mut reader = MetacacheReader::new(Cursor::new(stream)); + reader.skip(1).await.expect("complete entry must be skippable"); + assert_eq!(reader.peek().await.expect_err("skip must restore between-entry state"), Error::Unexpected); + } + + #[tokio::test] + async fn test_reader_truncated_entry_is_corruption_at_each_field() { + // Missing name marker, name bytes, bin marker, bin length, or bin payload. + for suffix in [vec![], vec![0xa1], vec![0xa0], vec![0xa0, 0xc4], vec![0xa0, 0xc4, 1]] { + let mut stream = vec![0xcc, 2, 0xc3]; + stream.extend(suffix); + let mut reader = MetacacheReader::new(Cursor::new(stream.clone())); + assert_eq!( + reader.peek().await.expect_err("incomplete entry must not become listing EOF"), + Error::FileCorrupt + ); + let mut reader = MetacacheReader::new(Cursor::new(stream)); + assert_eq!( + reader + .skip(1) + .await + .expect_err("incomplete skipped entry must not become listing EOF"), + Error::FileCorrupt + ); + } + } + fn corrupt_stream_with_metadata_len(len_marker: u8, len: u32) -> Vec { let mut data = Vec::new(); rmp::encode::write_u8(&mut data, METACACHE_STREAM_VERSION).unwrap(); @@ -1324,10 +1420,14 @@ mod tests { let data = corrupt_stream_with_metadata_len(0xc6, u32::MAX); let mut r = MetacacheReader::new(Cursor::new(data)); let err = r.peek().await.expect_err("corrupt bin length prefix must fail to decode"); - assert!(err.to_string().contains("exceeds"), "error should mention the exceeded limit, got: {err}"); + assert_eq!(err, Error::FileCorrupt); + assert!( + r.buf.capacity() <= 2 * METACACHE_READ_CHUNK_SIZE, + "a length prefix must not reserve its payload" + ); // The reader must stay in the error state instead of retrying. - assert!(r.peek().await.is_err()); + assert_eq!(r.peek().await.expect_err("decode errors must remain sticky"), err); } #[tokio::test] @@ -1341,35 +1441,150 @@ mod tests { let mut r = MetacacheReader::new(Cursor::new(data)); let err = r.peek().await.expect_err("corrupt str length prefix must fail to decode"); - assert!(err.to_string().contains("exceeds"), "error should mention the exceeded limit, got: {err}"); + assert_eq!(err, Error::FileCorrupt); + assert!( + r.buf.capacity() <= 2 * METACACHE_READ_CHUNK_SIZE, + "a length prefix must not reserve its payload" + ); + assert_eq!(r.peek().await.expect_err("decode errors must remain sticky"), err); } #[tokio::test] async fn test_reader_skip_rejects_corrupt_length_prefix() { let data = corrupt_stream_with_metadata_len(0xc6, u32::MAX); let mut r = MetacacheReader::new(Cursor::new(data)); - assert!(r.skip(1).await.is_err(), "skip over a corrupt length prefix must fail"); + let err = r.skip(1).await.expect_err("skip over a corrupt length prefix must fail"); + assert_eq!(err, Error::FileCorrupt); + assert!(r.buf.capacity() <= METACACHE_READ_CHUNK_SIZE); + assert_eq!(r.skip(1).await.expect_err("skip errors must remain sticky"), err); + assert_eq!(r.peek().await.expect_err("skip failure must prevent decoding another entry"), err); } - /// Large-but-legitimate records (well above typical sizes, below the - /// corruption guard) must still round-trip. + #[tokio::test] + async fn test_reader_truncated_payload_after_multiple_chunks() { + let mut data = corrupt_stream_with_metadata_len(0xc6, u32::MAX); + data.extend(std::iter::repeat_n(0xab, 2 * METACACHE_READ_CHUNK_SIZE + 7)); + let mut reader = MetacacheReader::new(Cursor::new(data)); + let err = reader.peek().await.expect_err("partial payload must fail"); + assert_eq!(err, Error::FileCorrupt); + assert!(reader.buf.capacity() <= 8 * METACACHE_READ_CHUNK_SIZE); + assert_eq!(reader.peek().await.expect_err("partial payload must leave a sticky error"), err); + } + + /// The stream carries a whole xl.meta, whose aggregate size can exceed + /// the limit for an individual value inside its metadata. #[tokio::test] async fn test_reader_accepts_large_legitimate_metadata() { + for size in [16 << 20, (16 << 20) + 1] { + let entry = MetaCacheEntry { + name: "big-object".to_string(), + metadata: vec![0xab; size], + ..Default::default() + }; + + let mut f = Cursor::new(Vec::new()); + let mut w = MetacacheWriter::new(&mut f); + w.write(std::slice::from_ref(&entry)).await.expect("large entry must encode"); + w.close().await.expect("large-entry stream must close"); + + let mut r = MetacacheReader::new(Cursor::new(f.into_inner())); + let decoded = r.read_all().await.expect("large entry must decode"); + assert_eq!(decoded, vec![entry]); + } + } + + #[tokio::test] + async fn test_reader_large_inline_version_history_remains_listable() { + const VERSIONS: u32 = 1700; + let payload = vec![0xab; 10 << 10]; + let mut meta = FileMeta::new(); + let mut inline = vec![1]; + rmp::encode::write_map_len(&mut inline, VERSIONS).expect("inline version map must encode"); + for i in 1..=VERSIONS { + let version_id = Uuid::from_u128(u128::from(i)); + let mut fi = FileInfo { + version_id: Some(version_id), + mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(i64::from(i))), + size: i64::try_from(payload.len()).expect("test payload size must fit"), + ..Default::default() + }; + fi.set_inline_data(); + meta.add_version(fi).expect("inline version metadata must be valid"); + rmp::encode::write_str(&mut inline, &version_id.to_string()).expect("inline version ID must encode"); + rmp::encode::write_bin(&mut inline, &payload).expect("inline payload must encode"); + } + // Encode the inline map once; adding payload with each version would + // repeatedly copy the growing map while constructing this fixture. + meta.data.update(&inline); + let metadata = meta.marshal_msg().expect("large xl.meta must encode"); + assert!(metadata.len() > 16 << 20); let entry = MetaCacheEntry { - name: "big-object".to_string(), - metadata: vec![0xab; 4 << 20], - cached: None, - reusable: false, + name: "versioned-object".to_string(), + metadata, + ..Default::default() }; + let mut output = Cursor::new(Vec::new()); + let mut writer = MetacacheWriter::new(&mut output); + writer + .write(std::slice::from_ref(&entry)) + .await + .expect("large xl.meta must enter the stream"); + writer.close().await.expect("stream must close"); + let mut reader = MetacacheReader::new(Cursor::new(output.into_inner())); + let decoded = reader + .peek() + .await + .expect("large xl.meta must decode") + .expect("object must be present"); + assert_eq!(decoded.metadata, entry.metadata); + let versions = decoded + .file_info_versions("bucket") + .expect("version listing must decode xl.meta"); + assert_eq!(versions.versions.len(), usize::try_from(VERSIONS).expect("version count must fit")); + let latest_id = Uuid::from_u128(u128::from(VERSIONS)); + assert_eq!(versions.versions[0].version_id, Some(latest_id)); + assert_eq!(versions.versions[0].size, i64::try_from(payload.len()).expect("payload size must fit")); + let loaded = FileMeta::load(&decoded.metadata).expect("stream must preserve valid xl.meta"); + assert_eq!( + loaded.data.find(&latest_id.to_string()).expect("inline payload must decode"), + Some(payload) + ); + assert!(reader.peek().await.expect("stream must terminate").is_none()); + } - let mut f = Cursor::new(Vec::new()); - let mut w = MetacacheWriter::new(&mut f); - w.write(std::slice::from_ref(&entry)).await.unwrap(); - w.close().await.unwrap(); - - let mut r = MetacacheReader::new(Cursor::new(f.into_inner())); - let decoded = r.read_all().await.unwrap(); - assert_eq!(decoded, vec![entry]); + #[tokio::test] + async fn test_reader_skip_large_entries_preserves_alignment_with_bounded_buffer() { + // Fixed v1/v2 framing: uint8 version, bool, fixstr name, bin32 payload. + // Build independently of MetacacheWriter to pin the existing wire shape. + for version in [1, 2] { + let mut stream = vec![0xcc, version]; + for size in [0, 2 * METACACHE_READ_CHUNK_SIZE, (16 << 20) + 1] { + stream.extend_from_slice(&[0xc3, 0xa1, b'x', 0xc6]); + stream.extend_from_slice(&u32::try_from(size).expect("payload length must fit bin32").to_be_bytes()); + stream.resize(stream.len() + size, 0xab); + } + stream.extend_from_slice(&[0xc3, 0xa1, b'y', 0xc4, 1, 0x42, 0xc2]); + let mut reader = MetacacheReader::new(Cursor::new(stream)); + reader.skip(3).await.expect("large entries must be skippable"); + assert!(reader.buf.is_empty()); + assert!( + reader.buf.capacity() <= METACACHE_READ_CHUNK_SIZE, + "skip must not retain discarded entries" + ); + let entry = reader + .peek() + .await + .expect("following entry must decode") + .expect("following entry must exist"); + assert_eq!(entry.name, "y"); + assert_eq!(entry.metadata, vec![0x42]); + reader + .skip(0) + .await + .expect("skipping zero with a current entry must be a no-op"); + reader.skip(1).await.expect("consume current entry without reading another"); + assert!(reader.peek().await.expect("stream must terminate").is_none()); + } } #[test] diff --git a/crates/filemeta/tests/xl_meta_codec_compat.rs b/crates/filemeta/tests/xl_meta_codec_compat.rs new file mode 100644 index 000000000..6a8cab5f9 --- /dev/null +++ b/crates/filemeta/tests/xl_meta_codec_compat.rs @@ -0,0 +1,258 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use bytes::Bytes; +use rustfs_filemeta::{ + Error, FileInfo, FileInfoOpts, FileMeta, FileMetaShallowVersion, FileMetaVersion, FileMetaVersionHeader, InlineData, + MetaDeleteMarker, VersionType, get_file_info, test_data::*, +}; +use rustfs_utils::http::{SUFFIX_FREE_VERSION, insert_bytes}; +use time::OffsetDateTime; +use uuid::Uuid; + +fn read_options(data: bool, include_free_versions: bool, include_part_checksums: bool) -> FileInfoOpts { + FileInfoOpts { + data, + include_free_versions, + include_part_checksums, + } +} + +#[test] +fn historical_codec_preserves_signatures_and_fileinfo() { + // Signatures were captured from the pre-optimization implementation at + // 6681dd12e39fda867c54cc4e576ecf7f4257aec4, not from a same-code round trip. + // MinIO fixtures pin RustFS recomputation, not MinIO's different hash algorithm. + let fixtures = [ + ( + "minio-small", + create_minio_small_object_xlmeta().expect("historical fixture"), + vec![[63, 254, 115, 180]], + ), + ( + "minio-versioned", + create_minio_versioned_object_xlmeta().expect("historical fixture"), + vec![[179, 89, 130, 162], [42, 145, 112, 214], [91, 28, 96, 74]], + ), + ( + "minio-large", + create_minio_large_object_xlmeta().expect("historical fixture"), + vec![[95, 172, 233, 51]], + ), + ( + "legacy-v1", + create_legacy_v1_object_xlmeta().expect("historical fixture"), + vec![[130, 178, 104, 208]], + ), + ( + "issue-2288", + create_issue_2288_legacy_xlmeta().expect("historical fixture"), + vec![[233, 25, 148, 246]], + ), + ( + "issue-2265-object", + create_issue_2265_legacy_meta_v2_object_xlmeta().expect("historical fixture"), + vec![[64, 154, 203, 216]], + ), + ( + "issue-2265-config", + create_issue_2265_legacy_meta_v2_config_xlmeta().expect("historical fixture"), + vec![[83, 146, 54, 27]], + ), + ( + "issue-2434-pool", + create_issue_2434_legacy_meta_v2_pool_xlmeta().expect("historical fixture"), + vec![[91, 227, 222, 255]], + ), + ( + "pre-metadata-keys", + create_pre_metadata_keys_xlmeta().expect("historical fixture"), + vec![[5, 84, 29, 43]], + ), + ]; + for (name, raw, expected_signatures) in fixtures { + let loaded = FileMeta::load(&raw).expect("load historical metadata"); + let signatures: Vec<_> = loaded + .versions + .iter() + .map(|version| version.parse_version_meta().expect("historical version").get_signature()) + .collect(); + assert_eq!(signatures, expected_signatures, "signature changed for {name}"); + + let mut selectors = vec![String::new(), Uuid::nil().to_string(), Uuid::from_u128(u128::MAX).to_string()]; + selectors.extend( + loaded + .versions + .iter() + .filter_map(|v| v.header.version_id.map(|id| id.to_string())), + ); + for selector in selectors { + for data in [false, true] { + for free in [false, true] { + for checksums in [false, true] { + let direct = get_file_info(&raw, "bucket", "object", &selector, read_options(data, free, checksums)); + let expected = if checksums { + loaded.into_fileinfo("bucket", "object", &selector, data, free, true) + } else { + loaded.into_fileinfo_without_part_checksums("bucket", "object", &selector, data, free) + }; + assert_eq!( + direct, expected, + "{name}: selector={selector}, data={data}, free={free}, checksums={checksums}" + ); + } + } + } + } + } +} + +#[test] +fn direct_fileinfo_reads_both_null_inline_keys() { + let mut meta = FileMeta::new(); + let payload = Bytes::from_static(b"legacy-inline"); + let mut fi = FileInfo::new("object", 2, 1); + fi.mod_time = Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp")); + fi.data = Some(payload.clone()); + fi.set_inline_data(); + meta.add_version(fi).expect("add inline null version"); + + for key in ["null".to_owned(), Uuid::nil().to_string()] { + meta.data = InlineData::default(); + meta.data.replace(&key, payload.to_vec()).expect("set inline key"); + let raw = meta.marshal_msg().expect("encode null version"); + for selector in ["", "00000000-0000-0000-0000-000000000000"] { + for data in [false, true] { + let fi = get_file_info(&raw, "bucket", "object", selector, read_options(data, false, true)) + .expect("read null version"); + assert_eq!(fi.version_id, None); + assert!(fi.is_latest); + assert_eq!(fi.num_versions, 1); + assert_eq!(fi.data, data.then(|| payload.clone()), "key={key}, selector={selector}"); + } + } + } +} + +#[test] +fn direct_fileinfo_preserves_free_version_selection() { + let mut marker = MetaDeleteMarker { + version_id: Some(Uuid::from_u128(1)), + mod_time: Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp")), + ..Default::default() + }; + insert_bytes(&mut marker.meta_sys, SUFFIX_FREE_VERSION, Vec::new()); + let mut meta = FileMeta::new(); + meta.versions.push( + FileMetaShallowVersion::try_from(FileMetaVersion { + version_type: VersionType::Delete, + delete_marker: Some(marker), + ..Default::default() + }) + .expect("free version"), + ); + let raw = meta.marshal_msg().expect("encode free version"); + assert_eq!( + get_file_info(&raw, "bucket", "object", "", read_options(false, false, true)), + Err(Error::FileNotFound) + ); + let free_info = get_file_info(&raw, "bucket", "object", "", read_options(false, true, true)).expect("include free version"); + assert!(free_info.tier_free_version()); + assert!(free_info.is_latest); + assert_eq!(free_info.num_versions, 0); + // Explicit addressing of the free record must retain the existing behavior. + let explicit = get_file_info( + &raw, + "bucket", + "object", + &Uuid::from_u128(1).to_string(), + read_options(false, false, true), + ) + .expect("explicit free version"); + assert_eq!(explicit, free_info); + + let object = FileMeta::load(&create_minio_small_object_xlmeta().expect("object fixture")).expect("load object"); + meta.versions.extend(object.versions); + let raw = meta.marshal_msg().expect("encode free and ordinary versions"); + for include_free in [false, true] { + let fi = get_file_info(&raw, "bucket", "object", "", read_options(false, include_free, true)) + .expect("ordinary object remains visible"); + assert!(!fi.deleted); + assert_eq!(fi.size, 19); + assert_eq!(fi.num_versions, 1); + } +} + +#[test] +fn direct_fileinfo_preserves_empty_history_and_invalid_selectors() { + let raw = create_empty_xlmeta().expect("empty history"); + for selector in [ + "", + "00000000-0000-0000-0000-000000000000", + "00000000-0000-0000-0000-000000000001", + ] { + let fi = + get_file_info(&raw, "bucket", "object", selector, read_options(true, true, true)).expect("empty history tombstone"); + assert_eq!( + fi.version_id, + if selector.is_empty() { + None + } else { + Some(Uuid::parse_str(selector).expect("UUID")) + } + ); + assert!(fi.deleted && fi.is_latest); + assert_eq!(fi.mod_time, Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp"))); + assert_eq!(fi.num_versions, 0); + } + for raw in [raw.as_slice(), &[]] { + assert!(matches!( + get_file_info(raw, "bucket", "object", "invalid", read_options(false, false, false)), + Err(Error::UuidParse(_)) + )); + } +} + +#[test] +fn large_version_histories_fit_initial_serialization_buffer() { + for count in [4, 128] { + for body_len in [255, 256, 65_535, 65_536] { + let mut meta = FileMeta::new(); + meta.versions = vec![ + FileMetaShallowVersion { + // Use the widest flags and erasure fields; version_type remains a fixint. + header: FileMetaVersionHeader { + version_id: Some(Uuid::nil()), + mod_time: Some(OffsetDateTime::from_unix_timestamp(1).expect("timestamp")), + version_type: VersionType::Object, + flags: u8::MAX, + ec_n: u8::MAX, + ec_m: u8::MAX, + ..Default::default() + }, + meta: vec![0; body_len], + }; + count + ]; + let encoded = meta.marshal_msg().expect("serialize large history"); + assert!( + encoded.capacity() - encoded.len() <= count * 4 + 32, + "unexpected buffer growth for {count} bodies of {body_len} bytes: length={}, capacity={}", + encoded.len(), + encoded.capacity() + ); + assert_eq!(FileMeta::load(&encoded).expect("load serialized history").versions, meta.versions); + } + } +} diff --git a/crates/heal/Cargo.toml b/crates/heal/Cargo.toml index a96a88510..bf91f8faa 100644 --- a/crates/heal/Cargo.toml +++ b/crates/heal/Cargo.toml @@ -73,7 +73,6 @@ hotpath-cpu = [ [dependencies] hotpath.workspace = true rustfs-config = { workspace = true } -rustfs-data-usage = { workspace = true } rustfs-concurrency = { workspace = true } rustfs-ecstore = { workspace = true } rustfs-lock = { workspace = true } diff --git a/crates/heal/src/error.rs b/crates/heal/src/error.rs index d1db7292a..f87a41112 100644 --- a/crates/heal/src/error.rs +++ b/crates/heal/src/error.rs @@ -90,6 +90,9 @@ pub enum Error { source: Box, }, + #[error("Bucket discovery incomplete for the requested heal scope; not all buckets could be traversed")] + BucketDiscoveryIncomplete, + #[error("Invalid heal type: {heal_type}")] InvalidHealType { heal_type: String }, @@ -118,81 +121,28 @@ impl Error { } /// Whether a heal operation can be retried without changing its inputs. - /// - /// Typed-first (backlog#1845): the primary classification reads error - /// variants and typed helpers (`is_quorum_error`, `LockError::is_fatal`); - /// the substring fallback in [`is_recoverable_heal_error_message`] only - /// catches errors whose typed identity was destroyed upstream. pub(crate) fn is_recoverable_heal(&self) -> bool { - match self { - Error::TaskCancelled | Error::TaskTimeout | Error::StaleBucketIncarnation { .. } => false, - Error::ReplacementTargetNotReady(_) | Error::DanglingDeleteDeferred { .. } => true, - Error::TransientSkip { .. } => true, - // Lock failures classify by LockError's own taxonomy: only the - // fatal variants (ResourceNotFound / PermissionDenied / - // Configuration) are terminal - retrying cannot fix them - while - // contention and transport variants (Timeout, Network, Internal, - // AlreadyLocked, QuorumNotReached, InsufficientNodes, ...) stay - // recoverable, as the previous blanket `Lock(_) => true` treated - // them. - Error::Storage(EcstoreError::Lock(lock_err)) => !lock_err.is_fatal(), - Error::Storage(err) => { - if err.is_dangling_delete_grace() { - return true; - } - err.is_quorum_error() - || matches!(err, EcstoreError::Io(error) if is_recoverable_io_error(error)) - || matches!( - err, - EcstoreError::DiskNotFound - | EcstoreError::VolumeNotFound - | EcstoreError::FaultyDisk - | EcstoreError::FaultyRemoteDisk - | EcstoreError::SlowDown - | EcstoreError::OperationCanceled - | EcstoreError::RemoteClientUnavailable(_) - ) - || is_recoverable_heal_error_message(&err.to_string()) + match classify_heal_error(self) { + // Only a typed grace window replays the task. A grace window + // rendered into a task message or an io::Error comes from an + // attempt that already finished; its owner (for example a durable + // MRF responsibility) schedules the next attempt. + HealErrorClass::DanglingGrace { .. } => { + matches!(self, Error::DanglingDeleteDeferred { .. } | Error::Storage(_) | Error::Disk(_)) } - // Nonblocking local CAS and replacement leases report lock - // contention as WouldBlock; retain the existing task retry budget. - Error::Disk(DiskError::Io(error)) if error.kind() == std::io::ErrorKind::WouldBlock => true, - Error::Disk(err) => { - if err.is_dangling_delete_grace() { - return true; - } - matches!( - err, - DiskError::DiskNotFound - | DiskError::ErasureReadQuorum - | DiskError::ErasureWriteQuorum - | DiskError::Timeout - | DiskError::SourceStalled - | DiskError::FaultyRemoteDisk - | DiskError::FaultyDisk - | DiskError::RemoteClientUnavailable(_) - ) || matches!(err, DiskError::Io(error) if is_recoverable_io_error(error)) - || is_recoverable_heal_error_message(&err.to_string()) - } - Error::TaskExecutionFailed { message } | Error::Other(message) => is_recoverable_heal_error_message(message), - Error::Io(err) => is_recoverable_io_error(err) || is_recoverable_heal_error_message(&err.to_string()), - _ => false, + class => class.is_retryable(), } } /// Retry an unproven missing object only within the per-object budget. /// Generic task retries retain the narrower classification above. pub(crate) fn is_recoverable_object_heal(&self) -> bool { - self.is_recoverable_heal() - || matches!( - self, - Error::Storage( - EcstoreError::FileNotFound - | EcstoreError::FileVersionNotFound - | EcstoreError::ObjectNotFound(_, _) - | EcstoreError::VersionNotFound(_, _, _) - ) | Error::Disk(DiskError::FileNotFound | DiskError::FileVersionNotFound) - ) + self.is_recoverable_heal() || self.is_object_absent() + } + + /// The locked storage read found no replica of the object or version. + pub(crate) fn is_object_absent(&self) -> bool { + classify_heal_error(self) == HealErrorClass::Absent } pub(crate) fn dangling_delete_retry_not_before(&self) -> Option { @@ -209,19 +159,157 @@ impl Error { } pub(crate) fn is_dangling_delete_grace(&self) -> bool { - match self { - Error::DanglingDeleteDeferred { .. } => true, - Error::Storage(err) => err.is_dangling_delete_grace(), - Error::Disk(err) => err.is_dangling_delete_grace(), - Error::Io(err) => DiskError::io_error_is_dangling_delete_grace(err), - Error::TaskExecutionFailed { message } | Error::Other(message) => { - message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE) - } - _ => false, - } + matches!(classify_heal_error(self), HealErrorClass::DanglingGrace { .. }) } } +/// Verdict classes shared by every heal error consumer (backlog#2900). +/// +/// One classifier replaces the per-call-site predicates that disagreed on the +/// same input. The class is decided from typed variants; display text is +/// consulted only by [`classify_error_display`] for errors whose type was +/// destroyed upstream. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum HealErrorClass { + /// The locked quorum read found no replica of the object or version: + /// nothing to repair, independent of request source or options. This is + /// a verdict, not a durable absence proof. + Absent, + /// Offline disk, missing volume, unmet quorum, transport failure, + /// slow-down or cancelled RPC: the same request can succeed later. + Transient, + /// Lock contention or a lock/RPC timeout (non-fatal `LockError`). + LockContention, + /// A storage grace window; `retry_not_before` is the absolute UNIX + /// deadline when storage reported one. + DanglingGrace { retry_not_before: Option }, + /// Storage reports nothing to heal. + NoHealRequired, + /// The task was cancelled, timed out or its admission became stale: the + /// object's state is unknown and no retry budget applies. + Aborted, + /// Retrying cannot help: corruption beyond parity, permission, fatal + /// lock state or invalid input. + Permanent, +} + +impl HealErrorClass { + /// Whether the same request can be retried unchanged. + pub(crate) const fn is_retryable(self) -> bool { + matches!(self, Self::Transient | Self::LockContention | Self::DanglingGrace { .. }) + } +} + +/// Classify a heal error from its typed identity. +pub(crate) fn classify_heal_error(error: &Error) -> HealErrorClass { + match error { + Error::TaskCancelled | Error::TaskTimeout | Error::StaleBucketIncarnation { .. } => HealErrorClass::Aborted, + Error::DanglingDeleteDeferred { retry_not_before } => HealErrorClass::DanglingGrace { + retry_not_before: Some(*retry_not_before), + }, + Error::ReplacementTargetNotReady(_) | Error::TransientSkip { .. } | Error::BucketDiscoveryIncomplete => { + HealErrorClass::Transient + } + Error::Storage(error) => classify_storage_error(error), + Error::Disk(error) => classify_disk_error(error), + Error::Io(error) => classify_io_error(error), + Error::TaskExecutionFailed { message } | Error::Other(message) => { + classify_error_display(message).unwrap_or(HealErrorClass::Permanent) + } + _ => HealErrorClass::Permanent, + } +} + +/// Classify a storage error. Lock failures follow `LockError`'s own taxonomy: +/// only its fatal variants are terminal. `StorageError::is_not_found()` is +/// deliberately not used: it lumps `DiskNotFound`/`VolumeNotFound` together +/// with object absence, the conflation that once let an offline drive be +/// recorded as absent and skipped (backlog#856 / #799 B7). +pub(crate) fn classify_storage_error(error: &EcstoreError) -> HealErrorClass { + if let EcstoreError::Lock(lock_error) = error { + return if lock_error.is_fatal() { + HealErrorClass::Permanent + } else { + HealErrorClass::LockContention + }; + } + if error.is_dangling_delete_grace() { + return HealErrorClass::DanglingGrace { + retry_not_before: dangling_deadline(error.dangling_delete_retry_after()), + }; + } + match error { + EcstoreError::FileNotFound + | EcstoreError::FileVersionNotFound + | EcstoreError::ObjectNotFound(_, _) + | EcstoreError::VersionNotFound(_, _, _) => HealErrorClass::Absent, + EcstoreError::NoHealRequired => HealErrorClass::NoHealRequired, + // An ambiguous legacy delete residue needs operator inspection; + // retrying the same heal cannot resolve it (backlog#2902). + EcstoreError::DeleteRecoveryLegacyAmbiguous => HealErrorClass::Permanent, + EcstoreError::Timeout => HealErrorClass::LockContention, + EcstoreError::DiskNotFound + | EcstoreError::VolumeNotFound + | EcstoreError::FaultyDisk + | EcstoreError::FaultyRemoteDisk + | EcstoreError::SlowDown + | EcstoreError::OperationCanceled + | EcstoreError::RemoteClientUnavailable(_) => HealErrorClass::Transient, + _ if error.is_quorum_error() => HealErrorClass::Transient, + EcstoreError::Io(error) => classify_io_error(error), + _ => classify_error_display(&error.to_string()).unwrap_or(HealErrorClass::Permanent), + } +} + +/// Classify a disk error. Nonblocking local CAS and replacement leases report +/// lock contention as `WouldBlock`. +pub(crate) fn classify_disk_error(error: &DiskError) -> HealErrorClass { + if error.is_dangling_delete_grace() { + return HealErrorClass::DanglingGrace { + retry_not_before: dangling_deadline(error.dangling_delete_retry_after()), + }; + } + match error { + DiskError::FileNotFound | DiskError::FileVersionNotFound => HealErrorClass::Absent, + DiskError::NoHealRequired => HealErrorClass::NoHealRequired, + DiskError::Timeout => HealErrorClass::LockContention, + DiskError::Io(error) if error.kind() == std::io::ErrorKind::WouldBlock => HealErrorClass::LockContention, + DiskError::DiskNotFound + | DiskError::VolumeNotFound + | DiskError::ErasureReadQuorum + | DiskError::ErasureWriteQuorum + | DiskError::SourceStalled + | DiskError::FaultyRemoteDisk + | DiskError::FaultyDisk + | DiskError::RemoteClientUnavailable(_) => HealErrorClass::Transient, + DiskError::Io(error) => classify_io_error(error), + _ => classify_error_display(&error.to_string()).unwrap_or(HealErrorClass::Permanent), + } +} + +fn classify_io_error(error: &std::io::Error) -> HealErrorClass { + if DiskError::io_error_is_dangling_delete_grace(error) { + return HealErrorClass::DanglingGrace { + retry_not_before: dangling_deadline(DiskError::io_error_dangling_delete_retry_after(error)), + }; + } + if is_recoverable_io_error(error) { + return HealErrorClass::Transient; + } + classify_error_display(&error.to_string()).unwrap_or(HealErrorClass::Permanent) +} + +/// Storage reports whole remaining seconds. Round the absolute deadline up so +/// subsecond grace boundaries cannot hot-loop. +fn dangling_deadline(retry_after: Option) -> Option { + std::time::SystemTime::now() + .checked_add(retry_after?)? + .duration_since(std::time::UNIX_EPOCH) + .ok()? + .as_secs() + .checked_add(1) +} + fn is_recoverable_io_error(error: &std::io::Error) -> bool { // ObjectIO preserves store errors as IO sources; their quorum identity must survive this boundary. if error @@ -259,43 +347,71 @@ fn is_recoverable_io_error(error: &std::io::Error) -> bool { && matches!(error.kind(), InternodeHttpErrorKind::HttpStatus(status) if matches!(status.as_u16(), 409 | 500))) } -/// Documented substring fallback for errors that reach heal with their typed -/// identity destroyed (stringified through `TaskExecutionFailed`/`Other`, or -/// boxed into `Io`). Every needle is annotated with the producer that emits -/// it; when a producer becomes typed end-to-end, delete its needle here -/// (backlog#1845 - this list only shrinks). -fn is_recoverable_heal_error_message(error: &str) -> bool { - let error = error.to_ascii_lowercase(); - [ - // set_disk/ops/locking.rs ns_loc lock failures rendered into messages. +/// Messages that reach heal with their typed identity destroyed: an older +/// peer's display text, a local `format!` into `TaskExecutionFailed`/`Other`, +/// or a boxed `io::Error`. Every needle names the producer that emits it; when +/// a producer becomes typed end-to-end, delete its needle here. This is the +/// only place in the heal crate that classifies error text. +// RUSTFS_COMPAT_TODO(backlog-2900-heal-error-display): map typed displays back to a class once, here. Remove after every heal producer and the minimum supported peer version return typed storage errors end-to-end. +pub(crate) fn classify_error_display(message: &str) -> Option { + // set_disk/ops/locking.rs ns_loc lock failures, ecstore remote_locker.rs + // and lock/src lock responses, tokio/tonic deadline and timeout rendering, + // and LockError::QuorumNotReached, when stringified before typing. + const LOCK_CONTENTION: [&str; 7] = [ "failed to acquire read lock", - // ecstore cluster/rpc/remote_locker.rs "Lock acquisition failed on - // remote server" and lock/src local lock responses. "lock acquisition failed", - // lock/src/distributed_lock.rs + client/local.rs LockResponse::failure. "lock acquisition timeout", - // remote_locker.rs RPC deadline wrapper. "remote lock rpc timed out", - // tokio/tonic deadline rendering, reaches heal via stringified RPC errors. "deadline has elapsed", - // generic io/tonic timeout rendering. "timed out", - // tonic transport failure rendering. - "transport error", - // LockError::Network display prefix. - "network error", - // io::Error ConnectionRefused rendering. - "connection refused", - // StorageError::OperationCanceled rendered through task messages. - "operation canceled", - // LockError::QuorumNotReached display, when stringified before typing. "quorum not reached", - // set_disk/ops/heal.rs HEAL_RENAME_INCOMPLETE - the one needle with no - // typed variant yet; the producer formats it into a plain message. + ]; + // tonic transport failure, LockError::Network and io ConnectionRefused + // rendering, StorageError::OperationCanceled rendered through task + // messages, and set_disk/ops/heal.rs HEAL_RENAME_INCOMPLETE, the one + // needle with no typed variant yet. + const TRANSIENT: [&str; 5] = [ + "transport error", + "network error", + "connection refused", + "operation canceled", "heal rename incomplete", - ] - .iter() - .any(|pattern| error.contains(pattern)) + ]; + // Canonical `Display` of DiskError/StorageError::NoHealRequired and the + // legacy heal-format wording. + const NO_HEAL_REQUIRED: [&str; 2] = ["No heal required", "No healing is required"]; + + if message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE) { + return Some(HealErrorClass::DanglingGrace { retry_not_before: None }); + } + // A contention or transient needle wins over an absence phrase: misreading + // a transient failure as absence would drop the object from repair. + let lower = message.to_ascii_lowercase(); + if LOCK_CONTENTION.iter().any(|needle| lower.contains(needle)) { + return Some(HealErrorClass::LockContention); + } + if TRANSIENT.iter().any(|needle| lower.contains(needle)) { + return Some(HealErrorClass::Transient); + } + // Only canonical errors and their type wrappers are absence evidence. + // Object names in contextual task messages can contain these same words. + let mut display = message; + while let Some(inner) = ["Storage error: ", "Disk error: ", "I/O error: ", "Other error: "] + .iter() + .find_map(|prefix| display.strip_prefix(*prefix)) + { + display = inner; + } + if matches!(display, "File not found" | "File version not found") + || display.starts_with("Object not found: ") + || display.starts_with("Version not found: ") + { + return Some(HealErrorClass::Absent); + } + if NO_HEAL_REQUIRED.iter().any(|display| message.contains(display)) { + return Some(HealErrorClass::NoHealRequired); + } + None } impl From for std::io::Error { @@ -489,7 +605,7 @@ mod tests { "RPC conversion and cloning must retain the typed transport source" ); assert!( - !super::is_recoverable_heal_error_message(&error.to_string()), + super::classify_error_display(&error.to_string()).is_none(), "GOAWAY regression must not be covered by the legacy message fallback: {error}" ); assert!(error.is_recoverable_heal(), "a peer GOAWAY must remain recoverable: {error}"); @@ -670,6 +786,176 @@ mod tests { } } + #[test] + fn classify_legacy_peer_display_not_found_is_object_level_absent() { + // backlog#2900: a typed display that lost its type at an old peer or a + // type wrapper converts back once, at the boundary. + for error in [ + Error::Other("File not found".to_string()), + Error::Other("File version not found".to_string()), + Error::Other("Object not found: bucket/object".to_string()), + Error::Other("Version not found: bucket/object/version".to_string()), + Error::Other("I/O error: Other error: Disk error: File version not found".to_string()), + Error::TaskExecutionFailed { + message: "Storage error: File not found".to_string(), + }, + ] { + assert!(error.is_recoverable_object_heal(), "{error:?}"); + assert!(!error.is_recoverable_heal(), "absence must not requeue the whole task: {error:?}"); + } + for error in [ + Error::Other("file not found".to_string()), + Error::Other("Disk not found".to_string()), + Error::Other("Volume not found".to_string()), + ] { + assert!(!error.is_recoverable_object_heal(), "{error:?}"); + } + } + + #[test] + fn classify_disk_volume_not_found_is_recoverable_like_storage() { + assert!(Error::Disk(DiskError::VolumeNotFound).is_recoverable_heal()); + assert!(Error::Disk(DiskError::VolumeNotFound).is_recoverable_object_heal()); + } + + #[test] + fn classify_heal_error_table() { + use super::{HealErrorClass as C, classify_heal_error}; + use rustfs_lock::LockError; + let grace = "dangling object deletion deferred by heal grace window; retry_after_secs=5; grace_secs=10"; + let cases = [ + (Error::TaskCancelled, C::Aborted), + (Error::TaskTimeout, C::Aborted), + ( + Error::StaleBucketIncarnation { + bucket: "b".into(), + expected: None, + }, + C::Aborted, + ), + (Error::Storage(EcstoreError::FileNotFound), C::Absent), + (Error::Storage(EcstoreError::FileVersionNotFound), C::Absent), + (Error::Storage(EcstoreError::ObjectNotFound("b".into(), "o".into())), C::Absent), + ( + Error::Storage(EcstoreError::VersionNotFound("b".into(), "o".into(), "v".into())), + C::Absent, + ), + (Error::Disk(DiskError::FileNotFound), C::Absent), + (Error::Disk(DiskError::FileVersionNotFound), C::Absent), + (Error::Storage(EcstoreError::DiskNotFound), C::Transient), + (Error::Storage(EcstoreError::VolumeNotFound), C::Transient), + (Error::Storage(EcstoreError::FaultyDisk), C::Transient), + (Error::Storage(EcstoreError::FaultyRemoteDisk), C::Transient), + (Error::Storage(EcstoreError::SlowDown), C::Transient), + (Error::Storage(EcstoreError::OperationCanceled), C::Transient), + (Error::Storage(EcstoreError::RemoteClientUnavailable("rebuild".into())), C::Transient), + (Error::Storage(EcstoreError::ErasureReadQuorum), C::Transient), + (Error::Storage(EcstoreError::InsufficientReadQuorum("b".into(), "o".into())), C::Transient), + (Error::Disk(DiskError::DiskNotFound), C::Transient), + (Error::Disk(DiskError::VolumeNotFound), C::Transient), + (Error::Disk(DiskError::ErasureWriteQuorum), C::Transient), + (Error::Disk(DiskError::SourceStalled), C::Transient), + (Error::TransientSkip { message: "skip".into() }, C::Transient), + (Error::ReplacementTargetNotReady("restarting".into()), C::Transient), + (Error::BucketDiscoveryIncomplete, C::Transient), + ( + Error::Storage(EcstoreError::Lock(LockError::timeout("b/o", std::time::Duration::from_secs(1)))), + C::LockContention, + ), + (Error::Storage(EcstoreError::Timeout), C::LockContention), + (Error::Disk(DiskError::Timeout), C::LockContention), + ( + Error::Disk(DiskError::Io(std::io::Error::from(std::io::ErrorKind::WouldBlock))), + C::LockContention, + ), + (Error::Other("remote lock rpc timed out".into()), C::LockContention), + (Error::Storage(EcstoreError::Lock(LockError::permission_denied("policy"))), C::Permanent), + (Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous), C::Permanent), + (Error::Disk(DiskError::FileCorrupt), C::Permanent), + (Error::Disk(DiskError::FileAccessDenied), C::Permanent), + (Error::Disk(DiskError::DiskFull), C::Permanent), + (Error::Other("boom".into()), C::Permanent), + (Error::Config("bad".into()), C::Permanent), + (Error::Storage(EcstoreError::NoHealRequired), C::NoHealRequired), + (Error::Disk(DiskError::NoHealRequired), C::NoHealRequired), + (Error::Other("No heal required".into()), C::NoHealRequired), + ( + Error::DanglingDeleteDeferred { retry_not_before: 42 }, + C::DanglingGrace { + retry_not_before: Some(42), + }, + ), + ( + Error::TaskExecutionFailed { message: grace.into() }, + C::DanglingGrace { retry_not_before: None }, + ), + ]; + for (error, expected) in cases { + assert_eq!(classify_heal_error(&error), expected, "{error:?}"); + } + assert!(matches!( + classify_heal_error(&Error::Disk(DiskError::other(grace))), + C::DanglingGrace { .. } + )); + } + + #[test] + fn rendered_grace_window_is_grace_but_does_not_replay_the_task() { + let grace = "dangling object deletion deferred by heal grace window; retry_after_secs=5; grace_secs=10"; + let rendered = Error::TaskExecutionFailed { + message: format!("Failed to recreate missing object b/o: Disk error: {grace}"), + }; + assert!(rendered.is_dangling_delete_grace()); + assert!(!rendered.is_recoverable_heal(), "the owner of a finished attempt schedules the next one"); + assert!(!rendered.is_recoverable_object_heal()); + let typed = Error::Disk(DiskError::other(grace)); + assert!(typed.is_dangling_delete_grace()); + assert!(typed.is_recoverable_heal()); + assert!(Error::DanglingDeleteDeferred { retry_not_before: 1 }.is_recoverable_heal()); + } + + #[test] + fn classify_error_display_prefers_contention_over_absence_text() { + use super::{HealErrorClass as C, classify_error_display}; + assert_eq!( + classify_error_display("File not found while waiting: lock acquisition timeout"), + Some(C::LockContention) + ); + assert_eq!(classify_error_display("Storage error: File not found"), Some(C::Absent)); + assert_eq!(classify_error_display("Object not found: b/o"), Some(C::Absent)); + assert_eq!(classify_error_display("file not found"), None); + assert_eq!(classify_error_display("Disk not found"), None); + assert_eq!(classify_error_display("operation was canceled"), None); + } + + #[test] + fn classify_error_display_does_not_take_absence_from_object_context() { + use super::{HealErrorClass, classify_error_display, classify_heal_error}; + for object in [ + "File not found", + "File version not found", + "Object not found: b/o", + "Version not found: b/o/v", + ] { + let message = format!( + "Heal left one or more drives unhealthy for bucket/{object}; retry after the missing drives are restored" + ); + assert_eq!( + classify_heal_error(&Error::TaskExecutionFailed { message }), + HealErrorClass::Permanent, + "{object}" + ); + } + for message in [ + "File not found.txt", + "File version not found.txt", + "Failed to heal object bucket/object: Storage error: File not found", + "Failed to heal object bucket/Object not found: b/o: access denied", + ] { + assert_eq!(classify_error_display(message), None, "{message}"); + } + } + #[test] fn remote_client_unavailable_is_recoverable_via_typed_variant() { // The detail deliberately avoids every substring needle: the typed diff --git a/crates/heal/src/heal/channel.rs b/crates/heal/src/heal/channel.rs index 20b37eff2..d212a5cf5 100644 --- a/crates/heal/src/heal/channel.rs +++ b/crates/heal/src/heal/channel.rs @@ -735,6 +735,9 @@ impl HealChannelProcessor { HealChannelPriority::Critical => HealPriority::Urgent, }; + // `recreate_missing` only reaches storage as the bucket-volume + // recreation flag. Whether a missing object is absent is decided by the + // locked storage read, never by this per-source default (backlog#2900). let recreate_missing = request.recreate_missing.unwrap_or(match request.source { HealRequestSource::Scanner => false, HealRequestSource::Admin @@ -853,6 +856,13 @@ mod tests { async fn get_bucket_info(&self, _bucket: &str) -> crate::Result> { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> crate::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> crate::Result> { Ok(vec![]) } diff --git a/crates/heal/src/heal/erasure_healer.rs b/crates/heal/src/heal/erasure_healer.rs index 5a02978ba..9fb8dee1f 100644 --- a/crates/heal/src/heal/erasure_healer.rs +++ b/crates/heal/src/heal/erasure_healer.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::error::{HealErrorClass, classify_heal_error}; use crate::heal::{ progress::{HealProgress, add_bytes, increment_counter}, resume::{ @@ -34,7 +35,7 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH}; use tokio::sync::{RwLock, Semaphore}; use tracing::{debug, error, warn}; -use super::{DiskStore, EcstoreError, POOL_META_NAME, RUSTFS_META_BUCKET}; +use super::{DiskStore, POOL_META_NAME, RUSTFS_META_BUCKET}; /// Outcome of classifying an error returned by [`HealStorageAPI::heal_object`]. enum HealObjectOutcome { @@ -112,6 +113,7 @@ mod admin; pub struct ErasureSetHealer { admin_task: Option, + unresolved_delete_recovery: bool, storage: Arc, progress: Arc>, cancel_token: tokio_util::sync::CancellationToken, @@ -310,70 +312,21 @@ impl ErasureSetHealer { } } - /// Classify an error returned by [`HealStorageAPI::heal_object`]. + /// Map the shared heal error class onto this healer's per-object actions. /// - /// Most heal object failures wrap `Error::Storage(StorageError)`, while - /// compatibility markers can also arrive through Disk/Io/task wrappers. + /// `classify_heal_error` is the only classifier; this adapter decides what + /// a replacement pass does with each class. fn classify_heal_object_error(err: &Error) -> HealObjectOutcome { - if let Error::DanglingDeleteDeferred { retry_not_before } = err { - return HealObjectOutcome::DanglingDeleteDeferred { - retry_not_before: *retry_not_before, - }; + match classify_heal_error(err) { + HealErrorClass::Absent => HealObjectOutcome::Absent, + HealErrorClass::DanglingGrace { + retry_not_before: Some(retry_not_before), + } => HealObjectOutcome::DanglingDeleteDeferred { retry_not_before }, + HealErrorClass::DanglingGrace { retry_not_before: None } + | HealErrorClass::Transient + | HealErrorClass::LockContention => HealObjectOutcome::Transient, + HealErrorClass::NoHealRequired | HealErrorClass::Aborted | HealErrorClass::Permanent => HealObjectOutcome::Failed, } - if err.is_dangling_delete_grace() { - if let Some(retry_not_before) = err - .dangling_delete_retry_not_before() - .and_then(|deadline| deadline.duration_since(UNIX_EPOCH).ok()) - .and_then(|deadline| deadline.as_secs().checked_add(1)) - { - // Storage reports whole remaining seconds. Round the absolute - // deadline up so subsecond grace boundaries cannot hot-loop. - return HealObjectOutcome::DanglingDeleteDeferred { retry_not_before }; - } - return HealObjectOutcome::Transient; - } - - let Error::Storage(se) = err else { - return HealObjectOutcome::Failed; - }; - - if let EcstoreError::Lock(lock_error) = se - && !lock_error.is_fatal() - { - return HealObjectOutcome::Transient; - } - - // Genuine object/version absence: nothing left to heal, treat as handled. - if matches!( - se, - EcstoreError::FileNotFound - | EcstoreError::FileVersionNotFound - | EcstoreError::ObjectNotFound(_, _) - | EcstoreError::VersionNotFound(_, _, _) - ) { - return HealObjectOutcome::Absent; - } - - // Transient infrastructure conditions — skip and retry on a later pass. - // NOTE: do NOT use `StorageError::is_not_found()` here: it lumps - // `DiskNotFound`/`VolumeNotFound` together with object absence, which is - // exactly the conflation that previously let an offline drive be recorded - // as "healed/absent" and permanently skipped (backlog#856 / #799 B7). - if se.is_quorum_error() - || matches!( - se, - EcstoreError::DiskNotFound - | EcstoreError::VolumeNotFound - | EcstoreError::FaultyDisk - | EcstoreError::FaultyRemoteDisk - | EcstoreError::SlowDown - | EcstoreError::OperationCanceled - ) - { - return HealObjectOutcome::Transient; - } - - HealObjectOutcome::Failed } pub fn new( @@ -386,6 +339,7 @@ impl ErasureSetHealer { ) -> Self { Self { admin_task: None, + unresolved_delete_recovery: false, storage, progress, cancel_token, @@ -406,6 +360,11 @@ impl ErasureSetHealer { self } + pub(crate) fn with_unresolved_delete_recovery(mut self, unresolved: bool) -> Self { + self.unresolved_delete_recovery = unresolved; + self + } + pub(crate) fn with_replacement_execution(mut self, execution: Option>) -> Self { self.replacement_execution = execution; self @@ -644,6 +603,22 @@ impl ErasureSetHealer { }; let state = resume_manager.get_state().await; + if state.delete_recovery_rescan_pending + && state.completed_buckets.is_empty() + && state.resume_cursor.is_none() + && state.processed_objects == 0 + && state.successful_objects == 0 + && state.failed_objects == 0 + && state.skipped_objects == 0 + && state.skipped_new_versions == 0 + && state.skipped_ilm_expired == 0 + && state.processed_bytes == 0 + { + // Resume is published before its companion checkpoint reset. + // Finish that rewind after a crash before clearing the obligation. + checkpoint_manager.reset_after_delete_recovery().await?; + resume_manager.set_delete_recovery_rescan_pending(false).await?; + } if self.admin_task.is_none() && (state.retry_count > 0 || state.dangling_delete_retry_not_before.is_some()) && state.completed_buckets.is_empty() @@ -706,6 +681,9 @@ impl ErasureSetHealer { resume_manager: &ResumeManager, checkpoint_manager: &CheckpointManager, ) -> Result<()> { + if self.unresolved_delete_recovery && !self.heal_opts.dry_run { + resume_manager.set_delete_recovery_rescan_pending(true).await?; + } // 1. get current state let state = resume_manager.get_state().await; if self.replacement_task_id.is_some() @@ -812,13 +790,27 @@ impl ErasureSetHealer { resume_manager.mark_counter_unknown().await?; } let mut failed_buckets = 0u64; + let mut unresolved_delete_recovery = self.unresolved_delete_recovery; // 4. process remaining buckets - for (bucket_idx, bucket) in buckets.iter().enumerate().skip(current_bucket_index) { - // check if completed - if state.completed_buckets.contains(bucket) { - checkpoint_manager.complete_bucket(bucket_idx.saturating_add(1)).await?; - current_object_index = 0; + let first_bucket_index = if self.admin_task.is_some() { 0 } else { current_bucket_index }; + for (bucket_idx, bucket) in buckets.iter().enumerate().skip(first_bucket_index) { + if bucket_idx < current_bucket_index || state.completed_buckets.contains(bucket) { + // Object progress does not certify hidden delete transactions. + // Retry recovery in the admitted incarnation without replaying + // payload work already acknowledged by the ordinary cursor. + if self.admin_task.is_some() { + let incarnation = self.admin_bucket_incarnation(checkpoint_manager, bucket).await?; + let unresolved = self.heal_admin_bucket_delete_recovery(bucket, incarnation).await?; + if unresolved && !self.heal_opts.dry_run { + resume_manager.set_delete_recovery_rescan_pending(true).await?; + } + unresolved_delete_recovery |= unresolved; + } + if bucket_idx >= current_bucket_index { + checkpoint_manager.complete_bucket(bucket_idx.saturating_add(1)).await?; + current_object_index = 0; + } continue; } @@ -843,7 +835,7 @@ impl ErasureSetHealer { .await; if matches!(bucket_result, Err(Error::TaskCancelled | Error::TaskTimeout)) { - return bucket_result; + return bucket_result.map(|_| ()); } // update progress @@ -899,7 +891,8 @@ impl ErasureSetHealer { // process bucket result match bucket_result { - Ok(_) => { + Ok(unresolved) => { + unresolved_delete_recovery |= unresolved; resume_manager.complete_bucket(bucket).await?; checkpoint_manager.complete_bucket(bucket_idx.saturating_add(1)).await?; debug!( @@ -970,8 +963,24 @@ impl ErasureSetHealer { .await?; } + if !unresolved_delete_recovery && resume_manager.get_state().await.delete_recovery_rescan_pending { + // Recovery can publish formerly hidden objects, including when a + // background worker settled and collected its proof before this + // retry. Aggregate counters only support a whole-set rewind. + resume_manager.restart_scan_after_delete_recovery().await?; + checkpoint_manager.reset_after_delete_recovery().await?; + resume_manager.set_delete_recovery_rescan_pending(false).await?; + return Err(Error::transient_skip("Delete recovery settled; ordinary coverage must be rescanned")); + } + if self.admin_task.is_some() { - return Ok(()); + return if unresolved_delete_recovery { + Err(Error::transient_skip( + "Delete recovery remains unresolved after ordinary erasure-set heal", + )) + } else { + Ok(()) + }; } // 5. finalize. Only declare the set healed when nothing failed AND @@ -1081,6 +1090,12 @@ impl ErasureSetHealer { ))); } + if unresolved_delete_recovery { + return Err(Error::transient_skip( + "Delete recovery remains unresolved after ordinary erasure-set heal", + )); + } + // No failures — ordinary heals are complete now. Replacement heals // atomically transition to Verified after the terminal identity fence. if self.replacement_task_id.is_none() { @@ -1353,7 +1368,40 @@ impl ErasureSetHealer { Ok(()) } - /// heal single bucket with resume + async fn heal_admin_bucket_delete_recovery(&self, bucket: &str, incarnation: Option) -> Result { + let Some(task) = &self.admin_task else { + return Ok(false); + }; + let bucket_opts = HealOpts { + recursive: false, + remove: false, + ..self.heal_opts + }; + let result = match incarnation { + Some(expected) => { + self.storage.validate_bucket_incarnation(bucket, Some(expected)).await?; + self.storage + .heal_bucket_at_incarnation(bucket, expected, &bucket_opts) + .await? + } + None => self.storage.heal_bucket(bucket, &bucket_opts).await?, + }; + task.record_result_item(result).await; + // Recovery retains the checkpoint's admitted incarnation, and the + // publication bucket must exist before adopting survivor delete proof. + if let Some(result) = self + .storage + .recover_delete_transactions(bucket, None, incarnation, &self.heal_opts) + .await? + { + let unresolved = result.delete_recovery.is_some_and(|status| status.has_unresolved()); + task.record_result_item(result).await; + return Ok(unresolved); + } + Ok(false) + } + + /// Heal one bucket's ordinary objects and report unresolved delete recovery separately. #[allow(clippy::too_many_arguments)] #[tracing::instrument(skip(self, current_object_index, processed_objects, successful_objects, failed_objects, skipped_objects, resume_manager, checkpoint_manager), fields(bucket = %bucket, bucket_index = bucket_index))] async fn heal_bucket_with_resume( @@ -1369,7 +1417,7 @@ impl ErasureSetHealer { resume_manager: &ResumeManager, checkpoint_manager: &CheckpointManager, started_at_secs: u64, - ) -> Result<()> { + ) -> Result { debug!( target: "rustfs::heal::erasure_healer", event = EVENT_HEAL_ERASURE_BUCKET_STATE, @@ -1384,22 +1432,9 @@ impl ErasureSetHealer { ); let incarnation = self.admin_bucket_incarnation(checkpoint_manager, bucket).await?; - if let Some(task) = &self.admin_task { - let bucket_opts = HealOpts { - recursive: false, - remove: false, - ..self.heal_opts - }; - let result = match incarnation { - Some(expected) => { - self.storage.validate_bucket_incarnation(bucket, Some(expected)).await?; - self.storage - .heal_bucket_at_incarnation(bucket, expected, &bucket_opts) - .await? - } - None => self.storage.heal_bucket(bucket, &bucket_opts).await?, - }; - task.record_result_item(result).await; + let unresolved_delete_recovery = self.heal_admin_bucket_delete_recovery(bucket, incarnation).await?; + if unresolved_delete_recovery && !self.heal_opts.dry_run { + resume_manager.set_delete_recovery_rescan_pending(true).await?; } // 1. get bucket info @@ -1416,7 +1451,7 @@ impl ErasureSetHealer { state = "missing", "Erasure set bucket heal skipped because bucket is missing" ); - return Ok(()); + return Ok(unresolved_delete_recovery); } }; @@ -1937,7 +1972,7 @@ impl ErasureSetHealer { previous_page_last = page_last; } - Ok(()) + Ok(unresolved_delete_recovery) } /// initialize progress tracking @@ -2124,8 +2159,8 @@ mod tests { // classification must not conflate an offline drive (or unmet quorum) with // genuine object absence, or transient failures get recorded as "healed" and // permanently skipped. - use super::{EcstoreError, Error, HealObjectOutcome}; - use crate::heal::DiskError; + use super::{Error, HealObjectOutcome}; + use crate::heal::{DiskError, EcstoreError}; fn classify(err: EcstoreError) -> HealObjectOutcome { ErasureSetHealer::classify_heal_object_error(&Error::Storage(err)) @@ -2219,6 +2254,82 @@ mod tests { HealObjectOutcome::Failed )); } + + // backlog#2900: one classifier for every wrapper. These inputs used to + // disagree between the erasure-set healer and the task-level predicates. + #[test] + fn classify_disk_level_object_absence_as_absent() { + assert!(matches!( + ErasureSetHealer::classify_heal_object_error(&Error::Disk(DiskError::FileNotFound)), + HealObjectOutcome::Absent + )); + assert!(matches!( + ErasureSetHealer::classify_heal_object_error(&Error::Disk(DiskError::FileVersionNotFound)), + HealObjectOutcome::Absent + )); + } + + #[test] + fn classify_wrapped_transient_errors_as_transient() { + for error in [ + Error::Storage(EcstoreError::RemoteClientUnavailable("rebuilding client".into())), + Error::Storage(EcstoreError::Timeout), + Error::Disk(DiskError::DiskNotFound), + Error::Disk(DiskError::VolumeNotFound), + Error::Disk(DiskError::ErasureReadQuorum), + Error::Disk(DiskError::Io(std::io::Error::from(std::io::ErrorKind::WouldBlock))), + Error::TransientSkip { + message: "skipped".into(), + }, + Error::Storage(EcstoreError::Io(std::io::Error::other( + "heal rename incomplete: 1 of 2 targets committed", + ))), + ] { + assert!( + matches!(ErasureSetHealer::classify_heal_object_error(&error), HealObjectOutcome::Transient), + "{error:?}" + ); + } + } + + #[test] + fn classify_partial_not_found_with_lost_quorum_as_transient_not_absent() { + // One disk answering NotFound while the others time out surfaces as a + // quorum failure from storage; it is never an absence verdict. + for error in [ + Error::Storage(EcstoreError::ErasureReadQuorum), + Error::Storage(EcstoreError::InsufficientReadQuorum("bucket".into(), "object".into())), + Error::Disk(DiskError::ErasureReadQuorum), + Error::Disk(DiskError::Timeout), + ] { + assert!( + matches!(ErasureSetHealer::classify_heal_object_error(&error), HealObjectOutcome::Transient), + "{error:?}" + ); + } + } + + #[test] + fn classify_legacy_peer_display_not_found_as_absent() { + for message in ["File not found", "File version not found", "Object not found: bucket/object"] { + assert!( + matches!( + ErasureSetHealer::classify_heal_object_error(&Error::Other(message.into())), + HealObjectOutcome::Absent + ), + "{message:?}" + ); + } + for message in ["file not found", "Disk not found", "Volume not found", "boom"] { + assert!( + !matches!( + ErasureSetHealer::classify_heal_object_error(&Error::Other(message.into())), + HealObjectOutcome::Absent + ), + "{message:?}" + ); + } + } } #[cfg(test)] @@ -2390,6 +2501,8 @@ mod resume_loop_tests { #[derive(Default)] struct FakeStorage { admin_disk: Mutex>, + bucket_discovery_incomplete: AtomicBool, + bucket_discovery_scopes: Mutex>>, bucket_pages: Mutex>, incarnations: Mutex>, receipts: Mutex>>, @@ -2469,6 +2582,17 @@ mod resume_loop_tests { ..Default::default() })) } + async fn list_buckets_for_admin_heal( + &self, + selector: Option, + ) -> Result> { + self.bucket_discovery_scopes.lock().expect("discovery scopes").push(selector); + if self.bucket_discovery_incomplete.load(Ordering::SeqCst) { + return Err(Error::BucketDiscoveryIncomplete); + } + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { Ok(self .bucket_pages @@ -2796,7 +2920,7 @@ mod resume_loop_tests { state.start_time, ) .await; - (processed, successful, failed, skipped, result) + (processed, successful, failed, skipped, result.map(|_| ())) } #[tokio::test] @@ -3866,6 +3990,112 @@ mod resume_loop_tests { ); } + #[tokio::test] + async fn unresolved_delete_recovery_rechecks_coverage_only_after_settlement() { + let mut env = make_env().await; + env.healer.unresolved_delete_recovery = true; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false)], + next: None, + truncated: false, + }, + ); + + let error = env + .healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("unresolved recovery cannot publish set completion"); + assert!(matches!(error, Error::TransientSkip { .. })); + assert_eq!(env.storage.calls(), [("healthy".to_owned(), None)]); + assert!(!env.resume.get_state().await.completed); + assert!(CheckpointManager::has_checkpoint(&env.healer.disk, &env.task_id).await); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("pending-only retries remain unresolved"); + assert_eq!( + env.storage.calls(), + [("healthy".to_owned(), None)], + "pending-only retries do not repeat payload scans" + ); + + env.healer.unresolved_delete_recovery = false; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false), item("restored", None, false)], + next: None, + truncated: false, + }, + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("settlement invalidates coverage even if another worker already collected the transaction"); + assert_eq!(env.storage.calls(), [("healthy".to_owned(), None)]); + assert_eq!( + env.resume.get_state().await.retry_count, + 0, + "coverage invalidation does not spend failure retries" + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect("fresh coverage includes the restored object"); + assert!(env.resume.get_state().await.completed); + assert_eq!(env.storage.calls().iter().filter(|(object, _)| object == "restored").count(), 1); + assert_eq!(env.resume.get_state().await.processed_objects, 2); + } + + #[tokio::test] + async fn delete_recovery_coverage_rewind_survives_interrupted_checkpoint_reset() { + let mut env = make_env().await; + env.healer.unresolved_delete_recovery = true; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false)], + next: None, + truncated: false, + }, + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &env.resume, &env.checkpoint) + .await + .expect_err("create overlapping coverage"); + assert_eq!(env.checkpoint.get_checkpoint().await.current_bucket_index, 1); + env.resume + .restart_scan_after_delete_recovery() + .await + .expect("publish rewind before its companion checkpoint"); + + let (resume, checkpoint) = env + .healer + .initialize_resume_state(&env.task_id, "pool_0_set_0", &["b".to_owned()]) + .await + .expect("restart must finish the interrupted checkpoint rewind"); + assert!(!resume.get_state().await.delete_recovery_rescan_pending); + assert_eq!(checkpoint.get_checkpoint().await.current_bucket_index, 0); + env.healer.unresolved_delete_recovery = false; + env.storage.set_page( + None, + Page { + items: vec![item("healthy", None, false), item("restored", None, false)], + next: None, + truncated: false, + }, + ); + env.healer + .execute_heal_with_resume(&["b".to_owned()], "pool_0_set_0", &resume, &checkpoint) + .await + .expect("restarted coverage must visit newly visible objects"); + assert!(resume.get_state().await.completed); + assert_eq!(env.storage.calls().iter().filter(|(object, _)| object == "restored").count(), 1); + } + #[tokio::test] async fn test_transient_skip_pass_not_marked_completed_state_survives() { // The finalize block must NOT declare a clean completion when a pass had diff --git a/crates/heal/src/heal/erasure_healer/admin.rs b/crates/heal/src/heal/erasure_healer/admin.rs index 5c9a686cd..bb88cdded 100644 --- a/crates/heal/src/heal/erasure_healer/admin.rs +++ b/crates/heal/src/heal/erasure_healer/admin.rs @@ -51,9 +51,10 @@ impl ErasureSetHealer { } let resolved; let buckets = if buckets.is_empty() { + let (pool_idx, set_idx) = crate::heal::utils::parse_set_disk_id(set_disk_id)?; resolved = self .storage - .list_buckets() + .list_buckets_for_admin_heal(Some(crate::heal::storage_api::storage::DiskSetSelector::new(pool_idx, set_idx))) .await? .into_iter() .map(|bucket| bucket.name) diff --git a/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs b/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs index e893f8eca..30a51fbe7 100644 --- a/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs +++ b/crates/heal/src/heal/erasure_healer/resume_loop_tests/admin_outcome.rs @@ -483,3 +483,50 @@ async fn admin_erasure_checkpoint_publication_cannot_undo_acknowledged_cancel() assert_eq!(outcome.coverage, HealTraversalCoverage::Partial); assert_eq!((outcome.counters.processed, outcome.counters.healed), (1, 1)); } + +#[tokio::test] +async fn admin_bucket_discovery_rejects_incomplete_checkpoint_and_freezes_complete_empty_scope() { + let env = make_env().await; + env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst); + let result = env + .healer + .prepare_admin_checkpoint(&env.checkpoint, &[], "pool_0_set_0") + .await; + assert!(matches!(result, Err(Error::BucketDiscoveryIncomplete))); + assert!(env.checkpoint.get_checkpoint().await.admin.is_none()); + let reloaded = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id) + .await + .expect("reload checkpoint"); + assert!(reloaded.get_checkpoint().await.admin.is_none(), "incomplete scope must not be persisted"); + assert_eq!( + env.storage.bucket_discovery_scopes.lock().expect("scopes").as_slice(), + &[Some(crate::heal::storage_api::storage::DiskSetSelector::new(0, 0))] + ); + + env.storage.bucket_discovery_incomplete.store(false, Ordering::SeqCst); + assert!( + env.healer + .prepare_admin_checkpoint(&env.checkpoint, &[], "pool_0_set_0") + .await + .expect("complete empty scope") + .is_empty() + ); + env.storage.bucket_discovery_incomplete.store(true, Ordering::SeqCst); + objects(&env.storage, "created-after-admission", &["object"]); + let reloaded = CheckpointManager::load_from_disk(env.healer.disk.clone(), &env.task_id) + .await + .expect("reload frozen scope"); + assert!( + env.healer + .prepare_admin_checkpoint(&reloaded, &[], "pool_0_set_0") + .await + .expect("resume frozen scope") + .is_empty() + ); + assert_eq!( + env.storage.bucket_discovery_scopes.lock().expect("scopes").len(), + 2, + "resume must not rediscover or expand scope" + ); + assert!(env.storage.calls().is_empty()); +} diff --git a/crates/heal/src/heal/manager.rs b/crates/heal/src/heal/manager.rs index cd091d8ce..dd17e7c87 100644 --- a/crates/heal/src/heal/manager.rs +++ b/crates/heal/src/heal/manager.rs @@ -16,8 +16,8 @@ use crate::heal::{ outcome::HealTaskOutcome, progress::{HealProgress, HealStatistics}, resume::{ - ReplacementPhase, ReplacementRecoveryCandidate, ResumeGc, ResumeManager, ResumeState, ResumeUtils, - merge_replacement_recovery_candidate, replacement_retry_is_exhausted_active, + ReplacementPhase, ReplacementRecoveryBlock, ReplacementRecoveryCandidate, ResumeGc, ResumeManager, ResumeState, + ResumeUtils, merge_replacement_recovery_candidate, replacement_retry_is_exhausted_active, }, storage::HealStorageAPI, task::{HealOptions, HealPriority, HealRequest, HealTask, HealTaskStatus, HealType, demote_to_debug_when}, @@ -988,8 +988,9 @@ pub struct HealManager { /// This is rebuilt from durable state after restart and never crosses the /// public request boundary. replacement_recovery_anchors: Arc>>, - /// Set IDs whose durable replacement metadata is corrupt or conflicting. - replacement_recovery_blocked_sets: Arc>>, + /// Durable replacement generations that discovery could not verify or + /// resolve, keyed by generation and re-evaluated on every pass. + replacement_recovery_blocks: ReplacementRecoveryBlocks, /// Durable handoff of interrupted administrator root traversals. root_recovery: Arc, /// Serialize admin starts and forceStart replacement with shutdown. @@ -1007,6 +1008,9 @@ pub struct HealManager { /// Bounded, low-cardinality admission telemetry exposed through the /// existing operations snapshot for cluster E2E assertions. admission_telemetry: Arc>, + /// Node-local memory of terminal per-object verdicts, consulted before + /// queue admission (backlog#2900). Not a receipt; empty after a restart. + disposition_cache: Arc>, } /// Where a task-id lookup resolved. The variants carry the resolved state @@ -1017,6 +1021,9 @@ enum TaskStateLookup { Retrying(HealTaskStatus, HealOptions), Completed(Arc), Queued(HealOptions), + /// A durable root heal record that cannot be admitted or quarantined; + /// it answers status queries with the reason until an operator repairs it. + Unrunnable(String), NotFound, } @@ -1036,6 +1043,7 @@ struct HealQueueContext<'a> { notify: &'a Arc, cancel_token: &'a CancellationToken, workload_provider: &'a Option, + disposition_cache: &'a Arc>, } impl HealManager { @@ -1059,6 +1067,37 @@ impl HealManager { } } + /// Requests answered from the disposition memory: best-effort object + /// heals. Administrator starts, forced starts, high and urgent priorities, + /// dry runs and durable MRF repairs always reach the queue. + fn disposition_gate(request: &HealRequest, mrf_notice_target: Option<&MrfRepairNoticeTarget>) -> Option { + let HealType::Object { + bucket, + object, + version_id, + } = &request.heal_type + else { + return None; + }; + let durable_mrf = request.expected_mrf_bucket_incarnation_id.is_some() + || mrf_notice_target.is_some_and(|target| target.kind.is_durable()); + if request.force_start + || request.source == HealRequestSource::Admin + || request.priority > HealPriority::Normal + || request.options.dry_run + || durable_mrf + { + return None; + } + Some(HealDispositionKey::new( + bucket, + object, + version_id.as_deref(), + request.options.pool_index, + request.options.set_index, + )) + } + fn can_displace_queued_work(request: &HealRequest) -> bool { matches!(request.source, HealRequestSource::Admin | HealRequestSource::Internal) } @@ -1450,28 +1489,33 @@ impl HealManager { remove_task_aliases_for_task(&self.task_aliases, task_id).await; } - fn block_replacement_recovery_set(&self, set_disk_id: &str) { - self.replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .insert(set_disk_id.to_string()); + /// Durable replacement generations this node could not verify or + /// resolve, with the reason and observation times, for the admin status + /// surface. Entries are node-local and re-evaluated by every discovery pass. + pub fn replacement_recovery_blocks(&self) -> Vec { + self.replacement_recovery_blocks.snapshot() } - fn replacement_recovery_set_is_blocked(&self, set_disk_id: &str) -> bool { - self.replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .contains(set_disk_id) + #[cfg(test)] + pub(super) fn replacement_recovery_generation_is_blocked(&self, task_id: &str) -> bool { + self.replacement_recovery_blocks.is_blocked(task_id) } - async fn validate_replacement_recovery_records(disk: &crate::heal::DiskStore) -> Result<()> { - if disk.endpoint().is_local { - ResumeUtils::migrate_legacy_replacement_records(disk).await?; - } - for task_id in ResumeUtils::get_replacement_intent_tasks(disk).await? { - ResumeManager::load_replacement_intent(disk.clone(), &task_id).await?; - } - Ok(()) + #[cfg(test)] + pub(super) fn block_replacement_recovery_generation_for_test( + &self, + task_id: &str, + set_disk_id: &str, + reason: &str, + reserves_targets: bool, + ) { + self.replacement_recovery_blocks + .block(task_id, set_disk_id, reason, reserves_targets); + } + + #[cfg(test)] + pub(super) fn replacement_recovery_set_has_reserving_block(&self, set_disk_id: &str) -> bool { + self.replacement_recovery_blocks.set_has_reserving_block(set_disk_id) } /// Start the bounded resume-state inspector. Destructive GC remains @@ -1594,7 +1638,7 @@ impl HealManager { retrying_heals: Arc::new(Mutex::new(HashMap::new())), mrf_repair_notice_targets: Arc::new(StdMutex::new(HashMap::new())), replacement_recovery_anchors: Arc::new(std::sync::Mutex::new(HashMap::new())), - replacement_recovery_blocked_sets: Arc::new(std::sync::Mutex::new(HashSet::new())), + replacement_recovery_blocks: ReplacementRecoveryBlocks::default(), root_recovery, admin_start_shutdown: Mutex::new(()), storage, @@ -1603,6 +1647,7 @@ impl HealManager { notify: Arc::new(Notify::new()), workload_provider, admission_telemetry: Arc::new(StdMutex::new(HealAdmissionTelemetry::default())), + disposition_cache: Arc::new(StdMutex::new(HealDispositionCache::default())), } } @@ -1937,17 +1982,17 @@ impl HealManager { }; // Decode all durable responsibilities before forceStart has side effects. // Do not hold runtime state locks while scanning recovery records. - let mut quarantined_task_ids = HashSet::new(); + let mut record_issue = None; let mut durable_owners = if source == HealRequestSource::Admin { let inventory = self.root_recovery.inventory().await?; - quarantined_task_ids = inventory.quarantined_task_ids; + record_issue = inventory.record_issue(&request.id); inventory.requests } else { Vec::new() }; - if quarantined_task_ids.contains(&request.id) { + if let Some(reason) = record_issue { return Err(Error::Other(format!( - "Root heal recovery task {} is quarantined and requires operator repair", + "Root heal recovery task {} requires operator repair: {reason}", request.id ))); } @@ -2250,6 +2295,50 @@ impl HealManager { }); } + // A best-effort resubmission of an object whose last task reached a + // terminal verdict is answered from the node-local disposition memory + // before it can occupy the queue again (backlog#2900). + if let Some(key) = Self::disposition_gate(&request, mrf_notice_target.as_ref()) { + let verdict = lock_disposition_cache(&self.disposition_cache).consult(&key, tokio::time::Instant::now()); + if let Some((disposition, recorded_task_id, hits, retry_after)) = verdict.rejection() { + let recorded_task_id = recorded_task_id.to_owned(); + drop(retrying_heals); + drop(queue); + drop(active_heals); + let lock_phase = lock_phase_start.elapsed(); + let admission = HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped); + Self::record_admission_metric(source, admission, "disposition_cache"); + self.record_admission_observation(HealAdmissionObservation { + source, + result: admission, + context: "disposition_cache", + force_start, + displaced: false, + start_duration: admission_start.elapsed(), + lock_phase, + }); + debug!( + target: "rustfs::heal::manager", + event = EVENT_HEAL_QUEUE_ADMISSION, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_MANAGER, + request_id = %request.id, + priority = ?request.priority, + source = source.as_str(), + disposition = disposition.as_str(), + disposition_task_id = %recorded_task_id, + disposition_hits = hits, + retry_after_ms = retry_after.map(|delay| u64::try_from(delay.as_millis()).unwrap_or(u64::MAX)), + result = "dropped_by_disposition", + "Heal queue admission answered from the disposition memory" + ); + return Ok(HealAdmissionReceipt { + result: admission, + task_id: recorded_task_id, + }); + } + } + let durable_handoff = root_recovery::is_admin_heal_recovery(&request.heal_type, request.source); let durable_handoff_required = durable_handoff && (request.force_start @@ -2404,10 +2493,15 @@ impl HealManager { terminal_completed = Some(Arc::new(completed)); } - Ok(match terminal_completed { - Some(completed) => TaskStateLookup::Completed(completed), - None => TaskStateLookup::NotFound, - }) + if let Some(completed) = terminal_completed { + return Ok(TaskStateLookup::Completed(completed)); + } + // A durable root record that can neither run nor be quarantined still + // answers status queries with its reason. + if let Some(reason) = self.root_recovery.record_issue(canonical_task_id).await { + return Ok(TaskStateLookup::Unrunnable(reason)); + } + Ok(TaskStateLookup::NotFound) } async fn publish_admin_terminal( @@ -2469,6 +2563,7 @@ impl HealManager { TaskStateLookup::Retrying(status, _) => Ok(status), TaskStateLookup::Completed(completed) => Ok(completed.status.clone()), TaskStateLookup::Queued(_) => Ok(HealTaskStatus::Pending), + TaskStateLookup::Unrunnable(error) => Ok(HealTaskStatus::Failed { error }), TaskStateLookup::NotFound => Err(Error::TaskNotFound { task_id: task_id.to_string(), }), @@ -2489,6 +2584,9 @@ impl HealManager { TaskStateLookup::Retrying(status, options) => Ok(empty_task_report_with_options(status, options)), TaskStateLookup::Completed(completed) => Ok(completed_task_report(&completed, since)), TaskStateLookup::Queued(options) => Ok(empty_task_report_with_options(HealTaskStatus::Pending, options)), + TaskStateLookup::Unrunnable(error) => { + Ok(empty_task_report_with_options(HealTaskStatus::Failed { error }, HealOptions::default())) + } TaskStateLookup::NotFound => Err(Error::TaskNotFound { task_id: task_id.to_string(), }), @@ -2512,6 +2610,9 @@ impl HealManager { TaskStateLookup::Retrying(status, options) => Ok(empty_task_report_with_options(status, options)), TaskStateLookup::Completed(completed) => Ok(completed_task_report(&completed, since)), TaskStateLookup::Queued(options) => Ok(empty_task_report_with_options(HealTaskStatus::Pending, options)), + TaskStateLookup::Unrunnable(error) => { + Ok(empty_task_report_with_options(HealTaskStatus::Failed { error }, HealOptions::default())) + } TaskStateLookup::NotFound => { if self.path_has_task(heal_path).await { return Err(Error::InvalidClientToken); @@ -2535,6 +2636,7 @@ impl HealManager { TaskStateLookup::Retrying(status, _) => Ok(status), TaskStateLookup::Completed(completed) => Ok(completed.status.clone()), TaskStateLookup::Queued(_) => Ok(HealTaskStatus::Pending), + TaskStateLookup::Unrunnable(error) => Ok(HealTaskStatus::Failed { error }), TaskStateLookup::NotFound => { if self.path_has_task(heal_path).await { return Err(Error::InvalidClientToken); @@ -2950,11 +3052,13 @@ impl std::fmt::Debug for HealManager { } mod auto_scan; +mod disposition; mod queue; pub(crate) mod root_recovery; mod scheduler; mod unclean_shutdown; +use disposition::{HealDispositionCache, HealDispositionKey, lock_disposition_cache}; use queue::*; use scheduler::*; use unclean_shutdown::*; diff --git a/crates/heal/src/heal/manager/auto_scan.rs b/crates/heal/src/heal/manager/auto_scan.rs index b09f337db..563f3f8b8 100644 --- a/crates/heal/src/heal/manager/auto_scan.rs +++ b/crates/heal/src/heal/manager/auto_scan.rs @@ -13,6 +13,18 @@ // limitations under the License. /// The automatic disk scanner: replacement discovery and unformatted-disk enqueue. use super::*; +use crate::heal::DiskStore; +use std::collections::BTreeMap; + +pub(super) fn extend_replacement_recovery_disks( + inventory: &mut BTreeMap, + disks: impl IntoIterator, +) { + for disk in disks { + // Preserve the local snapshot when later inventory repeats its endpoint. + inventory.entry(disk.endpoint().to_string()).or_insert(disk); + } +} impl HealManager { /// Start background task to auto scan local disks and enqueue erasure set heal requests @@ -25,7 +37,7 @@ impl HealManager { let mrf_repair_notice_targets = self.mrf_repair_notice_targets.clone(); let storage = self.storage.clone(); let replacement_recovery_anchors = self.replacement_recovery_anchors.clone(); - let replacement_recovery_blocked_sets = self.replacement_recovery_blocked_sets.clone(); + let blocks = self.replacement_recovery_blocks.clone(); let cancel_token = self.cancel_token.clone(); let notify = self.notify.clone(); let mut duration = { @@ -47,12 +59,6 @@ impl HealManager { tokio::spawn(async move { let mut interval = interval(duration); - let block_recovery_set = |set_disk_id: &str| { - replacement_recovery_blocked_sets - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()) - .insert(set_disk_id.to_string()); - }; loop { let mut candidate_count = 0usize; @@ -87,73 +93,32 @@ impl HealManager { let local_disk_map = local_disk_map_read().await; local_disk_map.values().flatten().cloned().collect::>() }; - let local_endpoints = local_disks.iter().map(|disk| disk.endpoint()).collect::>(); - let blocked_sets = replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .clone(); - let mut retry_recovery_disks = Vec::new(); - if !blocked_sets.is_empty() { - let mut retry_succeeded = HashSet::new(); - let mut retry_failed = HashSet::new(); - for set_disk_id in blocked_sets { - let disks = match storage.replacement_intent_disks(&set_disk_id).await { - Ok(disks) => disks, - Err(error) => { - retry_failed.insert(set_disk_id.clone()); - conflicted_recovery_sets.insert(set_disk_id.clone()); - warn!( - target: "rustfs::heal::manager", - event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_DISK_SCANNER, - set_disk_id, - error = %error, - "Replacement recovery disk retry failed" - ); - continue; - } - }; - let mut valid = true; - for disk in &disks { - if let Err(error) = Self::validate_replacement_recovery_records(disk).await { - valid = false; - warn!( - target: "rustfs::heal::manager", - event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_DISK_SCANNER, - set_disk_id, - endpoint = %disk.endpoint(), - error = %error, - "Replacement recovery retry failed" - ); - } - } - if valid { - retry_succeeded.insert(set_disk_id); - retry_recovery_disks.extend(disks); - } else { - retry_failed.insert(set_disk_id.clone()); - conflicted_recovery_sets.insert(set_disk_id); - } - } - let mut blocked = replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned"); - unblock_replacement_recovery_sets_after_validation(&mut blocked, retry_succeeded, &retry_failed); + let mut local_endpoints = HashMap::with_capacity(local_disks.len()); + for disk in &local_disks { + let endpoint = disk.endpoint(); + local_endpoints.entry(endpoint.to_string()).or_insert(endpoint); } + // Formatting removes a target from blank-disk discovery, but its + // unfinished intent may still live on a remote survivor. + let recovery_sets = local_disks + .iter() + .filter_map(|disk| { + let endpoint = disk.endpoint(); + crate::heal::utils::format_set_disk_id_from_i32(endpoint.pool_idx, endpoint.set_idx) + }) + .collect::>(); for disk in &local_disks { let endpoint = disk.endpoint(); let runtime_state = disk.runtime_state(); let set_disk_id = crate::heal::utils::format_set_disk_id_from_i32(endpoint.pool_idx, endpoint.set_idx); - if set_disk_id.as_ref().is_some_and(|set_disk_id| { - replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .contains(set_disk_id) - }) { + // A blocked generation that still reserves this + // set's slots defers fresh admission for the set; + // discovery below keeps re-evaluating that record. + if set_disk_id + .as_ref() + .is_some_and(|set_disk_id| blocks.set_has_reserving_block(set_disk_id)) + { skipped_invalid_count += 1; debug!( target: "rustfs::heal::manager", @@ -163,7 +128,7 @@ impl HealManager { endpoint = %endpoint, set_disk_id = set_disk_id.as_deref().unwrap_or_default(), disk_state = "replacement_recovery_blocked", - "Heal auto-scan replacement deferred because durable recovery is blocked" + "Heal auto-scan replacement deferred because a durable generation is blocked" ); continue; } @@ -235,11 +200,11 @@ impl HealManager { } } - let mut recovery_disks = local_disks.clone(); - recovery_disks.extend(retry_recovery_disks); - for set_disk_id in endpoints.keys() { + let mut recovery_disks = BTreeMap::new(); + extend_replacement_recovery_disks(&mut recovery_disks, local_disks.iter().cloned()); + for set_disk_id in &recovery_sets { match storage.replacement_intent_disks(set_disk_id).await { - Ok(disks) => recovery_disks.extend(disks), + Ok(disks) => extend_replacement_recovery_disks(&mut recovery_disks, disks), Err(error) => { conflicted_recovery_sets.insert(set_disk_id.clone()); warn!( @@ -254,10 +219,6 @@ impl HealManager { } } } - recovery_disks.sort_by_key(|disk| disk.endpoint().to_string()); - recovery_disks - .dedup_by(|left, right| left.endpoint().to_string() == right.endpoint().to_string()); - // Once formatting succeeds a replacement is no longer // discoverable as UnformattedDisk. Re-admit exactly one // incomplete durable generation per set within its @@ -265,7 +226,7 @@ impl HealManager { // verified terminal cleanup. Multiple generations are a // durable conflict: leave every marker/state intact and // require reconciliation rather than choosing one. - for disk in &recovery_disks { + for (anchor, disk) in &recovery_disks { let endpoint = disk.endpoint(); let disk_set_disk_id = crate::heal::utils::format_set_disk_id_from_i32(endpoint.pool_idx, endpoint.set_idx); @@ -309,9 +270,13 @@ impl HealManager { let resume_manager = match ResumeManager::load_replacement_intent(disk.clone(), &task_id).await { Ok(resume_manager) => resume_manager, Err(error) => { - if let Some(set_disk_id) = &disk_set_disk_id { - conflicted_recovery_sets.insert(set_disk_id.clone()); - } + conflicted_recovery_tasks.insert(task_id.clone()); + blocks.block( + &task_id, + disk_set_disk_id.as_deref().unwrap_or_default(), + format!("durable intent could not be loaded: {error}"), + true, + ); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -326,15 +291,19 @@ impl HealManager { } }; let state = resume_manager.get_state().await; + let reserves_targets = replacement_reserves_targets_on(&state, &local_disks).await; if disk_set_disk_id.as_deref().is_some_and(|disk_set| disk_set != state.set_disk_id) { let disk_set = disk_set_disk_id.as_deref().unwrap_or_default(); conflicted_recovery_tasks.insert(task_id.clone()); - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - conflicted_recovery_sets.insert(disk_set.to_string()); - block_recovery_set(&state.set_disk_id); - if !disk_set.is_empty() { - block_recovery_set(disk_set); - } + blocks.block( + &task_id, + &state.set_disk_id, + format!( + "durable intent binds to {} but was found on a disk of {disk_set}", + state.set_disk_id + ), + reserves_targets, + ); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -351,7 +320,7 @@ impl HealManager { // A remote survivor may own the durable intent, but only the // node mounting every replacement target may advance or replay it. if state.replacement_targets.iter().any(|target| { - !local_endpoints.iter().any(|endpoint| endpoint.to_string() == *target) + !local_endpoints.contains_key(target) }) { deferred_recovery_sets.insert(state.set_disk_id.clone()); continue; @@ -360,23 +329,16 @@ impl HealManager { continue; } let mut selected = durable_recovery_candidates.remove(&task_id); - let previous_set = selected.as_ref().map(|candidate| candidate.state.set_disk_id.clone()); + let candidate_set = state.set_disk_id.clone(); let candidate = match ReplacementRecoveryCandidate::new( - state.clone(), - disk.endpoint().to_string(), - disk.endpoint().is_local, + state, + anchor.clone(), + endpoint.is_local, ) { Ok(candidate) => candidate, Err(error) => { conflicted_recovery_tasks.insert(task_id.clone()); - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - if let Some(previous_set) = &previous_set { - conflicted_recovery_sets.insert(previous_set.clone()); - } - block_recovery_set(&state.set_disk_id); - if let Some(previous_set) = &previous_set { - block_recovery_set(previous_set); - } + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -390,17 +352,9 @@ impl HealManager { continue; } }; - let task_set = candidate.state.set_disk_id.clone(); if let Err(error) = merge_replacement_recovery_candidate(&mut selected, candidate) { conflicted_recovery_tasks.insert(task_id.clone()); - conflicted_recovery_sets.insert(task_set.clone()); - if let Some(previous_set) = &previous_set { - conflicted_recovery_sets.insert(previous_set.clone()); - } - block_recovery_set(&task_set); - if let Some(previous_set) = &previous_set { - block_recovery_set(previous_set); - } + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -432,20 +386,20 @@ impl HealManager { { continue; } - let Some(anchor_disk) = local_disks - .iter() - .chain(recovery_disks.iter()) - .find(|disk| disk.endpoint().to_string() == candidate.anchor) - .cloned() - else { + // Every copy merged cleanly: an earlier block for + // this generation no longer applies. Later + // failures re-block it. + blocks.unblock(&task_id); + let candidate_set = candidate.state.set_disk_id.clone(); + let reserves_targets = replacement_reserves_targets_on(&candidate.state, &local_disks).await; + let Some(anchor_disk) = recovery_disks.get(&candidate.anchor).cloned() else { deferred_recovery_sets.insert(candidate.state.set_disk_id.clone()); continue; }; let resume_manager = match ResumeManager::load_replacement_intent(anchor_disk, &task_id).await { Ok(resume_manager) => resume_manager, Err(error) => { - conflicted_recovery_sets.insert(candidate.state.set_disk_id.clone()); - block_recovery_set(&candidate.state.set_disk_id); + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -477,15 +431,21 @@ impl HealManager { // compatibility transient-skip marker // remains bounded; do not turn it into // a fresh retry. - if durable_replacement_reserves_targets(&state) { - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - block_recovery_set(&state.set_disk_id); + if reserves_targets { + blocks.block( + &task_id, + &candidate_set, + state + .error_message + .clone() + .unwrap_or_else(|| "replacement retry budget exhausted".to_string()), + true, + ); } continue; } Err(error) => { - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - block_recovery_set(&state.set_disk_id); + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -500,9 +460,16 @@ impl HealManager { } } if !durable_replacement_recovery_is_due(&state, &task_id) { - if durable_replacement_reserves_targets(&state) { - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - block_recovery_set(&state.set_disk_id); + if reserves_targets { + blocks.block( + &task_id, + &candidate_set, + state + .error_message + .clone() + .unwrap_or_else(|| "durable generation reserves its targets but cannot run".to_string()), + true, + ); } continue; } @@ -513,8 +480,7 @@ impl HealManager { continue; } Err(error) => { - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - block_recovery_set(&state.set_disk_id); + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_AUTO_SCAN_ENQUEUE, @@ -530,32 +496,26 @@ impl HealManager { let targets = state .replacement_targets .iter() - .filter_map(|target| { - local_endpoints - .iter() - .find(|endpoint| endpoint.to_string() == *target) - .cloned() - }) + .filter_map(|target| local_endpoints.get(target).cloned()) .collect::>(); if targets.len() != state.replacement_targets.len() { deferred_recovery_sets.insert(state.set_disk_id.clone()); continue; } - let Some(first_target) = targets.first() else { - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - block_recovery_set(&state.set_disk_id); - continue; - }; // A pending handoff may resolve the predecessor to // its durable successor. Queue the resolved owner, // not the stale filename that initiated replay. let resolved_task_id = state.task_id.clone(); + blocks.unblock(&resolved_task_id); + let Some(first_target) = targets.first() else { + blocks.block(&resolved_task_id, &candidate_set, "resolved generation has no local target", true); + continue; + }; let Some(set_disk_id) = crate::heal::utils::format_set_disk_id_from_i32( first_target.pool_idx, first_target.set_idx, ) else { - conflicted_recovery_sets.insert(state.set_disk_id.clone()); - block_recovery_set(&state.set_disk_id); + blocks.block(&resolved_task_id, &candidate_set, "resolved target has no erasure-set slot", true); continue; }; if state.set_disk_id != set_disk_id @@ -565,8 +525,12 @@ impl HealManager { != Some(set_disk_id.as_str()) }) { - block_recovery_set(&set_disk_id); - conflicted_recovery_sets.insert(set_disk_id); + blocks.block( + &resolved_task_id, + &candidate_set, + "resolved targets belong to a different erasure set than the generation", + true, + ); continue; } let resume_endpoint = candidate.anchor; @@ -575,7 +539,12 @@ impl HealManager { if existing_task_id != &resolved_task_id || existing_targets != &targets || existing_buckets != &state.replacement_buckets => { - block_recovery_set(&set_disk_id); + // Two active generations resolved for one + // set: block each on its own and leave the + // set alone for this pass. + let reason = "multiple active generations resolved for the same set"; + blocks.block(existing_task_id, &set_disk_id, reason, true); + blocks.block(&resolved_task_id, &set_disk_id, reason, true); conflicted_recovery_sets.insert(set_disk_id); } Some(_) => {} @@ -607,20 +576,9 @@ impl HealManager { } for (set_disk_id, (_, targets, _, _)) in &durable_recoveries { - let expected = targets.iter().map(ToString::to_string).collect::>(); - let observed = endpoints - .get(set_disk_id) - .map(|endpoints| endpoints.iter().map(ToString::to_string).collect::>()) - .unwrap_or_default(); - if !observed.is_subset(&expected) { - replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .insert(set_disk_id.clone()); - conflicted_recovery_sets.insert(set_disk_id.clone()); - continue; - } - endpoints.entry(set_disk_id.clone()).or_default().extend(targets.clone()); + // The durable owner must finish its exact target scope before + // newly discovered replacements can start another generation. + endpoints.insert(set_disk_id.clone(), targets.clone()); } for set_disk_id in &conflicted_recovery_sets { durable_recoveries.remove(set_disk_id); @@ -647,11 +605,11 @@ impl HealManager { // Admit one set task with every ready replacement target. Queue deduplication is // set-scoped, so admitting endpoints independently would silently drop later targets. for (set_disk_id, endpoints) in endpoints { - if replacement_recovery_blocked_sets - .lock() - .expect("replacement recovery blocked set lock poisoned") - .contains(&set_disk_id) - { + // A resolved durable generation replays within its + // own target scope and is not held back by another + // blocked generation of the set. Only fresh + // admission waits for a blocked reservation. + if !durable_recoveries.contains_key(&set_disk_id) && blocks.set_has_reserving_block(&set_disk_id) { skipped_invalid_count += 1; debug!( target: "rustfs::heal::manager", @@ -660,7 +618,7 @@ impl HealManager { subsystem = LOG_SUBSYSTEM_DISK_SCANNER, set_disk_id, result = "replacement_recovery_blocked", - "Heal auto-scan replacement admission deferred because durable recovery is blocked" + "Heal auto-scan replacement admission deferred because a durable generation is blocked" ); continue; } diff --git a/crates/heal/src/heal/manager/disposition.rs b/crates/heal/src/heal/manager/disposition.rs new file mode 100644 index 000000000..142b1d91f --- /dev/null +++ b/crates/heal/src/heal/manager/disposition.rs @@ -0,0 +1,475 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Node-local memory of terminal per-object heal dispositions. +//! +//! The queue releases its dedup key when a request leaves the queue, so a +//! best-effort producer (scanner, read repair, autoheal, MRF hints) can +//! resubmit the same object as soon as its task finished, and a task that +//! found nothing to repair is forgotten immediately. This cache remembers the +//! last terminal verdict per `(bucket, object, version, pool, set)` so admission can +//! answer a resubmission without queueing it again: +//! +//! - `Absent`: a locked quorum read classified the object as absent +//! ([`crate::error::HealErrorClass::Absent`]). Resubmissions are dropped +//! until the entry expires. +//! - `RetryExhausted` / `Permanent`: the task ended in a terminal failure. +//! Resubmissions are admitted again on an exponential schedule that is +//! capped at one hour, so a failing object keeps being retried but no longer +//! on every producer tick. +//! +//! The cache is bounded (least recently used eviction) and every entry +//! expires. It is process memory: a restart starts empty, and nothing here is +//! a receipt. It never releases a durable responsibility; durable MRF repairs +//! bypass it entirely, as do administrator starts, forced starts, high and +//! urgent priorities, and dry runs. + +use crate::error::{HealErrorClass, classify_heal_error}; +use crate::heal::outcome::HealObjectDisposition; +use crate::heal::task::{HealTask, HealType}; +use metrics::{counter, gauge}; +use std::collections::{HashMap, VecDeque}; +use std::sync::{Mutex as StdMutex, MutexGuard as StdMutexGuard}; +use std::time::Duration; +use tokio::time::Instant; + +/// Default upper bound on retained entries. +pub(crate) const DEFAULT_DISPOSITION_CACHE_CAPACITY: usize = 100_000; +/// Default lifetime of an entry after its last record. +pub(crate) const DEFAULT_DISPOSITION_CACHE_TTL: Duration = Duration::from_secs(24 * 60 * 60); +/// First re-admission delay after a terminal failure; doubles per repeat. +const RETRY_BACKOFF_BASE: Duration = Duration::from_secs(60); +/// Upper bound of the re-admission delay. +const RETRY_BACKOFF_MAX: Duration = Duration::from_secs(60 * 60); + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum HealDispositionKind { + Absent, + RetryExhausted, + Permanent, +} + +impl HealDispositionKind { + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::Absent => "absent", + Self::RetryExhausted => "retry_exhausted", + Self::Permanent => "permanent", + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub(crate) struct HealDispositionKey { + pub(crate) bucket: String, + pub(crate) object: String, + /// `None` is the unversioned/latest target; an empty wire string is the same thing. + pub(crate) version_id: Option, + pub(crate) pool_index: Option, + pub(crate) set_index: Option, +} + +impl HealDispositionKey { + pub(crate) fn new( + bucket: &str, + object: &str, + version_id: Option<&str>, + pool_index: Option, + set_index: Option, + ) -> Self { + Self { + bucket: bucket.to_owned(), + object: object.to_owned(), + version_id: version_id.filter(|version| !version.is_empty()).map(ToOwned::to_owned), + pool_index, + set_index, + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct HealDispositionEntry { + pub(crate) kind: HealDispositionKind, + /// Consecutive records of the same kind for this key. + pub(crate) count: u32, + pub(crate) recorded_at: Instant, + /// Task that produced the latest record; returned to producers as the receipt pointer. + pub(crate) task_id: String, + sequence: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum HealDispositionVerdict { + Admit, + /// The last locked quorum read found the object absent and the memory is still fresh. + Absent { + task_id: String, + count: u32, + }, + /// The last task failed terminally; re-admission waits for the backoff to elapse. + BackedOff { + kind: HealDispositionKind, + task_id: String, + count: u32, + retry_after: Duration, + }, +} + +impl HealDispositionVerdict { + pub(crate) fn rejection(&self) -> Option<(HealDispositionKind, &str, u32, Option)> { + match self { + Self::Admit => None, + Self::Absent { task_id, count } => Some((HealDispositionKind::Absent, task_id, *count, None)), + Self::BackedOff { + kind, + task_id, + count, + retry_after, + } => Some((*kind, task_id, *count, Some(*retry_after))), + } + } +} + +pub(crate) struct HealDispositionCache { + capacity: usize, + ttl: Duration, + entries: HashMap, + /// Recency order. A key can appear several times; only the pair whose + /// sequence matches the entry is live, older pairs are skipped on eviction. + recency: VecDeque<(u64, HealDispositionKey)>, + next_sequence: u64, +} + +impl Default for HealDispositionCache { + fn default() -> Self { + Self::with_limits(DEFAULT_DISPOSITION_CACHE_CAPACITY, DEFAULT_DISPOSITION_CACHE_TTL) + } +} + +impl HealDispositionCache { + pub(crate) fn with_limits(capacity: usize, ttl: Duration) -> Self { + Self { + capacity: capacity.max(1), + ttl, + entries: HashMap::new(), + recency: VecDeque::new(), + next_sequence: 0, + } + } + + #[cfg(test)] + pub(crate) fn len(&self) -> usize { + self.entries.len() + } + + #[cfg(test)] + pub(crate) fn get(&self, key: &HealDispositionKey) -> Option<&HealDispositionEntry> { + self.entries.get(key) + } + + /// Delay before a key with `count` consecutive terminal failures is admitted again. + pub(crate) fn retry_backoff(count: u32) -> Duration { + let exponent = count.saturating_sub(1).min(31); + RETRY_BACKOFF_BASE.saturating_mul(1_u32 << exponent).min(RETRY_BACKOFF_MAX) + } + + /// Decide whether a resubmission for `key` may enter the queue at `now`. + pub(crate) fn consult(&mut self, key: &HealDispositionKey, now: Instant) -> HealDispositionVerdict { + let Some(entry) = self.entries.get(key) else { + return HealDispositionVerdict::Admit; + }; + let age = now.saturating_duration_since(entry.recorded_at); + if age >= self.ttl { + self.entries.remove(key); + self.publish_entries(); + return HealDispositionVerdict::Admit; + } + let verdict = match entry.kind { + HealDispositionKind::Absent => HealDispositionVerdict::Absent { + task_id: entry.task_id.clone(), + count: entry.count, + }, + kind @ (HealDispositionKind::RetryExhausted | HealDispositionKind::Permanent) => { + let backoff = Self::retry_backoff(entry.count); + if age >= backoff { + return HealDispositionVerdict::Admit; + } + HealDispositionVerdict::BackedOff { + kind, + task_id: entry.task_id.clone(), + count: entry.count, + retry_after: backoff.saturating_sub(age), + } + } + }; + counter!("rustfs_heal_disposition_cache_hits_total", "disposition" => entry.kind.as_str()).increment(1); + self.touch(key); + verdict + } + + /// Remember the terminal verdict of `task_id` for `key`. + /// + /// A repeat of the same kind increments `count`; a different kind starts + /// a fresh count, because the backoff schedule belongs to one failure mode. + pub(crate) fn record(&mut self, key: HealDispositionKey, kind: HealDispositionKind, task_id: &str, now: Instant) { + let sequence = self.next_sequence(); + let count = match self.entries.get(&key) { + Some(existing) if existing.kind == kind && now.saturating_duration_since(existing.recorded_at) < self.ttl => { + existing.count.saturating_add(1) + } + _ => 1, + }; + self.entries.insert( + key.clone(), + HealDispositionEntry { + kind, + count, + recorded_at: now, + task_id: task_id.to_owned(), + sequence, + }, + ); + self.recency.push_back((sequence, key)); + self.evict_to_capacity(); + self.compact_recency(); + self.publish_entries(); + } + + /// Forget `key` after a task completed with a positive storage result. + pub(crate) fn clear(&mut self, key: &HealDispositionKey) { + if self.entries.remove(key).is_some() { + self.publish_entries(); + } + } + + fn next_sequence(&mut self) -> u64 { + self.next_sequence = self.next_sequence.wrapping_add(1); + self.next_sequence + } + + fn touch(&mut self, key: &HealDispositionKey) { + let sequence = self.next_sequence(); + if let Some(entry) = self.entries.get_mut(key) { + entry.sequence = sequence; + self.recency.push_back((sequence, key.clone())); + self.compact_recency(); + } + } + + fn evict_to_capacity(&mut self) { + while self.entries.len() > self.capacity { + let Some((sequence, key)) = self.recency.pop_front() else { + // Every live entry has a recency pair; an empty queue with + // entries left cannot happen, but never spin if it does. + break; + }; + if self.entries.get(&key).is_some_and(|entry| entry.sequence == sequence) { + self.entries.remove(&key); + } + } + } + + fn compact_recency(&mut self) { + if self.recency.len() > self.capacity.saturating_mul(2).saturating_add(16) { + let entries = &self.entries; + self.recency + .retain(|(sequence, key)| entries.get(key).is_some_and(|entry| entry.sequence == *sequence)); + } + } + + fn publish_entries(&self) { + gauge!("rustfs_heal_disposition_cache_entries").set(self.entries.len() as f64); + } +} + +pub(super) fn lock_disposition_cache(cache: &StdMutex) -> StdMutexGuard<'_, HealDispositionCache> { + match cache.lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + } +} + +/// Remember the terminal verdict of a finished single-object task, or forget +/// the key after a positive completion. Scheduled retries, cancellations, +/// deadlines, dry runs and durable MRF repairs leave the memory untouched. +pub(crate) async fn record_task_disposition( + cache: &StdMutex, + task: &HealTask, + result: &crate::Result<()>, + will_retry: bool, +) { + let HealType::Object { + bucket, + object, + version_id, + } = &task.heal_type + else { + return; + }; + if task.options.dry_run || task.expected_mrf_bucket_incarnation_id.is_some() { + return; + } + let key = HealDispositionKey::new(bucket, object, version_id.as_deref(), task.options.pool_index, task.options.set_index); + let kind = match result { + Ok(()) if task.object_absent() => HealDispositionKind::Absent, + Ok(()) => { + // A deferral (grace window, transient skip) is neither a repair + // nor a failure; only a positive completion forgets the key. + let deferred = task + .get_outcome() + .await + .objects + .iter() + .any(|object| matches!(object.disposition, HealObjectDisposition::Deferred { .. })); + if !deferred { + lock_disposition_cache(cache).clear(&key); + } + return; + } + Err(_) if will_retry => return, + Err(error) => match classify_heal_error(error) { + HealErrorClass::Aborted => return, + HealErrorClass::Absent => HealDispositionKind::Absent, + HealErrorClass::NoHealRequired => { + lock_disposition_cache(cache).clear(&key); + return; + } + HealErrorClass::Transient | HealErrorClass::LockContention | HealErrorClass::DanglingGrace { .. } => { + HealDispositionKind::RetryExhausted + } + HealErrorClass::Permanent => HealDispositionKind::Permanent, + }, + }; + lock_disposition_cache(cache).record(key, kind, &task.id, Instant::now()); +} + +#[cfg(test)] +mod tests { + use super::*; + + fn key(object: &str) -> HealDispositionKey { + HealDispositionKey::new("bucket", object, None, None, None) + } + + #[test] + fn disposition_cache_absent_hit_until_ttl_expires() { + let mut cache = HealDispositionCache::with_limits(10, Duration::from_secs(60)); + let now = Instant::now(); + cache.record(key("gone"), HealDispositionKind::Absent, "task-1", now); + assert_eq!( + cache.consult(&key("gone"), now + Duration::from_secs(59)), + HealDispositionVerdict::Absent { + task_id: "task-1".into(), + count: 1 + } + ); + assert_eq!(cache.consult(&key("gone"), now + Duration::from_secs(60)), HealDispositionVerdict::Admit); + assert_eq!(cache.len(), 0, "an expired entry is dropped on consult"); + assert_eq!(cache.consult(&key("other"), now), HealDispositionVerdict::Admit); + } + + #[test] + fn disposition_cache_retry_exhausted_backs_off_exponentially_up_to_one_hour() { + let mut cache = HealDispositionCache::with_limits(10, Duration::from_secs(24 * 60 * 60)); + let now = Instant::now(); + cache.record(key("flaky"), HealDispositionKind::RetryExhausted, "task-1", now); + // The fourth submission (first after the task's own three retries) is backed off. + assert_eq!( + cache.consult(&key("flaky"), now + Duration::from_secs(30)), + HealDispositionVerdict::BackedOff { + kind: HealDispositionKind::RetryExhausted, + task_id: "task-1".into(), + count: 1, + retry_after: Duration::from_secs(30), + } + ); + assert_eq!(cache.consult(&key("flaky"), now + Duration::from_secs(60)), HealDispositionVerdict::Admit); + let later = now + Duration::from_secs(61); + cache.record(key("flaky"), HealDispositionKind::RetryExhausted, "task-2", later); + assert_eq!(cache.get(&key("flaky")).map(|entry| entry.count), Some(2)); + assert!(matches!( + cache.consult(&key("flaky"), later + Duration::from_secs(119)), + HealDispositionVerdict::BackedOff { count: 2, .. } + )); + assert_eq!( + cache.consult(&key("flaky"), later + Duration::from_secs(120)), + HealDispositionVerdict::Admit + ); + for count in 1..=40 { + assert!(HealDispositionCache::retry_backoff(count) <= Duration::from_secs(3600), "count={count}"); + } + assert_eq!(HealDispositionCache::retry_backoff(6), Duration::from_secs(1920)); + assert_eq!(HealDispositionCache::retry_backoff(7), Duration::from_secs(3600)); + assert_eq!(HealDispositionCache::retry_backoff(u32::MAX), Duration::from_secs(3600)); + // A different terminal kind starts its own count. + cache.record(key("flaky"), HealDispositionKind::Permanent, "task-3", later); + assert_eq!(cache.get(&key("flaky")).map(|entry| entry.count), Some(1)); + } + + #[test] + fn disposition_cache_evicts_least_recently_used_when_full() { + let mut cache = HealDispositionCache::with_limits(3, Duration::from_secs(600)); + let now = Instant::now(); + for object in ["a", "b", "c"] { + cache.record(key(object), HealDispositionKind::Absent, "task", now); + } + // Touch "a" so "b" becomes the oldest. + assert!(matches!(cache.consult(&key("a"), now), HealDispositionVerdict::Absent { .. })); + cache.record(key("d"), HealDispositionKind::Absent, "task", now); + assert_eq!(cache.len(), 3); + assert!(cache.get(&key("b")).is_none(), "the oldest entry is evicted, not the new one"); + for object in ["a", "c", "d"] { + assert!(cache.get(&key(object)).is_some(), "{object}"); + } + // Stale recency pairs never evict a re-recorded key by mistake. + cache.record(key("a"), HealDispositionKind::Absent, "task", now); + cache.record(key("e"), HealDispositionKind::Absent, "task", now); + assert!(cache.get(&key("a")).is_some()); + assert!(cache.get(&key("c")).is_none()); + assert_eq!(cache.len(), 3); + } + + #[test] + fn disposition_cache_success_clears_the_entry() { + let mut cache = HealDispositionCache::with_limits(10, Duration::from_secs(600)); + let now = Instant::now(); + cache.record(key("fixed"), HealDispositionKind::Permanent, "task-1", now); + cache.clear(&key("fixed")); + assert_eq!(cache.consult(&key("fixed"), now), HealDispositionVerdict::Admit); + assert_eq!(cache.len(), 0); + } + + #[test] + fn disposition_cache_recency_queue_stays_bounded_under_repeated_hits() { + let mut cache = HealDispositionCache::with_limits(4, Duration::from_secs(600)); + let now = Instant::now(); + cache.record(key("hot"), HealDispositionKind::Absent, "task", now); + for _ in 0..10_000 { + assert!(matches!(cache.consult(&key("hot"), now), HealDispositionVerdict::Absent { .. })); + } + assert!(cache.recency.len() <= 4 * 2 + 16 + 1, "recency={}", cache.recency.len()); + } + + #[test] + fn disposition_key_normalizes_empty_version() { + assert_eq!( + HealDispositionKey::new("b", "o", Some(""), None, None), + HealDispositionKey::new("b", "o", None, None, None) + ); + assert_ne!( + HealDispositionKey::new("b", "o", Some("v1"), None, None), + HealDispositionKey::new("b", "o", None, None, None) + ); + } +} diff --git a/crates/heal/src/heal/manager/root_recovery.rs b/crates/heal/src/heal/manager/root_recovery.rs index 5eb51df8e..158064dd4 100644 --- a/crates/heal/src/heal/manager/root_recovery.rs +++ b/crates/heal/src/heal/manager/root_recovery.rs @@ -19,11 +19,13 @@ //! or deletion. use super::*; +use crate::heal::outcome::HealExecutionOutcome; use crate::heal::resume::CheckpointManager; use crate::heal::storage_api::owner::{EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes}; use crate::heal::{DiskStore, RUSTFS_META_BUCKET}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; +use std::collections::BTreeMap; use tokio::io::AsyncReadExt; use uuid::Uuid; @@ -36,12 +38,15 @@ const ROOT_RECOVERY_PREFIX: &str = "root-heal-"; const ROOT_TERMINAL_PREFIX: &str = "terminal-root-heal-"; const LEGACY_ROOT_RECOVERY_SCHEMA: u32 = 1; const SCOPED_ROOT_RECOVERY_SCHEMA: u32 = 2; -const ROOT_RECOVERY_SCHEMA: u32 = 3; +const BUCKET_ROOT_RECOVERY_SCHEMA: u32 = 3; +const ROOT_RECOVERY_SCHEMA: u32 = 4; const ROOT_TERMINAL_SCHEMA: u32 = 1; const ROOT_QUARANTINE_INTENT_PREFIX: &str = "quarantined-root-heal-intent-"; const ROOT_QUARANTINE_TERMINAL_PREFIX: &str = "quarantined-root-heal-terminal-"; const ROOT_QUARANTINE_SCHEMA: u32 = 1; pub(super) const MAX_ROOT_RECOVERY_BYTES: usize = 64 * 1024; +// Cluster checkpoints additionally retain a frozen inventory and the bounded outcome window. +pub(super) const MAX_CLUSTER_RECOVERY_BYTES: usize = 16 * 1024 * 1024; const ROOT_TERMINAL_GC_SCAN_BUDGET: usize = 1024; const ROOT_TERMINAL_GC_DELETE_BUDGET: usize = 64; @@ -237,7 +242,83 @@ fn default_recovery_heal_type() -> RecoveryHealType { RecoveryHealType::Cluster } -#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct AdminClusterScope { + #[serde(deserialize_with = "decode_cluster_buckets")] + pub(crate) buckets: std::collections::BTreeMap, + pub(crate) next_bucket_index: usize, + pub(crate) outcome: HealTaskOutcome, + pub(crate) progress: HealProgress, +} + +fn decode_cluster_buckets<'de, D: serde::Deserializer<'de>>( + deserializer: D, +) -> std::result::Result, D::Error> { + struct Buckets; + impl<'de> serde::de::Visitor<'de> for Buckets { + type Value = std::collections::BTreeMap; + fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { + formatter.write_str("unique bucket generation bindings") + } + fn visit_map>(self, mut map: M) -> std::result::Result { + let mut buckets = std::collections::BTreeMap::new(); + while let Some((bucket, incarnation)) = map.next_entry::()? { + if buckets.insert(bucket, incarnation).is_some() { + return Err(serde::de::Error::custom("duplicate administrator cluster bucket")); + } + } + Ok(buckets) + } + } + deserializer.deserialize_map(Buckets) +} + +impl AdminClusterScope { + pub(crate) fn checkpoint_interval(&self) -> usize { + // Cap full-inventory rewrites per traversal, including many empty buckets. + self.buckets.len().div_ceil(32).max(1) + } + + fn validate(&self) -> Result<()> { + if self.next_bucket_index > self.buckets.len() + || self.outcome.execution != HealExecutionOutcome::Running + || self.outcome.coverage != crate::heal::outcome::HealTraversalCoverage::Partial + || self.outcome.counters.failed != 0 + { + return Err(Error::InvalidCheckpoint("Invalid administrator cluster checkpoint progress".to_string())); + } + for (bucket, incarnation) in &self.buckets { + if bucket.contains('/') + || bucket.trim() != bucket + || crate::heal::storage_api::owner::ecstore_is_reserved_or_invalid_bucket(bucket, false) + { + return Err(Error::InvalidCheckpoint("Invalid administrator cluster bucket name".to_string())); + } + if incarnation.is_nil() { + return Err(Error::InvalidCheckpoint( + "Administrator cluster checkpoint lacks a bucket incarnation".to_string(), + )); + } + } + let completed_through = self + .next_bucket_index + .checked_sub(1) + .and_then(|index| self.buckets.keys().nth(index)); + if self.outcome.objects.iter().any(|item| { + completed_through.is_none_or(|last| item.identity.bucket > *last) + || self.buckets.get(&item.identity.bucket).copied() != item.identity.bucket_incarnation_id + }) || (self.next_bucket_index == 0 && self.outcome.counters.processed != 0) + { + return Err(Error::InvalidCheckpoint( + "Administrator cluster checkpoint outcome is outside its completed scope".to_string(), + )); + } + Ok(()) + } +} + +#[derive(Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct RootHealIntent { schema: u32, @@ -246,6 +327,8 @@ struct RootHealIntent { heal_type: RecoveryHealType, #[serde(default, skip_serializing_if = "Option::is_none")] bucket_incarnation_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + cluster_scope: Option, #[serde(deserialize_with = "decode_options")] options: HealOptions, priority: HealPriority, @@ -292,10 +375,46 @@ struct RootHealSource { source_oversized: bool, } +impl RootHealQuarantine { + fn matches_source(&self, source: &RootHealSource) -> bool { + if self.source_prefix_len == source.bytes.len() + && self.source_prefix_sha256 == source.source_prefix_sha256 + && self.source_oversized == source.source_oversized + { + return true; + } + // Before schema 4, oversized intents retained only the 64 KiB prefix. + // Keep that quarantine fence across upgrades without reinterpreting its source. + self.record_kind == RootHealRecordKind::Intent + && self.source_oversized + && self.source_prefix_len == MAX_ROOT_RECOVERY_BYTES + 1 + && source + .bytes + .get(..self.source_prefix_len) + .is_some_and(|prefix| Sha256::digest(prefix).as_slice() == self.source_prefix_sha256.as_slice()) + } +} + #[derive(Debug, Default)] pub(super) struct RootHealInventory { pub(super) requests: Vec, pub(super) quarantined_task_ids: HashSet, + /// Records that can neither be admitted nor quarantined, with the reason. + /// They are reported, refused for admission, and retained for operator + /// repair; every other record keeps replaying. + pub(super) conflicted: BTreeMap, +} + +impl RootHealInventory { + /// Why a durable record cannot run, for admission refusals and status. + pub(super) fn record_issue(&self, task_id: &str) -> Option { + if let Some(reason) = self.conflicted.get(task_id) { + return Some(reason.clone()); + } + self.quarantined_task_ids + .contains(task_id) + .then(|| "durable record is quarantined".to_string()) + } } impl RootHealTerminal { @@ -362,12 +481,38 @@ impl RootHealTerminal { } impl RootHealIntent { + fn encode(&self) -> Result { + let limit = match &self.cluster_scope { + // Reserve headroom for the bounded outcome window and progress so + // completing work cannot overflow an inventory admitted at the limit. + Some(scope) if scope.next_bucket_index == 0 => MAX_CLUSTER_RECOVERY_BYTES - 1024 * 1024, + Some(_) => MAX_CLUSTER_RECOVERY_BYTES, + None => MAX_ROOT_RECOVERY_BYTES, + }; + encode_root_record(self, limit) + } + + fn validate_task(&self, task: &HealTask) -> Result<()> { + let mut options = self.options.clone(); + options.timeout = task.options.timeout; + if HealType::from(self.heal_type.clone()) != task.heal_type + || self.bucket_incarnation_id != task.bucket_incarnation_id + || self.created_at != task.created_at + || self.priority != task.priority + || options != task.options + { + return Err(Error::InvalidCheckpoint(format!("Root heal recovery owner changed for {}", task.id))); + } + Ok(()) + } + fn from_request(request: &HealRequest) -> Result { Ok(Self { schema: ROOT_RECOVERY_SCHEMA, task_id: request.id.clone(), heal_type: RecoveryHealType::try_from(&request.heal_type)?, bucket_incarnation_id: request.bucket_incarnation_id, + cluster_scope: None, options: request.options.clone(), priority: request.priority, retry_attempts: request.retry_attempts, @@ -386,6 +531,32 @@ impl RootHealIntent { } } +fn encode_root_record(value: &impl Serialize, limit: usize) -> Result { + struct BoundedWriter { + bytes: Vec, + limit: usize, + } + impl std::io::Write for BoundedWriter { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if bytes.len() > self.limit.saturating_sub(self.bytes.len()) { + return Err(std::io::Error::other("root heal record exceeds its size limit")); + } + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + let mut writer = BoundedWriter { + bytes: Vec::new(), + limit, + }; + serde_json::to_writer(&mut writer, value) + .map_err(|error| Error::Serialization(format!("Serialize root heal record: {error}")))?; + Ok(writer.bytes.into()) +} + #[derive(Default)] pub(crate) struct RootHealRecovery { mutation: Mutex<()>, @@ -490,19 +661,27 @@ async fn read_bounded(disk: &DiskStore, path: &str) -> Result usize { + if path.starts_with(ROOT_RECOVERY_PREFIX) { + MAX_CLUSTER_RECOVERY_BYTES + } else { + MAX_ROOT_RECOVERY_BYTES + } +} + async fn read_source(disk: &DiskStore, path: &str) -> Result> { let Some(bytes) = read_bounded(disk, path).await? else { return Ok(None); }; Ok(Some(RootHealSource { source_prefix_sha256: Sha256::digest(&bytes).to_vec(), - source_oversized: bytes.len() > MAX_ROOT_RECOVERY_BYTES, + source_oversized: bytes.len() > root_record_read_limit(path), bytes, })) } @@ -544,7 +723,7 @@ fn encode_quarantine( fn decode_intent(task_id: &str, bytes: &[u8]) -> Result { let _ = intent_path(task_id)?; - if bytes.len() > MAX_ROOT_RECOVERY_BYTES { + if bytes.len() > MAX_CLUSTER_RECOVERY_BYTES { return Err(Error::Other(format!("Root heal recovery record exceeds its size limit {task_id}"))); } let mut intent: RootHealIntent = serde_json::from_slice(bytes) @@ -556,9 +735,25 @@ fn decode_intent(task_id: &str, bytes: &[u8]) -> Result { LEGACY_ROOT_RECOVERY_SCHEMA if intent.heal_type == RecoveryHealType::Cluster && intent.bucket_incarnation_id.is_none() => {} SCOPED_ROOT_RECOVERY_SCHEMA if intent.bucket_incarnation_id.is_none() => {} + BUCKET_ROOT_RECOVERY_SCHEMA if intent.cluster_scope.is_none() => {} ROOT_RECOVERY_SCHEMA => {} _ => return Err(Error::Other(format!("Unsupported or mismatched root heal recovery record {task_id}"))), } + if intent.heal_type == RecoveryHealType::Cluster && intent.schema != ROOT_RECOVERY_SCHEMA { + return Err(Error::InvalidCheckpoint( + "Legacy cluster heal has no frozen bucket generations; start a fresh task".to_string(), + )); + } + match &intent.cluster_scope { + Some(scope) if intent.schema == ROOT_RECOVERY_SCHEMA && intent.heal_type == RecoveryHealType::Cluster => { + scope.validate()?; + } + Some(_) => return Err(Error::InvalidCheckpoint("Unexpected administrator cluster scope".to_string())), + None if bytes.len() > MAX_ROOT_RECOVERY_BYTES => { + return Err(Error::InvalidCheckpoint("Root heal recovery record exceeds its size limit".to_string())); + } + None => {} + } if !matches!(intent.heal_type, RecoveryHealType::Bucket { .. }) && intent.bucket_incarnation_id.is_some() { return Err(Error::Other(format!( "Unexpected bucket incarnation in root heal recovery record {task_id}" @@ -723,17 +918,18 @@ impl RootHealRecovery { } }; let marker = decode_quarantine(task_id, record_kind, &marker_bytes)?; - if marker.source_prefix_len != source.bytes.len() - || marker.source_prefix_sha256 != source.source_prefix_sha256 - || marker.source_oversized != source.source_oversized - { + if !marker.matches_source(source) { return Err(Error::other(format!("Root heal quarantine source changed {task_id}"))); } Ok(marker) } + /// Index quarantine markers by record. A marker that cannot be read or + /// decoded, or that has several owners, marks its task as conflicted + /// instead of failing the whole inventory; such a task is never replayed. async fn quarantine_marker_map( disks: &[DiskStore], + conflicted: &mut BTreeMap, ) -> Result> { let mut markers = HashMap::new(); for disk in disks { @@ -747,15 +943,25 @@ impl RootHealRecovery { continue; }; let Some(bytes) = read_bounded(disk, &entry).await? else { - return Err(Error::other(format!("Root heal quarantine marker disappeared {task_id}"))); + conflicted.insert(task_id.to_owned(), format!("Root heal quarantine marker disappeared {task_id}")); + continue; + }; + let marker = match decode_quarantine(task_id, record_kind, &bytes) { + Ok(marker) => marker, + Err(error) => { + conflicted.insert(task_id.to_owned(), error.to_string()); + continue; + } }; - let marker = decode_quarantine(task_id, record_kind, &bytes)?; let key = (record_kind, task_id.to_owned()); if markers.insert(key, (disk.clone(), marker)).is_some() { - return Err(Error::other(format!("Multiple root heal quarantine owners for {task_id}"))); + conflicted.insert(task_id.to_owned(), format!("Multiple root heal quarantine owners for {task_id}")); } } } + for task_id in conflicted.keys() { + markers.retain(|(_, marked), _| marked != task_id); + } Ok(markers) } @@ -785,7 +991,8 @@ impl RootHealRecovery { } } - let markers = Self::quarantine_marker_map(disks).await?; + let mut conflicted = BTreeMap::new(); + let markers = Self::quarantine_marker_map(disks, &mut conflicted).await?; for (record_kind, task_id) in markers.keys() { match record_kind { RootHealRecordKind::Intent => { @@ -799,20 +1006,30 @@ impl RootHealRecovery { let mut inventory = RootHealInventory { requests: Vec::new(), quarantined_task_ids: markers.keys().map(|(_, task_id)| task_id.clone()).collect(), + conflicted, }; + // One record's conflict never postpones the others: it is recorded + // with its reason, refused for admission, and retained for repair. for task_id in intent_ids.union(&terminal_ids) { + if inventory.conflicted.contains_key(task_id) { + continue; + } let terminal_raw = Self::read_raw(disks, &terminal_path(task_id)?).await?; if terminal_raw.len() > 1 { - return Err(Error::other(format!("Multiple root heal terminal owners for {task_id}"))); + inventory + .conflicted + .insert(task_id.clone(), format!("Multiple root heal terminal owners for {task_id}")); + continue; } if let Some((owner, source)) = terminal_raw.into_iter().next() { if let Some(marker) = markers.get(&(RootHealRecordKind::Terminal, task_id.clone())) { if EcstoreDiskAPI::endpoint(marker.0.as_ref()) != EcstoreDiskAPI::endpoint(owner.as_ref()) - || marker.1.source_prefix_len != source.bytes.len() - || marker.1.source_prefix_sha256 != source.source_prefix_sha256 - || marker.1.source_oversized != source.source_oversized + || !marker.1.matches_source(&source) { - return Err(Error::other(format!("Root heal quarantine source changed {task_id}"))); + inventory + .conflicted + .insert(task_id.clone(), format!("Root heal quarantine source changed {task_id}")); + continue; } inventory.quarantined_task_ids.insert(task_id.clone()); continue; @@ -828,32 +1045,43 @@ impl RootHealRecovery { } } if let Some(marker) = markers.get(&(RootHealRecordKind::Terminal, task_id.clone())) { - return Err(Error::other(format!( - "Root heal quarantine marker has no retained terminal source {} ({} bytes)", - task_id, marker.1.source_prefix_len - ))); + inventory.conflicted.insert( + task_id.clone(), + format!( + "Root heal quarantine marker has no retained terminal source {} ({} bytes)", + task_id, marker.1.source_prefix_len + ), + ); + continue; } let intent_raw = Self::read_raw(disks, &intent_path(task_id)?).await?; if intent_raw.len() > 1 { - return Err(Error::other(format!("Multiple root heal recovery owners for {task_id}"))); + inventory + .conflicted + .insert(task_id.clone(), format!("Multiple root heal recovery owners for {task_id}")); + continue; } let Some((owner, source)) = intent_raw.into_iter().next() else { if let Some(marker) = markers.get(&(RootHealRecordKind::Intent, task_id.clone())) { - return Err(Error::other(format!( - "Root heal quarantine marker has no retained intent source {} ({} bytes)", - task_id, marker.1.source_prefix_len - ))); + inventory.conflicted.insert( + task_id.clone(), + format!( + "Root heal quarantine marker has no retained intent source {} ({} bytes)", + task_id, marker.1.source_prefix_len + ), + ); } continue; }; if let Some(marker) = markers.get(&(RootHealRecordKind::Intent, task_id.clone())) { if EcstoreDiskAPI::endpoint(marker.0.as_ref()) != EcstoreDiskAPI::endpoint(owner.as_ref()) - || marker.1.source_prefix_len != source.bytes.len() - || marker.1.source_prefix_sha256 != source.source_prefix_sha256 - || marker.1.source_oversized != source.source_oversized + || !marker.1.matches_source(&source) { - return Err(Error::other(format!("Root heal quarantine source changed {task_id}"))); + inventory + .conflicted + .insert(task_id.clone(), format!("Root heal quarantine source changed {task_id}")); + continue; } inventory.quarantined_task_ids.insert(task_id.clone()); continue; @@ -866,6 +1094,9 @@ impl RootHealRecovery { } } } + inventory + .quarantined_task_ids + .retain(|task_id| !inventory.conflicted.contains_key(task_id)); inventory .requests .sort_by(|left, right| left.created_at.cmp(&right.created_at).then_with(|| left.id.cmp(&right.id))); @@ -924,18 +1155,29 @@ impl RootHealRecovery { if request.options.no_lock { return Err(Error::Other("Administrator root heal cannot skip namespace locking".to_string())); } - let intent = RootHealIntent::from_request(request)?; + let mut intent = RootHealIntent::from_request(request)?; intent.heal_type.validate()?; - let bytes = - serde_json::to_vec(&intent).map_err(|error| Error::Other(format!("Serialize root heal recovery record: {error}")))?; let path = intent_path(&request.id)?; if let Some((disk, expected)) = existing { + let previous = decode_intent(&request.id, &expected)?; + let mut previous_options = previous.options.clone(); + previous_options.timeout = intent.options.timeout; + if previous.heal_type != intent.heal_type + || previous.bucket_incarnation_id != intent.bucket_incarnation_id + || previous.created_at != intent.created_at + || previous.priority != intent.priority + || previous_options != intent.options + { + return Err(Error::InvalidCheckpoint("Root heal recovery owner changed during handoff".to_string())); + } + intent.cluster_scope = previous.cluster_scope; + let bytes = intent.encode()?; return match EcstoreDiskAPI::compare_and_update_file( disk.as_ref(), RUSTFS_META_BUCKET, &path, Some(expected), - Some(bytes.into()), + Some(bytes), ) .await? { @@ -944,19 +1186,15 @@ impl RootHealRecovery { }; } + let bytes = intent.encode()?; + if disks.is_empty() { return Err(Error::Other("No local disk available for root heal shutdown recovery".to_string())); } let mut last_not_committed = None; for disk in &disks { - match EcstoreDiskAPI::compare_and_update_file( - disk.as_ref(), - RUSTFS_META_BUCKET, - &path, - None, - Some(bytes.clone().into()), - ) - .await + match EcstoreDiskAPI::compare_and_update_file(disk.as_ref(), RUSTFS_META_BUCKET, &path, None, Some(bytes.clone())) + .await { Ok(EcstoreConditionalFileUpdate::Updated) => return Ok(()), Ok(_) => return Err(Error::Other(format!("Root heal recovery record changed for {}", request.id))), @@ -970,6 +1208,63 @@ impl RootHealRecovery { } } + pub(crate) async fn cluster_scope(&self, task: &HealTask) -> Result> { + #[cfg(any(test, feature = "test-util"))] + if self.disabled_for_tests { + return Ok(None); + } + let _guard = self.mutation.lock().await; + let (_, bytes) = Self::find(&self.disks().await?, &task.id) + .await? + .ok_or_else(|| Error::InvalidCheckpoint("Administrator cluster intent is missing".to_string()))?; + let intent = decode_intent(&task.id, &bytes)?; + intent.validate_task(task)?; + Ok(intent.cluster_scope) + } + + pub(crate) async fn checkpoint_cluster_scope(&self, task: &HealTask, scope: &AdminClusterScope) -> Result<()> { + scope.validate()?; + #[cfg(any(test, feature = "test-util"))] + if self.disabled_for_tests { + return Ok(()); + } + let _guard = self.mutation.lock().await; + let (disk, expected) = Self::find(&self.disks().await?, &task.id) + .await? + .ok_or_else(|| Error::InvalidCheckpoint("Administrator cluster intent is missing".to_string()))?; + let mut intent = decode_intent(&task.id, &expected)?; + intent.validate_task(task)?; + if intent.heal_type != RecoveryHealType::Cluster { + return Err(Error::InvalidCheckpoint("Cluster scope requires a cluster task".to_string())); + } + if let Some(previous) = &intent.cluster_scope { + if previous.buckets != scope.buckets + || scope.next_bucket_index < previous.next_bucket_index + || scope.next_bucket_index > previous.next_bucket_index.saturating_add(scope.checkpoint_interval()) + { + return Err(Error::InvalidCheckpoint( + "Administrator cluster scope changed during recovery".to_string(), + )); + } + } else if scope.next_bucket_index != 0 { + return Err(Error::InvalidCheckpoint("Administrator cluster scope was not initialized".to_string())); + } + intent.cluster_scope = Some(scope.clone()); + let bytes = intent.encode()?; + match EcstoreDiskAPI::compare_and_update_file( + disk.as_ref(), + RUSTFS_META_BUCKET, + &intent_path(&task.id)?, + Some(expected), + Some(bytes), + ) + .await? + { + EcstoreConditionalFileUpdate::Updated => Ok(()), + _ => Err(Error::InvalidCheckpoint("Administrator cluster checkpoint owner changed".to_string())), + } + } + pub(super) async fn remove(&self, task_id: &str, heal_type: &HealType, source: HealRequestSource) -> Result { if !is_admin_heal_recovery(heal_type, source) { return Ok(false); @@ -1015,16 +1310,7 @@ impl RootHealRecovery { return Ok(()); }; let mut intent = decode_intent(&task.id, &expected)?; - if HealType::from(intent.heal_type.clone()) != task.heal_type - || intent.bucket_incarnation_id != task.bucket_incarnation_id - { - return Err(Error::Other(format!("Root heal recovery owner changed for {}", task.id))); - } - let mut expected_options = intent.options.clone(); - expected_options.timeout = task.options.timeout; - if intent.created_at != task.created_at || intent.priority != task.priority || expected_options != task.options { - return Err(Error::Other(format!("Root heal recovery owner changed for {}", task.id))); - } + intent.validate_task(task)?; // A terminal timeout leaves no runtime owner for stop() to snapshot. // Checkpoint its consumed budget before publishing terminal status; // never refund time if an earlier checkpoint is already stricter. @@ -1033,14 +1319,13 @@ impl RootHealRecovery { (previous, remaining) => previous.or(remaining), }; intent.retry_attempts = intent.retry_attempts.max(task.retry_attempts); - let bytes = serde_json::to_vec(&intent) - .map_err(|error| Error::Other(format!("Serialize root heal recovery checkpoint: {error}")))?; + let bytes = intent.encode()?; match EcstoreDiskAPI::compare_and_update_file( disk.as_ref(), RUSTFS_META_BUCKET, &intent_path(&task.id)?, Some(expected), - Some(bytes.into()), + Some(bytes), ) .await? { @@ -1360,6 +1645,17 @@ impl RootHealRecovery { pub(super) async fn pending(&self) -> Result> { Ok(self.inventory().await?.requests) } + + /// Why a durable record for `task_id` cannot run, if the inventory can be + /// read. Status queries are best-effort: an unreadable inventory reports + /// no issue rather than failing the query. + pub(crate) async fn record_issue(&self, task_id: &str) -> Option { + #[cfg(any(test, feature = "test-util"))] + if self.disabled_for_tests { + return None; + } + self.inventory().await.ok()?.record_issue(task_id) + } } impl HealManager { @@ -1367,7 +1663,20 @@ impl HealManager { // Decode every record before admitting anything. These are already // accepted responsibilities, so restore distinct IDs even when their // paths overlap or the configured admission capacity has changed. - let pending = self.root_recovery.inventory().await?.requests; + let inventory = self.root_recovery.inventory().await?; + for (task_id, reason) in &inventory.conflicted { + warn!( + target: "rustfs::heal::manager", + event = EVENT_HEAL_MANAGER_STATE, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_MANAGER, + task_id, + result = "conflicting_record_skipped", + reason, + "Root heal recovery record skipped; operator repair required" + ); + } + let pending = inventory.requests; let mut requests = Vec::with_capacity(pending.len()); for request in pending { if let HealType::Bucket { bucket } = &request.heal_type { diff --git a/crates/heal/src/heal/manager/root_recovery/report.rs b/crates/heal/src/heal/manager/root_recovery/report.rs index 923daf67b..fc646882e 100644 --- a/crates/heal/src/heal/manager/root_recovery/report.rs +++ b/crates/heal/src/heal/manager/root_recovery/report.rs @@ -14,7 +14,6 @@ use super::*; use crate::heal::outcome::{HealAbortReason, HealExecutionOutcome}; -use std::io::Write; use tokio::io::AsyncReadExt; // This prefix must not overlap either namespace understood by schema-1 readers. @@ -68,24 +67,7 @@ impl RootHealReport { } pub(super) fn encode(&self) -> Result { - struct BoundedWriter(Vec); - impl Write for BoundedWriter { - fn write(&mut self, bytes: &[u8]) -> std::io::Result { - if bytes.len() > MAX_ROOT_REPORT_BYTES.saturating_sub(self.0.len()) { - return Err(std::io::Error::other("root heal report exceeds its size limit")); - } - self.0.extend_from_slice(bytes); - Ok(bytes.len()) - } - - fn flush(&mut self) -> std::io::Result<()> { - Ok(()) - } - } - let mut writer = BoundedWriter(Vec::new()); - serde_json::to_writer(&mut writer, self) - .map_err(|error| Error::Serialization(format!("Serialize root heal report: {error}")))?; - Ok(writer.0.into()) + encode_root_record(self, MAX_ROOT_REPORT_BYTES) } } diff --git a/crates/heal/src/heal/manager/scheduler.rs b/crates/heal/src/heal/manager/scheduler.rs index 2a02fc6b8..fd9f8214f 100644 --- a/crates/heal/src/heal/manager/scheduler.rs +++ b/crates/heal/src/heal/manager/scheduler.rs @@ -11,6 +11,7 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. +use super::disposition::record_task_disposition; /// The heal scheduler: queue consumption loop and its skip/metric helpers. use super::*; @@ -47,6 +48,7 @@ impl HealManager { let storage = self.storage.clone(); let notify = self.notify.clone(); let workload_provider = self.workload_provider.clone(); + let disposition_cache = self.disposition_cache.clone(); tokio::spawn(async move { let mut interval = interval(config.read().await.heal_interval); @@ -82,6 +84,7 @@ impl HealManager { notify: ¬ify, cancel_token: &cancel_token, workload_provider: &workload_provider, + disposition_cache: &disposition_cache, }) .await; } @@ -102,6 +105,7 @@ impl HealManager { notify: ¬ify, cancel_token: &cancel_token, workload_provider: &workload_provider, + disposition_cache: &disposition_cache, }) .await; } @@ -131,6 +135,7 @@ impl HealManager { notify, cancel_token, workload_provider, + disposition_cache, } = context; let config = config.read().await; @@ -227,6 +232,7 @@ impl HealManager { let retrying_heals_clone = retrying_heals.clone(); let mrf_repair_notice_targets_clone = mrf_repair_notice_targets.clone(); let replacement_recovery_anchors_clone = replacement_recovery_anchors.clone(); + let disposition_cache_clone = disposition_cache.clone(); let statistics_clone = statistics.clone(); let notify_clone = notify.clone(); let root_recovery_clone = root_recovery.clone(); @@ -251,6 +257,7 @@ impl HealManager { ); let result = task.execute().await; let retry_request = retry_request_for_result_with_budget(task.as_ref(), &result).await; + record_task_disposition(&disposition_cache_clone, task.as_ref(), &result, retry_request.is_some()).await; match &result { Ok(_) => { debug!( diff --git a/crates/heal/src/heal/manager/tests.rs b/crates/heal/src/heal/manager/tests.rs index 2838c9d6c..c97340b90 100644 --- a/crates/heal/src/heal/manager/tests.rs +++ b/crates/heal/src/heal/manager/tests.rs @@ -29,6 +29,8 @@ use tempfile::TempDir; pub(super) mod admin_overlap; mod root_recovery; use uuid::Uuid; +mod auto_scan; +mod disposition; mod running_mainline; use super::super::{DiskOption, DiskStore, Endpoint, new_disk, storage_api::status::BucketInfo}; @@ -104,6 +106,7 @@ async fn process_manager_queue_once(manager: &HealManager) { notify: &manager.notify, cancel_token: &manager.cancel_token, workload_provider: &manager.workload_provider, + disposition_cache: &manager.disposition_cache, }) .await; } @@ -601,6 +604,13 @@ impl HealStorageAPI for MockStorage { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { if let Some(hook) = manager_recovery_test_hook() { *hook.listed.lock().expect("manager recovery listed lock should not poison") = true; @@ -3141,13 +3151,48 @@ fn replacement_target_reservation_ends_only_after_an_explicit_transfer() { } #[test] -fn replacement_recovery_blocker_is_set_scoped() { +fn replacement_recovery_blocker_is_generation_scoped() { let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None); + let conflicted = Uuid::new_v4().to_string(); + let sibling = Uuid::new_v4().to_string(); - manager.block_replacement_recovery_set("pool_0_set_0"); + manager.block_replacement_recovery_generation_for_test(&conflicted, "pool_0_set_0", "durable copies diverge", true); - assert!(manager.replacement_recovery_set_is_blocked("pool_0_set_0")); - assert!(!manager.replacement_recovery_set_is_blocked("pool_0_set_1")); + assert!(manager.replacement_recovery_generation_is_blocked(&conflicted)); + assert!( + !manager.replacement_recovery_generation_is_blocked(&sibling), + "a sibling generation of the same set is not blocked" + ); + assert!(manager.replacement_recovery_set_has_reserving_block("pool_0_set_0")); + assert!(!manager.replacement_recovery_set_has_reserving_block("pool_0_set_1")); + let blocks = manager.replacement_recovery_blocks(); + assert_eq!(blocks.len(), 1); + assert_eq!(blocks[0].task_id, conflicted); + assert_eq!(blocks[0].set_disk_id, "pool_0_set_0"); + assert_eq!(blocks[0].reason, "durable copies diverge"); + assert!(blocks[0].reserves_targets); + assert!(blocks[0].blocked_since > 0); + assert_eq!(blocks[0].blocked_since, blocks[0].last_observed); + + // A refreshed block keeps its first observation and follows the latest reason. + manager.block_replacement_recovery_generation_for_test(&conflicted, "pool_0_set_0", "retired", false); + let refreshed = manager.replacement_recovery_blocks(); + assert_eq!(refreshed[0].reason, "retired"); + assert!(!refreshed[0].reserves_targets); + assert_eq!(refreshed[0].blocked_since, blocks[0].blocked_since); + assert!( + !manager.replacement_recovery_set_has_reserving_block("pool_0_set_0"), + "a released block no longer defers fresh admission for the set" + ); +} + +#[test] +fn legacy_ambiguous_delete_residue_is_not_a_retryable_heal_error() { + // rustfs/rustfs#8608: the residue is deterministic; it must not be retried + // as request-rate pressure. + let error = Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous); + assert!(!error.is_recoverable_heal()); + assert!(Error::Storage(EcstoreError::SlowDown).is_recoverable_heal()); } #[test] @@ -3200,24 +3245,6 @@ fn replacement_recovery_discovery_unformatted_is_quiet_only_for_deferred_endpoin )); } -#[test] -fn replacement_recovery_retry_barrier_requires_all_set_records_to_validate() { - let mut blocked = HashSet::from(["pool_0_set_0".to_string(), "pool_0_set_1".to_string()]); - let retry_succeeded = HashSet::from(["pool_0_set_0".to_string(), "pool_0_set_1".to_string()]); - let retry_failed = HashSet::from(["pool_0_set_0".to_string()]); - - unblock_replacement_recovery_sets_after_validation(&mut blocked, retry_succeeded, &retry_failed); - - assert!( - blocked.contains("pool_0_set_0"), - "one failed disk record must keep the whole replacement set blocked" - ); - assert!( - !blocked.contains("pool_0_set_1"), - "a blocked set may resume only after every retried record validates" - ); -} - #[tokio::test] async fn scheduler_completes_cleanup_pending_recovery_from_manager_anchor() { let temp = TempDir::new().expect("temporary manager recovery directory should be created"); @@ -5695,7 +5722,7 @@ async fn configured_task_timeout_applies_only_when_request_timeout_is_absent() { process_manager_queue_once(&manager).await; let explicit_status = tokio::time::timeout(Duration::from_secs(1), async { loop { - if let Ok(status @ HealTaskStatus::Failed { .. }) = manager.get_task_status(&explicit_id).await { + if let Ok(status @ HealTaskStatus::Completed) = manager.get_task_status(&explicit_id).await { break status; } tokio::task::yield_now().await; @@ -5703,7 +5730,8 @@ async fn configured_task_timeout_applies_only_when_request_timeout_is_absent() { }) .await .expect("explicit timeout request should finish without using the zero default"); - assert!(matches!(explicit_status, HealTaskStatus::Failed { .. })); + // Generation-bound admin tasks reach the successful mock heal without a namespace preflight. + assert_eq!(explicit_status, HealTaskStatus::Completed); } #[tokio::test] @@ -5934,3 +5962,25 @@ fn test_heal_config_respects_feature_flags() { }, ); } + +#[test] +fn admin_bucket_discovery_retries_keep_the_existing_budget() { + let storage: Arc = Arc::new(MockStorage); + let mut request = HealRequest::new(HealType::Cluster, HealOptions::default(), HealPriority::Normal); + request.source = HealRequestSource::Admin; + let id = request.id.clone(); + let error = Err(Error::BucketDiscoveryIncomplete); + for attempt in 1..=MAX_RECOVERABLE_HEAL_RETRIES { + let task = HealTask::from_request(request, storage.clone()); + let (retry, delay, _) = retry_request_for_result(&task, &error).expect("discovery can recover within budget"); + assert_eq!(retry.id, id); + assert_eq!(retry.retry_attempts, attempt); + assert!(delay > Duration::ZERO); + request = retry; + } + let task = HealTask::from_request(request, storage); + assert!( + retry_request_for_result(&task, &error).is_none(), + "persistent incompleteness must terminate" + ); +} diff --git a/crates/heal/src/heal/manager/tests/auto_scan.rs b/crates/heal/src/heal/manager/tests/auto_scan.rs new file mode 100644 index 000000000..977cbfdb4 --- /dev/null +++ b/crates/heal/src/heal/manager/tests/auto_scan.rs @@ -0,0 +1,448 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::heal::storage::HealListItem; +use crate::heal::storage_api::{ + EcstoreEndpointServerPools as EndpointServerPools, EcstoreEndpoints as Endpoints, EcstorePoolEndpoints as PoolEndpoints, + ecstore_init_local_disks as init_local_disks, +}; +use std::{ + future::Future, + sync::atomic::{AtomicUsize, Ordering}, +}; + +const SET: &str = "pool_0_set_0"; +const SCAN_INTERVAL: Duration = Duration::from_secs(10); + +struct ScannerStorage { + survivor: DiskStore, + /// Additional survivors exposed through the same set inventory. + sibling_survivors: Vec, + identity: ReplacementTargetIdentity, + inventory_calls: AtomicUsize, +} + +#[async_trait::async_trait] +impl HealStorageAPI for ScannerStorage { + async fn replacement_intent_disks(&self, set: &str) -> Result> { + assert_eq!(set, SET, "discovery must use the target's erasure set"); + self.inventory_calls.fetch_add(1, Ordering::SeqCst); + // This disk is deliberately outside the node's local registry. The + // mock supplies the same inventory boundary as a remote survivor, + // while real disk I/O exercises durable intent discovery and loading. + let mut disks = vec![self.survivor.clone()]; + disks.extend(self.sibling_survivors.iter().cloned()); + Ok(disks) + } + + async fn replacement_target_identities(&self, targets: &[String]) -> Result> { + assert_eq!(targets, std::slice::from_ref(&self.identity.endpoint)); + Ok(vec![self.identity.clone()]) + } + + async fn get_object_meta(&self, _: &str, _: &str) -> Result> { + Err(Error::other("scanner admission must not inspect objects")) + } + + async fn ec_decode_rebuild(&self, _: &str, _: &str) -> Result> { + Err(Error::other("scanner admission must not rebuild objects")) + } + + async fn get_bucket_info(&self, _: &str) -> Result> { + Err(Error::other("scanner admission must not inspect buckets")) + } + + async fn list_buckets(&self) -> Result> { + Err(Error::other("same-incarnation discovery must reuse durable buckets")) + } + + async fn list_buckets_for_admin_heal( + &self, + _: Option, + ) -> Result> { + Err(Error::other("scanner admission must not start admin discovery")) + } + + async fn object_exists(&self, _: &str, _: &str) -> Result { + Err(Error::other("scanner admission must not inspect objects")) + } + + async fn heal_object(&self, _: &str, _: &str, _: Option<&str>, _: &HealOpts) -> Result<(HealResultItem, Option)> { + Err(Error::other("scanner admission must not execute object healing")) + } + + async fn heal_bucket(&self, _: &str, _: &HealOpts) -> Result { + Err(Error::other("scanner admission must not execute bucket healing")) + } + + async fn heal_format(&self, _: bool) -> Result<(HealResultItem, Option)> { + Err(Error::other("scanner admission must not execute format healing")) + } + + async fn list_objects_for_heal_page( + &self, + _: &str, + _: &str, + _: Option<&str>, + _: bool, + ) -> Result<(Vec, Option, bool)> { + Err(Error::other("scanner admission must not traverse objects")) + } + + async fn get_disk_for_resume(&self, _: &str) -> Result { + Err(Error::other("scanner admission must retain the durable survivor")) + } +} + +struct ScannerFixture { + temp: TempDir, + manager: HealManager, + storage: Arc, + owner: ResumeManager, + task_id: String, + local_targets: Vec, +} + +impl Drop for ScannerFixture { + fn drop(&mut self) { + self.manager.cancel_token.cancel(); + } +} + +impl ScannerFixture { + async fn new(unformatted_siblings: bool) -> Self { + let temp = TempDir::new().expect("scanner replacement fixture directory"); + let disk_ids = (0..3).map(|_| Uuid::new_v4()).collect::>(); + let deployment_id = Uuid::new_v4(); + let mut endpoints = Vec::new(); + for slot in 0..3 { + let path = temp.path().join(format!("target-{slot}")); + let metadata_path = path.join(crate::heal::RUSTFS_META_BUCKET); + std::fs::create_dir_all(&metadata_path).expect("scanner local target metadata directory"); + if !unformatted_siblings || slot == 0 { + let format = serde_json::json!({ + "version": "1", + "format": "xl", + "id": deployment_id, + "xl": { + "version": "3", + "this": disk_ids[slot], + "sets": [disk_ids], + "distributionAlgo": "SIPMOD+PARITY" + } + }); + std::fs::write( + metadata_path.join("format.json"), + serde_json::to_vec(&format).expect("scanner format fixture serialization"), + ) + .expect("scanner formatted target fixture"); + } + let mut endpoint = + Endpoint::try_from(path.to_str().expect("scanner target path is UTF-8")).expect("scanner local endpoint"); + endpoint.set_pool_index(0); + endpoint.set_set_index(0); + endpoint.set_disk_index(slot); + endpoints.push(endpoint); + } + let local_targets = endpoints.iter().map(ToString::to_string).collect::>(); + init_local_disks(EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 3, + endpoints: Endpoints::from(endpoints), + cmd_line: "replacement-scanner-recovery-test".into(), + platform: "test".into(), + }])) + .await + .expect("scanner local registry initialization"); + + let survivor = make_manager_resume_disk(&temp, "survivor").await; + let local_disks = crate::heal::local_disk_map_read().await; + assert_eq!(local_disks.len(), 3); + assert!( + !local_disks.contains_key(&survivor.endpoint().to_string()), + "the durable anchor must be discoverable only through the set inventory" + ); + for disk in local_disks.values().flatten() { + let is_owner = disk.endpoint().to_string() == local_targets[0]; + match disk.get_disk_id().await { + Ok(Some(id)) => assert!(!id.is_nil()), + Err(DiskError::UnformattedDisk) => assert!(unformatted_siblings && !is_owner), + result => panic!("unexpected fixture disk identity: {result:?}"), + } + assert_eq!(disk.runtime_state().as_str(), "online"); + } + drop(local_disks); + + let identity = ReplacementTargetIdentity { + endpoint: local_targets[0].clone(), + canonical_path: local_targets[0].clone(), + physical_device_ids: vec!["replacement-target-0".into()], + filesystem_identity: "replacement-target-0-incarnation".into(), + }; + let task_id = ResumeUtils::generate_task_id(); + let owner = ResumeManager::new_replacement_intent( + survivor.clone(), + task_id.clone(), + SET.into(), + vec!["seeded-bucket".into()], + vec![local_targets[0].clone()], + vec![identity.clone()], + ) + .await + .expect("durable remote-survivor replacement intent"); + owner + .mark_replacement_rebuilding(vec![identity.clone()]) + .await + .expect("durable replacement rebuild phase"); + owner + .record_replacement_failure(&Error::Storage(EcstoreError::FileNotFound), 1) + .await + .expect("durable in-budget FileNotFound failure"); + let storage = Arc::new(ScannerStorage { + survivor, + sibling_survivors: Vec::new(), + identity, + inventory_calls: AtomicUsize::new(0), + }); + let manager = HealManager::new_without_root_recovery_for_test( + storage.clone(), + Some(HealConfig { + heal_interval: SCAN_INTERVAL, + ..HealConfig::default() + }), + ); + Self { + temp, + manager, + storage, + owner, + task_id, + local_targets, + } + } + + async fn queued_request(&self) -> HealRequest { + // Keep the paused clock from advancing while filesystem workers + // finish. A wall-clock deadline also bounds a broken admission path. + let deadline = std::time::Instant::now() + Duration::from_secs(5); + loop { + let queue = self.manager.heal_queue.lock().await; + if let Some(item) = queue.heap.peek() { + assert_eq!(queue.len(), 1, "one set must have exactly one admitted owner"); + return item.request.clone(); + } + drop(queue); + assert!(std::time::Instant::now() < deadline, "scanner tick did not admit replacement recovery"); + tokio::task::yield_now().await; + } + } + + async fn assert_original_owner_is_queued(&self) { + let request = self.queued_request().await; + assert_eq!(request.id, self.task_id, "scanner must replay the durable UUID"); + assert_eq!(request.heal_endpoints, vec![self.local_targets[0].clone()]); + assert_eq!(request.source, HealRequestSource::AutoHeal); + assert!(matches!( + request.heal_type, + HealType::ErasureSet { set_disk_id, buckets } + if set_disk_id == SET && buckets == ["seeded-bucket"] + )); + assert_eq!( + self.manager + .replacement_recovery_anchors + .lock() + .expect("scanner anchor registry") + .get(&self.task_id), + Some(&self.storage.survivor.endpoint().to_string()) + ); + assert!(self.manager.replacement_recovery_blocks().is_empty()); + assert_eq!( + self.storage.inventory_calls.load(Ordering::SeqCst), + 1, + "three local slots in one set must share one inventory lookup per tick" + ); + let state = self.owner.get_state().await; + assert_eq!(state.retry_count, 1, "discovery must not reset the durable retry budget"); + assert_eq!(state.replacement_targets, request.heal_endpoints); + } +} + +async fn finish_without_advancing_clock(future: impl Future) -> T { + tokio::pin!(future); + loop { + tokio::select! { + biased; + result = &mut future => return result, + _ = tokio::task::yield_now() => {} + } + } +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial(replacement_auto_scan)] +async fn scanner_tick_recovers_remote_survivor_after_all_local_targets_are_formatted() { + let fixture = ScannerFixture::new(false).await; + fixture + .manager + .start_auto_disk_scanner() + .await + .expect("start only the periodic replacement scanner"); + fixture.assert_original_owner_is_queued().await; +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial(replacement_auto_scan)] +async fn scanner_tick_finishes_existing_owner_before_admitting_new_unformatted_targets() { + temp_env::async_with_vars([(rustfs_config::ENV_UNSAFE_BYPASS_DISK_CHECK, Some("true"))], async { + let fixture = ScannerFixture::new(true).await; + fixture + .manager + .start_auto_disk_scanner() + .await + .expect("start only the periodic replacement scanner"); + fixture.assert_original_owner_is_queued().await; + + // The real task's verified completion publishes the proof and + // removes its live intent before sibling admission is allowed. + let owner_removed = finish_without_advancing_clock(async { + fixture + .owner + .mark_replacement_completed_and_verified() + .await + .expect("durable owner completion proof"); + fixture + .owner + .mark_replacement_cleanup_pending() + .await + .expect("durable owner cleanup phase"); + fixture + .owner + .cleanup() + .await + .expect("completed owner releases its durable intent"); + !ResumeManager::has_replacement_intent(&fixture.storage.survivor, &fixture.task_id).await + }) + .await; + assert!(owner_removed, "completed ownership must be removed before the next generation"); + let finished = fixture + .manager + .heal_queue + .lock() + .await + .pop_next() + .expect("remove the completed queued owner"); + assert_eq!(finished.id, fixture.task_id); + tokio::time::advance(SCAN_INTERVAL).await; + + let next = fixture.queued_request().await; + assert_ne!(next.id, fixture.task_id, "the follow-up replacement gets a fresh generation"); + assert_eq!(next.heal_endpoints, fixture.local_targets[1..]); + assert_eq!(next.source, HealRequestSource::AutoHeal); + assert!(fixture.manager.replacement_recovery_blocks().is_empty()); + assert_eq!(fixture.storage.inventory_calls.load(Ordering::SeqCst), 2); + }) + .await; +} + +#[tokio::test] +async fn recovery_disk_index_keeps_first_snapshot_and_sorted_unique_endpoints() { + let temp = TempDir::new().expect("recovery disk index fixture"); + let local = make_manager_resume_disk(&temp, "a-local").await; + let duplicate = make_manager_resume_disk(&temp, "a-local").await; + let survivor = make_manager_resume_disk(&temp, "z-survivor").await; + assert!(!Arc::ptr_eq(&local, &duplicate), "duplicate must be a different disk snapshot"); + let local_endpoint = local.endpoint().to_string(); + let survivor_endpoint = survivor.endpoint().to_string(); + let mut inventory = std::collections::BTreeMap::new(); + super::super::auto_scan::extend_replacement_recovery_disks(&mut inventory, [local.clone()]); + super::super::auto_scan::extend_replacement_recovery_disks(&mut inventory, [survivor.clone(), duplicate, survivor]); + assert_eq!(inventory.len(), 2, "each endpoint is discovered once"); + assert!( + Arc::ptr_eq(inventory.get(&local_endpoint).expect("local endpoint remains indexed"), &local), + "later set inventories must not replace the local snapshot" + ); + assert_eq!( + inventory.keys().collect::>(), + vec![&local_endpoint, &survivor_endpoint], + "discovery order remains lexical regardless of insertion order" + ); +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial(replacement_auto_scan)] +async fn scanner_tick_blocks_only_the_conflicting_generation_of_a_set() { + // backlog#2902 step 3: two survivors hold the same revision of a sibling + // generation with different bytes. That generation is unverifiable, but + // the set's other generation must still be replayed. + let mut fixture = ScannerFixture::new(false).await; + let sibling_a = make_manager_resume_disk(&fixture.temp, "sibling-survivor-a").await; + let sibling_b = make_manager_resume_disk(&fixture.temp, "sibling-survivor-b").await; + let sibling_id = ResumeUtils::generate_task_id(); + let sibling_identity = ReplacementTargetIdentity { + endpoint: fixture.local_targets[1].clone(), + canonical_path: fixture.local_targets[1].clone(), + physical_device_ids: vec!["replacement-target-1".into()], + filesystem_identity: "replacement-target-1-incarnation".into(), + }; + for (disk, copy) in [(&sibling_a, "copy-a"), (&sibling_b, "copy-b")] { + let manager = ResumeManager::new_replacement_intent( + disk.clone(), + sibling_id.clone(), + SET.into(), + vec!["seeded-bucket".into()], + vec![fixture.local_targets[1].clone()], + vec![sibling_identity.clone()], + ) + .await + .expect("durable sibling generation copy"); + manager + .set_error(format!("divergent {copy}")) + .await + .expect("diverge the copies"); + } + let storage = Arc::new(ScannerStorage { + survivor: fixture.storage.survivor.clone(), + sibling_survivors: vec![sibling_a, sibling_b], + identity: fixture.storage.identity.clone(), + inventory_calls: AtomicUsize::new(0), + }); + fixture.storage = storage.clone(); + fixture.manager = HealManager::new_without_root_recovery_for_test( + storage, + Some(HealConfig { + heal_interval: SCAN_INTERVAL, + ..HealConfig::default() + }), + ); + fixture + .manager + .start_auto_disk_scanner() + .await + .expect("start only the periodic replacement scanner"); + + let request = fixture.queued_request().await; + assert_eq!(request.id, fixture.task_id, "the clean generation of the set is replayed"); + assert_eq!(request.heal_endpoints, vec![fixture.local_targets[0].clone()]); + assert_eq!(fixture.owner.get_state().await.retry_count, 1, "replay must not touch the durable budget"); + assert!(!fixture.manager.replacement_recovery_generation_is_blocked(&fixture.task_id)); + let blocks = fixture.manager.replacement_recovery_blocks(); + assert_eq!(blocks.len(), 1, "only the unverifiable generation is blocked: {blocks:?}"); + assert_eq!(blocks[0].task_id, sibling_id); + assert_eq!(blocks[0].set_disk_id, SET); + assert!(blocks[0].reason.contains("diverge"), "{}", blocks[0].reason); + assert!(blocks[0].reserves_targets, "an active intent still reserves its slot"); + assert!(blocks[0].blocked_since > 0); +} diff --git a/crates/heal/src/heal/manager/tests/disposition.rs b/crates/heal/src/heal/manager/tests/disposition.rs new file mode 100644 index 000000000..fe3d2761d --- /dev/null +++ b/crates/heal/src/heal/manager/tests/disposition.rs @@ -0,0 +1,392 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Admission-side behaviour of the per-object disposition cache (backlog#2900). + +use super::*; +use crate::heal::manager::disposition::{HealDispositionCache, HealDispositionKey, HealDispositionKind, record_task_disposition}; +use crate::heal::storage::HealListItem; +use rustfs_heal_contracts::heal_channel::{HealAdmissionDropReason, HealAdmissionResult}; +use std::time::Duration; +use tokio::time::Instant; + +const BUCKET: &str = "bucket-a"; + +fn object_request(object: &str, source: HealRequestSource, priority: HealPriority, version: Option<&str>) -> HealRequest { + let mut request = HealRequest::new( + HealType::Object { + bucket: BUCKET.to_string(), + object: object.to_string(), + version_id: version.map(str::to_string), + }, + HealOptions::default(), + priority, + ); + request.source = source; + request +} + +fn key(object: &str, version: Option<&str>) -> HealDispositionKey { + HealDispositionKey::new(BUCKET, object, version, None, None) +} + +fn seed(manager: &HealManager, object: &str, kind: HealDispositionKind, task_id: &str) { + manager + .disposition_cache + .lock() + .expect("disposition cache") + .record(key(object, None), kind, task_id, Instant::now()); +} + +fn cached_kind(manager: &HealManager, object: &str) -> Option { + manager + .disposition_cache + .lock() + .expect("disposition cache") + .get(&key(object, None)) + .map(|entry| entry.kind) +} + +#[tokio::test] +async fn disposition_absent_hit_drops_best_effort_object_requests_before_the_queue() { + let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None); + seed(&manager, "gone", HealDispositionKind::Absent, "absent-task"); + for source in [ + HealRequestSource::ReadRepair, + HealRequestSource::Scanner, + HealRequestSource::AutoHeal, + HealRequestSource::Internal, + HealRequestSource::Mrf, + ] { + for priority in [HealPriority::Low, HealPriority::Normal] { + let receipt = manager + .submit_heal_request_with_receipt(object_request("gone", source, priority, None)) + .await + .expect("admission decides"); + assert_eq!( + receipt.result, + HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped), + "{source:?} {priority:?}" + ); + assert_eq!(receipt.task_id, "absent-task", "the receipt points at the task that observed absence"); + } + } + assert!(manager.heal_queue.lock().await.is_empty(), "a cached absence never reaches the queue"); + // A different version of the same key is a different memory. + let receipt = manager + .submit_heal_request_with_receipt(object_request("gone", HealRequestSource::ReadRepair, HealPriority::Low, Some("v1"))) + .await + .expect("admission decides"); + assert_eq!(receipt.result, HealAdmissionResult::Accepted); +} + +#[tokio::test] +async fn disposition_scoped_absence_does_not_block_other_scopes() { + for (pool, set, expected) in [ + (Some(0), Some(0), HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped)), + (Some(1), Some(0), HealAdmissionResult::Accepted), + (Some(0), Some(1), HealAdmissionResult::Accepted), + (Some(0), None, HealAdmissionResult::Accepted), + (None, Some(0), HealAdmissionResult::Accepted), + (None, None, HealAdmissionResult::Accepted), + ] { + let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None); + let mut old_pool = object_request("moved", HealRequestSource::ReadRepair, HealPriority::Low, None); + old_pool.options.pool_index = Some(0); + old_pool.options.set_index = Some(0); + let absent = task_for(old_pool); + absent.mark_object_absent(); + record_task_disposition(&manager.disposition_cache, &absent, &Ok(()), false).await; + + let mut request = object_request("moved", HealRequestSource::ReadRepair, HealPriority::Low, None); + request.options.pool_index = pool; + request.options.set_index = set; + let receipt = manager + .submit_heal_request_with_receipt(request) + .await + .expect("scoped admission decides"); + assert_eq!(receipt.result, expected, "pool={pool:?}, set={set:?}"); + } +} + +#[tokio::test(start_paused = true)] +async fn disposition_unhealthy_admin_object_name_does_not_cache_absence() { + let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None); + let object = "reports/File not found.txt"; + let task = task_for(object_request(object, HealRequestSource::Admin, HealPriority::Normal, None)); + let failure = Error::TaskExecutionFailed { + message: format!( + "Heal left one or more drives unhealthy for {BUCKET}/{object}; retry after the missing drives are restored" + ), + }; + record_task_disposition(&manager.disposition_cache, &task, &Err(failure), false).await; + assert_eq!(cached_kind(&manager, object), Some(HealDispositionKind::Permanent)); + + tokio::time::advance(Duration::from_secs(61)).await; + let receipt = manager + .submit_heal_request_with_receipt(object_request(object, HealRequestSource::ReadRepair, HealPriority::Low, None)) + .await + .expect("repair can retry after the failure backoff"); + assert_eq!(receipt.result, HealAdmissionResult::Accepted); +} + +#[tokio::test] +async fn disposition_cache_is_bypassed_by_admin_forced_high_priority_dry_run_durable_mrf_and_other_kinds() { + let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None); + for object in ["admin", "forced", "high", "urgent", "dry", "durable", "metadata"] { + seed(&manager, object, HealDispositionKind::Absent, "absent-task"); + } + let admin = object_request("admin", HealRequestSource::Admin, HealPriority::Normal, None); + let mut forced = object_request("forced", HealRequestSource::Scanner, HealPriority::Low, None); + forced.force_start = true; + let high = object_request("high", HealRequestSource::Scanner, HealPriority::High, None); + let urgent = object_request("urgent", HealRequestSource::ReadRepair, HealPriority::Urgent, None); + let mut dry = object_request("dry", HealRequestSource::Scanner, HealPriority::Low, None); + dry.options.dry_run = true; + let mut durable = object_request("durable", HealRequestSource::Mrf, HealPriority::Normal, None); + durable.expected_mrf_bucket_incarnation_id = Some(Uuid::new_v4()); + let mut metadata = HealRequest::new( + HealType::Metadata { + bucket: BUCKET.to_string(), + object: "metadata".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + metadata.source = HealRequestSource::Scanner; + for (label, request) in [ + ("admin", admin), + ("forced", forced), + ("high", high), + ("urgent", urgent), + ("dry", dry), + ("durable", durable), + ("metadata", metadata), + ] { + let receipt = manager + .submit_heal_request_with_receipt(request) + .await + .expect("admission decides"); + assert_eq!(receipt.result, HealAdmissionResult::Accepted, "{label}"); + } + assert_eq!(manager.heal_queue.lock().await.len(), 7); +} + +#[tokio::test(start_paused = true)] +async fn disposition_backed_off_failure_is_dropped_then_admitted_after_the_backoff() { + let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None); + seed(&manager, "flaky", HealDispositionKind::RetryExhausted, "failed-task"); + let dropped = manager + .submit_heal_request_with_receipt(object_request("flaky", HealRequestSource::Scanner, HealPriority::Low, None)) + .await + .expect("admission decides"); + assert_eq!(dropped.result, HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped)); + assert_eq!(dropped.task_id, "failed-task"); + tokio::time::advance(Duration::from_secs(61)).await; + let admitted = manager + .submit_heal_request_with_receipt(object_request("flaky", HealRequestSource::Scanner, HealPriority::Low, None)) + .await + .expect("admission decides"); + assert_eq!(admitted.result, HealAdmissionResult::Accepted, "the backoff elapsed"); + assert_eq!( + cached_kind(&manager, "flaky"), + Some(HealDispositionKind::RetryExhausted), + "admission does not forget the failure" + ); +} + +fn task_for(request: HealRequest) -> HealTask { + HealTask::from_request(request, Arc::new(MockStorage)) +} + +#[tokio::test] +async fn disposition_record_task_result_maps_terminal_outcomes() { + let cache = StdMutex::new(HealDispositionCache::with_limits(16, Duration::from_secs(600))); + let kind = |object: &str| cache.lock().expect("cache").get(&key(object, None)).map(|entry| entry.kind); + + let absent = task_for(object_request("absent", HealRequestSource::ReadRepair, HealPriority::Low, None)); + absent.mark_object_absent(); + record_task_disposition(&cache, &absent, &Ok(()), false).await; + assert_eq!(kind("absent"), Some(HealDispositionKind::Absent)); + + let retried = task_for(object_request("retried", HealRequestSource::Scanner, HealPriority::Low, None)); + let quorum = || Error::Storage(EcstoreError::InsufficientReadQuorum(BUCKET.to_string(), "retried".to_string())); + record_task_disposition(&cache, &retried, &Err(quorum()), true).await; + assert_eq!(kind("retried"), None, "a scheduled retry is not terminal"); + record_task_disposition(&cache, &retried, &Err(quorum()), false).await; + assert_eq!(kind("retried"), Some(HealDispositionKind::RetryExhausted)); + + let denied = task_for(object_request("denied", HealRequestSource::AutoHeal, HealPriority::Normal, None)); + record_task_disposition(&cache, &denied, &Err(Error::Disk(DiskError::FileAccessDenied)), false).await; + assert_eq!(kind("denied"), Some(HealDispositionKind::Permanent)); + + let cancelled = task_for(object_request("cancelled", HealRequestSource::Scanner, HealPriority::Low, None)); + record_task_disposition(&cache, &cancelled, &Err(Error::TaskCancelled), false).await; + record_task_disposition(&cache, &cancelled, &Err(Error::TaskTimeout), false).await; + assert_eq!(kind("cancelled"), None); + + // A positive completion forgets an earlier failure. + record_task_disposition(&cache, &denied, &Ok(()), false).await; + assert_eq!(kind("denied"), None); + + // Durable MRF, dry runs and other task kinds never write the memory. + let mut durable_request = object_request("durable", HealRequestSource::Mrf, HealPriority::Normal, None); + durable_request.expected_mrf_bucket_incarnation_id = Some(Uuid::new_v4()); + let durable = task_for(durable_request); + durable.mark_object_absent(); + record_task_disposition(&cache, &durable, &Ok(()), false).await; + assert_eq!(kind("durable"), None); + let mut dry_request = object_request("dry", HealRequestSource::Scanner, HealPriority::Low, None); + dry_request.options.dry_run = true; + let dry = task_for(dry_request); + dry.mark_object_absent(); + record_task_disposition(&cache, &dry, &Ok(()), false).await; + assert_eq!(kind("dry"), None); + let mut metadata = HealRequest::new( + HealType::Metadata { + bucket: BUCKET.to_string(), + object: "metadata".to_string(), + }, + HealOptions::default(), + HealPriority::Low, + ); + metadata.source = HealRequestSource::Scanner; + let metadata = task_for(metadata); + record_task_disposition(&cache, &metadata, &Err(Error::Disk(DiskError::FileAccessDenied)), false).await; + assert_eq!(kind("metadata"), None); +} + +struct AbsentStorage; + +#[async_trait::async_trait] +impl HealStorageAPI for AbsentStorage { + async fn admit_bucket_incarnation(&self, _: &str) -> Result { + Ok(Uuid::from_u128(7)) + } + async fn heal_bucket_at_incarnation(&self, bucket: &str, expected: Uuid, opts: &HealOpts) -> Result { + self.validate_bucket_incarnation(bucket, Some(expected)).await?; + self.heal_bucket(bucket, opts).await + } + async fn heal_object_at_incarnation( + &self, + bucket: &str, + object: &str, + version_id: Option<&str>, + expected: Uuid, + opts: &HealOpts, + ) -> Result { + self.validate_bucket_incarnation(bucket, Some(expected)).await?; + self.heal_object_with_receipt(bucket, object, version_id, opts).await + } + async fn get_object_meta(&self, _: &str, _: &str) -> Result> { + Ok(None) + } + async fn ec_decode_rebuild(&self, _: &str, _: &str) -> Result> { + Ok(Vec::new()) + } + async fn get_bucket_info(&self, _: &str) -> Result> { + Ok(None) + } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + Ok(Vec::new()) + } + async fn list_buckets(&self) -> Result> { + Ok(Vec::new()) + } + async fn object_exists(&self, _: &str, _: &str) -> Result { + Ok(false) + } + async fn heal_bucket(&self, _: &str, _: &HealOpts) -> Result { + Ok(HealResultItem::default()) + } + async fn heal_format(&self, _: bool) -> Result<(HealResultItem, Option)> { + Ok((HealResultItem::default(), None)) + } + async fn get_disk_for_resume(&self, _: &str) -> Result { + Err(Error::other("no resume disk in this fixture")) + } + async fn list_objects_for_heal_page( + &self, + _: &str, + _: &str, + _: Option<&str>, + _: bool, + ) -> Result<(Vec, Option, bool)> { + Ok((Vec::new(), None, false)) + } + async fn heal_object(&self, _: &str, _: &str, _: Option<&str>, _: &HealOpts) -> Result<(HealResultItem, Option)> { + // The locked quorum read found no replica on any disk. + Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::FileNotFound)))) + } +} + +async fn wait_until_idle(manager: &HealManager) { + tokio::time::timeout(Duration::from_secs(10), async { + while !manager.active_heals.lock().await.is_empty() { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await + .expect("the absent object heal completes"); +} + +#[tokio::test] +async fn disposition_read_repair_absent_object_is_remembered_after_one_task() { + let manager = HealManager::new_without_root_recovery_for_test( + Arc::new(AbsentStorage), + Some(HealConfig { + max_concurrent_heals: 1, + ..HealConfig::default() + }), + ); + let first = object_request("deleted", HealRequestSource::ReadRepair, HealPriority::Low, None); + let first_id = first.id.clone(); + let receipt = manager + .submit_heal_request_with_receipt(first) + .await + .expect("admission decides"); + assert_eq!(receipt.result, HealAdmissionResult::Accepted); + process_manager_queue_once(&manager).await; + wait_until_idle(&manager).await; + assert_eq!( + manager + .get_task_status(&first_id) + .await + .expect("completed status is retained"), + HealTaskStatus::Completed, + "an absent object is a completed heal, not a failed one" + ); + assert_eq!(cached_kind(&manager, "deleted"), Some(HealDispositionKind::Absent)); + + // The producer's reservation expired and it resubmits: nothing is queued. + let second = manager + .submit_heal_request_with_receipt(object_request("deleted", HealRequestSource::ReadRepair, HealPriority::Low, None)) + .await + .expect("admission decides"); + assert_eq!(second.result, HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped)); + assert_eq!(second.task_id, first_id); + assert!(manager.heal_queue.lock().await.is_empty()); + + // A durable MRF responsibility for the same key still reaches the queue. + let mut durable = object_request("deleted", HealRequestSource::Mrf, HealPriority::Normal, None); + durable.expected_mrf_bucket_incarnation_id = Some(Uuid::from_u128(7)); + let durable = manager + .submit_heal_request_with_receipt(durable) + .await + .expect("admission decides"); + assert_eq!(durable.result, HealAdmissionResult::Accepted); +} diff --git a/crates/heal/src/heal/manager/tests/root_recovery.rs b/crates/heal/src/heal/manager/tests/root_recovery.rs index 732c83383..837260b58 100644 --- a/crates/heal/src/heal/manager/tests/root_recovery.rs +++ b/crates/heal/src/heal/manager/tests/root_recovery.rs @@ -1844,7 +1844,7 @@ async fn root_recovery_queued_non_root_admin_owner_is_not_priority_displaced() { } #[tokio::test] -async fn root_recovery_legacy_schema_replays_as_cluster() { +async fn root_recovery_legacy_cluster_requires_a_fresh_generation_bound_task() { #[derive(serde::Serialize)] struct LegacyRootHealIntent<'a> { schema: u32, @@ -1873,10 +1873,113 @@ async fn root_recovery_legacy_schema_replays_as_cluster() { let restarted = recovery_manager(vec![disk]); restarted.replay_root_heals().await.expect("replay legacy root record"); - let queued = restarted.heal_queue.lock().await.requests().cloned().collect::>(); - assert_eq!(queued.len(), 1); - assert_eq!(queued[0].id, request.id); - assert_eq!(queued[0].heal_type, HealType::Cluster); + assert!( + restarted.heal_queue.lock().await.requests().next().is_none(), + "unfenced legacy work must not bind to a later bucket generation" + ); + assert!( + restarted + .root_recovery + .inventory() + .await + .expect("quarantined legacy task") + .quarantined_task_ids + .contains(&request.id) + ); +} + +#[tokio::test] +async fn root_recovery_cluster_scope_preserves_generations_and_completed_prefix_on_handoff() { + use crate::heal::manager::root_recovery::AdminClusterScope; + use std::collections::BTreeMap; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk.clone()]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request.clone(), Arc::new(MockStorage)); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let mut scope = AdminClusterScope { + buckets: BTreeMap::from([ + ("alpha-bucket".to_owned(), Uuid::from_u128(42)), + ("bravo-bucket".to_owned(), Uuid::from_u128(43)), + ]), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("freeze bucket generations before mutation"); + scope.next_bucket_index = 1; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("checkpoint completed empty bucket"); + let retry = task.retry_request(); + manager + .root_recovery + .persist(&retry) + .await + .expect("shutdown/retry handoff retains scope"); + let mut changed_options = retry.clone(); + changed_options.options.recursive = !changed_options.options.recursive; + assert!( + manager.root_recovery.persist(&changed_options).await.is_err(), + "handoff cannot reinterpret completed work" + ); + drop(manager); + let restarted = recovery_manager(vec![disk]); + let replay = HealTask::from_request(retry, Arc::new(MockStorage)); + assert_eq!( + restarted + .root_recovery + .cluster_scope(&replay) + .await + .expect("read frozen scope"), + Some(scope.clone()) + ); + let mut changed = scope.clone(); + changed.buckets.insert("alpha-bucket".to_owned(), Uuid::from_u128(44)); + assert!( + restarted + .root_recovery + .checkpoint_cluster_scope(&replay, &changed) + .await + .is_err(), + "never rebind a saved generation" + ); + changed = scope.clone(); + changed.next_bucket_index = 0; + assert!( + restarted + .root_recovery + .checkpoint_cluster_scope(&replay, &changed) + .await + .is_err(), + "never rewind a committed prefix" + ); + changed = scope.clone(); + changed.buckets.insert("../invalid".to_owned(), Uuid::from_u128(45)); + assert!( + restarted + .root_recovery + .checkpoint_cluster_scope(&replay, &changed) + .await + .is_err(), + "reject decoded path aliases" + ); + assert_eq!( + restarted + .root_recovery + .cluster_scope(&replay) + .await + .expect("failed updates preserve scope"), + Some(scope) + ); } #[tokio::test] @@ -1899,6 +2002,155 @@ async fn root_recovery_rejected_admin_start_does_not_persist() { assert_eq!(restarted.get_queue_length().await, 0); } +#[tokio::test] +async fn root_recovery_cluster_replay_rejects_successor_and_preserves_completed_scope() { + use crate::heal::manager::root_recovery::AdminClusterScope; + use crate::heal::outcome::{HealObjectDisposition, HealObjectIdentity, HealObjectKind, HealObjectOutcome}; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request.clone(), Arc::new(MockStorage)); + let bucket = format!("pinned-{}", Uuid::new_v4().simple()); + let expected = Uuid::from_u128(42); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let mut scope = AdminClusterScope { + buckets: std::collections::BTreeMap::from([(bucket.clone(), expected)]), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("freeze generation"); + bucket_incarnations() + .lock() + .expect("generation fixture") + .insert(bucket.clone(), Some(Uuid::from_u128(99))); + let replay = + HealTask::from_request(request.clone(), Arc::new(MockStorage)).with_admin_recovery(manager.root_recovery.clone()); + assert!( + matches!(replay.execute().await, Err(Error::StaleBucketIncarnation { .. })), + "replay must reject a successor before traversal" + ); + assert_eq!(replay.get_outcome().await.counters.processed, 0); + bucket_incarnations() + .lock() + .expect("generation fixture") + .insert(bucket.clone(), Some(expected)); + scope.next_bucket_index = 1; + scope.outcome.record(HealObjectOutcome { + identity: HealObjectIdentity { + kind: HealObjectKind::Object, + bucket: bucket.clone(), + object: "completed-object".to_owned(), + version_id: None, + bucket_incarnation_id: Some(expected), + pool_index: None, + set_index: None, + }, + disposition: HealObjectDisposition::VerifiedHealthy, + detail: None, + }); + scope.progress.objects_scanned = 1; + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("completed bucket with retained outcome"); + let replay = HealTask::from_request(request, Arc::new(MockStorage)).with_admin_recovery(manager.root_recovery.clone()); + replay + .execute() + .await + .expect("completed frozen scope needs no rediscovery or repair"); + assert_eq!(replay.get_outcome().await.coverage, crate::heal::outcome::HealTraversalCoverage::Complete); + assert_eq!(replay.get_outcome().await.counters.processed, 1); + assert_eq!(replay.get_progress().await.objects_scanned, 1); + bucket_incarnations().lock().expect("generation fixture").remove(&bucket); +} + +#[tokio::test] +async fn root_recovery_cluster_inventory_exceeds_legacy_limit_without_losing_bindings() { + use crate::heal::manager::root_recovery::AdminClusterScope; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk.clone()]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request.clone(), Arc::new(MockStorage)); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let scope = AdminClusterScope { + buckets: (0..1500) + .map(|index| (format!("bucket-{index:05}"), Uuid::from_u128(42))) + .collect(), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + assert!(scope.checkpoint_interval() > 1, "large inventories must batch durable rewrites"); + manager + .root_recovery + .checkpoint_cluster_scope(&task, &scope) + .await + .expect("persist large frozen inventory"); + let path = format!("root-heal-{}.json", request.id); + let bytes = disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("scope bytes"); + assert!(bytes.len() > MAX_ROOT_RECOVERY_BYTES, "fixture must cross legacy size limit"); + assert_eq!( + manager + .root_recovery + .cluster_scope(&task) + .await + .expect("all bindings survive"), + Some(scope) + ); + let json = String::from_utf8(bytes.to_vec()).expect("UTF-8 scope"); + let binding = format!("\"bucket-00000\":\"{}\"", Uuid::from_u128(42)); + let duplicate = json.replacen(&binding, &format!("{binding},{binding}"), 1); + assert_ne!(duplicate, json, "inject a duplicate map key"); + disk.write_all(RUSTFS_META_BUCKET, &path, duplicate.into_bytes().into()) + .await + .expect("inject corrupt duplicate binding"); + assert!( + manager.root_recovery.cluster_scope(&task).await.is_err(), + "duplicate identity bindings fail closed" + ); +} + +#[tokio::test] +async fn root_recovery_cluster_rejects_inventory_without_checkpoint_headroom() { + use crate::heal::manager::root_recovery::AdminClusterScope; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk]); + let request = root_request(); + manager.root_recovery.persist(&request).await.expect("admit cluster task"); + let task = HealTask::from_request(request, Arc::new(MockStorage)); + let mut outcome = HealTaskOutcome::default(); + outcome.start(); + let scope = AdminClusterScope { + // This inventory fits the 16 MiB record limit but leaves too little + // space for future object outcomes. Reject it before authorizing repair. + buckets: (0..152_000) + .map(|index| (format!("bucket-{index:056}"), Uuid::from_u128(42))) + .collect(), + next_bucket_index: 0, + outcome, + progress: HealProgress::new(), + }; + assert!(manager.root_recovery.checkpoint_cluster_scope(&task, &scope).await.is_err()); + assert!( + manager + .root_recovery + .cluster_scope(&task) + .await + .expect("original admission survives") + .is_none() + ); +} + #[tokio::test] async fn root_recovery_queued_owner_is_not_priority_displaced() { let (_temp, disk) = recovery_disk().await; @@ -2127,7 +2379,16 @@ async fn root_recovery_force_start_replaces_fresh_queued_and_retrying_admin_root #[tokio::test] async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() { - for kind in ["truncated", "schema", "identity", "option", "no_lock", "oversized"] { + use crate::heal::manager::root_recovery::MAX_CLUSTER_RECOVERY_BYTES; + for kind in [ + "truncated", + "schema", + "identity", + "option", + "no_lock", + "unscoped_oversized", + "oversized", + ] { let (_temp, disk) = recovery_disk().await; let manager = recovery_manager(vec![disk.clone()]); let valid = root_request(); @@ -2142,7 +2403,7 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() let original = disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("read root record"); let mut value: serde_json::Value = serde_json::from_slice(&original).expect("record JSON"); match kind { - "schema" => value["schema"] = 4.into(), + "schema" => value["schema"] = u32::MAX.into(), "identity" => value["task_id"] = valid.id.clone().into(), "option" => value["options"]["future_delete_mode"] = true.into(), "no_lock" => value["options"]["no_lock"] = true.into(), @@ -2150,7 +2411,12 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() } let bytes = match kind { "truncated" => b"{".to_vec(), - "oversized" => vec![b' '; MAX_ROOT_RECOVERY_BYTES + 1], + "oversized" => vec![b' '; MAX_CLUSTER_RECOVERY_BYTES + 1], + "unscoped_oversized" => { + let mut bytes = serde_json::to_vec(&value).expect("valid unscoped record"); + bytes.resize(MAX_ROOT_RECOVERY_BYTES + 1, b' '); + bytes + } _ => serde_json::to_vec(&value).expect("modified record"), }; disk.write_all(RUSTFS_META_BUCKET, &path, bytes.clone().into()) @@ -2173,7 +2439,11 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() let marker: serde_json::Value = serde_json::from_slice(&marker).expect("quarantine marker JSON"); assert_eq!(marker["task_id"], invalid.id, "kind={kind}"); assert_eq!(marker["record_kind"], "intent", "kind={kind}"); - assert_eq!(marker["source_prefix_len"], bytes.len().min(MAX_ROOT_RECOVERY_BYTES + 1), "kind={kind}"); + assert_eq!( + marker["source_prefix_len"], + bytes.len().min(MAX_CLUSTER_RECOVERY_BYTES + 1), + "kind={kind}" + ); assert_eq!(marker["source_oversized"], kind == "oversized", "kind={kind}"); let mut reused = invalid.clone(); @@ -2196,7 +2466,63 @@ async fn root_recovery_invalid_records_are_quarantined_without_blocking_replay() } #[tokio::test] -async fn root_recovery_quarantine_source_change_fails_closed() { +async fn root_recovery_legacy_oversized_quarantine_survives_larger_intent_limit() { + use sha2::{Digest, Sha256}; + let (_temp, disk) = recovery_disk().await; + let manager = recovery_manager(vec![disk.clone()]); + let request = root_request(); + let path = format!("root-heal-{}.json", request.id); + let mut bytes = vec![b' '; MAX_ROOT_RECOVERY_BYTES + 100]; + disk.write_all(RUSTFS_META_BUCKET, &path, bytes.clone().into()) + .await + .expect("old oversized intent"); + let marker = serde_json::json!({ + "schema": 1, + "task_id": request.id, + "record_kind": "intent", + "source_prefix_len": MAX_ROOT_RECOVERY_BYTES + 1, + "source_prefix_sha256": Sha256::digest(&bytes[..MAX_ROOT_RECOVERY_BYTES + 1]).to_vec(), + "source_oversized": true, + "reason": "legacy size limit", + "quarantined_at": SystemTime::now(), + }); + disk.write_all( + RUSTFS_META_BUCKET, + &format!("quarantined-root-heal-intent-{}.json", request.id), + serde_json::to_vec(&marker).unwrap().into(), + ) + .await + .expect("old quarantine marker"); + let inventory = manager + .root_recovery + .inventory() + .await + .expect("upgrade preserves old quarantine"); + assert!(inventory.requests.is_empty()); + assert!(inventory.quarantined_task_ids.contains(&request.id)); + bytes[0] = b'{'; + disk.write_all(RUSTFS_META_BUCKET, &path, bytes.into()) + .await + .expect("alter retained prefix"); + let inventory = manager + .root_recovery + .inventory() + .await + .expect("a changed legacy source is reported per record"); + assert!(inventory.requests.is_empty(), "the changed source is never replayed"); + assert!(!inventory.quarantined_task_ids.contains(&request.id)); + assert!( + inventory + .conflicted + .get(&request.id) + .is_some_and(|reason| reason.contains("quarantine source changed")), + "{:?}", + inventory.conflicted + ); +} + +#[tokio::test] +async fn root_recovery_quarantine_source_change_is_fenced_per_record() { let (_temp, disk) = recovery_disk().await; let manager = recovery_manager(vec![disk.clone()]); let request = root_request(); @@ -2208,7 +2534,29 @@ async fn root_recovery_quarantine_source_change_fails_closed() { disk.write_all(RUSTFS_META_BUCKET, &path, b"changed".to_vec().into()) .await .expect("change retained source"); - assert!(manager.root_recovery.pending().await.is_err(), "marker must fence a changed source"); + assert!( + manager + .root_recovery + .pending() + .await + .expect("a changed source is reported, not a failed inventory") + .is_empty(), + "marker must fence a changed source" + ); + let inventory = manager.root_recovery.inventory().await.expect("per-record inventory"); + assert!( + inventory + .conflicted + .get(&request.id) + .is_some_and(|reason| reason.contains("quarantine source changed")) + ); + let mut reused = request.clone(); + reused.force_start = true; + let refused = manager + .submit_heal_request(reused) + .await + .expect_err("a changed source keeps the task id reserved"); + assert!(refused.to_string().contains("quarantine source changed"), "{refused}"); } #[tokio::test] @@ -2707,3 +3055,74 @@ async fn root_recovery_erasure_bucket_scope_marker_quarantines_conflicts() { ); } } + +#[tokio::test] +async fn root_recovery_conflicting_record_does_not_block_other_replays() { + // A record with two durable owners cannot be admitted or quarantined, but + // it must not postpone every other accepted responsibility. + let (_temp_a, disk_a) = recovery_disk().await; + let (_temp_b, disk_b) = recovery_disk().await; + let manager = recovery_manager(vec![disk_a.clone(), disk_b.clone()]); + let valid = root_request(); + manager.root_recovery.persist(&valid).await.expect("valid root record"); + let conflicting = root_request(); + let bytes = disk_a + .read_all(RUSTFS_META_BUCKET, &format!("root-heal-{}.json", valid.id)) + .await + .expect("read valid record bytes") + .to_vec(); + let duplicated = String::from_utf8(bytes) + .expect("record JSON") + .replace(&valid.id, &conflicting.id); + for disk in [&disk_a, &disk_b] { + disk.write_all( + RUSTFS_META_BUCKET, + &format!("root-heal-{}.json", conflicting.id), + duplicated.clone().into_bytes().into(), + ) + .await + .expect("write duplicated owner"); + } + + manager + .replay_root_heals() + .await + .expect("one conflicting record must not fail the whole replay"); + assert_eq!(manager.get_queue_length().await, 1, "the valid record replays"); + assert!( + manager + .heal_queue + .lock() + .await + .requests() + .any(|request| request.id == valid.id) + ); + let inventory = manager.root_recovery.inventory().await.expect("per-record inventory"); + assert!( + inventory.requests.iter().all(|request| request.id != conflicting.id), + "a conflicting record is never replayed" + ); + let mut reused = conflicting.clone(); + reused.force_start = true; + let refused = manager + .submit_heal_request(reused) + .await + .expect_err("a conflicting task id stays reserved"); + assert!(refused.to_string().contains("Multiple root heal recovery owners"), "{refused}"); + assert!( + matches!( + manager.get_task_status(&conflicting.id).await, + Ok(HealTaskStatus::Failed { error }) if error.contains("Multiple root heal recovery owners") + ), + "the admin status interface reports why the record cannot run" + ); + for disk in [&disk_a, &disk_b] { + assert_eq!( + disk.read_all(RUSTFS_META_BUCKET, &format!("root-heal-{}.json", conflicting.id)) + .await + .expect("conflicting record retained") + .as_ref(), + duplicated.as_bytes() + ); + } +} diff --git a/crates/heal/src/heal/manager/tests/running_mainline.rs b/crates/heal/src/heal/manager/tests/running_mainline.rs index d7b827ce6..4cc153d28 100644 --- a/crates/heal/src/heal/manager/tests/running_mainline.rs +++ b/crates/heal/src/heal/manager/tests/running_mainline.rs @@ -80,6 +80,13 @@ impl HealStorageAPI for RunningStorage { ..Default::default() })) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { Ok(Vec::new()) } diff --git a/crates/heal/src/heal/manager/unclean_shutdown.rs b/crates/heal/src/heal/manager/unclean_shutdown.rs index 45396568a..dba9915e6 100644 --- a/crates/heal/src/heal/manager/unclean_shutdown.rs +++ b/crates/heal/src/heal/manager/unclean_shutdown.rs @@ -13,21 +13,14 @@ // limitations under the License. /// Unclean-shutdown recovery: durable replacement-intent discovery and healing-marker rewrite. use super::*; +use crate::heal::DiskStore; +use crate::heal::resume::{ + ReplacementRecoveryBlock, abandoned_targets_are_owned_by_another_generation, replacement_generation_reserves_targets, +}; +use std::collections::BTreeMap; pub(super) fn durable_replacement_reserves_targets(state: &ResumeState) -> bool { - state.replacement_generation.as_deref() == Some(state.task_id.as_str()) - && !state.replacement_targets.is_empty() - && (matches!( - state.replacement_phase, - ReplacementPhase::Intent - | ReplacementPhase::OwnershipPending - | ReplacementPhase::HandoffPending - | ReplacementPhase::Rebuilding - | ReplacementPhase::Verified - | ReplacementPhase::CleanupPending - ) || (state.replacement_phase == ReplacementPhase::Abandoned - && state.replacement_handoff.is_none() - && state.replacement_legacy_successor.is_none())) + replacement_generation_reserves_targets(state) } pub(super) fn durable_replacement_recovery_is_due(state: &ResumeState, task_id: &str) -> bool { @@ -67,16 +60,80 @@ pub(super) fn replacement_targets_belong_to_local_node( !replacement_targets.is_empty() && replacement_targets.iter().all(|target| local_endpoints.contains(target)) } -pub(super) fn unblock_replacement_recovery_sets_after_validation( - blocked_sets: &mut HashSet, - retry_succeeded: HashSet, - retry_failed: &HashSet, -) { - for set_disk_id in retry_succeeded { - if !retry_failed.contains(&set_disk_id) { - blocked_sets.remove(&set_disk_id); +/// Node-local registry of blocked replacement generations, shared by startup +/// discovery and the periodic scanner. Entries are keyed by generation, carry +/// the reason and observation times, and are re-evaluated on every discovery +/// pass; a block never widens to the generation's set. +#[derive(Clone, Default)] +pub(super) struct ReplacementRecoveryBlocks { + entries: Arc>>, +} + +impl ReplacementRecoveryBlocks { + fn entries(&self) -> std::sync::MutexGuard<'_, BTreeMap> { + self.entries.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) + } + + /// Record or refresh a block. The first observation time is kept; the + /// reason and reservation flag follow the latest pass. + pub(super) fn block(&self, task_id: &str, set_disk_id: &str, reason: impl Into, reserves_targets: bool) { + let now = SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs(); + let reason = reason.into(); + let mut entries = self.entries(); + match entries.get_mut(task_id) { + Some(entry) => { + entry.set_disk_id = set_disk_id.to_string(); + entry.reason = reason; + entry.reserves_targets = reserves_targets; + entry.last_observed = now; + } + None => { + entries.insert( + task_id.to_string(), + ReplacementRecoveryBlock { + task_id: task_id.to_string(), + set_disk_id: set_disk_id.to_string(), + reason, + reserves_targets, + blocked_since: now, + last_observed: now, + }, + ); + } } } + + /// Clear a block after the generation merged and resolved cleanly. + pub(super) fn unblock(&self, task_id: &str) -> bool { + self.entries().remove(task_id).is_some() + } + + #[cfg(test)] + pub(super) fn is_blocked(&self, task_id: &str) -> bool { + self.entries().contains_key(task_id) + } + + /// Fresh admission for a set waits while one of its blocked generations + /// still reserves target slots; a retired or released block does not. + pub(super) fn set_has_reserving_block(&self, set_disk_id: &str) -> bool { + self.entries() + .values() + .any(|entry| entry.set_disk_id == set_disk_id && entry.reserves_targets) + } + + pub(super) fn snapshot(&self) -> Vec { + self.entries().values().cloned().collect() + } +} + +/// Whether a durable generation still reserves its target slots against this +/// node's disks. An orphaned `Abandoned` record releases them once every slot's +/// healing marker names another generation of the same set. +pub(super) async fn replacement_reserves_targets_on(state: &ResumeState, disks: &[DiskStore]) -> bool { + durable_replacement_reserves_targets(state) && !abandoned_targets_are_owned_by_another_generation(state, disks).await } impl HealManager { @@ -91,7 +148,10 @@ impl HealManager { let mut replacement_intents = HashMap::, Vec, String)>::new(); let mut replacement_recovery_candidates = HashMap::::new(); let mut conflicted_replacement_tasks = HashSet::new(); + // Sets whose records could not be inventoried in this pass. The skip is + // pass-local: the next discovery pass reads them again. let mut conflicted_replacement_sets = HashSet::new(); + let blocks = self.replacement_recovery_blocks.clone(); { let local_disks = { @@ -165,7 +225,7 @@ impl HealManager { }; if let Err(error) = migration { if let Some(set_disk_id) = &disk_set_disk_id { - self.block_replacement_recovery_set(set_disk_id); + conflicted_replacement_sets.insert(set_disk_id.clone()); } warn!( target: "rustfs::heal::manager", @@ -183,7 +243,7 @@ impl HealManager { match self.storage.replacement_intent_disks(set_disk_id).await { Ok(disks) => recovery_disks.extend(disks), Err(error) => { - self.block_replacement_recovery_set(set_disk_id); + conflicted_replacement_sets.insert(set_disk_id.clone()); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -206,7 +266,7 @@ impl HealManager { Ok(task_ids) => task_ids, Err(error) => { if let Some(set_disk_id) = &disk_set_disk_id { - self.block_replacement_recovery_set(set_disk_id); + conflicted_replacement_sets.insert(set_disk_id.clone()); } warn!( target: "rustfs::heal::manager", @@ -224,9 +284,13 @@ impl HealManager { let manager = match ResumeManager::load_replacement_intent(disk.clone(), &task_id).await { Ok(manager) => manager, Err(error) => { - if let Some(set_disk_id) = &disk_set_disk_id { - self.block_replacement_recovery_set(set_disk_id); - } + conflicted_replacement_tasks.insert(task_id.clone()); + blocks.block( + &task_id, + disk_set_disk_id.as_deref().unwrap_or_default(), + format!("durable intent could not be loaded: {error}"), + true, + ); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -241,18 +305,19 @@ impl HealManager { } }; let state = manager.get_state().await; + let reserves_targets = replacement_reserves_targets_on(&state, &local_disks).await; if disk_set_disk_id .as_deref() .is_some_and(|disk_set| disk_set != state.set_disk_id) { let disk_set = disk_set_disk_id.as_deref().unwrap_or_default(); conflicted_replacement_tasks.insert(task_id.clone()); - conflicted_replacement_sets.insert(state.set_disk_id.clone()); - conflicted_replacement_sets.insert(disk_set.to_string()); - self.block_replacement_recovery_set(&state.set_disk_id); - if !disk_set.is_empty() { - self.block_replacement_recovery_set(disk_set); - } + blocks.block( + &task_id, + &state.set_disk_id, + format!("durable intent binds to {} but was found on a disk of {disk_set}", state.set_disk_id), + reserves_targets, + ); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -266,7 +331,7 @@ impl HealManager { ); continue; } - if durable_replacement_reserves_targets(&state) { + if reserves_targets { reserved_replacement_sets.insert(state.set_disk_id.clone()); } if !replacement_targets_belong_to_local_node(&state.replacement_targets, &local_endpoints) { @@ -283,31 +348,35 @@ impl HealManager { && (state.retry_count < state.max_retries || replacement_retry_is_exhausted_active(&state)); let verified_replacement = state.completed && matches!(state.replacement_phase, ReplacementPhase::Verified | ReplacementPhase::CleanupPending); - if !active_replacement && !verified_replacement && durable_replacement_reserves_targets(&state) { - self.block_replacement_recovery_set(&state.set_disk_id); + if !active_replacement && !verified_replacement { + if reserves_targets { + // Exhausted or orphaned: the record keeps its + // slots reserved until an operator repairs it, but + // only this generation is held back. + blocks.block( + &task_id, + &state.set_disk_id, + state + .error_message + .clone() + .unwrap_or_else(|| "durable generation reserves its targets but cannot run".to_string()), + true, + ); + } + continue; } - if (active_replacement || verified_replacement) - && state.replacement_generation.as_deref() == Some(task_id.as_str()) - && !state.replacement_targets.is_empty() + if state.replacement_generation.as_deref() == Some(task_id.as_str()) && !state.replacement_targets.is_empty() { if conflicted_replacement_tasks.contains(&task_id) { continue; } let mut selected = replacement_recovery_candidates.remove(&task_id); - let previous_set = selected.as_ref().map(|candidate| candidate.state.set_disk_id.clone()); let candidate = match ReplacementRecoveryCandidate::new(state.clone(), endpoint.to_string(), endpoint.is_local) { Ok(candidate) => candidate, Err(error) => { conflicted_replacement_tasks.insert(task_id.clone()); - conflicted_replacement_sets.insert(state.set_disk_id.clone()); - if let Some(previous_set) = &previous_set { - conflicted_replacement_sets.insert(previous_set.clone()); - } - self.block_replacement_recovery_set(&state.set_disk_id); - if let Some(previous_set) = &previous_set { - self.block_replacement_recovery_set(previous_set); - } + blocks.block(&task_id, &state.set_disk_id, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -321,15 +390,9 @@ impl HealManager { continue; } }; - let task_set = candidate.state.set_disk_id.clone(); if let Err(error) = merge_replacement_recovery_candidate(&mut selected, candidate) { conflicted_replacement_tasks.insert(task_id.clone()); - conflicted_replacement_sets.insert(task_set.clone()); - self.block_replacement_recovery_set(&task_set); - if let Some(previous_set) = &previous_set { - conflicted_replacement_sets.insert(previous_set.clone()); - self.block_replacement_recovery_set(previous_set); - } + blocks.block(&task_id, &state.set_disk_id, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -350,20 +413,23 @@ impl HealManager { // Replay only the canonical durable copy. Replica locations are // observations of one generation and must never each advance it. for (task_id, candidate) in replacement_recovery_candidates { + // Every copy merged cleanly: an earlier block for this + // generation no longer applies. Later failures re-block it. + blocks.unblock(&task_id); + let candidate_set = candidate.state.set_disk_id.clone(); + let reserves_targets = replacement_reserves_targets_on(&candidate.state, &local_disks).await; let Some(anchor_disk) = recovery_disks .iter() .find(|disk| disk.endpoint().to_string() == candidate.anchor) .cloned() else { - conflicted_replacement_sets.insert(candidate.state.set_disk_id.clone()); - self.block_replacement_recovery_set(&candidate.state.set_disk_id); + blocks.block(&task_id, &candidate_set, "durable anchor is unavailable", reserves_targets); continue; }; let manager = match ResumeManager::load_replacement_intent(anchor_disk, &task_id).await { Ok(manager) => manager, Err(error) => { - conflicted_replacement_sets.insert(candidate.state.set_disk_id.clone()); - self.block_replacement_recovery_set(&candidate.state.set_disk_id); + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -376,7 +442,7 @@ impl HealManager { continue; } }; - let mut state = manager.get_state().await; + let state = manager.get_state().await; let now = SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .unwrap_or_default() @@ -388,18 +454,25 @@ impl HealManager { } if replacement_retry_is_exhausted_active(&state) { match manager.rearm_replacement_recovery_if_needed(self.storage.as_ref()).await { - Ok(true) => state = manager.get_state().await, + // Resolution below reloads the re-armed state. + Ok(true) => {} Ok(false) => { // A ready target with an exhausted all-skip pass // is terminal for this generation; preserve its // reservation and do not enqueue a fresh task. - conflicted_replacement_sets.insert(state.set_disk_id.clone()); - self.block_replacement_recovery_set(&state.set_disk_id); + blocks.block( + &task_id, + &candidate_set, + state + .error_message + .clone() + .unwrap_or_else(|| "replacement retry budget exhausted".to_string()), + reserves_targets, + ); continue; } Err(error) => { - conflicted_replacement_sets.insert(state.set_disk_id.clone()); - self.block_replacement_recovery_set(&state.set_disk_id); + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -417,8 +490,7 @@ impl HealManager { Ok(state) => state, Err(Error::ReplacementTargetNotReady(_)) => continue, Err(error) => { - conflicted_replacement_sets.insert(state.set_disk_id.clone()); - self.block_replacement_recovery_set(&state.set_disk_id); + blocks.block(&task_id, &candidate_set, error.to_string(), reserves_targets); warn!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -431,6 +503,7 @@ impl HealManager { continue; } }; + blocks.unblock(&state.task_id); let resolved = (state.set_disk_id, state.replacement_targets, state.replacement_buckets, candidate.anchor); match replacement_intents.entry(state.task_id.clone()) { std::collections::hash_map::Entry::Vacant(entry) => { @@ -439,10 +512,13 @@ impl HealManager { std::collections::hash_map::Entry::Occupied(entry) => { let existing = entry.get(); if existing.0 != resolved.0 || existing.1 != resolved.1 || existing.2 != resolved.2 { - conflicted_replacement_sets.insert(existing.0.clone()); - conflicted_replacement_sets.insert(resolved.0.clone()); - self.block_replacement_recovery_set(&resolved.0); - self.block_replacement_recovery_set(&existing.0); + blocks.block( + &state.task_id, + &resolved.0, + "durable copies resolve to different set, targets, or bucket plans", + true, + ); + entry.remove(); } } } @@ -465,7 +541,7 @@ impl HealManager { let Ok((pool_index, set_index)) = crate::heal::utils::parse_set_disk_id(&set_disk_id) else { continue; }; - if self.replacement_recovery_set_is_blocked(&set_disk_id) { + if conflicted_replacement_sets.contains(&set_disk_id) { debug!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -473,12 +549,17 @@ impl HealManager { subsystem = LOG_SUBSYSTEM_MANAGER, set_disk_id, recovery_count = recoveries.len(), - "Replacement recovery deferred because durable recovery validation is blocked" + "Replacement recovery deferred because durable recovery inventory is incomplete" ); continue; } - if conflicted_replacement_sets.contains(&set_disk_id) || recoveries.len() != 1 { - self.block_replacement_recovery_set(&set_disk_id); + if recoveries.len() != 1 { + // Two active generations resolved for one set. Neither may + // run until an operator reconciles them; each is blocked on + // its own so other sets keep recovering. + for (task_id, _, _, _) in &recoveries { + blocks.block(task_id, &set_disk_id, "multiple active generations resolved for the same set", true); + } debug!( target: "rustfs::heal::manager", event = EVENT_HEAL_UNCLEAN_SHUTDOWN, @@ -569,7 +650,10 @@ impl HealManager { }; for set_disk_id in set_disk_ids { - if reserved_replacement_sets.contains(&set_disk_id) || self.replacement_recovery_set_is_blocked(&set_disk_id) { + if reserved_replacement_sets.contains(&set_disk_id) + || conflicted_replacement_sets.contains(&set_disk_id) + || blocks.set_has_reserving_block(&set_disk_id) + { continue; } let mut req = HealRequest::new( diff --git a/crates/heal/src/heal/mrf_queue.rs b/crates/heal/src/heal/mrf_queue.rs index c14faba97..2e98b9865 100644 --- a/crates/heal/src/heal/mrf_queue.rs +++ b/crates/heal/src/heal/mrf_queue.rs @@ -27,6 +27,7 @@ use super::{DiskStore, HealDiskExt as _, local_disk_map_read}; use crate::heal::manager::{HealManager, MrfRepairNoticeTarget}; +use futures::{StreamExt, stream}; use metrics::{counter, gauge}; use rustfs_common::mrf_channel::{ MRF_MAX_ATTEMPTS, MRF_MAX_DELETE_MARKER_BYTES, MrfDeleteMarkerPurge, MrfDeleteMarkerPurgeIdentity, MrfDurableAdmissionError, @@ -48,6 +49,7 @@ pub use partial_write::{MrfLegacyResponsibility, MrfOperatorAcceptance}; use partial_write::{PartialWrites, ResponsibilityCheckpoint}; const MRF_LIFECYCLE_CONTROL_CAPACITY: usize = 128; +const MRF_JOURNAL_DISK_CONCURRENCY: usize = 4; const MRF_LIFECYCLE_LIST_MAX_LIMIT: usize = 256; static GLOBAL_MRF_LIFECYCLE_SENDER: std::sync::OnceLock> = std::sync::OnceLock::new(); @@ -712,7 +714,29 @@ fn intent_digest(intent: &MrfIntent) -> Option<[u8; 32]> { async fn journal_disks() -> Vec { let map = local_disk_map_read().await; - map.values().flatten().cloned().collect() + distinct_journal_disks(map.values().flatten()) +} + +fn distinct_journal_disks<'a>(disks: impl IntoIterator) -> Vec { + let mut roots = HashSet::new(); + disks + .into_iter() + .filter(|disk| { + !crate::heal::storage_api::owner::EcstoreDiskAPI::is_local(disk.as_ref()) + || roots.insert(crate::heal::storage_api::owner::EcstoreDiskAPI::path(disk.as_ref())) + }) + .cloned() + .collect() +} + +// Bound disk fan-out without serializing independent fsyncs. The caller keeps +// each disk's commit steps in one future and waits for every replica result. +fn journal_disk_io(operations: I) -> impl std::future::Future> +where + I: IntoIterator, + F: std::future::Future, +{ + stream::iter(operations).buffered(MRF_JOURNAL_DISK_CONCURRENCY).collect() } async fn read_journal(path: &str) -> Option> { @@ -750,9 +774,15 @@ async fn read_mrf_lifecycle_checkpoint( /// the next tick retries the persist. async fn write_journal(path: &str, data: &[u8]) -> bool { let payload = bytes::Bytes::copy_from_slice(data); + let disks = journal_disks().await; + let results = journal_disk_io(disks.into_iter().map(|disk| { + let payload = payload.clone(); + async move { disk.write_all(super::RUSTFS_META_BUCKET, path, payload).await } + })) + .await; let mut any_persisted = false; - for disk in journal_disks().await { - match disk.write_all(super::RUSTFS_META_BUCKET, path, payload.clone()).await { + for result in results { + match result { Ok(()) => any_persisted = true, Err(err) => warn_mrf_journal_write(&err), } @@ -766,15 +796,17 @@ async fn delete_journal(path: &str) -> bool { counter!("rustfs_heal_mrf_journal_delete_failures_total").increment(1); return false; } + let results = journal_disk_io(disks.into_iter().map(|disk| async move { + disk.delete( + super::RUSTFS_META_BUCKET, + path, + crate::heal::storage_api::owner::EcstoreDeleteOptions::default(), + ) + .await + })) + .await; let mut all_deleted = true; - for disk in disks { - let result = disk - .delete( - super::RUSTFS_META_BUCKET, - path, - crate::heal::storage_api::owner::EcstoreDeleteOptions::default(), - ) - .await; + for result in results { if let Err(err) = result { // Delete is idempotent: a compatibility mirror that was never // written (or was already removed) is clean, not a retry state. @@ -2024,6 +2056,60 @@ mod tests { use std::sync::Arc as StdArc; use std::time::{Duration as StdDuration, Instant}; + #[tokio::test(start_paused = true)] + async fn journal_disk_io_bounds_fanout_and_collects_every_result() { + use std::sync::atomic::{AtomicUsize, Ordering}; + + let active = AtomicUsize::new(0); + let peak = AtomicUsize::new(0); + let completed = AtomicUsize::new(0); + let results = journal_disk_io((0..9).map(|index| { + let (active, peak, completed) = (&active, &peak, &completed); + async move { + let running = active.fetch_add(1, Ordering::SeqCst) + 1; + peak.fetch_max(running, Ordering::SeqCst); + tokio::time::sleep(Duration::from_secs(1)).await; + active.fetch_sub(1, Ordering::SeqCst); + completed.fetch_add(1, Ordering::SeqCst); + if index == 0 || index == 8 { Err(index) } else { Ok(index) } + } + })) + .await; + + assert_eq!(peak.load(Ordering::SeqCst), MRF_JOURNAL_DISK_CONCURRENCY); + assert_eq!(active.load(Ordering::SeqCst), 0); + assert_eq!(completed.load(Ordering::SeqCst), 9, "an early failure must not skip later replicas"); + assert_eq!(results, vec![Err(0), Ok(1), Ok(2), Ok(3), Ok(4), Ok(5), Ok(6), Ok(7), Err(8)]); + } + + #[tokio::test(start_paused = true)] + async fn journal_disk_io_cancellation_drops_inflight_operations() { + use std::sync::atomic::{AtomicUsize, Ordering}; + + struct InFlight<'a>(&'a AtomicUsize); + impl Drop for InFlight<'_> { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::SeqCst); + } + } + let started = AtomicUsize::new(0); + let active = AtomicUsize::new(0); + let work = journal_disk_io((0..9).map(|_| { + let (started, active) = (&started, &active); + async move { + started.fetch_add(1, Ordering::SeqCst); + active.fetch_add(1, Ordering::SeqCst); + let _guard = InFlight(active); + std::future::pending::<()>().await; + } + })); + tokio::time::timeout(Duration::from_secs(1), work) + .await + .expect_err("the unfinished fan-out must remain cancellable"); + assert_eq!(started.load(Ordering::SeqCst), MRF_JOURNAL_DISK_CONCURRENCY); + assert_eq!(active.load(Ordering::SeqCst), 0, "fan-out must not detach replica tasks"); + } + const W13_EVIDENCE_DIR_ENV: &str = "RUSTFS_SCANNER_HEAL_W13_EVIDENCE_DIR"; const W13_SOURCE_REVISION_ENV: &str = "RUSTFS_SCANNER_HEAL_W13_SOURCE_REVISION"; const W13_SELECTION_ENV: &str = "RUSTFS_SCANNER_HEAL_W13_SELECTION"; diff --git a/crates/heal/src/heal/mrf_queue/snapshot.rs b/crates/heal/src/heal/mrf_queue/snapshot.rs index 15144a0e9..25b2b71d3 100644 --- a/crates/heal/src/heal/mrf_queue/snapshot.rs +++ b/crates/heal/src/heal/mrf_queue/snapshot.rs @@ -28,7 +28,7 @@ //! One surviving committed replica supports process restart recovery only; //! this reader does not establish a replication quorum or a power-loss policy. -use super::{MRF_JOURNAL_PATH, MRF_SCOPED_JOURNAL_PATH, decode_journal}; +use super::{MRF_JOURNAL_PATH, MRF_SCOPED_JOURNAL_PATH, decode_journal, distinct_journal_disks, journal_disk_io}; use crate::heal::RUSTFS_META_BUCKET; use crate::heal::storage_api::owner::{ EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError, EcstoreDiskStore, @@ -307,7 +307,7 @@ async fn read_committed_with_stats( async fn cas_replace( disk: &EcstoreDiskStore, path: &str, - replacement: &[u8], + replacement: EcstoreDiskBytes, limit: usize, ) -> Result { let expected = read_bounded(disk, path, limit).await?.map(EcstoreDiskBytes::from); @@ -318,17 +318,11 @@ async fn cas_replace_expected( disk: &EcstoreDiskStore, path: &str, expected: Option, - replacement: &[u8], + replacement: EcstoreDiskBytes, ) -> Result { - EcstoreDiskAPI::compare_and_update_file( - disk.as_ref(), - RUSTFS_META_BUCKET, - path, - expected, - Some(EcstoreDiskBytes::copy_from_slice(replacement)), - ) - .await - .map_err(SnapshotError::Disk) + EcstoreDiskAPI::compare_and_update_file(disk.as_ref(), RUSTFS_META_BUCKET, path, expected, Some(replacement)) + .await + .map_err(SnapshotError::Disk) } async fn cas_delete_expected( @@ -391,56 +385,58 @@ pub async fn publish_committed_snapshot_with_companion( if decode_journal(payload).1 != 0 { return Err(SnapshotError::Corrupt); } - let current = read_committed(disks, limit).await?; + // Endpoint aliases can share the same canonical local root and CAS lock. + let disks = distinct_journal_disks(disks.iter()); + let current = read_committed(&disks, limit).await?; if current.as_ref().is_some_and(|snapshot| snapshot.sequence() >= sequence) { return Err(SnapshotError::Conflict); } let slot = current.as_ref().map_or(0, |snapshot| 1usize.saturating_sub(snapshot.slot())); - let manifest = Manifest::encode(owner, sequence, payload)?; + let manifest = EcstoreDiskBytes::from(Manifest::encode(owner, sequence, payload)?); + let payload = EcstoreDiskBytes::copy_from_slice(payload); + let companion = companion.map(|(paths, bytes, limit)| (paths, EcstoreDiskBytes::copy_from_slice(bytes), limit)); + // Share encoded bodies across replicas; parallel fsync must not multiply + // the large payload/lifecycle allocations. CAS steps stay ordered per disk. + let manifest = &manifest; + let payload = &payload; + let companion = &companion; + let results = journal_disk_io(disks.into_iter().map(|disk| async move { + let mut payload_written = false; + let manifest_written: Result = async { + let expected_manifest = read_bounded(&disk, MANIFEST_PATHS[slot], MANIFEST_LEN).await?; + validate_reusable_manifest_slot(expected_manifest.as_deref(), sequence, limit)?; + match cas_replace(&disk, PAYLOAD_PATHS[slot], payload.clone(), limit).await? { + EcstoreConditionalFileUpdate::Updated => payload_written = true, + EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch => return Ok(false), + } + if let Some((companion_paths, companion_bytes, companion_limit)) = companion { + match cas_replace(&disk, companion_paths[slot], companion_bytes.clone(), *companion_limit).await? { + EcstoreConditionalFileUpdate::Updated => {} + EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch => return Ok(false), + } + } + Ok(matches!( + cas_replace_expected( + &disk, + MANIFEST_PATHS[slot], + expected_manifest.map(EcstoreDiskBytes::from), + manifest.clone(), + ) + .await?, + EcstoreConditionalFileUpdate::Updated + )) + } + .await; + (payload_written, manifest_written) + })) + .await; let mut payload_replicas = 0usize; let mut manifest_replicas = 0usize; let mut first_error = None; - for disk in disks { - let expected_manifest = match read_bounded(disk, MANIFEST_PATHS[slot], MANIFEST_LEN).await { - Ok(expected) => expected, - Err(error) => { - if first_error.is_none() { - first_error = Some(error); - } - continue; - } - }; - if let Err(error) = validate_reusable_manifest_slot(expected_manifest.as_deref(), sequence, limit) { - if first_error.is_none() { - first_error = Some(error); - } - continue; - } - match cas_replace(disk, PAYLOAD_PATHS[slot], payload, limit).await { - Ok(EcstoreConditionalFileUpdate::Updated) => payload_replicas += 1, - Ok(EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch) => continue, - Err(error) => { - if first_error.is_none() { - first_error = Some(error); - } - continue; - } - } - if let Some((companion_paths, companion_bytes, companion_limit)) = companion { - match cas_replace(disk, companion_paths[slot], companion_bytes, companion_limit).await { - Ok(EcstoreConditionalFileUpdate::Updated) => {} - Ok(EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch) => continue, - Err(error) => { - if first_error.is_none() { - first_error = Some(error); - } - continue; - } - } - } - match cas_replace_expected(disk, MANIFEST_PATHS[slot], expected_manifest.map(EcstoreDiskBytes::from), &manifest).await { - Ok(EcstoreConditionalFileUpdate::Updated) => manifest_replicas += 1, - Ok(EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch) => {} + for (payload_written, result) in results { + payload_replicas += usize::from(payload_written); + match result { + Ok(written) => manifest_replicas += usize::from(written), Err(error) => { if first_error.is_none() { first_error = Some(error); @@ -1450,6 +1446,62 @@ mod tests { ); } + #[tokio::test] + async fn committed_snapshot_writer_retains_payload_count_when_one_companion_fails() { + let root = TempDir::new().expect("test directory"); + let first = disk(&root, "first").await; + let second = disk(&root, "second").await; + let paths = super::super::MRF_LIFECYCLE_PATHS; + std::fs::create_dir(EcstoreDiskAPI::path(first.as_ref()).join(RUSTFS_META_BUCKET).join(paths[0])) + .expect("block only the first replica's companion"); + let owner = Uuid::new_v4(); + let bytes = payload("healthy-companion-replica"); + let disks = [first.clone(), second]; + let publication = + publish_committed_snapshot_with_companion(&disks, owner, 1, &bytes, 4096, Some((&paths, b"lifecycle", 4096))) + .await + .expect("a failed companion must not stop another replica's commit"); + assert_eq!(publication.payload_replicas, 2); + assert_eq!(publication.manifest_replicas, 1); + assert!( + read_bounded(&first, MANIFEST_PATHS[0], MANIFEST_LEN) + .await + .expect("read the failed replica's manifest") + .is_none(), + "a failed companion must not publish its commit manifest" + ); + assert_eq!( + read_committed_companion(&disks, owner, 1, &paths, 4096, 4096) + .await + .expect("read the healthy committed companion") + .as_deref(), + Some(b"lifecycle".as_slice()) + ); + } + + #[tokio::test] + async fn committed_snapshot_writer_deduplicates_disk_roots_before_parallel_cas() { + let root = TempDir::new().expect("test directory"); + let first = disk(&root, "first").await; + let alias = disk(&root, "first").await; + let second = disk(&root, "second").await; + let bytes = payload("distinct-replicas"); + let disks = [first.clone(), first, alias, second]; + let publication = publish_committed_snapshot(&disks, Uuid::new_v4(), 1, &bytes, 4096) + .await + .expect("duplicate handles must not contend with their own replica CAS"); + assert_eq!(publication.payload_replicas, 2, "count canonical disk roots, not handles"); + assert_eq!(publication.manifest_replicas, 2); + assert_eq!( + read_committed(&disks, 4096) + .await + .expect("read both independent replicas") + .expect("committed snapshot") + .payload(), + bytes + ); + } + #[tokio::test] async fn committed_snapshot_writer_publishes_100k_records_with_bounded_readback() { let root = TempDir::new().expect("test directory"); @@ -1521,16 +1573,17 @@ mod tests { .await .ok(); assert_eq!( - cas_replace(&store, PAYLOAD_PATHS[1], &next, 4096) + cas_replace(&store, PAYLOAD_PATHS[1], EcstoreDiskBytes::copy_from_slice(&next), 4096) .await .expect("successor payload CAS"), EcstoreConditionalFileUpdate::Updated ); install(&store, MANIFEST_PATHS[1], &damaged_manifest).await; - let manifest_update = cas_replace_expected(&store, MANIFEST_PATHS[1], expected_manifest, &manifest(owner, 2, &next)) - .await - .expect("successor manifest CAS"); + let manifest_update = + cas_replace_expected(&store, MANIFEST_PATHS[1], expected_manifest, manifest(owner, 2, &next).into()) + .await + .expect("successor manifest CAS"); assert_eq!(manifest_update, EcstoreConditionalFileUpdate::Mismatch); let reopened = disk(&root, "disk").await; diff --git a/crates/heal/src/heal/resume.rs b/crates/heal/src/heal/resume.rs index b97617b2d..af63e4e53 100644 --- a/crates/heal/src/heal/resume.rs +++ b/crates/heal/src/heal/resume.rs @@ -42,10 +42,12 @@ pub use legacy_handoff::{LegacyReplacementApproval, LegacyReplacementSource}; pub(crate) use replacement::replacement_target_identities_match; use replacement::replacement_targets_match_identities; pub use replacement::{ - ReplacementPhase, ReplacementRecoveryRecord, ReplacementRecoveryState, ReplacementTargetIdentity, compose_key, + ReplacementPhase, ReplacementRecoveryBlock, ReplacementRecoveryRecord, ReplacementRecoveryState, ReplacementTargetIdentity, + compose_key, }; pub(crate) use replacement::{ - ReplacementRecoveryCandidate, merge_replacement_recovery_candidate, replacement_retry_is_exhausted_active, + ReplacementRecoveryCandidate, abandoned_targets_are_owned_by_another_generation, merge_replacement_recovery_candidate, + replacement_generation_is_retired, replacement_generation_reserves_targets, replacement_retry_is_exhausted_active, replacement_retry_is_legacy_transient_skip, }; pub use utils::ResumeUtils; @@ -321,6 +323,10 @@ pub struct ResumeState { /// the version listing. `None` means "start from the beginning". #[serde(default)] pub resume_cursor: Option, + /// Ordinary coverage overlapped unresolved hidden delete recovery. The + /// scan must restart after recovery settles, even if another worker settled it. + #[serde(default)] + pub delete_recovery_rescan_pending: bool, /// task id pub task_id: String, /// task type @@ -364,6 +370,11 @@ pub struct ResumeState { /// and may receive one bounded compatibility re-arm. #[serde(default)] pub replacement_legacy_retry_compatibility_done: bool, + /// New generations cannot consume the one-shot compatibility handoff for + /// a pre-format intent stranded by historical rebalance topology. Missing + /// fields in older schema-seven snapshots remain eligible for validation. + #[serde(default)] + pub replacement_preformat_handoff_retry_done: bool, /// The current pass was rewound because the replacement mount was not /// ready. Task-level error persistence must not turn this readiness wait /// back into a consumed retry attempt. @@ -434,6 +445,7 @@ impl ResumeState { Self { schema_version: CURRENT_RESUME_SCHEMA, resume_cursor: None, + delete_recovery_rescan_pending: false, task_id, task_type, set_disk_id, @@ -450,6 +462,7 @@ impl ResumeState { replacement_legacy_import: None, replacement_legacy_successor: None, replacement_legacy_retry_compatibility_done: false, + replacement_preformat_handoff_retry_done: false, replacement_retry_waiting_for_target: false, dangling_delete_retry_not_before: None, start_time: SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(), @@ -495,6 +508,7 @@ impl ResumeState { // New generations already use readiness-aware persistence and must // never enter the one-shot compatibility path for legacy snapshots. state.replacement_legacy_retry_compatibility_done = true; + state.replacement_preformat_handoff_retry_done = true; state } @@ -1048,6 +1062,30 @@ impl ResumeManager { self.state.read().await.clone() } + pub(crate) async fn set_delete_recovery_rescan_pending(&self, pending: bool) -> Result<()> { + let mut state = self.state.write().await; + let previous = state.delete_recovery_rescan_pending; + if previous == pending { + return Ok(()); + } + state.delete_recovery_rescan_pending = pending; + drop(state); + if let Err(error) = self.save_state_strict().await { + self.state.write().await.delete_recovery_rescan_pending = previous; + return Err(error); + } + Ok(()) + } + + /// Keep the coverage obligation until the companion checkpoint rewind is durable. + pub(crate) async fn restart_scan_after_delete_recovery(&self) -> Result<()> { + let mut state = self.state.write().await; + state.delete_recovery_rescan_pending = true; + state.reset_for_retry(); + drop(state); + self.save_state_strict().await + } + /// update progress pub async fn update_progress(&self, processed: u64, successful: u64, failed: u64, skipped: u64) -> Result<()> { let mut state = self.state.write().await; @@ -1243,6 +1281,9 @@ impl ResumeManager { if state.max_retries == 0 { return Ok(false); } + if self.rearm_preformat_handoff_retry_if_needed(storage).await? { + return Ok(true); + } let legacy_rearm = replacement_retry_is_legacy_transient_skip(&state); let ready = storage .replacement_targets_ready_for_retry( diff --git a/crates/heal/src/heal/resume/checkpoint.rs b/crates/heal/src/heal/resume/checkpoint.rs index 50d27150c..c387ba034 100644 --- a/crates/heal/src/heal/resume/checkpoint.rs +++ b/crates/heal/src/heal/resume/checkpoint.rs @@ -18,9 +18,12 @@ use sha2::{Digest, Sha256}; use std::collections::HashSet; use std::path::Path; use std::sync::{Arc, Mutex}; -use std::time::{SystemTime, UNIX_EPOCH}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; use tokio::sync::{Mutex as AsyncMutex, RwLock}; -use tracing::{debug, warn}; +use tracing::{debug, info, warn}; + +mod integrity; +use integrity::CheckpointJson; use super::super::storage_api::owner::{EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes}; use super::super::{BUCKET_META_PREFIX, DiskStore, HealDiskExt, RUSTFS_META_BUCKET}; @@ -31,6 +34,53 @@ use super::{ const EVENT_HEAL_CHECKPOINT_STATE: &str = "heal_checkpoint_state"; const RESUME_CHECKPOINT_DIGEST_FILE: &str = "ahm_checkpoint.sha256"; +const BLOCKED_CHECKPOINT_MARKER_SCHEMA: u32 = 1; +/// A blocked snapshot is re-read with the current reader at most this often. +const BLOCKED_CHECKPOINT_REVALIDATION_INTERVAL: Duration = Duration::from_secs(60 * 60); +const BLOCKED_CHECKPOINT_MARKER_MAX_BYTES: usize = 4 * 1024; + +/// Content of `_ahm_checkpoint.blocked`. Older binaries wrote the +/// literal `blocked` and treat any content as blocked, so this record only +/// adds the revalidation schedule for the current reader. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +struct BlockedCheckpointMarker { + schema: u32, + blocked_at: u64, + revalidate_after: u64, + reason: String, +} + +impl BlockedCheckpointMarker { + fn new(now: u64, reason: &str) -> Self { + Self { + schema: BLOCKED_CHECKPOINT_MARKER_SCHEMA, + blocked_at: now, + revalidate_after: now.saturating_add(BLOCKED_CHECKPOINT_REVALIDATION_INTERVAL.as_secs()), + reason: reason.chars().take(512).collect(), + } + } + + /// A legacy or unreadable marker is due immediately; it was written by a + /// reader that could not schedule its own revalidation. + fn decode(bytes: &[u8]) -> Option { + if bytes.len() > BLOCKED_CHECKPOINT_MARKER_MAX_BYTES { + return None; + } + serde_json::from_slice::(bytes) + .ok() + .filter(|marker| marker.schema == BLOCKED_CHECKPOINT_MARKER_SCHEMA) + } + + fn encode(&self) -> Result { + serde_json::to_vec(self) + .map(EcstoreDiskBytes::from) + .map_err(|error| Error::Serialization(format!("Serialize blocked checkpoint marker: {error}"))) + } +} + +fn unix_now() -> u64 { + SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs() +} const CHECKPOINT_PER_VERSION_SCHEMA: u32 = 7; /// Current on-disk schema version for `ResumeCheckpoint`. Schema 5 could @@ -283,8 +333,10 @@ impl CheckpointManager { Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}")) } - /// Return whether a checkpoint was permanently isolated after a malformed - /// or unsupported snapshot was observed. + /// Return whether a checkpoint is isolated after a malformed or + /// unsupported snapshot was observed. A due marker is revalidated with the + /// current reader first, so a snapshot that only an older reader rejected + /// unblocks here instead of staying blocked forever. pub(crate) async fn is_blocked(disk: &DiskStore, task_id: &str) -> bool { if validate_resume_task_id(task_id).is_err() { return false; @@ -294,12 +346,97 @@ impl CheckpointManager { return false; }; match HealDiskExt::read_all(disk.as_ref(), RUSTFS_META_BUCKET, path).await { - Ok(_) => true, + Ok(marker) => Self::revalidate_blocked_snapshot(disk, task_id, marker).await, Err(crate::heal::DiskError::FileNotFound) => false, Err(_) => true, } } + /// Re-read a blocked snapshot with the current reader once its marker is + /// due. Returns whether the task is still blocked. Verification clears the + /// marker; a tampered or corrupt snapshot keeps it and advances the next + /// revalidation time. Both outcomes retain the snapshot bytes. + async fn revalidate_blocked_snapshot(disk: &DiskStore, task_id: &str, marker_bytes: EcstoreDiskBytes) -> bool { + let now = unix_now(); + let marker = BlockedCheckpointMarker::decode(&marker_bytes); + if marker.as_ref().is_some_and(|marker| marker.revalidate_after > now) { + return true; + } + let checkpoint_path = Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_FILE}")); + let Ok(path) = path_to_str(&checkpoint_path) else { + return true; + }; + let verification = match HealDiskExt::read_all(disk.as_ref(), RUSTFS_META_BUCKET, path).await { + // No snapshot remains to isolate; an operator already replaced or + // removed it and a fresh scan is the safe continuation. + Ok(bytes) if bytes.is_empty() => Ok(()), + Err(crate::heal::DiskError::FileNotFound) => Ok(()), + Ok(bytes) => Self::load_from_data(disk.clone(), task_id, bytes.to_vec()).await.map(|_| ()), + Err(error) => Err(Error::TaskExecutionFailed { + message: format!("Failed to read blocked checkpoint: {error}"), + }), + }; + let reason = match verification { + Ok(()) => { + if let Err(error) = delete_resume_file(disk, &Self::blocked_path(task_id)).await { + warn!( + target: "rustfs::heal::resume", + event = EVENT_HEAL_CHECKPOINT_STATE, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_RESUME, + task_id, + state = "unblock_marker_delete_failed", + error = %error, + "Heal checkpoint verified but its blocked marker could not be removed" + ); + return true; + } + info!( + target: "rustfs::heal::resume", + event = EVENT_HEAL_CHECKPOINT_STATE, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_RESUME, + task_id, + blocked_at = marker.as_ref().map(|marker| marker.blocked_at).unwrap_or_default(), + state = "unblocked_by_revalidation", + "Heal checkpoint verified by the current reader; blocked marker cleared" + ); + return false; + } + Err(error) => error.to_string(), + }; + let mut refreshed = BlockedCheckpointMarker::new(now, &reason); + if let Some(previous) = &marker { + refreshed.blocked_at = previous.blocked_at; + } + let update = match refreshed.encode() { + Ok(bytes) => { + EcstoreDiskAPI::compare_and_update_file( + disk.as_ref(), + RUSTFS_META_BUCKET, + Self::blocked_path(task_id).to_string_lossy().as_ref(), + Some(marker_bytes), + Some(bytes), + ) + .await + } + Err(error) => Err(crate::heal::DiskError::other(error)), + }; + warn!( + target: "rustfs::heal::resume", + event = EVENT_HEAL_CHECKPOINT_STATE, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_RESUME, + task_id, + revalidate_after = refreshed.revalidate_after, + marker_refreshed = matches!(update, Ok(EcstoreConditionalFileUpdate::Updated)), + state = "revalidation_failed", + reason, + "Heal checkpoint remains blocked after revalidation" + ); + true + } + /// Validate the checkpoint while enumerating resumable state. This reads /// the checkpoint once and also isolates malformed or unsupported data. pub(crate) async fn is_resumable(disk: &DiskStore, task_id: &str) -> Result { @@ -321,21 +458,31 @@ impl CheckpointManager { } } - async fn block_invalid_snapshot(disk: &DiskStore, task_id: &str) { - // This marker is intentionally version-agnostic: an unsupported reader - // must stop selector retries until an operator cleans up the snapshot. + async fn block_invalid_snapshot(disk: &DiskStore, task_id: &str, reason: &str) { + // Any marker content blocks older readers. The current reader keeps + // the reason and revalidates the snapshot once the schedule is due; + // an existing marker keeps its own schedule. let blocked_path = Self::blocked_path(task_id); let Ok(path) = path_to_str(&blocked_path) else { return; }; - let result = EcstoreDiskAPI::compare_and_update_file( - disk.as_ref(), - RUSTFS_META_BUCKET, - path, - None, - Some(EcstoreDiskBytes::from_static(b"blocked")), - ) - .await; + let marker = match BlockedCheckpointMarker::new(unix_now(), reason).encode() { + Ok(marker) => marker, + Err(error) => { + warn!( + target: "rustfs::heal::resume", + event = EVENT_HEAL_CHECKPOINT_STATE, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_RESUME, + task_id, + state = "blocked_marker_write_failed", + error = %error, + "Heal checkpoint could not persist its blocked marker" + ); + return; + } + }; + let result = EcstoreDiskAPI::compare_and_update_file(disk.as_ref(), RUSTFS_META_BUCKET, path, None, Some(marker)).await; match result { Ok(EcstoreConditionalFileUpdate::Updated | EcstoreConditionalFileUpdate::Mismatch) => {} Ok(EcstoreConditionalFileUpdate::Missing) => warn!( @@ -406,27 +553,32 @@ impl CheckpointManager { async fn load_from_data(disk: DiskStore, task_id: &str, checkpoint_data: Vec) -> Result { validate_resume_task_id(task_id)?; - let mut checkpoint: ResumeCheckpoint = match serde_json::from_slice(&checkpoint_data) { - Ok(checkpoint) => checkpoint, + let (mut persisted, mut checkpoint) = match serde_json::from_slice::(&checkpoint_data) + .and_then(|checkpoint| serde_json::from_slice::(&checkpoint_data).map(|value| (value, checkpoint))) + { + Ok(decoded) => decoded, Err(error) => { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::TaskExecutionFailed { + let error = Error::TaskExecutionFailed { message: format!("Failed to deserialize checkpoint: {error}"), - }); + }; + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } }; if checkpoint.task_id != task_id { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::TaskExecutionFailed { + let error = Error::TaskExecutionFailed { message: "Resume checkpoint task id does not match filename".to_string(), - }); + }; + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } if checkpoint .dangling_delete_retry_not_before .is_some_and(|deadline| !dangling_delete_deadline_is_representable(deadline)) { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::InvalidCheckpoint("Unrepresentable dangling-delete grace deadline".to_string())); + let error = Error::InvalidCheckpoint("Unrepresentable dangling-delete grace deadline".to_string()); + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } // Older checkpoints can contain identities that are not comparable to @@ -441,29 +593,36 @@ impl CheckpointManager { return Err(Error::InvalidCheckpoint("Administrator outcome requires checkpoint schema 8".to_string())); } if checkpoint.schema_version > supported_schema { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::TaskExecutionFailed { + let error = Error::TaskExecutionFailed { message: format!( "Checkpoint schema {} is newer than supported schema {supported_schema}", checkpoint.schema_version ), - }); + }; + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } let integrity_verified = if let Some(expected) = checkpoint.integrity_digest.as_deref() { - let actual = Self::checkpoint_digest(&Self::serialize_without_digest(&checkpoint)?); + // Verify the persisted fields: typed defaults can add fields that + // were absent from an otherwise intact older signed checkpoint. + let mut actual = Self::checkpoint_digest(&persisted.unsigned_bytes()?); if expected != actual { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::InvalidCheckpoint(format!( - "Resume checkpoint digest does not match task {task_id}" - ))); + // RUSTFS_COMPAT_TODO(#2871): retain until snapshots whose + // payload order differs from their digest order are retired. + persisted.sort_all_objects(); + actual = Self::checkpoint_digest(&persisted.unsigned_bytes()?); + } + if expected != actual { + let error = Error::InvalidCheckpoint(format!("Resume checkpoint digest does not match task {task_id}")); + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } true } else if checkpoint.schema_version >= CHECKPOINT_PER_VERSION_SCHEMA { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::InvalidCheckpoint(format!( - "Resume checkpoint digest is missing for task {task_id}" - ))); + let error = Error::InvalidCheckpoint(format!("Resume checkpoint digest is missing for task {task_id}")); + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } else { let digest_path = Self::digest_path(task_id); let digest_path = path_to_str(&digest_path)?; @@ -471,10 +630,9 @@ impl CheckpointManager { Ok(expected) => { let actual = Self::checkpoint_digest(&checkpoint_data); if expected.as_ref() != actual.as_bytes() { - Self::block_invalid_snapshot(&disk, task_id).await; - return Err(Error::InvalidCheckpoint(format!( - "Resume checkpoint digest does not match task {task_id}" - ))); + let error = Error::InvalidCheckpoint(format!("Resume checkpoint digest does not match task {task_id}")); + Self::block_invalid_snapshot(&disk, task_id, &error.to_string()).await; + return Err(error); } true } @@ -602,6 +760,18 @@ impl CheckpointManager { self.save_checkpoint_throttled().await } + pub(crate) async fn reset_after_delete_recovery(&self) -> Result<()> { + let mut checkpoint = self.checkpoint.write().await; + checkpoint.reset_for_retry(); + if let Some(admin) = checkpoint.admin.as_mut() { + admin.completed = false; + admin.outcome = Default::default(); + admin.outcome.start(); + } + drop(checkpoint); + self.save_checkpoint().await + } + /// Add a processed object. Called once per healed object, so persistence /// is batched (`PERSIST_EVERY_MUTATIONS` / `PERSIST_INTERVAL`); positions /// and page boundaries still persist unconditionally. @@ -792,10 +962,14 @@ impl CheckpointManager { let _save_guard = self.save_lock.lock().await; let checkpoint = self.checkpoint.read().await.clone(); validate_resume_task_id(&checkpoint.task_id)?; - let unsigned_checkpoint_data = Self::serialize_without_digest(&checkpoint)?; + let checkpoint_value = serde_json::to_value(&checkpoint).map_err(|e| Error::TaskExecutionFailed { + message: format!("Failed to serialize checkpoint: {e}"), + })?; + let mut persisted_checkpoint = CheckpointJson::from_value(checkpoint_value)?; + persisted_checkpoint.sort_all_objects(); + let unsigned_checkpoint_data = persisted_checkpoint.unsigned_bytes()?; let digest = Self::checkpoint_digest(&unsigned_checkpoint_data); - let mut persisted_checkpoint = checkpoint.clone(); - persisted_checkpoint.integrity_digest = Some(digest); + persisted_checkpoint.set_digest(Some(digest))?; let checkpoint_data = EcstoreDiskBytes::from(serde_json::to_vec(&persisted_checkpoint).map_err(|e| Error::TaskExecutionFailed { message: format!("Failed to serialize checkpoint: {e}"), @@ -854,17 +1028,19 @@ impl CheckpointManager { let current: ResumeCheckpoint = match serde_json::from_slice(&existing) { Ok(current) => current, Err(error) => { - Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id).await; - return Err(Error::TaskExecutionFailed { + let error = Error::TaskExecutionFailed { message: format!("Existing checkpoint is corrupt: {error}"), - }); + }; + Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id, &error.to_string()).await; + return Err(error); } }; if current.task_id != checkpoint.task_id { - Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id).await; - return Err(Error::TaskExecutionFailed { + let error = Error::TaskExecutionFailed { message: "Existing checkpoint task id does not match filename".to_string(), - }); + }; + Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id, &error.to_string()).await; + return Err(error); } let supported_schema = if current.admin.is_some() { ADMIN_CHECKPOINT_SCHEMA @@ -872,13 +1048,14 @@ impl CheckpointManager { CURRENT_CHECKPOINT_SCHEMA }; if current.schema_version > supported_schema { - Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id).await; - return Err(Error::TaskExecutionFailed { + let error = Error::TaskExecutionFailed { message: format!( "Existing checkpoint schema {} is newer than supported schema {supported_schema}", current.schema_version ), - }); + }; + Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id, &error.to_string()).await; + return Err(error); } if last_saved.as_ref().is_none_or(|saved| saved.as_ref() != existing.as_ref()) { return Err(Error::TaskExecutionFailed { @@ -943,25 +1120,6 @@ impl CheckpointManager { }) } - fn serialize_without_digest(checkpoint: &ResumeCheckpoint) -> Result> { - let mut unsigned = checkpoint.clone(); - unsigned.integrity_digest = None; - let mut value = serde_json::to_value(&unsigned).map_err(|e| Error::TaskExecutionFailed { - message: format!("Failed to serialize checkpoint: {e}"), - })?; - for field in ["processed_objects", "failed_objects", "skipped_objects"] { - let Some(values) = value.get_mut(field).and_then(serde_json::Value::as_array_mut) else { - return Err(Error::TaskExecutionFailed { - message: format!("Failed to canonicalize checkpoint field: {field}"), - }); - }; - values.sort_by(|left, right| left.as_str().cmp(&right.as_str())); - } - serde_json::to_vec(&value).map_err(|e| Error::TaskExecutionFailed { - message: format!("Failed to serialize checkpoint: {e}"), - }) - } - fn checkpoint_digest(checkpoint_data: &[u8]) -> String { base64_simd::STANDARD.encode_to_string(Sha256::digest(checkpoint_data)) } diff --git a/crates/heal/src/heal/resume/checkpoint/integrity.rs b/crates/heal/src/heal/resume/checkpoint/integrity.rs new file mode 100644 index 000000000..4752fa7c9 --- /dev/null +++ b/crates/heal/src/heal/resume/checkpoint/integrity.rs @@ -0,0 +1,219 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::{Error, Result}; +use serde::de::{MapAccess, SeqAccess, Visitor}; +use serde::ser::SerializeMap; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use serde_json::Value; +use std::fmt; + +// Checkpoint-local ordering preserves historical digests without affecting +// the server's serde_json feature set. +#[derive(Debug)] +pub(super) enum CheckpointJson { + Object(Vec<(String, Self)>), + Array(Vec), + Scalar(Value), +} + +impl CheckpointJson { + fn as_str(&self) -> Option<&str> { + match self { + Self::Scalar(Value::String(value)) => Some(value), + _ => None, + } + } + + pub(super) fn from_value(value: Value) -> Result { + fn convert(value: Value, depth: usize) -> Result { + if depth >= 128 { + return Err(Error::InvalidCheckpoint("Checkpoint JSON exceeds the nesting limit".to_string())); + } + match value { + Value::Object(fields) => fields + .into_iter() + .map(|(key, value)| convert(value, depth + 1).map(|value| (key, value))) + .collect::>() + .map(CheckpointJson::Object), + Value::Array(values) => values + .into_iter() + .map(|value| convert(value, depth + 1)) + .collect::>() + .map(CheckpointJson::Array), + scalar => Ok(CheckpointJson::Scalar(scalar)), + } + } + convert(value, 0) + } + + pub(super) fn sort_all_objects(&mut self) { + let mut pending = vec![self]; + while let Some(value) = pending.pop() { + match value { + Self::Object(fields) => { + fields.sort_unstable_by(|left, right| left.0.cmp(&right.0)); + pending.extend(fields.iter_mut().map(|(_, value)| value)); + } + Self::Array(values) => pending.extend(values), + Self::Scalar(_) => {} + } + } + } + + pub(super) fn set_digest(&mut self, digest: Option) -> Result<()> { + let Self::Object(fields) = self else { + return Err(Error::InvalidCheckpoint("Checkpoint must be a JSON object".to_string())); + }; + let (_, value) = fields + .iter_mut() + .find(|(key, _)| key == "integrity_digest") + .ok_or_else(|| Error::InvalidCheckpoint("Signed checkpoint is missing its integrity digest field".to_string()))?; + *value = Self::Scalar(digest.map_or(Value::Null, Value::String)); + Ok(()) + } + + pub(super) fn unsigned_bytes(&mut self) -> Result> { + self.set_digest(None)?; + let Self::Object(fields) = self else { + return Err(Error::InvalidCheckpoint("Checkpoint must be a JSON object".to_string())); + }; + for field in ["processed_objects", "failed_objects", "skipped_objects"] { + let Some((_, Self::Array(values))) = fields.iter_mut().find(|(key, _)| key == field) else { + return Err(Error::InvalidCheckpoint(format!("Checkpoint set is missing or invalid: {field}"))); + }; + values.sort_unstable_by(|left, right| left.as_str().cmp(&right.as_str())); + } + serde_json::to_vec(self).map_err(|error| Error::InvalidCheckpoint(format!("Failed to encode checkpoint digest: {error}"))) + } +} + +impl Serialize for CheckpointJson { + fn serialize(&self, serializer: S) -> std::result::Result { + match self { + Self::Object(fields) => { + let mut map = serializer.serialize_map(Some(fields.len()))?; + for (key, value) in fields { + map.serialize_entry(key, value)?; + } + map.end() + } + Self::Array(values) => values.serialize(serializer), + Self::Scalar(value) => value.serialize(serializer), + } + } +} + +impl<'de> Deserialize<'de> for CheckpointJson { + fn deserialize>(deserializer: D) -> std::result::Result { + struct JsonVisitor; + + impl<'de> Visitor<'de> for JsonVisitor { + type Value = CheckpointJson; + + fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("checkpoint JSON") + } + + fn visit_map>(self, mut map: A) -> std::result::Result { + let mut fields = Vec::new(); + while let Some(field) = map.next_entry()? { + fields.push(field); + } + Ok(CheckpointJson::Object(fields)) + } + + fn visit_seq>(self, mut seq: A) -> std::result::Result { + let mut values = Vec::new(); + while let Some(value) = seq.next_element()? { + values.push(value); + } + Ok(CheckpointJson::Array(values)) + } + + fn visit_bool(self, value: bool) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Bool(value))) + } + + fn visit_i64(self, value: i64) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Number(value.into()))) + } + + fn visit_u64(self, value: u64) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Number(value.into()))) + } + + fn visit_f64(self, value: f64) -> std::result::Result { + serde_json::Number::from_f64(value) + .map(|value| CheckpointJson::Scalar(Value::Number(value))) + .ok_or_else(|| E::custom("non-finite checkpoint number")) + } + + fn visit_str(self, value: &str) -> std::result::Result { + self.visit_string(value.to_owned()) + } + + fn visit_string(self, value: String) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::String(value))) + } + + fn visit_unit(self) -> std::result::Result { + Ok(CheckpointJson::Scalar(Value::Null)) + } + } + + deserializer.deserialize_any(JsonVisitor) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn checkpoint_json_preserves_nested_order_and_scalar_values() { + let wire = r#"{"z":{"z":18446744073709551615,"a":-9223372036854775808,"f":1.5,"s":"hello\nworld","b":true,"n":null},"other":[2,1],"processed_objects":["b","a"],"failed_objects":[],"skipped_objects":[],"integrity_digest":"signed"}"#; + let mut value: CheckpointJson = serde_json::from_str(wire).expect("parse ordered checkpoint JSON"); + assert_eq!(serde_json::to_string(&value).expect("encode ordered checkpoint JSON"), wire); + let unsigned = value.unsigned_bytes().expect("normalize only checkpoint sets and digest"); + assert_eq!( + String::from_utf8(unsigned).expect("unsigned checkpoint must be UTF-8"), + wire.replace("[\"b\",\"a\"]", "[\"a\",\"b\"]").replace("\"signed\"", "null") + ); + } + + #[test] + fn checkpoint_json_sorts_nested_objects_without_reordering_other_arrays() { + let mut value: CheckpointJson = + serde_json::from_str(r#"{"z":[{"z":2,"a":1},3,1],"a":false}"#).expect("parse nested checkpoint JSON"); + value.sort_all_objects(); + assert_eq!( + serde_json::to_string(&value).expect("encode sorted nested checkpoint JSON"), + r#"{"a":false,"z":[{"a":1,"z":2},3,1]}"# + ); + } + + #[test] + fn checkpoint_json_bounds_both_input_and_value_conversion_depth() { + for depth in [127, 128] { + let wire = format!("{}null{}", "[".repeat(depth), "]".repeat(depth)); + assert_eq!(serde_json::from_str::(&wire).is_ok(), depth == 127); + let mut value = Value::Null; + for _ in 0..depth { + value = Value::Array(vec![value]); + } + assert_eq!(CheckpointJson::from_value(value).is_ok(), depth == 127); + } + } +} diff --git a/crates/heal/src/heal/resume/gc.rs b/crates/heal/src/heal/resume/gc.rs index ab5ae9262..74366ce91 100644 --- a/crates/heal/src/heal/resume/gc.rs +++ b/crates/heal/src/heal/resume/gc.rs @@ -67,6 +67,10 @@ pub(crate) struct ResumeGcReport { /// Records that must be handed to #1927's quarantine owner. pub(crate) quarantine_required: usize, pub(crate) generation_skipped: usize, + /// Retired predecessors (committed handoff) whose successor has verified + /// or already cleaned up behind a completion proof. They are retention + /// candidates for the durable owner; this pass still retains them. + pub(crate) retired_predecessors: usize, pub(crate) clock_skew: usize, pub(crate) read_errors: usize, pub(crate) retained: usize, @@ -299,6 +303,19 @@ async fn inspect_task( return Ok(()); } + if replacement && super::replacement_generation_is_retired(&parsed) { + let successor = parsed + .replacement_handoff + .as_ref() + .map(|handoff| handoff.link.successor.clone()); + if let Some(successor) = successor + && super::validate_resume_task_id(&successor).is_ok() + && retired_successor_is_finished(options, disk, prefix, &successor, progress).await? + { + progress.report.retired_predecessors += 1; + } + } + if !replacement && artifacts.checkpoint { let checkpoint_path = artifact_path(prefix, task_id, RESUME_CHECKPOINT_FILE)?; match read_bounded(disk, &checkpoint_path, options.max_bytes, progress.bytes_read).await { @@ -340,6 +357,53 @@ async fn inspect_task( Ok(()) } +/// Whether a retired predecessor's successor has verified or has left a +/// completion proof behind. Unreadable or oversized successor records count as +/// unfinished, so the predecessor stays retained. +async fn retired_successor_is_finished( + options: InspectOptions, + disk: &DiskStore, + prefix: &str, + successor: &str, + progress: &mut InspectProgress<'_>, +) -> Result { + let intent_path = artifact_path(prefix, successor, REPLACEMENT_INTENT_FILE)?; + match read_bounded(disk, &intent_path, options.max_bytes, progress.bytes_read).await { + ReadOutcome::Bytes(bytes) => { + return Ok(serde_json::from_slice::(&bytes).is_ok_and(|state| { + state.task_id == successor + && state.completed + && matches!( + state.replacement_phase, + super::ReplacementPhase::Verified | super::ReplacementPhase::CleanupPending + ) + })); + } + ReadOutcome::Missing => {} + ReadOutcome::TooLarge => { + progress.report.budget_exhausted = true; + return Ok(false); + } + ReadOutcome::Error => { + progress.report.read_errors += 1; + return Ok(false); + } + } + let proof_path = artifact_path(prefix, successor, REPLACEMENT_COMPLETION_PROOF_FILE)?; + Ok(match read_bounded(disk, &proof_path, options.max_bytes, progress.bytes_read).await { + ReadOutcome::Bytes(bytes) => !bytes.is_empty(), + ReadOutcome::Missing => false, + ReadOutcome::TooLarge => { + progress.report.budget_exhausted = true; + false + } + ReadOutcome::Error => { + progress.report.read_errors += 1; + false + } + }) +} + enum ReadOutcome { Bytes(Vec), Missing, diff --git a/crates/heal/src/heal/resume/handoff.rs b/crates/heal/src/heal/resume/handoff.rs index 47f3a0bdf..4c42c4908 100644 --- a/crates/heal/src/heal/resume/handoff.rs +++ b/crates/heal/src/heal/resume/handoff.rs @@ -89,13 +89,71 @@ pub(super) fn validate_lineage(task_id: &str, set_disk_id: &str, lineage: &[Repl } impl ResumeManager { + // RUSTFS_COMPAT_TODO(8565): recover pre-format intents stranded by historical pool topology. Remove after supported upgrades no longer admit snapshots lacking replacement_preformat_handoff_retry_done and retained affected intents are resolved. + pub(super) async fn rearm_preformat_handoff_retry_if_needed( + &self, + storage: &dyn crate::heal::storage::HealStorageAPI, + ) -> Result { + let state = self.get_state().await; + if !preformat_handoff_retry_candidate(&state) { + return Ok(false); + } + let identities = storage.replacement_target_identities(&state.replacement_targets).await?; + if identities == state.replacement_target_identities { + return Ok(false); + } + // Lock order: target execution leases, the store's format fence during + // the probe, then the intent CAS after the probe releases its fence. + let execution = storage.replacement_execution(&state.replacement_targets).await?; + if execution.identities() != identities.as_slice() { + return Err(Error::ReplacementTargetNotReady( + "target changed before pre-format recovery admission".to_string(), + )); + } + // A cleared summary does not prove an unstarted pass. Any checkpoint, + // including an empty/corrupt file or an isolated checkpoint marker, + // prevents this compatibility admission. + for suffix in [RESUME_CHECKPOINT_FILE, RESUME_CHECKPOINT_BLOCKED_FILE] { + let path = Path::new(BUCKET_META_PREFIX).join(format!("{}_{suffix}", state.task_id)); + match self.disk.read_all(RUSTFS_META_BUCKET, path_to_str(&path)?).await { + Ok(_) => return Ok(false), + Err(DiskError::FileNotFound) => {} + Err(error) => return Err(error.into()), + } + } + if Self::replacement_completion_proof_if_present(self.disk.clone(), &state.task_id) + .await? + .is_some() + || !storage + .replacement_preformat_retry_is_safe(&state.set_disk_id, &state.replacement_targets) + .await? + { + return Ok(false); + } + let buckets = storage.list_buckets().await?.into_iter().map(|bucket| bucket.name).collect(); + if storage + .replacement_target_identities(&state.replacement_targets) + .await? + .as_slice() + != execution.identities() + { + return Err(Error::ReplacementTargetNotReady( + "target changed during pre-format recovery admission".to_string(), + )); + } + // Retain the execution leases through validation and the Prepared CAS. + // Budget admission, its one-shot marker, and its reserved successor are + // one durable decision, so a crash cannot grant an unbound retry. + self.prepare_replacement_handoff_inner(&execution, buckets, true).await?; + Ok(true) + } + /// Shared startup/scanner decision. A changed mount starts a full scan in /// one durable successor; an unfinished transfer always reuses that UUID. pub(crate) async fn resolve_replacement_recovery( &self, storage: &dyn crate::heal::storage::HealStorageAPI, ) -> Result { - let attempt = self.get_state().await.retry_count.saturating_add(1); match self.resolve_replacement_recovery_inner(storage).await { Ok(state) if !state.completed && state.retry_count >= state.max_retries => { Err(Error::ReplacementRetryBudgetExhausted) @@ -108,7 +166,9 @@ impl ResumeManager { // restarts. This is a readiness observation, not a failed // generation attempt, so do not consume the durable retry budget. Err(error @ Error::ReplacementTargetNotReady(_)) => Err(error), - Err(failure) => match self.record_replacement_failure(&failure, attempt).await { + // Resolution precedes any rebuild: keep the reason for status and + // the next discovery pass, but never spend a rebuild attempt. + Err(failure) => match self.record_replacement_resolution_failure(&failure).await { Ok(()) => Err(failure), Err(persistence) => Err(Error::ReplacementFailurePersistence { failure: Box::new(failure), @@ -184,11 +244,23 @@ impl ResumeManager { /// Select a successor only after every old executor is fenced by the target /// execution leases. The durable predecessor is the sole handoff authority. pub(crate) async fn prepare_replacement_handoff( + &self, + execution: &ReplacementExecution, + buckets: Vec, + ) -> Result { + self.prepare_replacement_handoff_inner(execution, buckets, false).await + } + + async fn prepare_replacement_handoff_inner( &self, execution: &ReplacementExecution, mut buckets: Vec, + preformat_retry: bool, ) -> Result { let state = self.get_state().await; + if preformat_retry && !preformat_handoff_retry_candidate(&state) { + return Err(replacement_recovery_conflict("pre-format recovery eligibility changed")); + } if let Some(handoff) = &state.replacement_handoff { Self::validate_handoff(handoff, &state)?; let raw = Self::read_state_file(&self.disk, &state.task_id, self.state_file).await?; @@ -219,7 +291,10 @@ impl ResumeManager { "replacement generation is not eligible for an online handoff", )); } - if state.retry_count.saturating_add(1) >= state.max_retries { + // A handoff is not a rebuild attempt. Only a generation whose rebuild + // budget is already spent stays terminal; the compatibility re-arm + // below is the single exception for stranded pre-format intents. + if !preformat_retry && state.retry_count >= state.max_retries { return Err(Error::ReplacementRetryBudgetExhausted); } if state.replacement_phase == ReplacementPhase::OwnershipPending { @@ -280,6 +355,12 @@ impl ResumeManager { if current.replacement_revision != state.replacement_revision { return Err(replacement_recovery_conflict("replacement changed during handoff preparation")); } + if preformat_retry { + current.replacement_preformat_handoff_retry_done = true; + // The successor inherits this count unchanged and must have + // exactly one actual rebuild attempt under the same cap. + current.retry_count = current.max_retries - 1; + } current.replacement_handoff = Some(ReplacementHandoff { phase: ReplacementHandoffPhase::Prepared, link, @@ -378,11 +459,12 @@ impl ResumeManager { link.targets.clone(), ); state.replacement_phase = ReplacementPhase::OwnershipPending; + // The constructor marks this compatibility grant consumed for every + // successor, including descendants of a recovered older generation. state.replacement_predecessor = Some(predecessor.task_id.clone()); - state.retry_count = predecessor - .retry_count - .checked_add(1) - .ok_or_else(|| replacement_recovery_conflict("handoff retry overflow"))?; + // The successor continues the same bounded rebuild; the handoff that + // created it is an interruption, not a spent attempt. + state.retry_count = predecessor.retry_count; state.max_retries = predecessor.max_retries; state.replacement_lineage = predecessor.replacement_lineage.clone(); state.replacement_legacy_import = predecessor.replacement_legacy_import.clone(); @@ -493,6 +575,31 @@ impl ResumeManager { } } +fn preformat_handoff_retry_candidate(state: &ResumeState) -> bool { + replacement_retry_is_exhausted_active(state) + && state.replacement_generation.as_deref() == Some(state.task_id.as_str()) + && !state.replacement_preformat_handoff_retry_done + && state.replacement_phase == ReplacementPhase::Intent + && state.replacement_execution_protocol == 1 + && state.max_retries >= 2 + && state.replacement_predecessor.is_none() + && state.replacement_handoff.is_none() + && state.replacement_lineage.is_empty() + && state.replacement_legacy_import.is_none() + && state.replacement_legacy_successor.is_none() + && state.resume_cursor.is_none() + && state.current_bucket.is_none() + && state.current_object.is_none() + && state.completed_buckets.is_empty() + && state.processed_objects == 0 + && state.successful_objects == 0 + && state.failed_objects == 0 + && state.skipped_objects == 0 + && state.skipped_new_versions == 0 + && state.skipped_ilm_expired == 0 + && state.processed_bytes == 0 +} + fn successor_can_be_rebound(state: &ResumeState, predecessor: &str) -> bool { state.replacement_predecessor.as_deref() == Some(predecessor) && successor_has_no_progress(state) } diff --git a/crates/heal/src/heal/resume/handoff/tests.rs b/crates/heal/src/heal/resume/handoff/tests.rs index a6aaf9442..6f690e854 100644 --- a/crates/heal/src/heal/resume/handoff/tests.rs +++ b/crates/heal/src/heal/resume/handoff/tests.rs @@ -13,10 +13,451 @@ // limitations under the License. use super::*; -use crate::heal::{HEALING_MARKER_PATH, resume::tests::schema_test_disk}; -use std::sync::LazyLock; +use crate::heal::{ + HEALING_MARKER_PATH, HealObjectInfo, + resume::tests::schema_test_disk, + storage::{HealListItem, HealStorageAPI}, + storage_api::storage::{BucketInfo, DiskSetSelector}, +}; +use rustfs_heal_contracts::heal_channel::HealOpts; +use rustfs_madmin::heal_commands::HealResultItem; +use std::sync::{ + LazyLock, + atomic::{AtomicUsize, Ordering}, +}; static FAILURES: LazyLock>> = LazyLock::new(|| Mutex::new(HashMap::new())); + +struct PreformatRecoveryStorage { + execution: Arc, + historical_topology: bool, + probe_calls: AtomicUsize, + identity_calls: AtomicUsize, + change_identity_on_call: usize, +} + +#[async_trait::async_trait] +impl HealStorageAPI for PreformatRecoveryStorage { + async fn replacement_target_identities(&self, _: &[String]) -> Result> { + let mut identities = self.execution.identities().to_vec(); + if self.identity_calls.fetch_add(1, Ordering::SeqCst) == self.change_identity_on_call { + identities[0].filesystem_identity = "another-mount:fs:root".to_string(); + } + Ok(identities) + } + + async fn replacement_execution(&self, _: &[String]) -> Result> { + Ok(self.execution.clone()) + } + + async fn replacement_targets_ready_for_retry( + &self, + _: &str, + _: &[String], + expected: &[ReplacementTargetIdentity], + ) -> Result { + if expected != self.execution.identities() { + return Ok(false); + } + for disk in self.execution.test_disks() { + if !matches!(disk.get_disk_id().await, Ok(Some(id)) if !id.is_nil()) { + return Ok(false); + } + } + Ok(true) + } + + async fn replacement_preformat_retry_is_safe(&self, _: &str, _: &[String]) -> Result { + self.probe_calls.fetch_add(1, Ordering::SeqCst); + if !self.historical_topology { + return Ok(false); + } + for disk in self.execution.test_disks() { + match disk.read_all(RUSTFS_META_BUCKET, "format.json").await { + Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => {} + Ok(_) => return Ok(false), + Err(error) => return Err(error.into()), + } + } + Ok(true) + } + + async fn list_buckets(&self) -> Result> { + Ok(Vec::new()) + } + + async fn get_object_meta(&self, _: &str, _: &str) -> Result> { + Err(Error::other("pre-format recovery must not inspect objects")) + } + + async fn ec_decode_rebuild(&self, _: &str, _: &str) -> Result> { + Err(Error::other("pre-format recovery must not rebuild objects")) + } + + async fn get_bucket_info(&self, _: &str) -> Result> { + Err(Error::other("pre-format recovery must not inspect buckets")) + } + + async fn list_buckets_for_admin_heal(&self, _: Option) -> Result> { + Err(Error::other("pre-format recovery must not start admin discovery")) + } + + async fn object_exists(&self, _: &str, _: &str) -> Result { + Err(Error::other("pre-format recovery must not inspect objects")) + } + + async fn heal_object(&self, _: &str, _: &str, _: Option<&str>, _: &HealOpts) -> Result<(HealResultItem, Option)> { + Err(Error::other("pre-format recovery must not heal objects")) + } + + async fn heal_bucket(&self, _: &str, _: &HealOpts) -> Result { + Err(Error::other("pre-format recovery must not heal buckets")) + } + + async fn heal_format(&self, _: bool) -> Result<(HealResultItem, Option)> { + Err(Error::other("pre-format recovery must not write formats")) + } + + async fn list_objects_for_heal_page( + &self, + _: &str, + _: &str, + _: Option<&str>, + _: bool, + ) -> Result<(Vec, Option, bool)> { + Err(Error::other("pre-format recovery must not traverse objects")) + } + + async fn get_disk_for_resume(&self, _: &str) -> Result { + Err(Error::other("pre-format recovery must retain its durable anchor")) + } +} + +async fn preformat_recovery_fixture() -> (Vec, ResumeManager, PreformatRecoveryStorage) { + let (dirs, manager, execution) = fixture(1).await; + execution.test_disks()[0] + .delete(RUSTFS_META_BUCKET, HEALING_MARKER_PATH, Default::default()) + .await + .expect("remove the fixture marker from the blank replacement"); + { + let mut state = manager.state.write().await; + state.retry_count = state.max_retries; + state.replacement_preformat_handoff_retry_done = false; + state.replacement_retry_waiting_for_target = true; + } + manager.save_state_strict().await.expect("persist exhausted older intent"); + ( + dirs, + manager, + PreformatRecoveryStorage { + execution, + historical_topology: true, + probe_calls: AtomicUsize::new(0), + identity_calls: AtomicUsize::new(0), + change_identity_on_call: usize::MAX, + }, + ) +} + +#[tokio::test] +async fn preformat_recovery_same_mount_reuses_existing_readiness_rearm() { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + let initial = manager.get_state().await; + storage.execution = + ReplacementExecution::for_test(storage.execution.test_disks().to_vec(), initial.replacement_target_identities.clone()); + assert!(matches!( + storage.execution.test_disks()[0].get_disk_id().await, + Err(DiskError::UnformattedDisk) + )); + assert!( + manager + .rearm_replacement_recovery_if_needed(&storage) + .await + .expect("rearm blank same mount") + ); + let resumed = manager + .resolve_replacement_recovery(&storage) + .await + .expect("resume the same generation"); + assert_eq!(resumed.task_id, initial.task_id); + assert_eq!(resumed.retry_count, resumed.max_retries - 1); + assert!(resumed.replacement_handoff.is_none()); + assert!(!resumed.replacement_preformat_handoff_retry_done); + assert_eq!(storage.probe_calls.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn preformat_recovery_reserves_one_handoff_and_one_rebuild() { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + assert!( + manager + .rearm_replacement_recovery_if_needed(&storage) + .await + .expect("admit historical pre-format recovery") + ); + let prepared = manager.get_state().await; + assert_eq!(prepared.replacement_phase, ReplacementPhase::HandoffPending); + assert!(prepared.replacement_preformat_handoff_retry_done); + let child_state = manager + .resolve_replacement_recovery(&storage) + .await + .expect("resolve the reserved successor"); + assert_eq!(child_state.retry_count, child_state.max_retries - 1); + assert!(child_state.replacement_preformat_handoff_retry_done); + assert_eq!(child_state.replacement_lineage.len(), 1); + let child = ResumeManager::load_replacement_intent(manager.disk.clone(), &child_state.task_id) + .await + .expect("reload the admitted successor"); + // Unlike a generic schema-test disk, an admitted replacement has an exact + // set/slot binding that get_disk_id validates against its persisted format. + let mut endpoint = storage.execution.test_disks()[0].endpoint(); + endpoint.set_pool_index(0); + endpoint.set_set_index(0); + endpoint.set_disk_index(0); + let disk = crate::heal::new_disk( + &endpoint, + &crate::heal::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("open the replacement with its real erasure-set slot"); + storage.execution = ReplacementExecution::for_test(vec![disk.clone()], storage.execution.identities().to_vec()); + let disk_id = Uuid::new_v4(); + let format = serde_json::json!({ + "version": "1", "format": "xl", "id": Uuid::new_v4(), + "xl": {"version": "3", "this": disk_id, "sets": [[disk_id]], "distributionAlgo": "SIPMOD+PARITY"} + }); + storage.execution.test_disks()[0] + .write_all( + RUSTFS_META_BUCKET, + "format.json", + serde_json::to_vec(&format).expect("serialize target format").into(), + ) + .await + .expect("format the admitted target before rebuilding"); + crate::heal::storage_api::EcstoreDiskAPI::set_disk_id(disk.as_ref(), Some(disk_id)) + .await + .expect("publish the formatted disk identity as save_format_file does"); + assert_eq!(disk.get_disk_id().await.expect("formatted replacement must be ready"), Some(disk_id)); + child + .acquire_replacement_markers(&storage.execution) + .await + .expect("commit successor ownership"); + child + .mark_replacement_rebuilding(storage.execution.identities().to_vec()) + .await + .expect("admit actual rebuilding"); + child + .record_replacement_failure(&Error::other("actual rebuild failed"), child_state.max_retries) + .await + .expect("exhaust final attempt"); + assert!( + !child + .rearm_replacement_recovery_if_needed(&storage) + .await + .expect("never regrant the successor") + ); + let reloaded = ResumeManager::load_replacement_intent(manager.disk.clone(), &prepared.task_id) + .await + .expect("reload predecessor"); + assert!( + !reloaded + .rearm_preformat_handoff_retry_if_needed(&storage) + .await + .expect("never regrant the predecessor") + ); + assert_eq!(storage.probe_calls.load(Ordering::SeqCst), 1); +} + +#[tokio::test] +async fn preformat_recovery_replays_prepared_and_published_crashes_without_new_credit() { + for boundary in ["prepared", "successor_published"] { + let (_dirs, manager, storage) = preformat_recovery_fixture().await; + let source = manager.get_state().await.task_id; + FAILURES + .lock() + .expect("failure injection lock") + .insert(source.clone(), boundary); + assert!(manager.rearm_replacement_recovery_if_needed(&storage).await.is_err()); + let reloaded = ResumeManager::load_replacement_intent(manager.disk.clone(), &source) + .await + .expect("reload crashed grant"); + let prepared = reloaded.get_state().await; + assert!(prepared.replacement_preformat_handoff_retry_done); + let successor = prepared + .replacement_handoff + .expect("grant and successor are atomic") + .link + .successor; + assert!( + !reloaded + .rearm_preformat_handoff_retry_if_needed(&storage) + .await + .expect("grant already used") + ); + let recovered = reloaded + .resolve_replacement_recovery(&storage) + .await + .expect("replay the reserved handoff"); + assert_eq!(recovered.task_id, successor); + assert_eq!(recovered.retry_count, recovered.max_retries - 1); + assert_eq!(storage.probe_calls.load(Ordering::SeqCst), 1); + } +} + +#[tokio::test] +async fn preformat_recovery_rejects_progress_checkpoints_formats_and_unrelated_topology() { + for rejection in [ + "progress", + "checkpoint", + "blocked", + "completion_proof", + "format", + "topology", + "new_generation", + "single_retry", + ] { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + let source = manager.get_state().await.task_id; + match rejection { + "progress" => manager.state.write().await.processed_objects = 1, + "checkpoint" | "blocked" => { + let suffix = if rejection == "checkpoint" { + RESUME_CHECKPOINT_FILE + } else { + RESUME_CHECKPOINT_BLOCKED_FILE + }; + manager + .disk + .write_all(RUSTFS_META_BUCKET, &format!("{BUCKET_META_PREFIX}/{source}_{suffix}"), Vec::new().into()) + .await + .expect("persist checkpoint evidence, including an empty file"); + } + "format" => storage.execution.test_disks()[0] + .write_all(RUSTFS_META_BUCKET, "format.json", b"existing format".to_vec().into()) + .await + .expect("persist actual format evidence"), + "completion_proof" => { + let proof = super::super::replacement::ReplacementCompletionProof::from_state(&manager.get_state().await, 1) + .expect("construct a prior completion proof"); + manager + .disk + .write_all( + RUSTFS_META_BUCKET, + path_to_str(&replacement_completion_proof_path(&source)).expect("proof path"), + serde_json::to_vec(&proof).expect("serialize completion proof").into(), + ) + .await + .expect("persist existing completion proof"); + } + "topology" => storage.historical_topology = false, + "new_generation" => manager.state.write().await.replacement_preformat_handoff_retry_done = true, + "single_retry" => manager.state.write().await.max_retries = 1, + _ => unreachable!("known test rejection"), + } + manager.save_state_strict().await.expect("persist rejection fixture"); + let before = ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes before admission"); + assert!( + !manager + .rearm_preformat_handoff_retry_if_needed(&storage) + .await + .expect("reject unsafe recovery"), + "{rejection}" + ); + assert_eq!( + ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes after rejection"), + before, + "{rejection}" + ); + } +} + +#[tokio::test] +async fn preformat_recovery_rejects_mount_changes_before_or_during_probe() { + for change_identity_on_call in [0, 1] { + let (_dirs, manager, mut storage) = preformat_recovery_fixture().await; + storage.change_identity_on_call = change_identity_on_call; + let source = manager.get_state().await.task_id; + let before = ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes before mount change"); + assert!(matches!( + manager.rearm_preformat_handoff_retry_if_needed(&storage).await, + Err(Error::ReplacementTargetNotReady(_)) + )); + assert_eq!( + ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source bytes after mount change"), + before + ); + } +} + +#[tokio::test] +async fn preformat_recovery_rejects_unknown_marker_and_stale_revision() { + let (_dirs, manager, storage) = preformat_recovery_fixture().await; + let source = manager.get_state().await.task_id; + storage.execution.test_disks()[0] + .write_all(RUSTFS_META_BUCKET, HEALING_MARKER_PATH, b"unknown".to_vec().into()) + .await + .expect("publish unknown marker owner"); + let before = ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source before owner conflict"); + assert!(matches!( + manager.rearm_preformat_handoff_retry_if_needed(&storage).await, + Err(Error::ReplacementOwnershipConflict(_)) + )); + assert_eq!( + ResumeManager::read_state_file(&manager.disk, &source, manager.state_file) + .await + .expect("source after owner conflict"), + before + ); + storage.execution.test_disks()[0] + .delete(RUSTFS_META_BUCKET, HEALING_MARKER_PATH, Default::default()) + .await + .expect("remove fixture owner conflict"); + let stale = ResumeManager::load_replacement_intent(manager.disk.clone(), &source) + .await + .expect("open stale source"); + manager.save_state_strict().await.expect("advance source revision"); + assert!(stale.rearm_preformat_handoff_retry_if_needed(&storage).await.is_err()); + let unchanged = ResumeManager::load_replacement_intent(manager.disk.clone(), &source) + .await + .expect("reload authoritative source") + .get_state() + .await; + assert!(!unchanged.replacement_preformat_handoff_retry_done); + assert!(unchanged.replacement_handoff.is_none()); +} + +#[test] +fn preformat_recovery_marker_defaults_only_old_snapshots_to_eligible() { + let state = ResumeState::replacement_intent( + Uuid::new_v4().to_string(), + "erasure_set".into(), + "pool_0_set_0".into(), + Vec::new(), + vec!["replacement-0".into()], + vec![identity("replacement-0", "mount:fs:root")], + ); + assert!(state.replacement_preformat_handoff_retry_done); + let mut wire = serde_json::to_value(state).expect("serialize new intent"); + wire.as_object_mut() + .expect("intent object") + .remove("replacement_preformat_handoff_retry_done"); + let old: ResumeState = serde_json::from_value(wire).expect("deserialize older schema-seven intent"); + assert!(!old.replacement_preformat_handoff_retry_done); +} + #[tokio::test] async fn handoff_rebinding_never_resets_an_exhausted_successor_budget() { let (_dirs, parent, execution) = fixture(1).await; @@ -388,3 +829,92 @@ async fn handoff_gc_retains_pending_and_committed_authority() { assert!(ResumeManager::has_replacement_intent(&parent.disk, &parent.get_state().await.task_id).await); assert!(ResumeManager::has_replacement_intent(&parent.disk, &child.get_state().await.task_id).await); } + +#[tokio::test] +async fn handoff_chain_does_not_consume_the_rebuild_retry_budget() { + // rustfs/rustfs#8664: every mount change hands the generation off again. + // Interruptions are not rebuild failures, so three consecutive handoffs + // must leave the full `max_retries` budget for the eventual rebuild. + let (_dirs, root, execution) = fixture(1).await; + let max_retries = root.get_state().await.max_retries; + let mut current = root; + let mut mount = 0; + for interruption in 0..3 { + mount += 1; + let next_mount = ReplacementExecution::for_test( + execution.test_disks().to_vec(), + vec![identity("replacement-0", &format!("mount-{mount}:fs-1:root-1"))], + ); + let child = current + .prepare_replacement_handoff(&next_mount, Vec::new()) + .await + .unwrap_or_else(|error| panic!("interruption {interruption} must still hand off: {error}")); + child + .acquire_replacement_markers(&next_mount) + .await + .expect("successor takes the markers"); + let state = child.get_state().await; + assert_eq!(state.retry_count, 0, "handoff {interruption} must not spend a rebuild attempt"); + assert_eq!(state.max_retries, max_retries); + assert_eq!(state.replacement_lineage.len(), interruption + 1); + current = child; + } + let final_mount = ReplacementExecution::for_test( + execution.test_disks().to_vec(), + vec![identity("replacement-0", &format!("mount-{mount}:fs-1:root-1"))], + ); + current + .mark_replacement_rebuilding(final_mount.identities().to_vec()) + .await + .expect("the last successor rebuilds"); + current + .mark_replacement_completed_and_verified() + .await + .expect("the last successor completes"); + assert_eq!(current.get_state().await.replacement_phase, ReplacementPhase::Verified); +} + +#[tokio::test] +async fn committed_predecessor_is_collectible_once_its_successor_is_verified() { + let (_dirs, parent, execution) = fixture(1).await; + let parent_id = parent.get_state().await.task_id; + let child = parent + .prepare_replacement_handoff(&execution, Vec::new()) + .await + .expect("prepare"); + child.acquire_replacement_markers(&execution).await.expect("commit"); + let child_id = child.get_state().await.task_id; + // The commit advanced the predecessor through its own handle; reload the + // durable revision before aging both records. + let parent = ResumeManager::load_replacement_intent(parent.disk.clone(), &parent_id) + .await + .expect("reload committed predecessor"); + for manager in [&parent, &child] { + manager.state.write().await.last_update = 1; + manager.save_state_strict().await.expect("age the record"); + } + ResumeUtils::cleanup_expired_states(&parent.disk, 0) + .await + .expect("GC before verification"); + assert!( + ResumeManager::has_replacement_intent(&parent.disk, &parent_id).await, + "a committed authority is retained while its successor may still need it" + ); + + child + .mark_replacement_rebuilding(execution.identities().to_vec()) + .await + .expect("rebuild"); + child.mark_replacement_completed_and_verified().await.expect("verify"); + ResumeUtils::cleanup_expired_states(&parent.disk, 0) + .await + .expect("GC after verification"); + assert!( + !ResumeManager::has_replacement_intent(&parent.disk, &parent_id).await, + "a committed predecessor of a verified successor is collectible" + ); + assert!( + ResumeManager::has_replacement_intent(&parent.disk, &child_id).await, + "the verified successor keeps its own record until cleanup" + ); +} diff --git a/crates/heal/src/heal/resume/replacement.rs b/crates/heal/src/heal/resume/replacement.rs index 2c920a739..c04ec6e75 100644 --- a/crates/heal/src/heal/resume/replacement.rs +++ b/crates/heal/src/heal/resume/replacement.rs @@ -168,6 +168,91 @@ pub(super) fn replacement_targets_match_identities(targets: &[String], identitie && identities.iter().map(|identity| &identity.endpoint).eq(targets.iter()) } +/// Whether a durable generation still reserves its target slots. Active and +/// verified generations do; so does an `Abandoned` record that never +/// transferred its responsibility. A committed handoff or an approved legacy +/// migration has moved the responsibility to the successor, so the retired +/// record neither owns the set nor blocks fresh admission for other slots. +pub(crate) fn replacement_generation_reserves_targets(state: &ResumeState) -> bool { + state.replacement_generation.as_deref() == Some(state.task_id.as_str()) + && !state.replacement_targets.is_empty() + && (matches!( + state.replacement_phase, + ReplacementPhase::Intent + | ReplacementPhase::OwnershipPending + | ReplacementPhase::HandoffPending + | ReplacementPhase::Rebuilding + | ReplacementPhase::Verified + | ReplacementPhase::CleanupPending + ) || replacement_generation_is_orphaned_abandoned(state)) +} + +/// An `Abandoned` record whose responsibility was never transferred. +pub(crate) fn replacement_generation_is_orphaned_abandoned(state: &ResumeState) -> bool { + state.replacement_phase == ReplacementPhase::Abandoned + && state.replacement_handoff.is_none() + && state.replacement_legacy_successor.is_none() +} + +/// A retired predecessor: committed handoff or approved legacy migration. +pub(crate) fn replacement_generation_is_retired(state: &ResumeState) -> bool { + state.replacement_phase == ReplacementPhase::Abandoned + && (state + .replacement_handoff + .as_ref() + .is_some_and(|handoff| handoff.phase == super::ReplacementHandoffPhase::Committed) + || state.replacement_legacy_successor.is_some()) +} + +/// An orphaned `Abandoned` record releases its target slots only once every +/// slot's healing marker names another generation of the same set. A missing +/// marker, a marker this generation still owns, or an unreadable marker keeps +/// the reservation. +pub(crate) async fn abandoned_targets_are_owned_by_another_generation(state: &ResumeState, disks: &[DiskStore]) -> bool { + if !replacement_generation_is_orphaned_abandoned(state) { + return false; + } + let own_marker = format!("{}:{}", state.set_disk_id, state.task_id); + let set_prefix = format!("{}:", state.set_disk_id); + for target in &state.replacement_targets { + let Some(disk) = disks.iter().find(|disk| disk.endpoint().to_string() == *target) else { + return false; + }; + let marker = match disk.read_all(RUSTFS_META_BUCKET, super::super::HEALING_MARKER_PATH).await { + Ok(bytes) if bytes.len() <= 256 => String::from_utf8(bytes.to_vec()).ok(), + _ => None, + }; + let Some(marker) = marker else { + return false; + }; + if marker == own_marker || !marker.starts_with(&set_prefix) { + return false; + } + if super::validate_resume_task_id(&marker[set_prefix.len()..]).is_err() { + return false; + } + } + true +} + +/// One durable generation that discovery could not verify or resolve. Blocks +/// are node-local, keyed by generation, and re-evaluated on every discovery +/// pass; they never widen to the generation's set. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct ReplacementRecoveryBlock { + pub task_id: String, + pub set_disk_id: String, + pub reason: String, + /// Whether the blocked record still reserves its target slots, which + /// defers fresh admission for the same set until it is repaired. + pub reserves_targets: bool, + /// UNIX seconds of the first discovery pass that blocked this generation. + pub blocked_since: u64, + /// UNIX seconds of the latest discovery pass that kept it blocked. + pub last_observed: u64, +} + /// A durable intent can be visible through more than one survivor after a /// restart. The endpoint that exposes the record is a replica location, not /// a generation identity, so discovery must merge those observations before it @@ -537,13 +622,34 @@ impl ResumeManager { return self.save_state_strict().await; } state.error_message = Some(error.to_string()); - state.retry_count = state.retry_count.max(retry_attempts); - if matches!( - error, - Error::ReplacementRetryBudgetExhausted - | Error::ReplacementOwnershipConflict(_) - | Error::ReplacementGenerationConflict { .. } - ) { + // `max_retries` bounds rebuild attempts. Only a failure after the + // target was formatted and the generation entered `Rebuilding` is one; + // format fences, handoffs, and ownership or generation conflicts keep + // their durable reason without spending an attempt. + if matches!(error, Error::ReplacementRetryBudgetExhausted) { + state.retry_count = state.max_retries; + } else if state.replacement_phase == ReplacementPhase::Rebuilding + && !matches!( + error, + Error::ReplacementOwnershipConflict(_) | Error::ReplacementGenerationConflict { .. } + ) + { + state.retry_count = state.retry_count.max(retry_attempts); + } + state.last_update = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(); + drop(state); + self.save_state_strict().await + } + + /// Persist why resolving this generation's durable record failed. Resolution + /// runs before any rebuild, so it never spends a rebuild attempt. + pub(super) async fn record_replacement_resolution_failure(&self, error: &Error) -> Result<()> { + let mut state = self.state.write().await; + if state.replacement_generation.as_deref() != Some(state.task_id.as_str()) { + return Err(replacement_recovery_conflict("replacement failure has no matching generation")); + } + state.error_message = Some(error.to_string()); + if matches!(error, Error::ReplacementRetryBudgetExhausted) { state.retry_count = state.max_retries; } state.last_update = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(); diff --git a/crates/heal/src/heal/resume/tests.rs b/crates/heal/src/heal/resume/tests.rs index bc4cdc303..42fd4119c 100644 --- a/crates/heal/src/heal/resume/tests.rs +++ b/crates/heal/src/heal/resume/tests.rs @@ -2339,6 +2339,162 @@ async fn test_checkpoint_schema_v5_discarded_on_load() { temp_dir.close().expect("remove schema test directory"); } +// Keep these signed wire fixtures independent of today's struct and its defaults. +const LEGACY_SIGNED_CHECKPOINTS: [&str; 3] = [ + r#"{"schema_version":5,"task_id":"00000000-0000-4000-8000-000000000002","checkpoint_time":1700000000,"current_bucket_index":2,"current_object_index":500,"processed_objects":["a","b"],"failed_objects":["c"],"skipped_objects":["d"],"integrity_digest":"/AZPOA1TZttBQ0ieKM0QBVMDnr5s2YWZqb+3rmg+vds="}"#, + r#"{"schema_version":6,"task_id":"00000000-0000-4000-8000-000000000002","checkpoint_time":1700000000,"current_bucket_index":2,"current_object_index":500,"processed_objects":["a","b"],"failed_objects":["c"],"skipped_objects":["d"],"integrity_digest":"4IBOlSw4ln91DXRnVZD0D7qDt+DKSp205Y6eFwNpm80="}"#, + r#"{"schema_version":6,"task_id":"00000000-0000-4000-8000-000000000002","checkpoint_time":1700000000,"current_bucket_index":2,"current_object_index":500,"processed_objects":["a","b"],"failed_objects":["c"],"skipped_objects":["d"],"integrity_digest":"mkGdi8fQRM6f1yS9NIyf+O8F9FRL4/pBXL79CD/il7k="}"#, +]; + +#[tokio::test] +async fn signed_legacy_checkpoint_verifies_before_schema_upgrade() { + for legacy in LEGACY_SIGNED_CHECKPOINTS { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + disk.write_all(RUSTFS_META_BUCKET, &path, legacy.as_bytes().to_vec().into()) + .await + .expect("write signed legacy checkpoint"); + + assert!( + CheckpointManager::is_resumable(&disk, task_id) + .await + .expect("discovery must accept an intact signed legacy checkpoint") + ); + let manager = CheckpointManager::load_from_disk(disk.clone(), task_id) + .await + .expect("verify the original signed fields before filling defaults"); + let checkpoint = manager.get_checkpoint().await; + assert_eq!(checkpoint.schema_version, CURRENT_CHECKPOINT_SCHEMA); + assert_eq!(checkpoint.current_bucket_index, 0); + assert_eq!(checkpoint.current_object_index, 0); + assert!(checkpoint.processed_objects.is_empty()); + assert!(checkpoint.failed_objects.is_empty()); + assert!(checkpoint.skipped_objects.is_empty()); + assert_eq!(checkpoint.successful_objects, 0); + assert_eq!(checkpoint.failed_object_count, 0); + assert_eq!(checkpoint.skipped_object_count, 0); + assert!(!checkpoint.baseline_known); + assert!(!CheckpointManager::is_blocked(&disk, task_id).await); + + manager + .update_position(1, 3) + .await + .expect("save upgraded checkpoint with CAS"); + let restored = CheckpointManager::load_from_disk(disk, task_id) + .await + .expect("reload upgraded checkpoint") + .get_checkpoint() + .await; + assert_eq!(restored.current_bucket_index, 1); + assert_eq!(restored.current_object_index, 3); + temp_dir.close().expect("remove signed legacy checkpoint fixture"); + } +} + +#[tokio::test] +async fn signed_legacy_checkpoint_rejects_tampering() { + for legacy in LEGACY_SIGNED_CHECKPOINTS { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let tampered = legacy.replace("\"current_object_index\":500", "\"current_object_index\":501"); + disk.write_all(RUSTFS_META_BUCKET, &path, tampered.as_bytes().to_vec().into()) + .await + .expect("write modified signed legacy checkpoint"); + + assert!(matches!( + CheckpointManager::load_from_disk(disk.clone(), task_id).await, + Err(Error::InvalidCheckpoint(message)) if message.contains("digest does not match") + )); + assert!(CheckpointManager::is_blocked(&disk, task_id).await); + assert_eq!( + disk.read_all(RUSTFS_META_BUCKET, &path) + .await + .expect("invalid snapshot must remain available"), + tampered.as_bytes() + ); + temp_dir.close().expect("remove modified legacy checkpoint fixture"); + } +} + +#[tokio::test] +async fn signed_checkpoint_rejects_duplicate_fields() { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let duplicate = LEGACY_SIGNED_CHECKPOINTS[2].replace( + "\"current_object_index\":500", + "\"current_object_index\":501,\"current_object_index\":500", + ); + disk.write_all(RUSTFS_META_BUCKET, &path, duplicate.as_bytes().to_vec().into()) + .await + .expect("write checkpoint with conflicting duplicate fields"); + + assert!(matches!( + CheckpointManager::load_from_disk(disk.clone(), task_id).await, + Err(Error::TaskExecutionFailed { message }) if message.contains("duplicate field") + )); + assert!(CheckpointManager::is_blocked(&disk, task_id).await); + temp_dir.close().expect("remove duplicate field checkpoint fixture"); +} + +#[tokio::test] +async fn checkpoint_writer_preserves_base_reader_canonicalization() { + use sha2::{Digest, Sha256}; + + for admin in [false, true] { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = ResumeUtils::generate_task_id(); + let manager = CheckpointManager::new(disk.clone(), task_id.clone()) + .await + .expect("create rollback-compatible checkpoint"); + if admin { + manager + .set_admin(AdminErasureCheckpoint { + set_disk_id: "pool_0_set_0".to_string(), + options: Default::default(), + buckets: [("bucket".to_string(), Some(uuid::Uuid::new_v4()))].into(), + outcome: Default::default(), + completed: false, + }) + .await + .expect("save nested administrator checkpoint"); + } + manager.add_processed_object("b".to_string()).await.expect("record object b"); + manager.add_processed_object("a".to_string()).await.expect("record object a"); + manager.update_position(2, 9).await.expect("flush rollback fixture"); + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let bytes = disk + .read_all(RUSTFS_META_BUCKET, &path) + .await + .expect("read real writer output"); + let mut wire: serde_json::Value = serde_json::from_slice(&bytes).expect("decode writer output"); + wire.sort_all_objects(); + assert_eq!( + bytes.as_ref(), + serde_json::to_vec(&wire).expect("encode recursively sorted payload"), + "persisted keys must have the production base writer's order, including nested admin fields" + ); + let signature = wire["integrity_digest"].take(); + let unsigned = serde_json::to_vec(&wire).expect("encode base reader signing representation"); + assert_eq!( + signature, + base64_simd::STANDARD.encode_to_string(Sha256::digest(&unsigned)), + "the production base reader must accept this writer's signature" + ); + let restored = CheckpointManager::load_from_disk(disk, &task_id) + .await + .expect("same-version control must also load") + .get_checkpoint() + .await; + assert_eq!(restored.current_bucket_index, 2); + assert_eq!(restored.current_object_index, 9); + assert_eq!(restored.admin.is_some(), admin); + temp_dir.close().expect("remove rollback canonicalization fixture"); + } +} + #[tokio::test] async fn downgraded_unsigned_checkpoint_resets_untrusted_progress() { let (temp_dir, disk) = schema_test_disk().await; @@ -2770,34 +2926,38 @@ async fn checkpoint_cleanup_leaves_no_task_specific_lock_artifact() { } #[tokio::test] -async fn an_empty_blocked_marker_still_blocks_resume_selection() { +async fn an_empty_blocked_marker_over_a_missing_snapshot_is_cleared_by_revalidation() { + // A legacy marker is due immediately. With no snapshot left to isolate, + // a fresh scan is the safe continuation, so the marker is cleared. let (temp_dir, disk) = schema_test_disk().await; let task_id = ResumeUtils::generate_task_id(); - let manager = CheckpointManager::new(disk.clone(), task_id.clone()) - .await - .expect("create checkpoint manager"); let blocked_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}"); disk.write_all(RUSTFS_META_BUCKET, &blocked_path, EcstoreDiskBytes::new()) .await .expect("write empty blocked marker fixture"); - assert!(CheckpointManager::is_blocked(&disk, &task_id).await); - assert!(CheckpointManager::is_resumable(&disk, &task_id).await.is_err()); - // Recovery requires replacing/cleaning the snapshot, then removing the - // marker; ordinary selector retries are intentionally not an unlock path. - manager.cleanup().await.expect("clean blocked checkpoint"); assert!(!CheckpointManager::is_blocked(&disk, &task_id).await); + assert!(disk.read_all(RUSTFS_META_BUCKET, &blocked_path).await.is_err()); + assert!( + CheckpointManager::is_resumable(&disk, &task_id) + .await + .expect("no snapshot and no marker") + ); temp_dir.close().expect("remove empty blocked marker test directory"); } #[tokio::test] -async fn resumable_selector_skips_healthy_tasks_with_blocked_markers() { +async fn resumable_selector_revalidates_legacy_markers_per_task() { + // One task carries a legacy marker over an intact snapshot: it unblocks. + // The other carries a legacy marker over a corrupt snapshot: it stays + // blocked and keeps failing the selector, while its bytes are retained. let (temp_dir, disk) = schema_test_disk().await; - let tasks = [ - (ResumeUtils::generate_task_id(), EcstoreDiskBytes::new()), - (ResumeUtils::generate_task_id(), EcstoreDiskBytes::from_static(b"blocked")), - ]; - for (task_id, marker) in &tasks { + let healthy = ResumeUtils::generate_task_id(); + let corrupt = ResumeUtils::generate_task_id(); + for (task_id, marker) in [ + (&healthy, EcstoreDiskBytes::new()), + (&corrupt, EcstoreDiskBytes::from_static(b"blocked")), + ] { ResumeManager::new( disk.clone(), task_id.clone(), @@ -2806,28 +2966,35 @@ async fn resumable_selector_skips_healthy_tasks_with_blocked_markers() { vec!["bucket".to_string()], ) .await - .expect("create healthy resume state"); + .expect("create resume state"); CheckpointManager::new(disk.clone(), task_id.clone()) .await - .expect("create healthy checkpoint"); - let checkpoint_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); - let checkpoint_bytes = disk - .read_all(RUSTFS_META_BUCKET, &checkpoint_path) - .await - .expect("read healthy checkpoint before blocking"); + .expect("create checkpoint"); let marker_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}"); - disk.write_all(RUSTFS_META_BUCKET, &marker_path, marker.clone()) + disk.write_all(RUSTFS_META_BUCKET, &marker_path, marker) .await - .expect("write blocked marker"); - - assert!(ResumeUtils::get_resumable_tasks(&disk).await.is_err()); - assert_eq!( - disk.read_all(RUSTFS_META_BUCKET, &checkpoint_path) - .await - .expect("read healthy checkpoint after blocking"), - checkpoint_bytes - ); + .expect("write legacy blocked marker"); } + let corrupt_path = format!("{BUCKET_META_PREFIX}/{corrupt}_{RESUME_CHECKPOINT_FILE}"); + disk.write_all(RUSTFS_META_BUCKET, &corrupt_path, b"{truncated".to_vec().into()) + .await + .expect("corrupt one snapshot"); + + assert!( + CheckpointManager::is_resumable(&disk, &healthy) + .await + .expect("intact snapshot unblocks") + ); + assert!(!CheckpointManager::is_blocked(&disk, &healthy).await); + assert!(CheckpointManager::is_resumable(&disk, &corrupt).await.is_err()); + assert!(CheckpointManager::is_blocked(&disk, &corrupt).await); + assert!(ResumeUtils::get_resumable_tasks(&disk).await.is_err()); + assert_eq!( + disk.read_all(RUSTFS_META_BUCKET, &corrupt_path) + .await + .expect("corrupt snapshot retained"), + b"{truncated".as_slice() + ); temp_dir.close().expect("remove blocked selector test directory"); } @@ -3221,3 +3388,217 @@ async fn resume_state_rejects_filename_and_json_task_id_mismatch() { Error::TaskExecutionFailed { message } if message == "Resume checkpoint task id does not match filename" )); } + +async fn budget_fixture() -> (tempfile::TempDir, DiskStore, ResumeManager) { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = ResumeUtils::generate_task_id(); + let manager = ResumeManager::new_replacement_intent( + disk.clone(), + task_id, + "pool_0_set_0".to_string(), + vec!["bucket".to_string()], + vec!["replacement-a".to_string()], + vec![ReplacementTargetIdentity { + endpoint: "replacement-a".to_string(), + canonical_path: "/mnt/replacement-a".to_string(), + physical_device_ids: vec!["device-a".to_string()], + filesystem_identity: "1:2:3".to_string(), + }], + ) + .await + .expect("replacement intent should persist"); + (temp_dir, disk, manager) +} + +#[tokio::test] +async fn preformat_failures_do_not_consume_the_rebuild_budget() { + // rustfs/rustfs#8565: a format fence rejection before `format.json` + // exists is not a rebuild attempt. + let (_temp_dir, disk, manager) = budget_fixture().await; + let task_id = manager.get_state().await.task_id; + for attempt in 1..=5 { + manager + .record_replacement_failure(&Error::Storage(crate::heal::EcstoreError::SlowDown), attempt) + .await + .expect("pre-format failure persists"); + } + let state = ResumeManager::load_replacement_intent(disk.clone(), &task_id) + .await + .expect("reload") + .get_state() + .await; + assert_eq!(state.replacement_phase, ReplacementPhase::Intent); + assert_eq!(state.retry_count, 0, "pre-format failures must not spend the rebuild budget"); + assert!( + state + .error_message + .as_deref() + .is_some_and(|m| m.contains("reduce your request rate")) + ); + assert_ne!( + ReplacementRecoveryRecord::from_state(state.clone()).expect("record").state, + ReplacementRecoveryState::Unrecoverable + ); + + // Genuine rebuild failures after formatting still exhaust the budget. + manager + .mark_replacement_rebuilding(state.replacement_target_identities.clone()) + .await + .expect("format admitted"); + for attempt in 1..=state.max_retries { + manager + .record_replacement_failure(&Error::other("rebuild failed"), attempt) + .await + .expect("rebuild failure persists"); + } + let exhausted = manager.get_state().await; + assert_eq!(exhausted.retry_count, exhausted.max_retries); + assert_eq!( + ReplacementRecoveryRecord::from_state(exhausted).expect("record").state, + ReplacementRecoveryState::Unrecoverable + ); +} + +#[tokio::test] +async fn repeated_target_outages_never_consume_the_rebuild_budget() { + let (_temp_dir, _disk, manager) = budget_fixture().await; + let identities = manager.get_state().await.replacement_target_identities; + manager + .mark_replacement_rebuilding(identities) + .await + .expect("format admitted"); + manager + .record_replacement_failure(&Error::other("one real failure"), 1) + .await + .expect("rebuild failure persists"); + for attempt in 2..=11 { + manager + .record_replacement_failure(&Error::ReplacementTargetNotReady("target offline".to_string()), attempt) + .await + .expect("outage persists as a deferral"); + } + let state = manager.get_state().await; + assert_eq!(state.retry_count, 1, "ten outages must leave the single real failure"); + assert!(state.replacement_retry_waiting_for_target); + assert!(state.can_retry()); +} + +fn blocked_marker_path(task_id: &str) -> String { + format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}") +} + +#[tokio::test] +async fn blocked_checkpoint_is_unblocked_when_the_current_reader_accepts_it() { + // rustfs/rustfs#8645: an older binary blocked a checkpoint whose digest it + // could not verify. The compatible reader accepts it, so a revalidation + // must clear the legacy marker instead of blocking the task forever. + for legacy in LEGACY_SIGNED_CHECKPOINTS { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + disk.write_all(RUSTFS_META_BUCKET, &path, legacy.as_bytes().to_vec().into()) + .await + .expect("write signed legacy checkpoint"); + disk.write_all( + RUSTFS_META_BUCKET, + &blocked_marker_path(task_id), + EcstoreDiskBytes::from_static(b"blocked"), + ) + .await + .expect("write legacy blocked marker"); + + assert!( + CheckpointManager::is_resumable(&disk, task_id) + .await + .expect("revalidation must accept the intact checkpoint") + ); + assert!(!CheckpointManager::is_blocked(&disk, task_id).await, "revalidation clears the marker"); + assert!( + disk.read_all(RUSTFS_META_BUCKET, &blocked_marker_path(task_id)) + .await + .is_err(), + "the marker file is removed" + ); + assert_eq!( + disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("checkpoint retained"), + legacy.as_bytes() + ); + temp_dir.close().expect("remove revalidation fixture"); + } +} + +#[tokio::test] +async fn tampered_checkpoint_stays_blocked_after_revalidation() { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + let tampered = LEGACY_SIGNED_CHECKPOINTS[2].replace("\"current_object_index\":500", "\"current_object_index\":501"); + disk.write_all(RUSTFS_META_BUCKET, &path, tampered.as_bytes().to_vec().into()) + .await + .expect("write tampered checkpoint"); + disk.write_all( + RUSTFS_META_BUCKET, + &blocked_marker_path(task_id), + EcstoreDiskBytes::from_static(b"blocked"), + ) + .await + .expect("write legacy blocked marker"); + + assert!(CheckpointManager::is_resumable(&disk, task_id).await.is_err()); + assert!(CheckpointManager::is_blocked(&disk, task_id).await); + let marker = disk + .read_all(RUSTFS_META_BUCKET, &blocked_marker_path(task_id)) + .await + .expect("marker retained"); + let marker: serde_json::Value = serde_json::from_slice(&marker).expect("revalidated marker is structured"); + let now = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(); + let revalidate_after = marker["revalidate_after"].as_u64().expect("next revalidation time"); + assert!(revalidate_after > now + 3000, "revalidation is rescheduled about an hour later"); + assert!(marker["reason"].as_str().is_some_and(|reason| reason.contains("digest"))); + + // Not due yet: the second call must not rewrite the marker. + assert!(CheckpointManager::is_resumable(&disk, task_id).await.is_err()); + let unchanged = disk + .read_all(RUSTFS_META_BUCKET, &blocked_marker_path(task_id)) + .await + .expect("marker retained"); + assert_eq!(serde_json::from_slice::(&unchanged).expect("marker JSON"), marker); + assert_eq!( + disk.read_all(RUSTFS_META_BUCKET, &path).await.expect("checkpoint retained"), + tampered.as_bytes() + ); + temp_dir.close().expect("remove tampered revalidation fixture"); +} + +#[tokio::test] +async fn blocked_checkpoint_is_not_revalidated_before_its_schedule() { + let (temp_dir, disk) = schema_test_disk().await; + let task_id = "00000000-0000-4000-8000-000000000002"; + let path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}"); + disk.write_all(RUSTFS_META_BUCKET, &path, LEGACY_SIGNED_CHECKPOINTS[2].as_bytes().to_vec().into()) + .await + .expect("write intact checkpoint"); + let now = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(); + let marker = serde_json::json!({ + "schema": 1, + "blocked_at": now, + "revalidate_after": now + 3600, + "reason": "digest mismatch observed by a previous reader", + }); + disk.write_all( + RUSTFS_META_BUCKET, + &blocked_marker_path(task_id), + serde_json::to_vec(&marker).expect("marker JSON").into(), + ) + .await + .expect("write scheduled marker"); + + assert!(CheckpointManager::is_resumable(&disk, task_id).await.is_err()); + assert!(CheckpointManager::is_blocked(&disk, task_id).await); + let retained = disk + .read_all(RUSTFS_META_BUCKET, &blocked_marker_path(task_id)) + .await + .expect("marker retained"); + assert_eq!(serde_json::from_slice::(&retained).expect("marker JSON"), marker); + temp_dir.close().expect("remove scheduled revalidation fixture"); +} diff --git a/crates/heal/src/heal/resume/utils.rs b/crates/heal/src/heal/resume/utils.rs index f3b39f563..903846d8b 100644 --- a/crates/heal/src/heal/resume/utils.rs +++ b/crates/heal/src/heal/resume/utils.rs @@ -22,7 +22,7 @@ use super::super::{BUCKET_META_PREFIX, DiskError, DiskStore, HealDiskExt as _, R use super::replacement::{ReplacementPhase, ReplacementRecoveryRecord}; use super::{ CheckpointManager, EVENT_HEAL_RESUME_STATE, LOG_COMPONENT_HEAL, LOG_SUBSYSTEM_RESUME, REPLACEMENT_COMPLETION_PROOF_FILE, - REPLACEMENT_INTENT_FILE, RESUME_STATE_FILE, ResumeManager, ResumeStateFile, is_replacement_intent, path_to_str, + REPLACEMENT_INTENT_FILE, RESUME_STATE_FILE, ResumeManager, ResumeState, ResumeStateFile, is_replacement_intent, path_to_str, replacement_recovery_corruption_for_state_load, replacement_recovery_dir, validate_resume_task_id, }; @@ -226,6 +226,37 @@ impl ResumeUtils { Ok(records) } + /// A predecessor that committed its handoff is collectible once its + /// successor has verified the rebuild or already cleaned up behind a + /// completion proof. Uncommitted authority and predecessors of unfinished + /// successors are retained; so is any record whose successor cannot be read. + pub(crate) async fn retired_predecessor_is_collectible(disk: &DiskStore, state: &ResumeState) -> bool { + if !super::replacement_generation_is_retired(state) { + return false; + } + let Some(successor) = state + .replacement_handoff + .as_ref() + .map(|handoff| handoff.link.successor.as_str()) + else { + return false; + }; + if ResumeManager::has_replacement_intent(disk, successor).await { + return match ResumeManager::load_replacement_intent(disk.clone(), successor).await { + Ok(manager) => { + let successor = manager.get_state().await; + successor.completed + && matches!(successor.replacement_phase, ReplacementPhase::Verified | ReplacementPhase::CleanupPending) + } + Err(_) => false, + }; + } + matches!( + ResumeManager::replacement_completion_proof_if_present(disk.clone(), successor).await, + Ok(Some(_)) + ) + } + /// cleanup expired resume states pub async fn cleanup_expired_states(disk: &DiskStore, max_age_hours: u64) -> Result<()> { let task_ids = Self::get_resumable_tasks(disk).await?; @@ -291,11 +322,13 @@ impl ResumeUtils { let state = resume_manager.get_state().await; let age_hours = current_time.saturating_sub(state.last_update) / 3600; - // Retain handoff authority even after commit: a successor may still - // need it to resume marker ownership. Completion proofs carry the + // Retain handoff authority until its successor no longer needs + // it to resume marker ownership. Completion proofs carry the // lineage so a future archival policy can verify the whole chain. - if state.replacement_handoff.is_some() - || state.replacement_legacy_successor.is_some() + if state.replacement_handoff.is_some() && !Self::retired_predecessor_is_collectible(disk, &state).await { + continue; + } + if state.replacement_legacy_successor.is_some() || (!state.completed && matches!( state.replacement_phase, diff --git a/crates/heal/src/heal/storage.rs b/crates/heal/src/heal/storage.rs index 35facd16b..95ce385ef 100644 --- a/crates/heal/src/heal/storage.rs +++ b/crates/heal/src/heal/storage.rs @@ -491,6 +491,10 @@ pub trait HealStorageAPI: Send + Sync { /// Get all buckets async fn list_buckets(&self) -> Result>; + /// Return only a complete bucket inventory for an administrator task. + /// Incomplete discovery must fail before traversal or scope checkpointing. + async fn list_buckets_for_admin_heal(&self, selector: Option) -> Result>; + /// Check object exists async fn object_exists(&self, bucket: &str, object: &str) -> Result; @@ -732,6 +736,13 @@ pub trait HealStorageAPI: Send + Sync { Ok(true) } + /// Admit a one-shot recovery of an old pre-format intent only when the + /// current format fence permits its set, rebalance has a quiescent older + /// topology, and every target is a blank, unformatted replacement. + async fn replacement_preformat_retry_is_safe(&self, _set_disk_id: &str, _targets: &[String]) -> Result { + Ok(false) + } + async fn replacement_execution(&self, _targets: &[String]) -> Result> { Err(Error::other("replacement execution lease acquisition is unsupported")) } @@ -911,35 +922,11 @@ impl ECStoreHealStorage { } } -fn is_transient_object_exists_message(message: &str) -> bool { - let message = message.to_ascii_lowercase(); - - [ - "failed to acquire read lock", - "lock acquisition failed", - "lock acquisition timeout", - "quorum not reached", - "deadline has elapsed", - "timed out", - "network error", - "transport error", - "connection refused", - ] - .iter() - .any(|pattern| message.contains(pattern)) -} - +/// A transient or contended existence check is skipped and retried; it is +/// never reported as object absence or as a failure. One classifier decides +/// (backlog#2900). fn is_transient_object_exists_error(err: &StorageError) -> bool { - if err.is_quorum_error() { - return true; - } - - match err { - StorageError::Lock(lock_err) => lock_err.is_retryable() || is_transient_object_exists_message(&lock_err.to_string()), - StorageError::Io(io_err) => is_transient_object_exists_message(&io_err.to_string()), - StorageError::SlowDown | StorageError::OperationCanceled => true, - _ => false, - } + crate::error::classify_storage_error(err).is_retryable() } #[async_trait] @@ -1335,6 +1322,14 @@ impl HealStorageAPI for ECStoreHealStorage { } } + async fn list_buckets_for_admin_heal(&self, selector: Option) -> Result> { + let listing = self.ecstore.list_buckets_for_admin_heal(selector).await?; + if !listing.topology_complete { + return Err(Error::BucketDiscoveryIncomplete); + } + Ok(listing.buckets) + } + async fn list_buckets(&self) -> Result> { debug!( target: "rustfs::heal::storage", @@ -1993,6 +1988,7 @@ impl HealStorageAPI for ECStoreHealStorage { ) -> Result { let disks = self.resume_disk_inventory(set_disk_id).await?; let mut selected = None; + let mut inspected_other_tasks = HashSet::new(); let mut observed_other_tasks = HashSet::new(); for disk_store in &disks { if excluded_targets.contains(&disk_store.endpoint().to_string()) @@ -2014,7 +2010,7 @@ impl HealStorageAPI for ECStoreHealStorage { // generation while another durable intent already owns this // set. Inspect one copy of each other task and let the // caller reconcile it through the normal recovery scanner. - if candidate_task_id != task_id && !observed_other_tasks.insert(candidate_task_id.clone()) { + if candidate_task_id != task_id && !inspected_other_tasks.insert(candidate_task_id.clone()) { continue; } let manager = @@ -2027,6 +2023,16 @@ impl HealStorageAPI for ECStoreHealStorage { }); } if candidate_task_id != task_id { + // A retired record (committed handoff or approved legacy + // migration) transferred its responsibility and no longer + // owns the set (rustfs/rustfs#8664). An orphaned abandoned + // record releases its slots once another generation's + // markers own them. + if super::resume::replacement_generation_reserves_targets(&state) + && !super::resume::abandoned_targets_are_owned_by_another_generation(&state, &disks).await + { + observed_other_tasks.insert(candidate_task_id); + } continue; } let candidate = @@ -2097,6 +2103,14 @@ impl HealStorageAPI for ECStoreHealStorage { async fn replacement_execution(&self, targets: &[String]) -> Result> { ReplacementExecution::acquire(targets).await } + + async fn replacement_preformat_retry_is_safe(&self, set_disk_id: &str, targets: &[String]) -> Result { + let (pool_index, set_index) = crate::heal::utils::parse_set_disk_id(set_disk_id)?; + self.ecstore + .replacement_preformat_retry_is_safe(pool_index, set_index, targets) + .await + .map_err(Error::Storage) + } } #[cfg(test)] @@ -2104,7 +2118,7 @@ mod tests { use super::super::StorageError; use super::{ decode_disk_walk_token, decode_heal_token, encode_disk_walk_token, encode_heal_token, is_transient_object_exists_error, - is_transient_object_exists_message, next_heal_listing_token, replacement_resume_conflict_for_other_tasks, + next_heal_listing_token, replacement_resume_conflict_for_other_tasks, }; use std::collections::HashSet; @@ -2337,20 +2351,23 @@ mod tests { } #[test] - fn transient_object_exists_message_matches_lock_quorum_failures() { - assert!(is_transient_object_exists_message( - "Failed to acquire read lock: ns_loc: read lock acquisition failed on bucket/object: Quorum not reached: required 2, achieved 0" - )); - assert!(is_transient_object_exists_message("deadline has elapsed")); - } - - #[test] - fn transient_object_exists_error_matches_quorum_variants() { + fn transient_object_exists_error_matches_quorum_lock_and_offline_variants() { assert!(is_transient_object_exists_error(&StorageError::ErasureReadQuorum)); assert!(is_transient_object_exists_error(&StorageError::InsufficientReadQuorum( "bucket".to_string(), "object".to_string(), ))); + assert!(is_transient_object_exists_error(&StorageError::Io(std::io::Error::other( + "Failed to acquire read lock: ns_loc: read lock acquisition failed on bucket/object: Quorum not reached: required 2, achieved 0" + )))); + assert!(is_transient_object_exists_error(&StorageError::Io(std::io::Error::other( + "deadline has elapsed" + )))); + assert!(is_transient_object_exists_error(&StorageError::DiskNotFound)); + assert!(is_transient_object_exists_error(&StorageError::VolumeNotFound)); + assert!(is_transient_object_exists_error(&StorageError::RemoteClientUnavailable( + "rebuild".to_string() + ))); } #[test] diff --git a/crates/heal/src/heal/storage_api.rs b/crates/heal/src/heal/storage_api.rs index 24a1e8cf0..c1640b757 100644 --- a/crates/heal/src/heal/storage_api.rs +++ b/crates/heal/src/heal/storage_api.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +pub(crate) use rustfs_ecstore::api::capacity::is_reserved_or_invalid_bucket as ecstore_is_reserved_or_invalid_bucket; pub(crate) use rustfs_ecstore::api::data_usage::{ DATA_USAGE_CACHE_NAME as ECSTORE_DATA_USAGE_CACHE_NAME, load_admin_data_usage_from_backend_cached as ecstore_load_admin_data_usage_from_backend_cached, @@ -26,7 +27,13 @@ pub(crate) use rustfs_ecstore::api::disk::{ }; pub(crate) use rustfs_ecstore::api::disk::{DiskOption as EcstoreDiskOption, new_disk as ecstore_new_disk}; pub(crate) use rustfs_ecstore::api::error::{Error as EcstoreErrorType, StorageError as EcstoreStorageError}; +#[cfg(test)] +pub(crate) use rustfs_ecstore::api::layout::{ + EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints, +}; pub(crate) use rustfs_ecstore::api::runtime::local_disk_map_read as ecstore_local_disk_map_read; +#[cfg(test)] +pub(crate) use rustfs_ecstore::api::storage::init_local_disks as ecstore_init_local_disks; pub(crate) use rustfs_ecstore::api::storage::{ ECStore as EcstoreStore, HealLifecycleExpiryContext as EcstoreHealLifecycleExpiryContext, HealObjectStorageResult as EcstoreHealObjectStorageResult, POOL_META_NAME as ECSTORE_POOL_META_NAME, @@ -40,8 +47,8 @@ pub(crate) mod owner { ECSTORE_BUCKET_META_PREFIX, ECSTORE_DATA_USAGE_CACHE_NAME, ECSTORE_HEALING_MARKER_PATH, ECSTORE_POOL_META_NAME, ECSTORE_RUSTFS_META_BUCKET, EcstoreConditionalFileUpdate, EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError, EcstoreDiskResult, EcstoreDiskStore, EcstoreEndpoint, EcstoreErrorType, - EcstoreHealLifecycleExpiryContext, EcstoreStorageError, EcstoreStore, ecstore_load_admin_data_usage_from_backend_cached, - ecstore_local_disk_map_read, + EcstoreHealLifecycleExpiryContext, EcstoreStorageError, EcstoreStore, ecstore_is_reserved_or_invalid_bucket, + ecstore_load_admin_data_usage_from_backend_cached, ecstore_local_disk_map_read, }; pub(crate) use super::{EcstoreDiskOption, ecstore_new_disk}; diff --git a/crates/heal/src/heal/task.rs b/crates/heal/src/heal/task.rs index 55a999e3f..61e024652 100644 --- a/crates/heal/src/heal/task.rs +++ b/crates/heal/src/heal/task.rs @@ -12,8 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::error::{HealErrorClass, classify_heal_error}; use crate::heal::{ - DiskError, EcstoreError, ErasureSetHealer, HealDiskExt as _, + DiskError, ErasureSetHealer, HealDiskExt as _, erasure_healer::target_outcomes_complete, outcome::{ HealAbortReason, HealDeferredReason, HealExecutionOutcome, HealFailureClass, HealObjectDisposition, HealObjectIdentity, @@ -167,17 +168,10 @@ impl HealType { } } -fn is_object_level_not_found_error(err: &Error) -> bool { - match err { - Error::Disk(DiskError::FileNotFound | DiskError::FileVersionNotFound) => true, - Error::Storage(EcstoreError::FileNotFound | EcstoreError::FileVersionNotFound) => true, - Error::Other(message) => matches!(message.as_str(), "File not found" | "File version not found"), - _ => false, - } -} - +/// A trailing-slash object-dir candidate whose locked storage read found no +/// object is a listing artifact, not a missing object. pub(crate) fn is_missing_object_dir_heal_result(object: &str, err: &Error) -> bool { - object.ends_with(SLASH_SEPARATOR) && is_object_level_not_found_error(err) + object.ends_with(SLASH_SEPARATOR) && err.is_object_absent() } /// Sample cap for per-object failure logs during a sweep: returns true (and @@ -440,6 +434,8 @@ pub struct HealTask { replacement_execution: Arc>>>, replacement_running: Arc, replacement_start_retry_count: Arc, + /// Set when the locked storage read classified the single target as absent. + object_absent: Arc, /// Task status pub status: Arc>, /// Progress tracking @@ -511,6 +507,7 @@ impl HealTask { replacement_execution: Arc::new(RwLock::new(None)), replacement_running: Arc::new(AtomicBool::new(false)), replacement_start_retry_count: Arc::new(AtomicU32::new(0)), + object_absent: Arc::new(AtomicBool::new(false)), status: Arc::new(RwLock::new(HealTaskStatus::Pending)), progress: Arc::new(RwLock::new(HealProgress::new())), result_items: Arc::new(RwLock::new(VecDeque::with_capacity(MAX_RETAINED_HEAL_RESULT_ITEMS))), @@ -605,6 +602,16 @@ impl HealTask { self.outcome.read().await.clone() } + /// Record that the locked storage read found the single target absent. + pub(crate) fn mark_object_absent(&self) { + self.object_absent.store(true, Ordering::Release); + } + + /// Whether this task's single target was classified absent. + pub(crate) fn object_absent(&self) -> bool { + self.object_absent.load(Ordering::Acquire) + } + fn outcome_identity( &self, bucket: &str, @@ -885,20 +892,7 @@ impl HealTask { } fn is_transient_lock_or_timeout_error(err: &Error) -> bool { - // Typed-first (backlog#1845): trust the lock taxonomy and timeout - // variants before falling back to message needles for errors whose - // typed identity was stringified upstream. - if let Error::Storage(EcstoreError::Lock(lock_err)) = err { - return lock_err.is_retryable() || matches!(lock_err, rustfs_lock::LockError::QuorumNotReached { .. }); - } - if matches!(err, Error::Disk(DiskError::Timeout) | Error::Storage(EcstoreError::Timeout)) { - return true; - } - let message = err.to_string().to_ascii_lowercase(); - message.contains("lock acquisition timeout") - || message.contains("lock acquisition failed") - || message.contains("timed out") - || message.contains("deadline has elapsed") + matches!(classify_heal_error(err), HealErrorClass::LockContention) } fn should_skip_data_usage_cache_heal_error(bucket: &str, object: &str, err: &Error) -> bool { @@ -910,32 +904,7 @@ impl HealTask { } fn is_no_heal_required_error(err: &Error) -> bool { - match err { - Error::Storage(EcstoreError::NoHealRequired) | Error::Disk(DiskError::NoHealRequired) => true, - Error::Other(message) => matches!(message.as_str(), "No heal required" | "No healing is required"), - _ => matches!(err.to_string().as_str(), "No heal required" | "No healing is required"), - } - } - - fn is_object_not_found_heal_error(err: &Error) -> bool { - match err { - Error::Disk(DiskError::FileNotFound | DiskError::FileVersionNotFound) => true, - Error::Storage( - EcstoreError::FileNotFound - | EcstoreError::FileVersionNotFound - | EcstoreError::ObjectNotFound(_, _) - | EcstoreError::VersionNotFound(_, _, _), - ) => true, - Error::Other(message) => { - message.contains("File not found") - || message.contains("file not found") - || message.contains("File version not found") - || message.contains("file version not found") - || message.contains("Object not found") - || message.contains("object not found") - } - _ => false, - } + matches!(classify_heal_error(err), HealErrorClass::NoHealRequired) } pub(super) fn bucket_object_retry_delay(task_id: &str, retry_attempt: u32) -> Duration { @@ -1022,30 +991,6 @@ impl HealTask { true } - async fn skip_scanner_synthetic_object_dir_missing(&self, bucket: &str, object: &str, err: &Error) -> bool { - if self.source != HealRequestSource::Scanner || !is_missing_object_dir_heal_result(object, err) { - return false; - } - - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_RESULT, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - source = self.source.as_str(), - result = "synthetic_object_dir_missing", - error = %err, - "Heal recreate skipped scanner synthetic object-dir candidate after object-level not-found" - ); - let mut progress = self.progress.write().await; - progress.set_current_object(Some(format!("skipped: {bucket}/{object}"))); - progress.update_stage(4, 4); - true - } - #[tracing::instrument(skip(self), fields(task_id = %self.id, heal_type = ?self.heal_type))] #[hotpath::measure] pub async fn execute(&self) -> Result<()> { diff --git a/crates/heal/src/heal/task/heal_bucket.rs b/crates/heal/src/heal/task/heal_bucket.rs index 55a592627..bbe7da030 100644 --- a/crates/heal/src/heal/task/heal_bucket.rs +++ b/crates/heal/src/heal/task/heal_bucket.rs @@ -132,6 +132,21 @@ fn unavailable_recreate_error(result: &HealResultItem, opts: &HealOpts) -> Optio } impl HealTask { + async fn recover_bucket_delete_transactions(&self, bucket: &str, opts: &HealOpts) -> Result { + if let Some(result) = self + .await_with_control( + self.storage + .recover_delete_transactions(bucket, None, self.bucket_incarnation_id, opts), + ) + .await? + { + let unresolved = result.delete_recovery.is_some_and(|status| status.has_unresolved()); + self.record_result_item(result).await; + return Ok(unresolved); + } + Ok(false) + } + pub(super) async fn heal_bucket(&self, bucket: &str) -> Result<()> { self.pace_mainline().await?; if self.source == HealRequestSource::Admin && matches!(self.heal_type, HealType::Bucket { .. }) { @@ -169,8 +184,10 @@ impl HealTask { "Heal bucket stage entered" ); self.check_control_flags().await?; - let bucket_exists = self.await_with_control(self.storage.get_bucket_info(bucket)).await?.is_some(); - if !bucket_exists { + // Bound administrator repair validates existence and generation under the + // lifecycle fence. Ordinary read quorum can reject recoverable directory loss. + let bound_admin_repair = self.source == HealRequestSource::Admin && self.bucket_incarnation_id.is_some(); + if !bound_admin_repair && self.await_with_control(self.storage.get_bucket_info(bucket)).await?.is_none() { warn!( target: "rustfs::heal::task", event = EVENT_HEAL_BUCKET_RESULT, @@ -224,18 +241,11 @@ impl HealTask { remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted, ..heal_opts }; - if let Some(result) = self - .await_with_control(self.storage.recover_delete_transactions( - bucket, - None, - self.bucket_incarnation_id, - &recovery_opts, - )) - .await? - { - self.record_result_item(result).await; + // Removal must settle survivor journals before deleting an empty + // replica's publication directory; repair must establish it first. + if heal_opts.remove && self.recover_bucket_delete_transactions(bucket, &recovery_opts).await? { + return Err(Error::transient_skip("Delete recovery must resolve before bucket removal")); } - let heal_result = match self.bucket_incarnation_id { Some(expected) => { self.await_with_control(self.storage.heal_bucket_at_incarnation(bucket, expected, &heal_opts)) @@ -261,9 +271,15 @@ impl HealTask { ); self.record_result_item(result).await; + let unresolved_delete_recovery = + !heal_opts.remove && self.recover_bucket_delete_transactions(bucket, &recovery_opts).await?; + if self.options.recursive { self.heal_bucket_objects(bucket, "").await?; } + if unresolved_delete_recovery { + return Err(Error::transient_skip("Delete recovery remains unresolved after ordinary bucket heal")); + } if !self.options.recursive { let mut progress = self.progress.write().await; @@ -308,17 +324,76 @@ impl HealTask { "Heal cluster started" ); - let bucket_infos = self.await_with_control(self.storage.list_buckets()).await?; + let mut cluster_scope = if self.source == HealRequestSource::Admin && !self.options.dry_run { + let saved = if let Some(recovery) = &self.admin_recovery { + recovery.cluster_scope(self).await? + } else { + None + }; + let scope = if let Some(scope) = saved { + scope + } else { + let mut buckets = std::collections::BTreeMap::new(); + for bucket in self + .await_with_control(self.storage.list_buckets_for_admin_heal(None)) + .await? + { + let incarnation = self + .await_with_control(self.storage.admit_bucket_incarnation(&bucket.name)) + .await?; + buckets.insert(bucket.name, incarnation); + } + let scope = super::super::manager::root_recovery::AdminClusterScope { + buckets, + next_bucket_index: 0, + outcome: self.get_outcome().await, + progress: self.progress.read().await.clone(), + }; + if let Some(recovery) = &self.admin_recovery { + recovery.checkpoint_cluster_scope(self, &scope).await?; + } + scope + }; + // Validate the frozen generations on every replay, including completed buckets. + for (bucket, incarnation) in &scope.buckets { + self.await_with_control(self.storage.validate_bucket_incarnation(bucket, Some(*incarnation))) + .await?; + } + self.restore_outcome(scope.outcome.clone()).await; + *self.progress.write().await = scope.progress.clone(); + Some(scope) + } else { + None + }; + let buckets = if let Some(scope) = &cluster_scope { + scope + .buckets + .iter() + .map(|(name, incarnation)| (name.clone(), Some(*incarnation))) + .collect::>() + } else { + let listed = if self.source == HealRequestSource::Admin { + self.await_with_control(self.storage.list_buckets_for_admin_heal(None)) + .await? + } else { + self.await_with_control(self.storage.list_buckets()).await? + }; + listed.into_iter().map(|bucket| (bucket.name, None)).collect() + }; let mut failed = 0_u64; let mut retryable = 0_u64; let mut permanent = 0_u64; let mut first_object = None; let mut first_error = None; - for bucket_info in bucket_infos { + let start = cluster_scope.as_ref().map_or(0, |scope| scope.next_bucket_index); + let mut checkpointed = start; + for (index, (bucket, incarnation)) in buckets.into_iter().enumerate().skip(start) { self.check_control_flags().await?; + let mut bucket_task = self.clone(); + bucket_task.bucket_incarnation_id = incarnation; let mut retry_attempt = 0_u32; loop { - match self.heal_bucket(&bucket_info.name).await { + match bucket_task.heal_bucket(&bucket).await { Ok(()) => break, Err(Error::TaskCancelled) => return Err(Error::TaskCancelled), Err(Error::TaskTimeout) => return Err(Error::TaskTimeout), @@ -347,15 +422,37 @@ impl HealTask { } else { permanent = permanent.saturating_add(1); } - first_object.get_or_insert(bucket_info.name.clone()); + first_object.get_or_insert(bucket.clone()); first_error.get_or_insert_with(|| err.to_string()); break; } } } + // Retain only the successful contiguous prefix. A later retry replays + // any work after a failure without inheriting its transient counters. + if failed == 0 + && let Some(scope) = &mut cluster_scope + { + scope.next_bucket_index = index.saturating_add(1); + scope.outcome = self.get_outcome().await; + scope.progress = self.progress.read().await.clone(); + if (scope.next_bucket_index.saturating_sub(checkpointed) >= scope.checkpoint_interval() + || scope.next_bucket_index == scope.buckets.len()) + && let Some(recovery) = &self.admin_recovery + { + recovery.checkpoint_cluster_scope(self, scope).await?; + checkpointed = scope.next_bucket_index; + } + } } if failed > 0 { + if let Some(scope) = &cluster_scope + && scope.next_bucket_index > checkpointed + && let Some(recovery) = &self.admin_recovery + { + recovery.checkpoint_cluster_scope(self, scope).await?; + } let failure = BatchHealFailure { scope: "cluster".to_string(), failed, @@ -470,6 +567,7 @@ impl HealTask { set: self.options.set_index, ..Default::default() }; + let mut unresolved_delete_recovery = false; if let Some(result) = self .await_with_control(self.storage.recover_delete_prefix_transactions( bucket, @@ -479,10 +577,15 @@ impl HealTask { )) .await? { + unresolved_delete_recovery = result.delete_recovery.is_some_and(|status| status.has_unresolved()); self.record_result_item(result).await; } - self.heal_bucket_objects(bucket, prefix).await + self.heal_bucket_objects(bucket, prefix).await?; + if unresolved_delete_recovery { + return Err(Error::transient_skip("Delete recovery remains unresolved after ordinary prefix heal")); + } + Ok(()) } #[hotpath::measure] diff --git a/crates/heal/src/heal/task/heal_erasure_set.rs b/crates/heal/src/heal/task/heal_erasure_set.rs index 376fc5314..ba32340e8 100644 --- a/crates/heal/src/heal/task/heal_erasure_set.rs +++ b/crates/heal/src/heal/task/heal_erasure_set.rs @@ -460,21 +460,7 @@ impl HealTask { set: self.options.set_index, }; - let recovery_opts = HealOpts { - remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted, - ..bucket_heal_opts - }; - // Include metadata-less transactions for both startup and admin heals. - for bucket in &buckets { - self.check_control_flags().await?; - if let Some(result) = self - .await_with_control(self.storage.recover_delete_transactions(bucket, None, None, &recovery_opts)) - .await? - { - self.record_result_item(result).await; - } - } - + let mut unresolved_delete_recovery = false; for bucket in buckets.iter().filter(|_| self.source != HealRequestSource::Admin) { // Check control flags before starting each bucket heal self.check_control_flags().await?; @@ -505,6 +491,17 @@ impl HealTask { return Err(err); } } + // Adopt delete proofs only after their publication bucket exists. + if let Some(result) = self + .await_with_control( + self.storage + .recover_delete_transactions(bucket, None, None, &bucket_heal_opts), + ) + .await? + { + unresolved_delete_recovery |= result.delete_recovery.is_some_and(|status| status.has_unresolved()); + self.record_result_item(result).await; + } } // Create erasure set healer with resume support @@ -526,6 +523,7 @@ impl HealTask { heal_opts, self.source, ) + .with_unresolved_delete_recovery(unresolved_delete_recovery) .with_replacement_targets(replacement_targets, is_auto_replacement.then(|| self.id.clone())) .with_pool_metadata_targets(if self.options.recreate_missing && !self.options.dry_run { self.heal_endpoints.clone() diff --git a/crates/heal/src/heal/task/heal_object.rs b/crates/heal/src/heal/task/heal_object.rs index e20d4f315..c20b533f3 100644 --- a/crates/heal/src/heal/task/heal_object.rs +++ b/crates/heal/src/heal/task/heal_object.rs @@ -11,7 +11,7 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. -/// object-level heal: metadata dir canonicalization and missing-object recreation +/// object-level heal: metadata dir canonicalization and the storage-decided absent verdict use super::*; impl HealTask { @@ -179,36 +179,22 @@ impl HealTask { recreate_missing = self.options.recreate_missing, "Heal target object is missing" }); - if self.options.recreate_missing { - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_STAGE, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - stage = "recreate_missing", - "Heal object recreate requested" - ); - return self.recreate_missing_object(bucket, object, version_id).await; - } else if self.source == HealRequestSource::Scanner { - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_STAGE, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - stage = "scanner_missing_probe", - "Heal scanner missing object will be checked by storage layer" - ); - } else { - return Err(Error::TaskExecutionFailed { - message: format!("Object not found: {bucket}/{object}"), - }); - } + // The unlocked existence check is only a hint. The locked storage + // heal below reads every disk; its object-level not-found is the + // absence verdict for every source and every `recreate_missing` + // value. A recreate from parity happens only on the durable MRF + // path above, which carries the write anchor. + debug!( + target: "rustfs::heal::task", + event = EVENT_HEAL_OBJECT_STAGE, + component = LOG_COMPONENT_HEAL, + subsystem = LOG_SUBSYSTEM_OBJECT, + task_id = %self.id, + bucket, + object, + stage = "missing_probe", + "Heal missing object will be checked by storage layer" + ); } { @@ -267,23 +253,8 @@ impl HealTask { return Ok(()); } - if Self::is_object_not_found_heal_error(&e) { - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_RESULT, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - result = "treated_as_deleted", - "Heal missing object treated as deleted" - ); - { - let mut progress = self.progress.write().await; - progress.update_stage(3, 3); - } - return Ok(()); + if e.is_object_absent() { + return self.complete_absent_object(bucket, object, &e).await; } error!( @@ -373,23 +344,8 @@ impl HealTask { return Ok(()); } - if Self::is_object_not_found_heal_error(&e) { - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_RESULT, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - result = "treated_as_deleted", - "Heal missing object treated as deleted" - ); - { - let mut progress = self.progress.write().await; - progress.update_stage(3, 3); - } - return Ok(()); + if e.is_object_absent() { + return self.complete_absent_object(bucket, object, &e).await; } error!( @@ -455,133 +411,11 @@ impl HealTask { } } - /// Recreate missing object (for EC decode scenarios) - async fn recreate_missing_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> { - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_STAGE, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - version_id = ?version_id, - stage = "recreate_missing", - "Heal object recreate started" - ); - - // Use ecstore's heal_object with recreate option - let heal_opts = HealOpts { - recursive: false, - dry_run: self.options.dry_run, - remove: false, - recreate: true, - scan_mode: HealScanMode::Deep, - update_parity: true, - no_lock: self.options.no_lock, - read_repair: false, - pool: None, - set: None, - }; - - match self - .await_with_control(self.storage.heal_object(bucket, object, version_id, &heal_opts)) - .await - { - Ok((result, error)) => { - if let Some(e) = error { - if self.skip_missing_usage_observation(bucket, object, version_id, &e).await - || self.skip_scanner_synthetic_object_dir_missing(bucket, object, &e).await - { - return Ok(()); - } - - error!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_RESULT, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - result = "recreate_failed", - error = %e, - "Heal object recovery failed" - ); - return Err(Error::TaskExecutionFailed { - message: format!("Failed to recreate missing object {bucket}/{object}: {e}"), - }); - } - - let object_size = result.object_size as u64; - debug!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_RESULT, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - object_size, - result = "recreated", - "Heal object recreated" - ); - - { - let mut progress = self.progress.write().await; - progress.update_object_progress(1, 1, 0, 0, object_size); - } - self.record_result_item(result).await; - Ok(()) - } - Err(Error::TaskCancelled) => Err(Error::TaskCancelled), - Err(Error::TaskTimeout) => Err(Error::TaskTimeout), - Err(e) => { - if self.skip_missing_usage_observation(bucket, object, version_id, &e).await - || self.skip_scanner_synthetic_object_dir_missing(bucket, object, &e).await - { - return Ok(()); - } - - error!( - target: "rustfs::heal::task", - event = EVENT_HEAL_OBJECT_RESULT, - component = LOG_COMPONENT_HEAL, - subsystem = LOG_SUBSYSTEM_OBJECT, - task_id = %self.id, - bucket, - object, - result = "recreate_failed", - error = %e, - "Heal object recovery failed" - ); - Err(Error::TaskExecutionFailed { - message: format!("Failed to recreate missing object {bucket}/{object}: {e}"), - }) - } - } - } - - async fn skip_missing_usage_observation(&self, bucket: &str, object: &str, version_id: Option<&str>, err: &Error) -> bool { - // Usage publication may delete an obsolete observation after a degraded - // GET has queued read repair. Its absence is not lost user data. Keep - // durable MRF completion on its separate, proof-bearing storage path. - if self.source != HealRequestSource::ReadRepair - || version_id.is_some() - || bucket != RUSTFS_META_BUCKET - || object - .strip_prefix(BUCKET_META_PREFIX) - .and_then(|suffix| suffix.strip_prefix('/')) - != Some(rustfs_data_usage::DATA_USAGE_OBSERVED_OBJECT_NAME) - || !matches!( - err, - Error::Disk(DiskError::FileNotFound) - | Error::Storage(EcstoreError::FileNotFound | EcstoreError::ObjectNotFound(_, _)) - ) - { - return false; - } - + /// The locked storage read found no replica: there is nothing to repair + /// and the heal completes for every source. No absence proof is recorded; + /// only storage's incarnation-fenced receipt can carry one. + async fn complete_absent_object(&self, bucket: &str, object: &str, err: &Error) -> Result<()> { + self.mark_object_absent(); debug!( target: "rustfs::heal::task", event = EVENT_HEAL_OBJECT_RESULT, @@ -591,14 +425,14 @@ impl HealTask { bucket, object, source = self.source.as_str(), - result = "usage_observation_missing", - "Heal skipped an absent usage observation" + result = "treated_as_deleted", + error = %err, + "Heal missing object treated as deleted" ); let mut progress = self.progress.write().await; - progress.set_current_object(Some(format!("skipped: {bucket}/{object}"))); progress.update_object_progress(1, 0, 0, 1, 0); - progress.update_stage(4, 4); - true + progress.update_stage(3, 3); + Ok(()) } /// Durable MRF partial writes may complete only with an incarnation-fenced storage proof. diff --git a/crates/heal/src/heal/task/tests.rs b/crates/heal/src/heal/task/tests.rs index 25bb0a143..327a8ef45 100644 --- a/crates/heal/src/heal/task/tests.rs +++ b/crates/heal/src/heal/task/tests.rs @@ -14,8 +14,10 @@ use super::super::{DiskOption, DiskStore, Endpoint, new_disk}; use super::*; +use crate::heal::EcstoreError; use crate::heal::storage::HealStorageObjectResult; +mod absent_verdict; mod concurrent_delete; mod deferred_retry; mod usage_observation; @@ -24,6 +26,59 @@ mod canonical_outcome { use super::*; use crate::heal::outcome::{HealExecutionOutcome, HealTraversalCoverage}; + #[tokio::test] + async fn admin_bucket_discovery_incomplete_cannot_finish_an_empty_cluster() { + for dry_run in [false, true] { + let storage = Arc::new(MockStorage { + listed_buckets: Mutex::new(Some(Vec::new())), + bucket_discovery_incomplete: true, + ..Default::default() + }); + let mut request = HealRequest::new( + HealType::Cluster, + HealOptions { + dry_run, + recursive: true, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage.clone()); + assert!(matches!(task.execute().await, Err(Error::BucketDiscoveryIncomplete))); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Aborted(HealAbortReason::Untraversable)); + assert_eq!(outcome.coverage, HealTraversalCoverage::Partial); + assert_eq!(outcome.counters.processed, 0); + assert_eq!(outcome.counters.failed, 0); + assert!(storage.bucket_heal_calls.lock().expect("bucket calls").is_empty()); + assert!(storage.heal_object_calls.lock().expect("object calls").is_empty()); + let wire = serde_json::to_value(&outcome).expect("outcome serializes"); + assert_eq!( + crate::heal::outcome::legacy_wire_status("finished", &wire, false) + .expect("legacy status") + .0, + "stopped" + ); + } + } + + #[tokio::test] + async fn admin_bucket_discovery_complete_empty_cluster_finishes() { + let storage = Arc::new(MockStorage { + listed_buckets: Mutex::new(Some(Vec::new())), + ..Default::default() + }); + let mut request = HealRequest::new(HealType::Cluster, HealOptions::default(), HealPriority::Normal); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage); + task.execute().await.expect("complete empty namespace is valid"); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed); + assert_eq!(outcome.coverage, HealTraversalCoverage::Complete); + assert_eq!(outcome.counters.processed, 0); + } + fn bucket_task(storage: Arc) -> HealTask { HealTask::from_request( HealRequest::new( @@ -1560,9 +1615,12 @@ struct MockStorage { truncate_without_token: Mutex, include_object_dir_candidate: Mutex, listed_buckets: Mutex>>, + bucket_discovery_incomplete: bool, bucket_heal_errors: Mutex>>, typed_bucket_heal_errors: Mutex>>, bucket_heal_calls: Mutex>, + delete_recovery_results: Mutex>>>, + delete_recovery_calls: Mutex>, block_heal_object: Mutex, resume_disk: Mutex>, replacement_resume_disk: Mutex>, @@ -1570,6 +1628,7 @@ struct MockStorage { usage_baseline_error: Mutex, erasure_set_scopes: Mutex>, disk_walk_calls: Mutex>, + disk_walk_items: Mutex>>, } #[test] @@ -2036,6 +2095,27 @@ fn test_missing_object_dir_heal_result_matches_only_object_level_not_found() { assert!(!is_missing_object_dir_heal_result("x.rnd", &Error::Disk(DiskError::FileNotFound))); } +#[test] +fn missing_object_dir_heal_result_not_found_accepts_typed_object_and_version_absence() { + assert!(is_missing_object_dir_heal_result( + "x.rnd/", + &Error::Storage(EcstoreError::ObjectNotFound("bucket".to_string(), "x.rnd/".to_string())) + )); + assert!(is_missing_object_dir_heal_result( + "x.rnd/", + &Error::Storage(EcstoreError::VersionNotFound( + "bucket".to_string(), + "x.rnd/".to_string(), + "version".to_string() + )) + )); + assert!(!is_missing_object_dir_heal_result("x.rnd/", &Error::Storage(EcstoreError::DiskNotFound))); + assert!(!is_missing_object_dir_heal_result( + "x.rnd/", + &Error::Storage(EcstoreError::VolumeNotFound) + )); +} + #[async_trait::async_trait] impl HealStorageAPI for MockStorage { async fn get_object_meta(&self, _bucket: &str, _object: &str) -> Result> { @@ -2060,6 +2140,16 @@ impl HealStorageAPI for MockStorage { Ok(*self.usage_baseline.lock().unwrap()) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result> { + if self.bucket_discovery_incomplete { + return Err(Error::BucketDiscoveryIncomplete); + } + self.list_buckets().await + } + async fn list_buckets(&self) -> Result> { let buckets = self .listed_buckets @@ -2348,6 +2438,34 @@ impl HealStorageAPI for MockStorage { Ok(HealResultItem::default()) } + async fn recover_delete_transactions( + &self, + bucket: &str, + _object: Option<&str>, + _expected: Option, + _opts: &HealOpts, + ) -> Result> { + self.delete_recovery_calls + .lock() + .expect("recovery calls") + .push(bucket.to_owned()); + self.delete_recovery_results + .lock() + .expect("recovery results") + .remove(bucket) + .unwrap_or(Ok(None)) + } + + async fn recover_delete_prefix_transactions( + &self, + bucket: &str, + _prefix: &str, + expected: Option, + opts: &HealOpts, + ) -> Result> { + self.recover_delete_transactions(bucket, None, expected, opts).await + } + async fn heal_format(&self, _dry_run: bool) -> Result<(HealResultItem, Option)> { *self.global_format_calls.lock().unwrap() += 1; if let Some(error) = self.format_error.lock().unwrap().take() { @@ -2513,7 +2631,15 @@ impl HealStorageAPI for MockStorage { if continuation_token.is_some() { return Ok((Vec::new(), None, false)); } - Ok((vec![heal_item(&format!("{set_disk_id}-object"))], None, false)) + Ok(( + self.disk_walk_items + .lock() + .expect("disk-walk items") + .clone() + .unwrap_or_else(|| vec![heal_item(&format!("{set_disk_id}-object"))]), + None, + false, + )) } async fn get_disk_for_resume(&self, _set_disk_id: &str) -> Result { @@ -2701,12 +2827,15 @@ async fn replacement_failure_persistence_retains_both_errors() { } #[tokio::test] -async fn replacement_format_failures_consume_the_durable_budget_across_new_executors() { +async fn replacement_format_failures_keep_the_durable_budget_across_new_executors() { + // backlog#2902: the rebuild budget counts attempts after the target is + // formatted. A failed format keeps its durable reason and leaves the + // budget intact, whether one executor retries or a new one is admitted. let directory = TempDir::new().expect("survivor root"); let disk = make_resume_disk(&directory).await; let mut request = directory_backed_replacement_request(); request.heal_endpoints = vec!["replacement-a".to_string()]; - for attempt in 1..=3 { + for _attempt in 1..=3 { let storage = Arc::new(MockStorage { replacement_target_identities_ready: Mutex::new(true), resume_disk: Mutex::new(Some(disk.clone())), @@ -2726,7 +2855,16 @@ async fn replacement_format_failures_consume_the_durable_budget_across_new_execu .expect("durable failure") .get_state() .await; - assert_eq!(state.retry_count, attempt, "a new executor must not reset or double-charge the attempt"); + assert_eq!(state.retry_count, 0, "a pre-format failure must not spend a rebuild attempt"); + assert!(state.can_retry()); + assert!( + state + .error_message + .as_deref() + .is_some_and(|message| message.contains("format failure")), + "the durable reason is retained for status: {:?}", + state.error_message + ); assert_eq!(state.replacement_phase, ReplacementPhase::Intent); assert_eq!(storage.replacement_format_calls.lock().unwrap().len(), 1); assert!(storage.bucket_heal_calls.lock().unwrap().is_empty()); @@ -3884,12 +4022,13 @@ async fn test_heal_admin_missing_object_dir_does_not_canonicalize_plain_object() request.source = HealRequestSource::Admin; let task = HealTask::from_request(request, storage.clone()); - let err = task - .execute() + task.execute() .await - .expect_err("admin object-dir request must not be canonicalized"); + .expect("an absent admin object-dir target completes under the general not-found rule"); - assert!(matches!(err, Error::TaskExecutionFailed { .. })); + assert!(matches!(task.get_status().await, HealTaskStatus::Completed)); + // The invariant under test: an administrator request names an exact key + // and is never rewritten to the plain object that happens to exist. assert_eq!(storage.heal_object_calls.lock().unwrap().as_slice(), ["x.rnd/".to_string()]); } @@ -4057,38 +4196,6 @@ async fn test_heal_recreate_scanner_synthetic_object_dir_skips_err_not_found() { assert!(matches!(task.get_status().await, HealTaskStatus::Completed)); } -#[tokio::test] -async fn test_heal_recreate_scanner_non_dir_not_found_fails() { - let storage = Arc::new(MockStorage { - object_exists: Mutex::new(Some(false)), - heal_object_outcome: Mutex::new(Some(MockHealObjectOutcome::ErrOther("File not found"))), - ..Default::default() - }); - let mut request = HealRequest::new( - HealType::Object { - bucket: "bucket-a".to_string(), - object: "x.rnd".to_string(), - version_id: None, - }, - HealOptions { - recreate_missing: true, - timeout: None, - ..Default::default() - }, - HealPriority::Normal, - ); - request.source = HealRequestSource::Scanner; - let task = HealTask::from_request(request, storage); - - let err = task - .execute() - .await - .expect_err("scanner non-dir missing object should still fail recreate"); - - assert!(matches!(err, Error::TaskExecutionFailed { .. })); - assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. })); -} - #[tokio::test] async fn mrf_recreate_missing_object_records_exact_absence_receipt_with_scope() { let incarnation = Uuid::new_v4(); @@ -4245,38 +4352,6 @@ async fn test_heal_recreate_scanner_synthetic_object_dir_disk_not_found_fails() assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. })); } -#[tokio::test] -async fn test_heal_recreate_admin_synthetic_object_dir_not_found_fails() { - let storage = Arc::new(MockStorage { - object_exists: Mutex::new(Some(false)), - heal_object_outcome: Mutex::new(Some(MockHealObjectOutcome::ErrOther("File not found"))), - ..Default::default() - }); - let mut request = HealRequest::new( - HealType::Object { - bucket: "bucket-a".to_string(), - object: "x.rnd/".to_string(), - version_id: None, - }, - HealOptions { - recreate_missing: true, - timeout: None, - ..Default::default() - }, - HealPriority::Normal, - ); - request.source = HealRequestSource::Admin; - let task = HealTask::from_request(request, storage); - - let err = task - .execute() - .await - .expect_err("admin object-dir not-found recreate should not be skipped"); - - assert!(matches!(err, Error::TaskExecutionFailed { .. })); - assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. })); -} - #[tokio::test] async fn test_heal_recreate_existing_trailing_slash_object_records_normal_result() { let storage = Arc::new(MockStorage { @@ -4457,6 +4532,336 @@ async fn erasure_set_bucket_prepass_failure_stops_before_object_heal() { assert!(storage.object_heal_opts.lock().unwrap().is_empty()); } +#[tokio::test] +async fn erasure_set_unresolved_delete_recovery_preserves_other_buckets_and_ordinary_heal() { + use rustfs_madmin::heal_commands::DeleteRecoveryStatus; + + for status in [ + DeleteRecoveryStatus { + pending: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + blocked: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }, + ] { + let temp = TempDir::new().expect("temporary directory should be created"); + let disk = make_resume_disk(&temp).await; + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + bucket: "bucket-a".to_owned(), + delete_recovery: Some(status), + ..Default::default() + })), + )])), + resume_disk: Mutex::new(Some(disk)), + list_each_bucket: true, + ..Default::default() + }); + let buckets = vec!["bucket-a".to_owned(), "bucket-b".to_owned()]; + let request = HealRequest::new( + HealType::ErasureSet { + buckets: buckets.clone(), + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + let task = HealTask::from_request(request, storage.clone()); + + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("unresolved recovery must not be reported as complete"); + + assert!(matches!(error, Error::TransientSkip { .. }), "recovery must remain retryable: {error}"); + assert_eq!(*storage.delete_recovery_calls.lock().expect("recovery calls"), buckets); + assert_eq!(*storage.bucket_heal_calls.lock().expect("bucket calls"), buckets); + assert_eq!( + storage.object_heal_opts.lock().expect("object calls").len(), + 4, + "ordinary object heal must run for both buckets" + ); + assert!( + task.get_result_items() + .await + .iter() + .any(|item| item.bucket == "bucket-a" && item.delete_recovery == Some(status)), + "the result retains the exact unresolved recovery category" + ); + } +} + +#[tokio::test] +async fn erasure_set_delete_recovery_fatal_errors_stop_before_other_buckets() { + for error in [ + Error::TaskCancelled, + Error::TaskTimeout, + Error::Storage(EcstoreError::PreconditionFailed), + Error::Storage(EcstoreError::SlowDown), + Error::Storage(EcstoreError::InvalidPath), + ] { + let expected = error.to_string(); + let temp = TempDir::new().expect("temporary directory should be created"); + let disk = make_resume_disk(&temp).await; + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([("bucket-a".to_owned(), Err(error))])), + resume_disk: Mutex::new(Some(disk)), + ..Default::default() + }); + let buckets = vec!["bucket-a".to_owned(), "bucket-b".to_owned()]; + let request = HealRequest::new( + HealType::ErasureSet { + buckets: buckets.clone(), + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + let task = HealTask::from_request(request, storage.clone()); + + let actual = task + .heal_erasure_set(buckets, "pool_0_set_0".to_owned()) + .await + .expect_err("fence and control errors must terminate the prepass"); + + assert_eq!(actual.to_string(), expected); + assert_eq!(*storage.delete_recovery_calls.lock().expect("recovery calls"), ["bucket-a"]); + assert_eq!(*storage.bucket_heal_calls.lock().expect("bucket calls"), ["bucket-a"]); + assert!(storage.object_heal_opts.lock().expect("object calls").is_empty()); + } +} + +#[tokio::test] +async fn bucket_and_prefix_delete_recovery_stays_unresolved_after_ordinary_heal() { + for prefix in [None, Some("logs/")] { + let status = rustfs_madmin::heal_commands::DeleteRecoveryStatus { + pending: 1, + ..Default::default() + }; + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + bucket: "bucket-a".to_owned(), + delete_recovery: Some(status), + ..Default::default() + })), + )])), + ..Default::default() + }); + let heal_type = prefix.map_or_else( + || HealType::Bucket { + bucket: "bucket-a".to_owned(), + }, + |prefix| HealType::Prefix { + bucket: "bucket-a".to_owned(), + prefix: prefix.to_owned(), + }, + ); + let task = HealTask::from_request( + HealRequest::new( + heal_type, + HealOptions { + recursive: true, + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ), + storage.clone(), + ); + + let result = match prefix { + Some(prefix) => task.heal_prefix("bucket-a", prefix).await, + None => task.heal_bucket("bucket-a").await, + }; + assert!( + matches!(result, Err(Error::TransientSkip { .. })), + "unfinished recovery must remain retryable: {result:?}" + ); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 2); + assert!( + task.get_result_items() + .await + .iter() + .any(|item| item.delete_recovery == Some(status)) + ); + } +} + +#[tokio::test] +async fn unresolved_delete_recovery_prevents_bucket_removal() { + let storage = Arc::new(MockStorage { + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + delete_recovery: Some(rustfs_madmin::heal_commands::DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }), + ..Default::default() + })), + )])), + ..Default::default() + }); + let task = HealTask::from_request( + HealRequest::new( + HealType::Bucket { + bucket: "bucket-a".to_owned(), + }, + HealOptions { + remove_corrupted: true, + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ), + storage.clone(), + ); + + assert!(matches!(task.heal_bucket("bucket-a").await, Err(Error::TransientSkip { .. }))); + assert!( + storage.bucket_heal_calls.lock().expect("bucket calls").is_empty(), + "publication directories must remain until recovery resolves" + ); +} + +#[tokio::test] +async fn admin_erasure_set_delete_recovery_rescans_only_after_settlement() { + let temp = TempDir::new().expect("temporary directory should be created"); + let disk = make_resume_disk(&temp).await; + let incarnation = Uuid::new_v4(); + let storage = Arc::new(MockStorage { + bucket_incarnation_id: Mutex::new(Some(incarnation)), + delete_recovery_results: Mutex::new(HashMap::from([( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + delete_recovery: Some(rustfs_madmin::heal_commands::DeleteRecoveryStatus { + blocked: 1, + ..Default::default() + }), + ..Default::default() + })), + )])), + resume_disk: Mutex::new(Some(disk.clone())), + disk_walk_items: Mutex::new(Some(vec![heal_item("object-a"), heal_item("object-b")])), + ..Default::default() + }); + let buckets = vec!["bucket-a".to_owned(), "bucket-b".to_owned()]; + let mut request = HealRequest::new( + HealType::ErasureSet { + buckets: buckets.clone(), + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage.clone()); + + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("administrator recovery cannot complete a set with unresolved recovery"); + assert!(matches!(error, Error::TransientSkip { .. }), "typed unresolved recovery: {error}"); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 4); + let checkpoint = CheckpointManager::load_from_disk(disk.clone(), &task.id) + .await + .expect("retained administrator checkpoint") + .get_checkpoint() + .await; + assert_eq!( + checkpoint.current_bucket_index, 2, + "ordinary work advances independently from hidden recovery" + ); + assert!(!checkpoint.admin.expect("administrator outcome").completed); + assert_eq!(*storage.delete_recovery_calls.lock().expect("recovery calls"), buckets); + storage.delete_recovery_results.lock().expect("recovery results").insert( + "bucket-a".to_owned(), + Ok(Some(HealResultItem { + delete_recovery: Some(rustfs_madmin::heal_commands::DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }), + ..Default::default() + })), + ); + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("completed ordinary buckets must still retry their hidden recovery"); + assert!(matches!(error, Error::TransientSkip { .. })); + assert_eq!( + storage.object_heal_opts.lock().expect("object calls").len(), + 4, + "retry must not replay payload healing" + ); + + *storage.bucket_incarnation_id.lock().expect("bucket identity") = Some(Uuid::new_v4()); + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("recovery-only retries must retain the admitted bucket generation"); + assert!(matches!(error, Error::StaleBucketIncarnation { .. })); + *storage.bucket_incarnation_id.lock().expect("bucket identity") = Some(incarnation); + + // A background worker may restore and collect the hidden transaction + // before this retry, so the recovery result can now be None. + *storage.disk_walk_items.lock().expect("disk-walk items") = + Some(vec![heal_item("object-a"), heal_item("object-b"), heal_item("restored")]); + let error = task + .heal_erasure_set(buckets.clone(), "pool_0_set_0".to_owned()) + .await + .expect_err("settlement must invalidate overlapping ordinary coverage"); + assert!(matches!(error, Error::TransientSkip { .. })); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 4); + task.heal_erasure_set(buckets, "pool_0_set_0".to_owned()) + .await + .expect("a fresh coverage pass can complete"); + assert_eq!(storage.object_heal_opts.lock().expect("object calls").len(), 10); + assert_eq!( + storage + .healed_objects + .lock() + .expect("healed objects") + .iter() + .filter(|object| object.as_str() == "restored") + .count(), + 2 + ); + assert_eq!( + *storage.delete_recovery_calls.lock().expect("recovery calls"), + [ + "bucket-a", "bucket-b", "bucket-a", "bucket-b", "bucket-a", "bucket-b", "bucket-a", "bucket-b" + ] + ); + let checkpoint = CheckpointManager::load_from_disk(disk, &task.id) + .await + .expect("completed administrator receipt") + .get_checkpoint() + .await; + let admin = checkpoint.admin.expect("administrator outcome"); + assert!(admin.completed); + assert_eq!(admin.outcome.counters.processed, 6, "the fresh coverage pass owns its counters"); +} + #[tokio::test] async fn erasure_set_heal_applies_usage_baseline_to_progress() { let temp = TempDir::new().expect("temporary directory should be created"); diff --git a/crates/heal/src/heal/task/tests/absent_verdict.rs b/crates/heal/src/heal/task/tests/absent_verdict.rs new file mode 100644 index 000000000..703647b55 --- /dev/null +++ b/crates/heal/src/heal/task/tests/absent_verdict.rs @@ -0,0 +1,221 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! The absent verdict for a heal target is decided by the storage quorum read, +//! not by the request source or the `recreate_missing` option (backlog#2900; +//! rustfs/rustfs#7997, #8132, #8235, #5950). + +use super::*; + +const OBSERVATION: &str = "buckets/.usage.observed.json"; + +fn missing_storage(outcome: MockHealObjectOutcome) -> Arc { + Arc::new(MockStorage { + object_exists: Mutex::new(Some(false)), + heal_object_outcome: Mutex::new(Some(outcome)), + ..Default::default() + }) +} + +fn object_task( + bucket: &str, + object: &str, + version: Option<&str>, + source: HealRequestSource, + recreate_missing: bool, + storage: Arc, +) -> HealTask { + let mut request = HealRequest::new( + HealType::Object { + bucket: bucket.to_string(), + object: object.to_string(), + version_id: version.map(str::to_string), + }, + HealOptions { + recreate_missing, + timeout: None, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = source; + HealTask::from_request(request, storage) +} + +const NON_MRF_SOURCES: [HealRequestSource; 5] = [ + HealRequestSource::Admin, + HealRequestSource::Internal, + HealRequestSource::AutoHeal, + HealRequestSource::ReadRepair, + HealRequestSource::Scanner, +]; + +#[tokio::test] +async fn absent_object_completes_for_every_source_regardless_of_recreate_missing() { + for source in NON_MRF_SOURCES { + for recreate_missing in [false, true] { + for outer in [false, true] { + let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer }); + let task = object_task("bucket-a", "x.rnd", None, source, recreate_missing, storage.clone()); + task.execute() + .await + .unwrap_or_else(|err| panic!("{source:?} recreate={recreate_missing} outer={outer}: {err}")); + assert_eq!( + task.get_status().await, + HealTaskStatus::Completed, + "{source:?} recreate={recreate_missing}" + ); + let outcome = task.get_outcome().await; + assert_eq!(outcome.counters.failed, 0, "{source:?} recreate={recreate_missing}"); + // One locked storage probe decides the verdict; no second recreate call follows. + assert_eq!( + storage.heal_object_calls.lock().unwrap().as_slice(), + ["x.rnd".to_string()], + "{source:?} recreate={recreate_missing}" + ); + let opts = storage.object_heal_opts.lock().unwrap(); + assert_eq!(opts.len(), 1); + assert_eq!(opts[0].recreate, recreate_missing, "the option is still parsed and forwarded"); + } + } + } +} + +#[tokio::test] +async fn absent_object_with_explicit_version_is_absent_for_read_repair() { + for outcome in [ + MockHealObjectOutcome::MissingObject { outer: false }, + MockHealObjectOutcome::MissingVersion, + ] { + let storage = missing_storage(outcome); + let task = object_task( + "bucket-a", + "x.rnd", + Some("version-a"), + HealRequestSource::ReadRepair, + true, + storage.clone(), + ); + task.execute() + .await + .expect("a missing version is absent, not recreate_failed"); + assert_eq!(task.get_status().await, HealTaskStatus::Completed); + assert_eq!(task.get_outcome().await.counters.failed, 0); + } +} + +#[tokio::test] +async fn absent_usage_observation_follows_the_general_not_found_rule() { + // rustfs/rustfs#8132: previously only the exact (ReadRepair, meta bucket, + // unversioned) tuple was excused through a path whitelist. + for (bucket, object, source, version) in [ + (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, None), + (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, Some("version-a")), + (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Admin, None), + (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Scanner, None), + (RUSTFS_META_BUCKET, "buckets/.usage.json", HealRequestSource::ReadRepair, None), + ("user-bucket", OBSERVATION, HealRequestSource::AutoHeal, None), + ] { + let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer: false }); + let task = object_task(bucket, object, version, source, true, storage); + task.execute() + .await + .unwrap_or_else(|err| panic!("{bucket}/{object} {source:?}: {err}")); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed, "{bucket}/{object} {source:?}"); + assert_eq!(outcome.counters.failed, 0, "{bucket}/{object} {source:?}"); + } +} + +#[tokio::test] +async fn absent_object_dir_candidate_follows_the_general_not_found_rule() { + // rustfs/rustfs#3839 excused only scanner-sourced trailing-slash candidates. + for source in NON_MRF_SOURCES { + for outcome in [ + MockHealObjectOutcome::MissingObject { outer: false }, + MockHealObjectOutcome::OkWithOtherError("File not found"), + MockHealObjectOutcome::ErrOther("File not found"), + ] { + let storage = missing_storage(outcome); + let task = object_task("bucket-a", "x.rnd/", None, source, true, storage); + task.execute().await.unwrap_or_else(|err| panic!("{source:?}: {err}")); + assert_eq!(task.get_status().await, HealTaskStatus::Completed, "{source:?}"); + } + } +} + +#[tokio::test] +async fn absent_legacy_display_not_found_is_absent_for_read_repair() { + // An old peer can still render the typed error as its display text. + for message in ["File not found", "File version not found", "Object not found: bucket-a/x.rnd"] { + let storage = missing_storage(MockHealObjectOutcome::ErrOther(message)); + let task = object_task("bucket-a", "x.rnd", None, HealRequestSource::ReadRepair, true, storage); + task.execute().await.unwrap_or_else(|err| panic!("{message:?}: {err}")); + assert_eq!(task.get_status().await, HealTaskStatus::Completed, "{message:?}"); + } +} + +#[tokio::test] +async fn not_found_verdict_is_not_granted_to_disk_or_quorum_failures() { + // backlog#856 / #799 B7: an offline drive or a lost quorum is never absence. + for (outcome, recoverable) in [ + (MockHealObjectOutcome::FaultyStorageDisk(false), true), + (MockHealObjectOutcome::FaultyStorageDisk(true), true), + (MockHealObjectOutcome::OkWithReadQuorum, true), + (MockHealObjectOutcome::RetryableReadQuorum, true), + (MockHealObjectOutcome::RetryableSlowDown, true), + (MockHealObjectOutcome::ErrOther("Disk not found"), false), + (MockHealObjectOutcome::PermissionDenied, false), + ] { + for source in [ + HealRequestSource::ReadRepair, + HealRequestSource::Admin, + HealRequestSource::Scanner, + ] { + let storage = missing_storage(outcome.clone()); + let task = object_task("bucket-a", "x.rnd", None, source, true, storage); + let err = task + .execute() + .await + .expect_err("non-absence failures must keep failing the attempt"); + assert_eq!(err.is_recoverable_heal(), recoverable, "{source:?}: {err}"); + assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. }), "{source:?}"); + assert_eq!(task.get_outcome().await.counters.failed, 1, "{source:?}"); + } + } +} + +#[tokio::test] +async fn not_found_from_lowercase_free_text_is_not_absent() { + // Only the canonical typed displays convert back; arbitrary text stays a failure. + let storage = missing_storage(MockHealObjectOutcome::ErrOther("file not found while listing")); + let task = object_task("bucket-a", "x.rnd", None, HealRequestSource::ReadRepair, true, storage); + task.execute() + .await + .expect_err("free text must not become an absence verdict"); + assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. })); +} + +#[tokio::test] +async fn recreate_missing_mrf_durable_repair_still_requires_storage_proof() { + // The durable MRF path is unchanged: absence without an exact receipt + // cannot retire the responsibility. + let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer: false }); + *storage.bucket_incarnation_id.lock().expect("bucket incarnation") = Some(Uuid::new_v4()); + let task = object_task(RUSTFS_META_BUCKET, OBSERVATION, None, HealRequestSource::Mrf, true, storage); + task.execute() + .await + .expect_err("MRF responsibility cannot retire without proof"); + assert_eq!(task.get_outcome().await.counters.failed, 1); +} diff --git a/crates/heal/src/heal/task/tests/usage_observation.rs b/crates/heal/src/heal/task/tests/usage_observation.rs index 24455348d..d3b0a41da 100644 --- a/crates/heal/src/heal/task/tests/usage_observation.rs +++ b/crates/heal/src/heal/task/tests/usage_observation.rs @@ -43,56 +43,25 @@ fn missing_storage(error: MockHealObjectOutcome) -> Arc { #[tokio::test] async fn missing_usage_observation_completes_without_claiming_repair_or_absence_proof() { + // rustfs/rustfs#8132: the observation follows the general not-found rule + // (see `absent_verdict`); the completion still records no proof. for outer in [false, true] { let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer }); let task = task(RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, None, storage.clone()); task.execute() .await - .expect("a removed observation is a normal read-repair skip"); + .expect("a removed observation is a normal read-repair completion"); assert_eq!(task.get_status().await, HealTaskStatus::Completed); let outcome = task.get_outcome().await; assert_eq!(outcome.execution, HealExecutionOutcome::Completed); assert_eq!(outcome.counters.failed, 0); - assert_eq!(outcome.counters.skipped, 1); assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::Unknown); let progress = task.get_progress().await; assert_eq!(progress.objects_healed, 0); - assert_eq!(progress.skipped_objects, 1); + assert_eq!(progress.objects_failed, 0); let calls = storage.object_heal_opts.lock().expect("heal calls"); assert_eq!(calls.len(), 1); - assert!(calls[0].recreate, "the test must reach the actual recreation branch"); - } -} - -#[tokio::test] -async fn missing_usage_observation_exclusion_is_limited_to_the_exact_read_repair_target() { - for (bucket, object, source, version) in [ - ("user-bucket", OBSERVATION, HealRequestSource::ReadRepair, None), - (RUSTFS_META_BUCKET, "buckets/.usage.json", HealRequestSource::ReadRepair, None), - ( - RUSTFS_META_BUCKET, - "buckets/.usage.observed.json.bkp", - HealRequestSource::ReadRepair, - None, - ), - ( - RUSTFS_META_BUCKET, - "buckets/nested/.usage.observed.json", - HealRequestSource::ReadRepair, - None, - ), - (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Admin, None), - (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Internal, None), - (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::Scanner, None), - (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::AutoHeal, None), - (RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, Some("version-a")), - ] { - let storage = missing_storage(MockHealObjectOutcome::MissingObject { outer: false }); - let task = task(bucket, object, source, version, storage); - task.execute() - .await - .expect_err("other requests must retain their missing-object failure"); - assert_eq!(task.get_outcome().await.counters.failed, 1, "{bucket}/{object}: {source:?}"); + assert!(calls[0].read_repair, "the request still reaches the locked storage read once"); } } @@ -102,12 +71,13 @@ async fn usage_observation_does_not_hide_non_absence_errors() { MockHealObjectOutcome::PermissionDenied, MockHealObjectOutcome::OkWithReadQuorum, MockHealObjectOutcome::FaultyStorageDisk(false), - MockHealObjectOutcome::MissingVersion, - MockHealObjectOutcome::ErrOther("File not found"), + MockHealObjectOutcome::ErrOther("Disk not found"), ] { let storage = missing_storage(error); let task = task(RUSTFS_META_BUCKET, OBSERVATION, HealRequestSource::ReadRepair, None, storage); - task.execute().await.expect_err("only a typed object absence may be skipped"); + task.execute() + .await + .expect_err("only an object-level absence completes the heal"); assert_eq!(task.get_outcome().await.counters.failed, 1); } } diff --git a/crates/heal/src/lib.rs b/crates/heal/src/lib.rs index 8a4fdf8b5..a64dc7353 100644 --- a/crates/heal/src/lib.rs +++ b/crates/heal/src/lib.rs @@ -23,7 +23,7 @@ pub use heal::{ HealSourceCounts, HealType, channel::HealChannelProcessor, progress::{HealProgress, aggregate_heal_progress}, - resume::{ReplacementRecoveryRecord, ReplacementRecoveryState, ResumeUtils}, + resume::{ReplacementRecoveryBlock, ReplacementRecoveryRecord, ReplacementRecoveryState, ResumeUtils}, }; use rustfs_concurrency::WorkloadAdmissionSnapshotProvider; use std::collections::BTreeMap; @@ -268,6 +268,15 @@ pub async fn current_heal_progress_snapshot() -> Option { } } +/// Durable replacement generations that this node's discovery could not +/// verify or resolve, with their reasons. Node-local; empty when the heal +/// runtime is not initialized. +pub fn current_replacement_recovery_blocks() -> Vec { + get_heal_manager() + .map(|manager| manager.replacement_recovery_blocks()) + .unwrap_or_default() +} + /// Read all local survivor-disk replacement records without conflating an I/O /// failure or conflicting copies with successful completion. pub async fn current_replacement_recovery_snapshot() -> ReplacementRecoverySnapshot { @@ -398,6 +407,13 @@ mod tests { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> Result, Error> { + self.list_buckets().await + } + async fn list_buckets(&self) -> Result, Error> { Ok(Vec::new()) } diff --git a/crates/heal/tests/heal_bug_fixes_test.rs b/crates/heal/tests/heal_bug_fixes_test.rs index d2690b6be..832c71cf7 100644 --- a/crates/heal/tests/heal_bug_fixes_test.rs +++ b/crates/heal/tests/heal_bug_fixes_test.rs @@ -110,6 +110,13 @@ fn test_heal_task_status_atomic_update() { async fn get_bucket_info(&self, _bucket: &str) -> rustfs_heal::Result> { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(vec![]) } @@ -205,6 +212,13 @@ async fn test_heal_task_transient_object_exists_skip_avoids_recreate() { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(Vec::new()) } diff --git a/crates/heal/tests/heal_concurrent_delete_test.rs b/crates/heal/tests/heal_concurrent_delete_test.rs index 2770295dc..c0808f7c8 100644 --- a/crates/heal/tests/heal_concurrent_delete_test.rs +++ b/crates/heal/tests/heal_concurrent_delete_test.rs @@ -79,6 +79,13 @@ impl HealStorageAPI for PausedTraversal { self.inner.get_bucket_info(bucket).await } + async fn list_buckets_for_admin_heal( + &self, + selector: Option, + ) -> Result> { + self.inner.list_buckets_for_admin_heal(selector).await + } + async fn list_buckets(&self) -> Result> { self.inner.list_buckets().await } @@ -95,6 +102,10 @@ impl HealStorageAPI for PausedTraversal { self.inner.admit_bucket_incarnation(bucket).await } + async fn heal_bucket_at_incarnation(&self, bucket: &str, expected: Uuid, opts: &HealOpts) -> Result { + self.inner.heal_bucket_at_incarnation(bucket, expected, opts).await + } + async fn heal_object_at_incarnation( &self, bucket: &str, diff --git a/crates/heal/tests/heal_integration_test.rs b/crates/heal/tests/heal_integration_test.rs index 9559aa242..b9b93460c 100644 --- a/crates/heal/tests/heal_integration_test.rs +++ b/crates/heal/tests/heal_integration_test.rs @@ -17,10 +17,11 @@ use http::HeaderMap; use rustfs_heal::heal::{ manager::{HealConfig, HealManager}, + outcome::{HealExecutionOutcome, HealTraversalCoverage}, storage::{ECStoreHealStorage, HealObjectOptions as ObjectOptions, HealPutObjReader as PutObjReader, HealStorageAPI}, - task::{HealOptions, HealPriority, HealRequest, HealTaskStatus, HealType}, + task::{HealOptions, HealPriority, HealRequest, HealTask, HealTaskStatus, HealType}, }; -use rustfs_heal_contracts::heal_channel::{HealOpts, HealScanMode}; +use rustfs_heal_contracts::heal_channel::{HealOpts, HealRequestSource, HealScanMode}; use serial_test::serial; use std::{ path::{Path, PathBuf}, @@ -33,7 +34,7 @@ use walkdir::WalkDir; mod storage_api; -use storage_api::integration::{BucketOperations, ECStore, ObjectIO as _, ObjectOperations as _}; +use storage_api::integration::{BucketOperations, ECStore, ObjectIO as _, ObjectOperations as _, WriteCompletion}; const HEAL_FORMAT_WAIT_TIMEOUT: Duration = Duration::from_secs(25); const HEAL_FORMAT_WAIT_INTERVAL: Duration = Duration::from_millis(250); @@ -107,16 +108,183 @@ async fn create_test_bucket(ecstore: &Arc, bucket_name: &str) { async fn upload_test_object(ecstore: &Arc, bucket: &str, object: &str, data: &[u8]) { let mut reader = PutObjReader::from_vec(data.to_vec()); let object_info = (**ecstore) - .put_object(bucket, object, &mut reader, &ObjectOptions::default()) + .put_object( + bucket, + object, + &mut reader, + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) .await .expect("Failed to upload test object"); info!("Uploaded test object: {}/{} ({} bytes)", bucket, object, object_info.size); } +async fn empty_replacement_with_committed_delete( + ecstore: &Arc, + disk_paths: &[PathBuf], + bucket: &str, + object: &str, +) -> (uuid::Uuid, uuid::Uuid) { + use rustfs_filemeta::FileInfo; + use storage_api::integration::{ + DeleteOptions, DeleteRecoveryAction, DeleteRecoveryState, DiskAPI as _, DiskSetSelector, StorageAdminApi as _, + }; + assert_eq!(disk_paths.len(), 4, "replacement fixture needs four paths"); + let incarnation = ecstore + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("admit the survivor bucket generation"); + let disks = ecstore + .disk_set_inventory(DiskSetSelector::new(0, 0)) + .await + .expect("read the live fixture set"); + assert_eq!(disks.len(), 4); + let transaction = uuid::Uuid::new_v4(); + // Retain a certified survivor decision without the normal delete's + // immediate recovery/cleanup consuming the fault input. + for disk in &disks[1..] { + let disk = disk.as_ref().expect("all three preparation survivors are online"); + disk.delete_version_recoverable( + bucket, + object, + FileInfo { + name: object.to_owned(), + ..Default::default() + }, + false, + DeleteOptions { + old_data_dir: Some(transaction), + delete_recovery_bucket_incarnation: Some(incarnation), + ..Default::default() + }, + ) + .await + .expect("prepare the same delete on every survivor"); + } + for disk in &disks[1..] { + let disk = disk.as_ref().expect("all three survivors are online"); + let prepared = disk + .inspect_delete_transaction(bucket, object, transaction) + .await + .expect("inspect prepared survivor decision"); + assert_eq!(prepared.state, DeleteRecoveryState::NewPrepared); + let committed = disk + .settle_delete_transaction(bucket, object, transaction, prepared.fingerprint, DeleteRecoveryAction::PublishCommit) + .await + .expect("publish the retained survivor decision"); + assert_eq!(committed.state, DeleteRecoveryState::QuorumCommitted); + } + // Linux disk I/O pins the root inode. Clearing its contents models the + // empty target on every platform; renaming it leaves Linux handles on + // the retired data and does not simulate a replacement for this test. + for entry in std::fs::read_dir(&disk_paths[0]).expect("read target root") { + let entry = entry.expect("read target entry"); + if entry.file_type().expect("target entry type").is_dir() { + std::fs::remove_dir_all(entry.path()).expect("clear target directory"); + } else { + std::fs::remove_file(entry.path()).expect("clear target file"); + } + } + (incarnation, transaction) +} + mod serial_tests { use super::*; + async fn bucket_directory_loss_task(cluster: bool, missing: usize, dry_run: bool, source: HealRequestSource) { + let temp = tempfile::tempdir().expect("isolated bucket recovery fixture"); + let env = rustfs_test_utils::TestECStoreEnv::builder() + .base_dir(temp.path()) + .build() + .await; + let bucket = "task-directory-recovery"; + env.make_bucket(bucket, false).await; + let storage = Arc::new(ECStoreHealStorage::new(env.ecstore.clone())); + let expected = storage + .admit_bucket_incarnation(bucket) + .await + .expect("committed bucket generation"); + for disk in env.disk_paths.iter().rev().take(missing) { + fs::remove_dir(disk.join(bucket)) + .await + .expect("remove only an empty bucket directory"); + } + assert_eq!(env.disk_paths.iter().filter(|disk| disk.join(bucket).exists()).count(), 4 - missing); + let mut request = HealRequest::new( + if cluster { + HealType::Cluster + } else { + HealType::Bucket { + bucket: bucket.to_owned(), + } + }, + HealOptions { + recursive: true, + dry_run, + recreate_missing: false, + timeout: Some(Duration::from_secs(30)), + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = source; + if !cluster && source == HealRequestSource::Admin { + request.bucket_incarnation_id = Some(expected); + } + let task = HealTask::from_request(request, storage.clone()); + let result = task.execute().await; + if missing == 4 || source != HealRequestSource::Admin { + assert!( + result.is_err(), + "unproven or unbound recovery must fail: missing={missing}, source={source:?}" + ); + } else { + result.unwrap_or_else(|error| { + panic!( + "administrator task must reach generation-bound reconstruction: cluster={cluster}, missing={missing}: {error}" + ) + }); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed); + assert_eq!(outcome.coverage, HealTraversalCoverage::Complete); + } + let remaining = if dry_run || missing == 4 || source != HealRequestSource::Admin { + 4 - missing + } else { + 4 + }; + assert_eq!(env.disk_paths.iter().filter(|disk| disk.join(bucket).is_dir()).count(), remaining); + assert_eq!(storage.admit_bucket_incarnation(bucket).await.expect("retained metadata"), expected); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn test_admin_bucket_directory_recovery_from_task_entry() { + for cluster in [false, true] { + // Two surviving directories satisfy ordinary reads; one does not. + for missing in [2, 3] { + bucket_directory_loss_task(cluster, missing, false, HealRequestSource::Admin).await; + } + } + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn test_admin_bucket_directory_recovery_dry_run_from_task_entry() { + bucket_directory_loss_task(false, 3, true, HealRequestSource::Admin).await; + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + #[serial] + async fn test_admin_bucket_directory_recovery_rejects_unproven_scope() { + bucket_directory_loss_task(false, 4, false, HealRequestSource::Admin).await; + bucket_directory_loss_task(false, 3, false, HealRequestSource::Internal).await; + } + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[serial] async fn test_heal_object_basic() { @@ -533,3 +701,383 @@ mod serial_tests { info!("Direct heal storage API test passed"); } } + +#[test] +#[serial] +fn historical_rebalance_topology_allows_format_and_object_rebuild() { + const STACK_SIZE: usize = 8 * 1024 * 1024; + std::thread::Builder::new() + .name("historical-rebalance-rebuild".to_owned()) + .stack_size(STACK_SIZE) + .spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(4) + .thread_stack_size(STACK_SIZE) + .enable_all() + .build() + .expect("historical rebalance recovery runtime"); + runtime.block_on(historical_rebalance_topology_allows_format_and_object_rebuild_case()); + }) + .expect("historical rebalance recovery test thread") + .join() + .expect("historical rebalance recovery test result"); +} + +async fn historical_rebalance_topology_allows_format_and_object_rebuild_case() { + use storage_api::integration::{ + DiskAPI as _, DiskError, DiskOption, RUSTFS_META_BUCKET, RebalStatus, RebalanceMeta, RebalanceStats, new_disk, + }; + + let temp = tempfile::tempdir().expect("isolated historical rebalance fixture"); + let env = rustfs_test_utils::TestECStoreEnv::builder() + .base_dir(temp.path()) + .build() + .await; + let ecstore = env.ecstore; + let disks = env.disk_paths; + let storage = Arc::new(ECStoreHealStorage::new(ecstore.clone())); + assert_eq!(ecstore.pools.len(), 1); + let bucket = "historical-rebalance-rebuild"; + let object = "non-inline-object"; + let expected = non_inline_test_data(); + create_test_bucket(&ecstore, bucket).await; + upload_test_object(&ecstore, bucket, object, &expected).await; + wait_for_object_copies(&disks, bucket, object).await; + + // Model the persisted post-removal state. This test does not execute a + // rebalance worker, decommission lifecycle, or mount-identity handoff. + let mut historical = RebalanceMeta { + id: uuid::Uuid::new_v4().to_string(), + pool_stats: vec![RebalanceStats::default(), RebalanceStats::default()], + ..Default::default() + }; + for pool in &mut historical.pool_stats { + pool.participating = true; + pool.info.status = RebalStatus::Completed; + } + historical + .save_with_opts( + ecstore.pools[0].clone(), + ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("persist completed historical two-pool metadata on the current metadata pool"); + let historical_bytes = ecstore.pools[0] + .get_object_reader(RUSTFS_META_BUCKET, "rebalance.bin", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open persisted historical metadata") + .read_all() + .await + .expect("read persisted historical metadata"); + let old_format: serde_json::Value = serde_json::from_slice( + &fs::read(disks[0].join(RUSTFS_META_BUCKET).join("format.json")) + .await + .expect("read target format"), + ) + .expect("decode target format"); + let target_object = disks[0].join(bucket).join(object); + let old_shard = fs::read(find_part_file(&target_object).expect("original target shard")) + .await + .expect("read original target shard bytes"); + + // Preserve the root inode held by disk I/O, while replacing its contents + // with an empty target, as in the existing fresh-target recovery fixture. + for entry in std::fs::read_dir(&disks[0]).expect("read replacement target root") { + let entry = entry.expect("replacement target entry"); + if entry.file_type().expect("replacement entry type").is_dir() { + fs::remove_dir_all(entry.path()).await.expect("empty target directory"); + } else { + fs::remove_file(entry.path()).await.expect("empty target file"); + } + } + assert!(!disks[0].join(RUSTFS_META_BUCKET).join("format.json").exists()); + assert!(!target_object.exists()); + // A replacement reconnects with a fresh handle. Reusing the old handle + // would retain its cached format bytes despite the emptied directory. + let endpoint = ecstore.pools[0].endpoints.endpoints.as_ref()[0].clone(); + let replacement = new_disk( + &endpoint, + &DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("open a fresh replacement disk handle"); + assert!(matches!(replacement.get_disk_id().await, Err(DiskError::UnformattedDisk))); + ecstore.pools[0].get_disks(0).disks.write().await[0] = Some(replacement); + let target = endpoint.to_string(); + let (_, format_error) = storage + .heal_replacement_format(false, 0, 0, std::slice::from_ref(&target)) + .await + .expect("historical topology must admit target-scoped format repair"); + assert!(format_error.is_none(), "target format repair: {format_error:?}"); + let repaired_format: serde_json::Value = serde_json::from_slice( + &fs::read(disks[0].join(RUSTFS_META_BUCKET).join("format.json")) + .await + .expect("read repaired target format"), + ) + .expect("decode repaired target format"); + assert_eq!( + repaired_format["xl"]["this"], old_format["xl"]["this"], + "format repair must retain the original slot UUID" + ); + assert!( + !target_object.exists(), + "format repair alone must not be mistaken for object reconstruction" + ); + + let mut request = HealRequest::new( + HealType::ErasureSet { + buckets: vec![bucket.to_owned()], + set_disk_id: "pool_0_set_0".to_owned(), + }, + HealOptions { + recursive: true, + pool_index: Some(0), + set_index: Some(0), + scan_mode: HealScanMode::Deep, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + let task = HealTask::from_request(request, storage); + task.execute() + .await + .expect("rebuild objects after historical-topology format repair"); + let outcome = task.get_outcome().await; + assert_eq!(outcome.execution, HealExecutionOutcome::Completed); + assert_eq!(outcome.coverage, HealTraversalCoverage::Complete); + // Verify target-side physical repair before a GET could perform read repair. + assert!(target_object.join("xl.meta").exists()); + let rebuilt_shard = fs::read(find_part_file(&target_object).expect("rebuilt target shard")) + .await + .expect("read rebuilt target shard bytes"); + assert_eq!(rebuilt_shard, old_shard, "the replacement shard must contain the original encoded bytes"); + let actual = ecstore + .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open rebuilt object") + .read_all() + .await + .expect("read rebuilt object bytes"); + assert_eq!(actual, expected, "reconstructed object bytes must be unchanged"); + let retained_history = ecstore.pools[0] + .get_object_reader(RUSTFS_META_BUCKET, "rebalance.bin", None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open retained historical metadata") + .read_all() + .await + .expect("read retained historical metadata"); + assert_eq!(retained_history, historical_bytes, "healing must retain the historical rebalance payload"); +} + +#[test] +#[serial] +fn replacement_delete_recovery_on_fresh_target() { + // Match the debug server's worker stack for the composed Heal future. + // Libtest otherwise polls it on its smaller calling-thread stack. + const STACK_SIZE: usize = 8 * 1024 * 1024; + std::thread::Builder::new() + .name("replacement-delete-recovery".to_owned()) + .stack_size(STACK_SIZE) + .spawn(|| { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(4) + .thread_stack_size(STACK_SIZE) + .enable_all() + .build() + .expect("replacement recovery runtime"); + runtime.block_on(replacement_delete_recovery_on_fresh_target_case()); + }) + .expect("replacement recovery test thread") + .join() + .expect("replacement recovery test result"); +} + +async fn replacement_delete_recovery_on_fresh_target_case() { + use rustfs_heal::Error; + use rustfs_heal::heal::task::HealTask; + use rustfs_heal_contracts::heal_channel::HealRequestSource; + use storage_api::integration::{ + DeleteRecoveryState, DiskAPI as _, DiskError, DiskSetSelector, EcstoreStorageError, StorageAdminApi as _, + }; + + for (source, erasure_set) in [ + (HealRequestSource::AutoHeal, true), + (HealRequestSource::Admin, true), + (HealRequestSource::Admin, false), + ] { + let (disk_paths, ecstore, storage) = heal_env().await; + let bucket = "replacement-delete-prepass"; + create_test_bucket(&ecstore, bucket).await; + let original = non_inline_test_data(); + for key in ["live", "deleted"] { + upload_test_object(&ecstore, bucket, key, &original).await; + wait_for_object_copies(&disk_paths, bucket, key).await; + } + let (incarnation, transaction) = empty_replacement_with_committed_delete(&ecstore, &disk_paths, bucket, "deleted").await; + let updated = vec![17; NON_INLINE_TEST_DATA_SIZE]; + upload_test_object(&ecstore, bucket, "live", &updated).await; + upload_test_object(&ecstore, bucket, "outage-created", &original).await; + let (_, error) = storage.heal_format(false).await.expect("format target"); + assert!(error.is_none(), "{error:?}"); + assert_eq!( + ecstore + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("survivor incarnation remains authoritative"), + incarnation + ); + let current_disks = ecstore + .disk_set_inventory(DiskSetSelector::new(0, 0)) + .await + .expect("read the formatted live set"); + assert!( + matches!( + current_disks[0] + .as_ref() + .expect("formatted target is attached") + .stat_volume(bucket) + .await, + Err(DiskError::VolumeNotFound) + ), + "the formatted target must still lack its publication bucket" + ); + for disk in ¤t_disks[1..] { + let retained = disk + .as_ref() + .expect("survivor remains attached") + .inspect_delete_transaction(bucket, "deleted", transaction) + .await + .expect("retained committed survivor proof"); + assert_eq!(retained.state, DeleteRecoveryState::QuorumCommitted); + } + // A committed survivor journal must be adopted, but publishing its + // witness cannot recreate a missing bucket outside lifecycle ownership. + let premature = storage + .recover_delete_transactions(bucket, None, None, &HealOpts::default()) + .await; + assert!( + matches!(&premature, Err(Error::Storage(EcstoreStorageError::FileNotFound))), + "{source:?} erasure_set={erasure_set}: {premature:?}" + ); + let heal_type = if erasure_set { + HealType::ErasureSet { + buckets: vec![bucket.to_owned()], + set_disk_id: "pool_0_set_0".to_owned(), + } + } else { + HealType::Bucket { + bucket: bucket.to_owned(), + } + }; + let mut request = HealRequest::new( + heal_type, + HealOptions { + recursive: true, + pool_index: Some(0), + set_index: Some(0), + scan_mode: HealScanMode::Deep, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = source; + if !erasure_set { + request.bucket_incarnation_id = Some(incarnation); + } + HealTask::from_request(request, storage.clone()) + .execute() + .await + .unwrap_or_else(|err| panic!("{source:?} erasure_set={erasure_set}: {err}")); + // Check physical repair before GET can hide missing shards via read repair. + for key in ["live", "outage-created"] { + assert!(disk_paths[0].join(bucket).join(key).join("xl.meta").exists()); + assert!(find_part_file(&disk_paths[0].join(bucket).join(key)).is_some()); + } + for (key, expected) in [("live", updated.as_slice()), ("outage-created", original.as_slice())] { + let mut reader = ecstore + .get_object_reader(bucket, key, None, HeaderMap::new(), &ObjectOptions::default()) + .await + .expect("open rebuilt object"); + assert_eq!(reader.read_all().await.expect("read rebuilt bytes"), expected); + } + assert!(matches!( + ecstore.get_object_info(bucket, "deleted", &ObjectOptions::default()).await, + Err(EcstoreStorageError::ObjectNotFound(_, _)) + )); + assert_eq!( + ecstore + .bucket_incarnation_id_from_disk(bucket) + .await + .expect("retained incarnation"), + incarnation + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn bucket_removal_recovers_delete_proofs_before_removing_empty_volumes() { + use rustfs_heal::heal::task::HealTask; + use rustfs_heal_contracts::heal_channel::HealRequestSource; + use storage_api::integration::{DeleteRecoveryState, DiskAPI as _, DiskSetSelector, StorageAdminApi as _}; + let (disk_paths, ecstore, storage) = heal_env().await; + let bucket = "delete-proof-before-volume-removal"; + create_test_bucket(&ecstore, bucket).await; + upload_test_object(&ecstore, bucket, "deleted", &non_inline_test_data()).await; + wait_for_object_copies(&disk_paths, bucket, "deleted").await; + let (incarnation, transaction) = empty_replacement_with_committed_delete(&ecstore, &disk_paths, bucket, "deleted").await; + let (_, error) = storage.heal_format(false).await.expect("format target"); + assert!(error.is_none()); + storage + .heal_bucket( + bucket, + &HealOpts { + recreate: true, + ..Default::default() + }, + ) + .await + .expect("empty target volume"); + assert!(disk_paths[0].join(bucket).exists()); + let disks = ecstore + .disk_set_inventory(DiskSetSelector::new(0, 0)) + .await + .expect("live removal fixture set"); + for disk in &disks[1..] { + let retained = disk + .as_ref() + .expect("removal survivor remains attached") + .inspect_delete_transaction(bucket, "deleted", transaction) + .await + .expect("removal must still adopt the retained delete proof"); + assert_eq!(retained.state, DeleteRecoveryState::QuorumCommitted); + } + let mut request = HealRequest::new( + HealType::Bucket { + bucket: bucket.to_owned(), + }, + HealOptions { + recursive: false, + remove_corrupted: true, + ..Default::default() + }, + HealPriority::Normal, + ); + request.source = HealRequestSource::Admin; + request.bucket_incarnation_id = Some(incarnation); + HealTask::from_request(request, storage) + .execute() + .await + .expect("settle journals before deleting empty volumes"); + assert!( + disk_paths.iter().all(|path| !path.join(bucket).exists()), + "all empty replicas should be removed after proof settlement" + ); +} diff --git a/crates/heal/tests/storage_api.rs b/crates/heal/tests/storage_api.rs index b47737203..171ad216b 100644 --- a/crates/heal/tests/storage_api.rs +++ b/crates/heal/tests/storage_api.rs @@ -2,7 +2,7 @@ pub(crate) mod bug_fixes { pub(crate) use rustfs_ecstore::api::disk::DiskStore; pub(crate) use rustfs_ecstore::api::disk::endpoint::Endpoint; - pub(crate) use rustfs_storage_api::BucketInfo; + pub(crate) use rustfs_storage_api::{BucketInfo, DiskSetSelector}; } #[allow(unused_imports)] @@ -21,12 +21,14 @@ pub(crate) mod endpoint_index { #[allow(unused_imports)] pub(crate) mod integration { pub(crate) use rustfs_ecstore::api::disk::{ - DiskAPI, DiskError, DiskOption, DiskStore, Endpoint, RUSTFS_META_BUCKET, ReadOptions, new_disk, + DeleteOptions, DeleteRecoveryAction, DeleteRecoveryState, DiskAPI, DiskError, DiskOption, DiskStore, Endpoint, + RUSTFS_META_BUCKET, ReadOptions, new_disk, }; pub(crate) use rustfs_ecstore::api::error::StorageError as EcstoreStorageError; pub(crate) use rustfs_ecstore::api::object::{ ObjectOptions, PutObjReader, ShardIntegrityWriteMode, WriteCommitGuard, WriteCompletion, }; + pub(crate) use rustfs_ecstore::api::rebalance::{RebalStatus, RebalanceMeta, RebalanceStats}; pub(crate) use rustfs_ecstore::api::storage::ECStore; pub(crate) use rustfs_storage_api::BucketOperations; pub(crate) use rustfs_storage_api::MakeBucketOptions; diff --git a/crates/heal/tests/usage_observation_read_repair_test.rs b/crates/heal/tests/usage_observation_read_repair_test.rs index d08ecda27..23cfbd248 100644 --- a/crates/heal/tests/usage_observation_read_repair_test.rs +++ b/crates/heal/tests/usage_observation_read_repair_test.rs @@ -77,7 +77,7 @@ async fn usage_observation_read_repair_survives_snapshot_cleanup() { .await .expect("the real degraded GET must emit a repair request"); assert_eq!(request.source, HealRequestSource::ReadRepair); - assert_eq!(request.recreate_missing, Some(true)); + assert_eq!(request.recreate_missing, None, "a degraded read does not instruct recreation"); // Authoritative usage publication deletes the obsolete observation with // these prefix options. Delay admission until that deletion has finished. diff --git a/crates/iam/src/federation/mapper.rs b/crates/iam/src/federation/mapper.rs index 1dca51e5c..6680a2471 100644 --- a/crates/iam/src/federation/mapper.rs +++ b/crates/iam/src/federation/mapper.rs @@ -12,12 +12,123 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::{FederatedAuthorization, VerifiedFederatedIdentity}; +use super::{FederatedClaims, VerifiedFederatedIdentity}; use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive}; +use rustfs_utils::HashAlgorithm; use std::{collections::HashMap, sync::Arc}; use tracing::debug; +#[cfg(any(test, feature = "test-util"))] +#[path = "test_support.rs"] +pub(super) mod test_support; + +/// Read-only authorization result produced by core federation mapping. +/// +/// The claims representation stays inside IAM: +/// +/// ```compile_fail +/// use rustfs_iam::federation::FederatedClaims; +/// ``` +/// +/// Callers cannot mutate mapped fields: +/// +/// ```compile_fail +/// use rustfs_iam::federation::FederatedAuthorization; +/// +/// fn mutate(authorization: &mut FederatedAuthorization) { +/// authorization.groups.clear(); +/// } +/// ``` +/// +/// Callers cannot construct authorization results: +/// +/// ```compile_fail +/// use rustfs_iam::federation::FederatedAuthorization; +/// +/// fn construct(authorization: FederatedAuthorization) -> FederatedAuthorization { +/// FederatedAuthorization { ..authorization } +/// } +/// ``` +#[derive(Debug, Clone)] +pub struct FederatedAuthorization { + provider_id: String, + claims: FederatedClaims, + policies: Vec, + group_claim_policies: Vec, + groups: Vec, + roles_claim_key: Option, + roles: Vec, +} + +impl FederatedAuthorization { + pub fn provider_id(&self) -> &str { + &self.provider_id + } + + pub fn subject(&self) -> &str { + &self.claims.sub + } + + pub fn email(&self) -> &str { + &self.claims.email + } + + pub fn username(&self) -> &str { + &self.claims.username + } + + pub fn session_identity(&self) -> String { + self.claims.session_identity() + } + + pub fn policies(&self) -> &[String] { + &self.policies + } + + /// Subset of `policies` derived only from the groups claim (including merged roles claim + /// values); these may be ignored when no policy of that name exists. Everything else in + /// `policies` must resolve. + pub fn group_claim_policies(&self) -> &[String] { + &self.group_claim_policies + } + + pub fn mapped_iam_groups(&self) -> &[String] { + &self.groups + } + + pub fn roles_claim_key(&self) -> Option<&str> { + self.roles_claim_key.as_deref() + } + + pub fn roles(&self) -> &[String] { + &self.roles + } + + pub fn has_authorization_context(&self) -> bool { + !self.policies.is_empty() || !self.groups.is_empty() + } + + pub fn oidc_virtual_parent(&self) -> Option { + let issuer = self.claims.raw.get("iss")?.as_str()?; + let subject = self.claims.sub.as_str(); + if issuer.is_empty() || subject.is_empty() { + return None; + } + + let subject_len = u64::try_from(subject.len()).ok()?; + let issuer_len = u64::try_from(issuer.len()).ok()?; + let mut source = Vec::with_capacity(23 + subject.len() + issuer.len()); + source.extend_from_slice(b"openid:"); + source.extend_from_slice(&subject_len.to_be_bytes()); + source.extend_from_slice(subject.as_bytes()); + source.extend_from_slice(&issuer_len.to_be_bytes()); + source.extend_from_slice(issuer.as_bytes()); + let digest = HashAlgorithm::SHA256.hash_encode(&source); + Some(base64_simd::URL_SAFE_NO_PAD.encode_to_string(digest.as_ref())) + } +} + #[derive(Clone)] pub(crate) struct FederatedAuthorizationRule { provider_id: String, @@ -308,8 +419,136 @@ fn claim_value_type(value: &serde_json::Value) -> &'static str { #[cfg(test)] mod tests { use super::*; - use crate::federation::{FederatedClaims, FederatedProviderRef, VerifiedFederatedIdentity}; - use serde_json::json; + use crate::federation::{FederatedAuthorizationFixture, FederatedClaims, FederatedProviderRef, VerifiedFederatedIdentity}; + use serde_json::{Value, json}; + + fn authorization(policies: Vec, groups: Vec) -> FederatedAuthorization { + FederatedAuthorizationFixture { + provider_id: "standard_oidc".to_string(), + sub: "subject".to_string(), + attributes: HashMap::from([("iss".to_string(), Value::String("https://idp.example.test".to_string()))]), + policies, + mapped_iam_groups: groups, + ..Default::default() + } + .into_authorization() + } + + #[test] + fn authorization_context_accepts_policy_or_group() { + assert!(!authorization(Vec::new(), Vec::new()).has_authorization_context()); + assert!(authorization(vec!["consoleAdmin".to_string()], Vec::new()).has_authorization_context()); + assert!(authorization(Vec::new(), vec!["RustFS.ConsoleAdmin".to_string()]).has_authorization_context()); + } + + #[test] + fn oidc_virtual_parent_is_issuer_scoped() { + let first = authorization(Vec::new(), Vec::new()); + let mut second = first.clone(); + second + .claims + .raw + .insert("iss".to_string(), Value::String("https://other-idp.example.test".to_string())); + + assert_eq!( + first.oidc_virtual_parent().as_deref(), + Some("HwDfWftzOy4jiuS3WjKytC_Sg_A2hKhrRAFtBDhoBr0") + ); + assert!(!rustfs_policy::auth::contains_reserved_chars( + first.oidc_virtual_parent().as_deref().expect("virtual parent") + )); + assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); + } + + #[test] + fn oidc_virtual_parent_length_delimits_identity_parts() { + let mut first = authorization(Vec::new(), Vec::new()); + first.claims.sub = "subject".to_string(); + first.claims.raw.insert( + "iss".to_string(), + Value::String("https://issuer.example/path:https://other.example".to_string()), + ); + let mut second = authorization(Vec::new(), Vec::new()); + second.claims.sub = "subject:https://issuer.example/path".to_string(); + second + .claims + .raw + .insert("iss".to_string(), Value::String("https://other.example".to_string())); + + assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); + } + + #[test] + fn oidc_virtual_parent_requires_verified_identity_parts() { + let mut missing_issuer = authorization(Vec::new(), Vec::new()); + missing_issuer.claims.raw.clear(); + let mut missing_subject = authorization(Vec::new(), Vec::new()); + missing_subject.claims.sub.clear(); + + assert!(missing_issuer.oidc_virtual_parent().is_none()); + assert!(missing_subject.oidc_virtual_parent().is_none()); + } + + #[test] + fn oidc_virtual_parent_preserves_exact_subject() { + let plain = authorization(Vec::new(), Vec::new()); + let mut padded = plain.clone(); + padded.claims.sub = format!(" {} ", plain.claims.sub); + + assert_ne!(plain.oidc_virtual_parent(), padded.oidc_virtual_parent()); + } + + #[test] + fn authorization_accessors_preserve_exact_values_and_distinct_groups() { + let attributes = HashMap::from([("custom".to_string(), json!(["value", null, 7]))]); + let authorization = FederatedAuthorizationFixture { + provider_id: " Corp Provider ".to_string(), + sub: " subject ".to_string(), + email: " email@example.test ".to_string(), + username: " username ".to_string(), + source_groups: vec!["different-source-group".to_string()], + attributes: attributes.clone(), + policies: vec![" policy-b ".to_string(), "policy-a".to_string(), " policy-b ".to_string()], + group_claim_policies: vec![" policy-b ".to_string(), " policy-b ".to_string()], + mapped_iam_groups: vec!["mapped-a".to_string(), "mapped-b".to_string()], + roles_claim_key: Some(" Roles ".to_string()), + roles: vec!["role-b".to_string(), " role-a ".to_string(), "role-b".to_string()], + } + .into_authorization(); + + assert_eq!(authorization.provider_id(), " Corp Provider "); + assert_eq!(authorization.subject(), " subject "); + assert_eq!(authorization.email(), " email@example.test "); + assert_eq!(authorization.username(), " username "); + assert_eq!(authorization.session_identity(), " username "); + assert_eq!(authorization.policies(), [" policy-b ", "policy-a", " policy-b "]); + assert_eq!(authorization.group_claim_policies(), [" policy-b ", " policy-b "]); + assert_eq!(authorization.claims.groups, ["different-source-group"]); + assert_eq!(authorization.claims.raw, attributes); + assert_eq!(authorization.mapped_iam_groups(), ["mapped-a", "mapped-b"]); + assert_eq!(authorization.roles_claim_key(), Some(" Roles ")); + assert_eq!(authorization.roles(), ["role-b", " role-a ", "role-b"]); + } + + #[test] + fn default_authorization_fixture_has_no_identity_or_mapping() { + let authorization = FederatedAuthorizationFixture::default().into_authorization(); + + assert_eq!(authorization.provider_id(), ""); + assert_eq!(authorization.subject(), ""); + assert_eq!(authorization.email(), ""); + assert_eq!(authorization.username(), ""); + assert_eq!(authorization.session_identity(), "oidc-user-unknown"); + assert!(authorization.claims.groups.is_empty()); + assert!(authorization.claims.raw.is_empty()); + assert!(authorization.policies().is_empty()); + assert!(authorization.group_claim_policies().is_empty()); + assert!(authorization.mapped_iam_groups().is_empty()); + assert!(authorization.roles_claim_key().is_none()); + assert!(authorization.roles().is_empty()); + assert!(!authorization.has_authorization_context()); + assert!(authorization.oidc_virtual_parent().is_none()); + } fn mapper(rule: FederatedAuthorizationRule) -> CoreFederatedAuthorizationMapper { CoreFederatedAuthorizationMapper::new(FederatedAuthorizationRules::new([rule])) diff --git a/crates/iam/src/federation/mod.rs b/crates/iam/src/federation/mod.rs index b7498cf0a..6b54f62e4 100644 --- a/crates/iam/src/federation/mod.rs +++ b/crates/iam/src/federation/mod.rs @@ -24,11 +24,13 @@ pub use error::{FederatedSessionBindingError, FederationError, Result}; pub(crate) use mapper::FederatedAuthorizationRule; #[cfg(test)] pub(crate) use mapper::FederatedAuthorizationRuleRef; -pub use mapper::{CoreFederatedAuthorizationMapper, FederatedAuthorizationRules}; -pub(crate) use model::OpaqueLogoutContinuation; +#[cfg(any(test, feature = "test-util"))] +pub use mapper::test_support::FederatedAuthorizationFixture; +pub use mapper::{CoreFederatedAuthorizationMapper, FederatedAuthorization, FederatedAuthorizationRules}; +pub(crate) use model::{FederatedClaims, OpaqueLogoutContinuation}; pub use model::{ - FederatedAuthorization, FederatedClaims, FederatedLoginSession, FederatedProviderRef, FederatedSession, - FederatedSessionTransaction, OIDC_VIRTUAL_PARENT_CLAIM, VerifiedFederatedIdentity, + FederatedLoginSession, FederatedProviderRef, FederatedSession, FederatedSessionTransaction, OIDC_VIRTUAL_PARENT_CLAIM, + VerifiedFederatedIdentity, }; pub(crate) use provider::{FederatedProviderQuery, StandardOidcAuthentication, VerifiedFederatedCodeExchange}; pub use provider::{FederatedProviderView, FederatedRedirectPolicy}; diff --git a/crates/iam/src/federation/model.rs b/crates/iam/src/federation/model.rs index 76b8a9816..54b6725b9 100644 --- a/crates/iam/src/federation/model.rs +++ b/crates/iam/src/federation/model.rs @@ -13,23 +13,24 @@ // limitations under the License. use rustfs_credentials::Credentials; -use rustfs_utils::HashAlgorithm; use serde_json::Value; use std::{collections::HashMap, fmt}; +use super::FederatedAuthorization; + pub const OIDC_VIRTUAL_PARENT_CLAIM: &str = "x-rustfs-internal-oidc-parent"; #[derive(Debug, Clone)] -pub struct FederatedClaims { - pub sub: String, - pub email: String, - pub username: String, - pub groups: Vec, - pub raw: HashMap, +pub(crate) struct FederatedClaims { + pub(crate) sub: String, + pub(crate) email: String, + pub(crate) username: String, + pub(crate) groups: Vec, + pub(crate) raw: HashMap, } impl FederatedClaims { - pub fn session_identity(&self) -> String { + pub(crate) fn session_identity(&self) -> String { if !self.username.is_empty() { self.username.clone() } else if !self.email.is_empty() { @@ -125,45 +126,6 @@ impl VerifiedFederatedIdentity { } } -#[derive(Debug, Clone)] -pub struct FederatedAuthorization { - pub provider_id: String, - pub claims: FederatedClaims, - pub policies: Vec, - /// Subset of `policies` derived only from the groups claim (including merged roles claim - /// values); these may be ignored when no policy of that name exists. Everything else in - /// `policies` must resolve. - pub group_claim_policies: Vec, - pub groups: Vec, - pub roles_claim_key: Option, - pub roles: Vec, -} - -impl FederatedAuthorization { - pub fn has_authorization_context(&self) -> bool { - !self.policies.is_empty() || !self.groups.is_empty() - } - - pub fn oidc_virtual_parent(&self) -> Option { - let issuer = self.claims.raw.get("iss")?.as_str()?; - let subject = self.claims.sub.as_str(); - if issuer.is_empty() || subject.is_empty() { - return None; - } - - let subject_len = u64::try_from(subject.len()).ok()?; - let issuer_len = u64::try_from(issuer.len()).ok()?; - let mut source = Vec::with_capacity(23 + subject.len() + issuer.len()); - source.extend_from_slice(b"openid:"); - source.extend_from_slice(&subject_len.to_be_bytes()); - source.extend_from_slice(subject.as_bytes()); - source.extend_from_slice(&issuer_len.to_be_bytes()); - source.extend_from_slice(issuer.as_bytes()); - let digest = HashAlgorithm::SHA256.hash_encode(&source); - Some(base64_simd::URL_SAFE_NO_PAD.encode_to_string(digest.as_ref())) - } -} - /// Carries the provider-bound ID token between code exchange and logout token creation. pub(crate) struct OpaqueLogoutContinuation { provider: FederatedProviderRef, @@ -224,21 +186,6 @@ mod tests { } } - fn authorization(policies: Vec, groups: Vec) -> FederatedAuthorization { - FederatedAuthorization { - provider_id: "standard_oidc".to_string(), - claims: FederatedClaims { - raw: HashMap::from([("iss".to_string(), Value::String("https://idp.example.test".to_string()))]), - ..claims("", "", "subject") - }, - policies, - group_claim_policies: Vec::new(), - groups, - roles_claim_key: None, - roles: Vec::new(), - } - } - #[test] fn session_identity_preserves_existing_fallback_order() { assert_eq!(claims("john", "john@example.com", "sub-1").session_identity(), "john"); @@ -247,70 +194,6 @@ mod tests { assert_eq!(claims("", "", "").session_identity(), "oidc-user-unknown"); } - #[test] - fn authorization_context_accepts_policy_or_group() { - assert!(!authorization(Vec::new(), Vec::new()).has_authorization_context()); - assert!(authorization(vec!["consoleAdmin".to_string()], Vec::new()).has_authorization_context()); - assert!(authorization(Vec::new(), vec!["RustFS.ConsoleAdmin".to_string()]).has_authorization_context()); - } - - #[test] - fn oidc_virtual_parent_is_issuer_scoped() { - let first = authorization(Vec::new(), Vec::new()); - let mut second = first.clone(); - second - .claims - .raw - .insert("iss".to_string(), Value::String("https://other-idp.example.test".to_string())); - - assert_eq!( - first.oidc_virtual_parent().as_deref(), - Some("HwDfWftzOy4jiuS3WjKytC_Sg_A2hKhrRAFtBDhoBr0") - ); - assert!(!rustfs_policy::auth::contains_reserved_chars( - first.oidc_virtual_parent().as_deref().expect("virtual parent") - )); - assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); - } - - #[test] - fn oidc_virtual_parent_length_delimits_identity_parts() { - let mut first = authorization(Vec::new(), Vec::new()); - first.claims.sub = "subject".to_string(); - first.claims.raw.insert( - "iss".to_string(), - Value::String("https://issuer.example/path:https://other.example".to_string()), - ); - let mut second = authorization(Vec::new(), Vec::new()); - second.claims.sub = "subject:https://issuer.example/path".to_string(); - second - .claims - .raw - .insert("iss".to_string(), Value::String("https://other.example".to_string())); - - assert_ne!(first.oidc_virtual_parent(), second.oidc_virtual_parent()); - } - - #[test] - fn oidc_virtual_parent_requires_verified_identity_parts() { - let mut missing_issuer = authorization(Vec::new(), Vec::new()); - missing_issuer.claims.raw.clear(); - let mut missing_subject = authorization(Vec::new(), Vec::new()); - missing_subject.claims.sub.clear(); - - assert!(missing_issuer.oidc_virtual_parent().is_none()); - assert!(missing_subject.oidc_virtual_parent().is_none()); - } - - #[test] - fn oidc_virtual_parent_preserves_exact_subject() { - let plain = authorization(Vec::new(), Vec::new()); - let mut padded = plain.clone(); - padded.claims.sub = format!(" {} ", plain.claims.sub); - - assert_ne!(plain.oidc_virtual_parent(), padded.oidc_virtual_parent()); - } - #[test] fn verified_identity_preserves_exact_values_and_distinct_groups() { let attributes = HashMap::from([ @@ -341,24 +224,6 @@ mod tests { assert!(!debug.contains(" username ")); assert!(!debug.contains(" source-b ")); assert!(!debug.contains("identity-attribute-poison")); - - let authorization = FederatedAuthorization { - provider_id: " Corp Provider ".to_string(), - claims: FederatedClaims { - sub: "subject".to_string(), - email: String::new(), - username: String::new(), - groups: vec!["different-source-group".to_string()], - raw: HashMap::new(), - }, - policies: Vec::new(), - group_claim_policies: Vec::new(), - groups: vec!["mapped-a".to_string(), "mapped-b".to_string()], - roles_claim_key: None, - roles: Vec::new(), - }; - assert_eq!(authorization.claims.groups, ["different-source-group"]); - assert_eq!(authorization.groups, ["mapped-a", "mapped-b"]); } #[test] diff --git a/crates/iam/src/federation/test_support.rs b/crates/iam/src/federation/test_support.rs new file mode 100644 index 000000000..3933ae910 --- /dev/null +++ b/crates/iam/src/federation/test_support.rs @@ -0,0 +1,54 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::{FederatedAuthorization, FederatedClaims}; +use serde_json::Value; +use std::collections::HashMap; + +/// Exact authorization inputs for tests that exercise the session binding boundary. +#[derive(Default)] +pub struct FederatedAuthorizationFixture { + pub provider_id: String, + pub sub: String, + pub email: String, + pub username: String, + pub source_groups: Vec, + pub attributes: HashMap, + pub policies: Vec, + pub group_claim_policies: Vec, + pub mapped_iam_groups: Vec, + pub roles_claim_key: Option, + pub roles: Vec, +} + +impl FederatedAuthorizationFixture { + /// Preserves supplied values without authentication, validation, or policy mapping. + pub fn into_authorization(self) -> FederatedAuthorization { + FederatedAuthorization { + provider_id: self.provider_id, + claims: FederatedClaims { + sub: self.sub, + email: self.email, + username: self.username, + groups: self.source_groups, + raw: self.attributes, + }, + policies: self.policies, + group_claim_policies: self.group_claim_policies, + groups: self.mapped_iam_groups, + roles_claim_key: self.roles_claim_key, + roles: self.roles, + } + } +} diff --git a/crates/iam/src/federation/transaction.rs b/crates/iam/src/federation/transaction.rs index 3f7ec15dc..84f4c2969 100644 --- a/crates/iam/src/federation/transaction.rs +++ b/crates/iam/src/federation/transaction.rs @@ -17,6 +17,7 @@ use super::{ FederatedProviderView, FederatedRedirectPolicy, FederatedSession, FederatedSessionBinding, FederatedSessionTransaction, FederationError, OpaqueLogoutContinuation, Result, StandardOidcAuthentication, }; +use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; use std::sync::Arc; const DEFAULT_OIDC_PROVIDER_ID: &str = "default"; @@ -133,22 +134,28 @@ impl FederatedIdentityService { let authorization = self.verify_identity(jwt).await?; if !authorization.has_authorization_context() { tracing::warn!( - provider_id = %authorization.provider_id, - username = %authorization.claims.username, - sub = %authorization.claims.sub, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), + username = %authorization.username(), + sub = %authorization.subject(), + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), "AssumeRoleWithWebIdentity has no mapped policies or groups" ); return Err(FederationError::NoAuthorizationContext); } tracing::debug!( - provider_id = %authorization.provider_id, - username = %authorization.claims.username, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), - policies = ?authorization.policies, - groups = ?authorization.groups, + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), + username = %authorization.username(), + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), + policies = ?authorization.policies(), + groups = ?authorization.mapped_iam_groups(), "AssumeRoleWithWebIdentity mapped OIDC policies and groups" ); @@ -379,19 +386,19 @@ mod tests { ) -> core::result::Result { self.events.lock().expect("event log should not be poisoned").push("bind"); self.transactions.lock().expect("transactions should not be poisoned").push(( - transaction.authorization.provider_id.clone(), + transaction.authorization.provider_id().to_string(), transaction.duration_seconds, transaction.session_policy.clone(), )); self.mapped_policies .lock() .expect("mapped policies should not be poisoned") - .push(transaction.authorization.policies.clone()); + .push(transaction.authorization.policies().to_vec()); if self.fail { return Err(FederatedSessionBindingError::Internal("binding failed".to_string())); } Ok(Credentials { - access_key: transaction.authorization.claims.session_identity(), + access_key: transaction.authorization.session_identity(), ..Default::default() }) } @@ -487,7 +494,7 @@ mod tests { .await .expect("callback flow should complete"); assert_eq!(login.session.credentials.access_key, "user"); - assert_eq!(login.session.authorization.provider_id, "corp"); + assert_eq!(login.session.authorization.provider_id(), "corp"); assert_eq!(login.redirect_after.as_deref(), Some("/browser")); assert_eq!(login.logout_token, "logout-token"); assert_eq!( @@ -501,7 +508,7 @@ mod tests { .await .expect("web identity flow should complete"); assert_eq!(web_identity.credentials.access_key, "user"); - assert_eq!(web_identity.authorization.provider_id, "partner"); + assert_eq!(web_identity.authorization.provider_id(), "partner"); assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]); assert_eq!( binding @@ -530,8 +537,8 @@ mod tests { .complete_authorization_code("state", "code", "https://console.example/callback", 3600, binding.as_ref()) .await .expect("standard callback flow should complete"); - assert_eq!(login.session.authorization.provider_id, "corp"); - assert_eq!(login.session.authorization.policies, ["readwrite"]); + assert_eq!(login.session.authorization.provider_id(), "corp"); + assert_eq!(login.session.authorization.policies(), ["readwrite"]); assert_eq!( events.lock().expect("event log should not be poisoned").as_slice(), ["exchange", "bind", "logout"] @@ -542,8 +549,8 @@ mod tests { .assume_role_with_web_identity("jwt", 7200, Some("session-policy".to_string()), binding.as_ref()) .await .expect("standard web identity flow should complete"); - assert_eq!(web_identity.authorization.provider_id, "partner"); - assert_eq!(web_identity.authorization.policies, ["readwrite"]); + assert_eq!(web_identity.authorization.provider_id(), "partner"); + assert_eq!(web_identity.authorization.policies(), ["readwrite"]); assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]); assert_eq!( binding @@ -586,8 +593,8 @@ mod tests { .await .expect("a mapped group is an authorization context"); - assert_eq!(session.authorization.policies, ["developers"]); - assert_eq!(session.authorization.groups, ["developers"]); + assert_eq!(session.authorization.policies(), ["developers"]); + assert_eq!(session.authorization.mapped_iam_groups(), ["developers"]); assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]); } diff --git a/crates/iam/src/manager.rs b/crates/iam/src/manager.rs index 7f6477f0f..c97c7e49d 100644 --- a/crates/iam/src/manager.rs +++ b/crates/iam/src/manager.rs @@ -56,6 +56,10 @@ use tokio::{ use tracing::warn; use tracing::{debug, error}; +const EVENT_USER_UPDATE_EXISTING: &str = "iam.user.update_existing"; +const LOG_COMPONENT_IAM: &str = "iam"; +const LOG_SUBSYSTEM_USERS: &str = "users"; + const IAM_FORMAT_FILE: &str = "format.json"; const IAM_FORMAT_VERSION_1: i32 = 1; #[cfg(not(test))] @@ -1485,7 +1489,13 @@ where let cache = self.cache.snapshot(); let users = Arc::clone(&cache.users); if let Some(x) = users.get(access_key) { - warn!(error = ?x, "IAM user already exists"); + debug!( + event = EVENT_USER_UPDATE_EXISTING, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_USERS, + state = "existing", + "Updating existing IAM user" + ); if x.credentials.is_temp() { return Err(Error::AccessKeyAlreadyExists); } @@ -2921,6 +2931,60 @@ mod tests { } } + #[tokio::test] + async fn credential_boundary_existing_user_logs_omit_credentials() { + use tracing::instrument::WithSubscriber; + + let cache = build_test_iam_cache(FailingInitialLoadStore); + let identity = UserIdentity::new(Credentials { + access_key: "LOGACCESS-PRIVATE-TAIL".to_string(), + secret_key: "Qstored-secret-valueZ".to_string(), + status: STATUS_ENABLED.to_string(), + ..Default::default() + }); + cache + .cache + .add_or_update_user(&identity.credentials.access_key, &identity, OffsetDateTime::now_utc()); + let log = tempfile::NamedTempFile::new().expect("capture file"); + let writer = log.reopen().expect("capture writer"); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::DEBUG) + .with_ansi(false) + .without_time() + .with_writer(move || writer.try_clone().expect("clone capture writer")) + .finish(); + let result = cache + .add_user( + &identity.credentials.access_key, + &AddOrUpdateUserReq { + secret_key: "replacement-secret".to_string(), + policy: None, + status: AccountStatus::Enabled, + }, + ) + .with_subscriber(subscriber) + .await; + assert!( + matches!(result, Err(Error::InvalidArgument)), + "the test store rejects persistence after the existing-user path" + ); + let captured = std::fs::read_to_string(log.path()).expect("read captured logs"); + assert!( + captured.contains(EVENT_USER_UPDATE_EXISTING) || captured.contains("IAM user already exists"), + "the existing-user diagnostic must be captured" + ); + for forbidden in [ + &identity.credentials.access_key, + &identity.credentials.secret_key, + "Q***Z|20", + "secret_key", + "session_token", + "credentials", + ] { + assert!(!captured.contains(forbidden), "credential material must not appear in logs: {forbidden}"); + } + } + #[tokio::test] async fn sts_policy_lookup_ignores_colliding_regular_user_mapping() { let iam = build_test_iam_cache(FailingInitialLoadStore); diff --git a/crates/iam/src/oidc/config.rs b/crates/iam/src/oidc/config.rs index deceb35e1..c8c48bebe 100644 --- a/crates/iam/src/oidc/config.rs +++ b/crates/iam/src/oidc/config.rs @@ -11,15 +11,20 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. +use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC}; use base64_simd::URL_SAFE_NO_PAD; use rustfs_config::oidc::*; use rustfs_config::server_config::{Config as ServerConfig, KVS}; -use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState}; +use rustfs_config::{ + CONSOLE_OIDC_SESSION_DURATION_MAX_SECS, DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS, DEFAULT_DELIMITER, ENABLE_KEY, + ENV_CONSOLE_OIDC_SESSION_DURATION, EnableState, +}; use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::collections::HashMap; use std::fmt; +use tracing::warn; use url::Url; pub(super) const REDACTED_SECRET: &str = "***redacted***"; @@ -815,6 +820,38 @@ pub fn load_oidc_provider_configs_from_env() -> Vec { parse_env_configs() } +/// Credential lifetime, in seconds, for Console credentials issued by the OIDC callback. +/// +/// `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` overrides the default for every provider. An unset, +/// unparseable, or zero value keeps [`DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS`], so a +/// misconfigured variable can never issue credentials that are already expired. Values above +/// [`CONSOLE_OIDC_SESSION_DURATION_MAX_SECS`] are clamped to that maximum with a warning. +/// +/// Both the credential lifetime and the Console logout-token retention derive from this single +/// value, so they stay in step whichever way the variable is set. +pub fn console_session_duration_secs() -> usize { + let configured = rustfs_utils::get_env_opt_u64(ENV_CONSOLE_OIDC_SESSION_DURATION) + .and_then(|seconds| usize::try_from(seconds).ok()) + .filter(|seconds| *seconds > 0) + .unwrap_or(DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS); + + if configured > CONSOLE_OIDC_SESSION_DURATION_MAX_SECS { + warn!( + event = EVENT_OIDC_DIAGNOSTICS, + component = LOG_COMPONENT_IAM, + subsystem = LOG_SUBSYSTEM_OIDC, + state = "console_session_duration_clamped", + config_key = ENV_CONSOLE_OIDC_SESSION_DURATION, + configured, + max = CONSOLE_OIDC_SESSION_DURATION_MAX_SECS, + "Console OIDC session duration exceeds the maximum" + ); + return CONSOLE_OIDC_SESSION_DURATION_MAX_SECS; + } + + configured +} + pub fn load_oidc_provider_configs_from_server_config(cfg: &ServerConfig) -> Vec { parse_persisted_configs(cfg) } diff --git a/crates/iam/src/oidc/config_tests.rs b/crates/iam/src/oidc/config_tests.rs index 7f79c7a96..37370a434 100644 --- a/crates/iam/src/oidc/config_tests.rs +++ b/crates/iam/src/oidc/config_tests.rs @@ -17,6 +17,28 @@ use crate::oidc::test_support::*; use rustfs_config::server_config::{Config as ServerConfig, KVS}; use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY}; +#[test] +#[serial_test::serial] +fn console_session_duration_follows_the_environment() { + for (configured, expected) in [ + (None, DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("1"), 1), + (Some("7200"), 7200), + (Some("604800"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("604801"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("999999999"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("18446744073709551615"), CONSOLE_OIDC_SESSION_DURATION_MAX_SECS), + (Some("18446744073709551616"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("0"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("-1"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some("60s"), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + (Some(""), DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS), + ] { + let observed = temp_env::with_var(ENV_CONSOLE_OIDC_SESSION_DURATION, configured, console_session_duration_secs); + assert_eq!(observed, expected, "Console OIDC session duration for {configured:?}"); + } +} + #[test] fn test_parse_single_provider_no_config_url() { let config = parse_single_provider("_TEST_EMPTY", "test_empty"); diff --git a/crates/iam/src/oidc/mod.rs b/crates/iam/src/oidc/mod.rs index 7452dcd19..a171b776e 100644 --- a/crates/iam/src/oidc/mod.rs +++ b/crates/iam/src/oidc/mod.rs @@ -25,9 +25,10 @@ mod test_support; pub use config::{ OidcConfigError, OidcConfigQuery, OidcConfigSnapshot, OidcProviderConfig, OidcProviderConfigInput, OidcProviderConfigSource, OidcProviderValidationInput, OidcProviderValidationResult, OidcSiteReplicationProvider, OidcSiteReplicationSnapshot, - SourcedOidcProviderConfig, build_upsert_provider_config, build_validation_provider_config, delete_persisted_provider_config, - load_oidc_config_snapshot, load_oidc_provider_configs_from_env, load_oidc_provider_configs_from_server_config, - persisted_provider_secret, upsert_persisted_provider_config, validate_mutable_provider_id, + SourcedOidcProviderConfig, build_upsert_provider_config, build_validation_provider_config, console_session_duration_secs, + delete_persisted_provider_config, load_oidc_config_snapshot, load_oidc_provider_configs_from_env, + load_oidc_provider_configs_from_server_config, persisted_provider_secret, upsert_persisted_provider_config, + validate_mutable_provider_id, }; pub use provider::validate_oidc_provider_config_with_extra_root_ca; pub use runtime::{OidcSys, StandardOidcAdapter}; diff --git a/crates/iam/src/oidc/recovery_tests.rs b/crates/iam/src/oidc/recovery_tests.rs index f1b500b85..96fece5bc 100644 --- a/crates/iam/src/oidc/recovery_tests.rs +++ b/crates/iam/src/oidc/recovery_tests.rs @@ -177,9 +177,9 @@ async fn web_identity_recovers_transport_failure_and_preserves_verification() { .await .expect("verify recovered workload token"); let authorization = mapper.map(identity); - assert_eq!(authorization.provider_id, "workload"); - assert_eq!(authorization.claims.sub, "recovered-user"); - assert_eq!(authorization.policies, vec!["team/readonly"]); + assert_eq!(authorization.provider_id(), "workload"); + assert_eq!(authorization.subject(), "recovered-user"); + assert_eq!(authorization.policies(), ["team/readonly"]); assert!( sys.authorize_url("workload", "https://console.example.com/callback", None) .await diff --git a/crates/iam/src/oidc/runtime_tests.rs b/crates/iam/src/oidc/runtime_tests.rs index 84dd95943..52181e530 100644 --- a/crates/iam/src/oidc/runtime_tests.rs +++ b/crates/iam/src/oidc/runtime_tests.rs @@ -371,12 +371,12 @@ async fn assert_complete_console_login(initially_unavailable: bool) { .await .unwrap_or_else(|err| panic!("hidden={hidden}, explicit={explicit}: {err}")); let (identity, redirect_after, _) = exchange.into_parts(); + assert_eq!(identity.source_groups(), ["readwrite"]); let authorization = mapper.map(identity); - assert_eq!(authorization.provider_id, "console"); - assert_eq!(authorization.claims.sub, "existing-user"); - assert_eq!(authorization.claims.email, "user@example.com"); - assert_eq!(authorization.claims.groups, vec!["readwrite"]); - assert_eq!(authorization.policies, vec!["team/readwrite"]); + assert_eq!(authorization.provider_id(), "console"); + assert_eq!(authorization.subject(), "existing-user"); + assert_eq!(authorization.email(), "user@example.com"); + assert_eq!(authorization.policies(), ["team/readwrite"]); assert_eq!(redirect_after.as_deref(), Some("/buckets")); assert!(matches!(sys.exchange_code(&state, "test-authorization-code", redirect).await, Err(error) if error == "invalid or expired OIDC state")); diff --git a/crates/iam/src/oidc/state.rs b/crates/iam/src/oidc/state.rs index fa9b90f3e..48af714b8 100644 --- a/crates/iam/src/oidc/state.rs +++ b/crates/iam/src/oidc/state.rs @@ -18,6 +18,8 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; use tracing::warn; +use super::config::console_session_duration_secs; + const OIDC_STATE_CAPACITY: u64 = 10_000; const OIDC_STATE_CAPACITY_WARNING: u64 = 9_000; const OIDC_STATE_CAPACITY_CRITICAL: u64 = 10_000; @@ -57,7 +59,10 @@ impl OidcStateStore { .build(); let logout_cache = Cache::builder() .max_capacity(OIDC_STATE_CAPACITY) - .time_to_live(Duration::from_secs(3600)) // 1 hour TTL to match console OIDC sessions + // Keep the logout token alive for as long as the credentials it belongs to: if it + // expired sooner, a session longer than the old hard-coded hour could no longer + // trigger RP-initiated logout. + .time_to_live(Duration::from_secs(console_session_duration_secs() as u64)) .build(); Self { cache, @@ -140,6 +145,8 @@ impl Default for OidcStateStore { #[cfg(test)] mod tests { use super::*; + use rustfs_config::ENV_CONSOLE_OIDC_SESSION_DURATION; + use serial_test::serial; use std::sync::Arc; use tokio::sync::Barrier; @@ -302,4 +309,26 @@ mod tests { assert!(store.take_logout("logout_abc").await.is_none()); } + + #[test] + #[serial] + fn logout_retention_follows_the_bounded_session_duration() { + for (configured, expected_seconds) in [ + (None, 3600), + (Some("1"), 1), + (Some("7200"), 7200), + (Some("604801"), 604_800), + (Some("18446744073709551615"), 604_800), + ] { + temp_env::with_var(ENV_CONSOLE_OIDC_SESSION_DURATION, configured, || { + let store = OidcStateStore::new(); + assert_eq!( + store.logout_cache.policy().time_to_live(), + Some(Duration::from_secs(expected_seconds)), + "logout retention for {configured:?} must match the issued session" + ); + assert_eq!(store.cache.policy().time_to_live(), Some(Duration::from_secs(300))); + }); + } + } } diff --git a/crates/iam/src/sys.rs b/crates/iam/src/sys.rs index 7830f9ea4..39202c1b5 100644 --- a/crates/iam/src/sys.rs +++ b/crates/iam/src/sys.rs @@ -5293,6 +5293,45 @@ mod tests { ); } + /// A session policy that requires conditional writes must decode and gate + /// PutObject on the `If-None-Match` header; an unknown condition key would fail + /// to parse and deny the session everything. + #[tokio::test] + async fn session_policy_enforces_if_none_match_condition_key() { + let session_policy_json = r#"{ + "Version":"2012-10-17", + "Statement":[{"Effect":"Allow","Action":["s3:PutObject"],"Resource":["arn:aws:s3:::bucket/*"],"Condition":{"Null":{"s3:if-none-match":"false"}}}] +}"#; + let encoded = base64_simd::URL_SAFE_NO_PAD.encode_to_string(session_policy_json.as_bytes()); + decode_session_policy(&encoded).expect("session policy with s3:if-none-match must decode"); + + let mut claims = HashMap::new(); + claims.insert(SESSION_POLICY_NAME.to_string(), Value::String(encoded)); + let groups: Option> = None; + let with_header = HashMap::from([("if-none-match".to_string(), vec!["*".to_string()])]); + let without_header = HashMap::new(); + + let mut outcomes = Vec::new(); + for conditions in [&with_header, &without_header] { + let args = Args { + account: "sts-conditional-write-user", + groups: &groups, + action: Action::S3Action(S3Action::PutObjectAction), + bucket: "bucket", + conditions, + is_owner: false, + object: "obj", + claims: &claims, + deny_only: false, + }; + let prepared = prepare_session_policy(&args, false); + assert!(matches!(prepared, PreparedSessionPolicy::Policy(_))); + outcomes.push(evaluate_prepared_session_policy(&prepared, &args).await); + } + + assert_eq!(outcomes, [Some(true), Some(false)]); + } + #[tokio::test] async fn test_prepare_auth_service_account_inherited_ignores_session_policy_tag_hint() { let store = StsTestMockStore::new(false); diff --git a/crates/io-metrics/Cargo.toml b/crates/io-metrics/Cargo.toml index 0591aa4e7..5cfd022a5 100644 --- a/crates/io-metrics/Cargo.toml +++ b/crates/io-metrics/Cargo.toml @@ -30,6 +30,7 @@ harness = false [features] default = [] +diagnostic-read-version-caller = [] hotpath = [ "hotpath/hotpath", "hotpath/tokio", diff --git a/crates/io-metrics/benches/metrics_pipeline.rs b/crates/io-metrics/benches/metrics_pipeline.rs index fb1aa6857..3f213e586 100644 --- a/crates/io-metrics/benches/metrics_pipeline.rs +++ b/crates/io-metrics/benches/metrics_pipeline.rs @@ -1,13 +1,15 @@ use criterion::{Criterion, criterion_group, criterion_main}; -use rustfs_io_metrics::{MetricsCollector, PerformanceMetrics, record_get_object_request_started}; +use rustfs_io_metrics::{MetricsCollector, PerformanceMetrics, record_get_object_timeout_operation_start}; use rustfs_io_metrics::{record_s3_op, s3_http_metrics::S3HttpRequestGuard}; use rustfs_s3_ops::S3Operation; use std::hint::black_box; use std::sync::Arc; use std::time::Duration; -fn bench_record_get_object_request_started(c: &mut Criterion) { - c.bench_function("record_get_object_request_started", |b| b.iter(record_get_object_request_started)); +fn bench_record_get_object_timeout_operation_start(c: &mut Criterion) { + c.bench_function("record_get_object_timeout_operation_start", |b| { + b.iter(record_get_object_timeout_operation_start) + }); } fn bench_s3_http_outcomes(c: &mut Criterion) { @@ -49,7 +51,7 @@ fn bench_metrics_collector_record_io_operation(c: &mut Criterion) { criterion_group!( benches, - bench_record_get_object_request_started, + bench_record_get_object_timeout_operation_start, bench_s3_http_outcomes, bench_update_concurrent_requests, bench_metrics_collector_record_io_operation diff --git a/crates/io-metrics/src/internode_metrics.rs b/crates/io-metrics/src/internode_metrics.rs index 2bc9b1ac1..d677cba5a 100644 --- a/crates/io-metrics/src/internode_metrics.rs +++ b/crates/io-metrics/src/internode_metrics.rs @@ -12,7 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::s3_http_metrics::S3HttpOperationContext; use metrics::{counter, gauge}; +#[cfg(not(test))] +use rustfs_s3_ops::S3Operation; use std::collections::HashMap; use std::sync::{ Arc, LazyLock, OnceLock, RwLock, @@ -65,6 +68,340 @@ pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_DECODE: &str = "batch_read pub const INTERNODE_STAGE_BATCH_READ_VERSION_COALESCER_WAIT: &str = "batch_read_version_coalescer_wait"; pub const INTERNODE_STAGE_BATCH_READ_VERSION_RESPONSE_MAP: &str = "batch_read_version_response_map"; +const INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL: &str = + "rustfs_system_network_internode_read_version_requests_by_caller_total"; + +/// An attribution dimension is absent, common to all items, or heterogeneous. +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadVersionAttribution { + Absent, + Common(T), + Mixed, +} + +/// Storage entry names describe call paths, not proof of an external request. +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadVersionEntrySource { + Unclassified, + ConfigRead, + ConfigWrite, + ScannerConfigWrite, + WritePreconditionRead, + ObjectInfoRead, + AppPutOwner, + PutCommitOwner, + LifecycleRecovery, + RecoveryDispositionMaintenance, + StoreObjectReader, + DataUsageCacheRead, + ServerInfoPools, + ScannerBacklogRead, + ScannerRevisionRead, + ScannerCycleStateRead, + TierConfigReload, + ObjectRead, + ListObjects, + MetadataRead, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionEntrySource { + pub const fn as_str(self) -> &'static str { + match self { + Self::Unclassified => "unclassified", + Self::ConfigRead => "config_read", + Self::ConfigWrite => "config_write", + Self::ScannerConfigWrite => "scanner_config_write", + Self::WritePreconditionRead => "write_precondition_read", + Self::ObjectInfoRead => "object_info_read", + Self::AppPutOwner => "app_put_owner", + Self::PutCommitOwner => "put_commit_owner", + Self::LifecycleRecovery => "lifecycle_recovery", + Self::RecoveryDispositionMaintenance => "recovery_disposition_maintenance", + Self::StoreObjectReader => "store_object_reader", + Self::DataUsageCacheRead => "data_usage_cache_read", + Self::ServerInfoPools => "server_info_pools", + Self::ScannerBacklogRead => "scanner_backlog_read", + Self::ScannerRevisionRead => "scanner_revision_read", + Self::ScannerCycleStateRead => "scanner_cycle_state_read", + Self::TierConfigReload => "tier_config_reload", + Self::ObjectRead => "object_read", + Self::ListObjects => "list_objects", + Self::MetadataRead => "metadata_read", + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +#[derive(Clone, Copy, Debug)] +pub enum ReadVersionContextBoundary { + Entry, + FullWait, + EarlyStop, + ListProducer, + ListGather, + CoalescerEnqueue, + UnaryFallback, + EntryToRpc, + RemoteRpc, + ScannerCacheLoad, + ScannerCycleStateLoad, + ConfigWrite, + ScannerConfigWrite, + WritePreconditionLookup, + ObjectInfoRead, + AppPutOwner, + PutCommitOwner, + TierFreeVersionSourceLookup, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +impl ReadVersionContextBoundary { + const fn as_str(self) -> &'static str { + match self { + Self::Entry => "entry", + Self::FullWait => "full_wait", + Self::EarlyStop => "early_stop", + Self::ListProducer => "list_producer", + Self::ListGather => "list_gather", + Self::CoalescerEnqueue => "coalescer_enqueue", + Self::UnaryFallback => "unary_fallback", + Self::EntryToRpc => "entry_to_rpc", + Self::RemoteRpc => "remote_rpc", + Self::ScannerCacheLoad => "scanner_cache_load", + Self::ScannerCycleStateLoad => "scanner_cycle_state_load", + Self::ConfigWrite => "config_write", + Self::ScannerConfigWrite => "scanner_config_write", + Self::WritePreconditionLookup => "write_precondition_lookup", + Self::ObjectInfoRead => "object_info_read", + Self::AppPutOwner => "app_put_owner", + Self::PutCommitOwner => "put_commit_owner", + Self::TierFreeVersionSourceLookup => "tier_free_version_source_lookup", + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +fn context_transition(parent: Option, child: Option) -> &'static str { + match (parent, child) { + (None, None) => "absent", + (Some(_), None) => "lost", + (None, Some(_)) => "gained", + (Some(parent), Some(child)) if parent == child => "preserved", + (Some(_), Some(_)) => "changed", + } +} + +/// Count observations separately from RPCs, using only bounded context labels. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn record_read_version_context_transition( + source: ReadVersionEntrySource, + boundary: ReadVersionContextBoundary, + parent_api: Option, + child_api: Option, + parent_caller: Option, + child_caller: Option, + mode: Option, +) { + if !crate::get_stage_metrics_enabled() { + return; + } + counter!( + "rustfs_storage_diagnostic_context_transitions_total", + "entry_source" => source.as_str(), + "boundary" => boundary.as_str(), + "parent_api_operation" => parent_api.map(S3HttpOperationContext::operation_name).unwrap_or("unattributed"), + "child_api_operation" => child_api.map(S3HttpOperationContext::operation_name).unwrap_or("unattributed"), + "parent_caller" => parent_caller.map(ReadVersionRequestCaller::as_str).unwrap_or("unattributed"), + "child_caller" => child_caller.map(ReadVersionRequestCaller::as_str).unwrap_or("unattributed"), + "mode" => mode.map(ReadVersionRequestMode::as_str).unwrap_or("not_applicable"), + "api_context" => if source == ReadVersionEntrySource::Unclassified && matches!(boundary, ReadVersionContextBoundary::RemoteRpc) { + "unobserved" + } else { context_transition(parent_api, child_api) }, + "caller_context" => if source == ReadVersionEntrySource::Unclassified && matches!(boundary, ReadVersionContextBoundary::RemoteRpc) { + "unobserved" + } else { context_transition(parent_caller, child_caller) }, + ) + .increment(1); +} + +/// Count a native batch RPC once; item observations and unary fallback are separate. +#[cfg(feature = "diagnostic-read-version-caller")] +pub fn record_batch_read_version_request_by_caller( + operation: ReadVersionAttribution, + caller: ReadVersionAttribution, + source: ReadVersionAttribution, + mode: ReadVersionRequestMode, +) { + let api_label = match operation { + ReadVersionAttribution::Absent => "unattributed", + ReadVersionAttribution::Common(operation) => operation.operation_name(), + ReadVersionAttribution::Mixed => "mixed", + }; + let caller_label = match caller { + ReadVersionAttribution::Absent => "unattributed", + ReadVersionAttribution::Common(caller) => caller.as_str(), + ReadVersionAttribution::Mixed => "mixed", + }; + let source_label = match source { + ReadVersionAttribution::Absent => "unclassified", + ReadVersionAttribution::Common(source) => source.as_str(), + ReadVersionAttribution::Mixed => "mixed", + }; + counter!( + "rustfs_system_network_internode_batch_read_version_requests_by_caller_total", + "api_operation" => api_label, + "caller" => caller_label, + "entry_source" => source_label, + "mode" => mode.as_str(), + ) + .increment(1); +} +#[cfg(not(test))] +const READ_VERSION_REQUEST_MODE_COUNT: usize = 5; +const READ_VERSION_REQUEST_CALLER_COUNT: usize = 18; + +#[cfg(not(test))] +static READ_VERSION_CALLER_COUNTERS: LazyLock]>> = LazyLock::new(|| { + std::iter::repeat_with(OnceLock::new) + .take((S3Operation::ALL.len() + 1) * READ_VERSION_REQUEST_MODE_COUNT * READ_VERSION_REQUEST_CALLER_COUNT) + .collect::>() + .into_boxed_slice() +}); + +/// Coarse read mode for diagnostic attribution of remote metadata reads. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ReadVersionRequestMode { + MetadataOnly, + Data, + Healing, + IncludeFreeVersions, + Combined, +} + +/// Bounded internal call-site class for an outgoing ReadVersion RPC. +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +#[repr(usize)] +pub enum ReadVersionRequestCaller { + #[default] + Unattributed, + Peer, + Startup, + Heal, + ListObjects, + Object, + MetadataRead, + Multipart, + DiskStore, + DiskDispatch, + SetDiskRead, + SetDiskIo, + Rebalance, + Decommission, + Lifecycle, + TierRecovery, + CacheValue, + Other, +} + +impl ReadVersionRequestCaller { + pub const ALL: [Self; READ_VERSION_REQUEST_CALLER_COUNT] = [ + Self::Unattributed, + Self::Peer, + Self::Startup, + Self::Heal, + Self::ListObjects, + Self::Object, + Self::MetadataRead, + Self::Multipart, + Self::DiskStore, + Self::DiskDispatch, + Self::SetDiskRead, + Self::SetDiskIo, + Self::Rebalance, + Self::Decommission, + Self::Lifecycle, + Self::TierRecovery, + Self::CacheValue, + Self::Other, + ]; + + pub const fn as_str(self) -> &'static str { + match self { + Self::Unattributed => "unattributed", + Self::Peer => "peer", + Self::Startup => "startup", + Self::Heal => "heal", + Self::ListObjects => "list_objects", + Self::Object => "object", + Self::MetadataRead => "metadata_read", + Self::Multipart => "multipart", + Self::DiskStore => "disk_store", + Self::DiskDispatch => "disk_dispatch", + Self::SetDiskRead => "set_disk_read", + Self::SetDiskIo => "set_disk_io", + Self::Rebalance => "rebalance", + Self::Decommission => "decommission", + Self::Lifecycle => "lifecycle", + Self::TierRecovery => "tier_recovery", + Self::CacheValue => "cache_value", + Self::Other => "other", + } + } +} + +impl ReadVersionRequestMode { + const fn as_str(self) -> &'static str { + match self { + Self::MetadataOnly => "metadata_only", + Self::Data => "data", + Self::Healing => "healing", + Self::IncludeFreeVersions => "include_free_versions", + Self::Combined => "combined", + } + } +} + +/// Record a bounded diagnostic caller class for one outgoing ReadVersion RPC. +/// The caller label is either a fixed S3 operation name or `unattributed`. +pub fn record_read_version_request_by_caller( + operation: Option, + caller: ReadVersionRequestCaller, + mode: ReadVersionRequestMode, +) { + let api_operation = operation + .map(S3HttpOperationContext::operation_name) + .unwrap_or("unattributed"); + + #[cfg(test)] + counter!( + INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL, + "api_operation" => api_operation, + "caller" => caller.as_str(), + "mode" => mode.as_str(), + ) + .increment(1); + + #[cfg(not(test))] + { + let caller_index = operation.map_or(S3Operation::ALL.len(), |context| context.operation_index()); + let counter_index = (caller_index * READ_VERSION_REQUEST_MODE_COUNT + mode as usize) * READ_VERSION_REQUEST_CALLER_COUNT + + caller as usize; + READ_VERSION_CALLER_COUNTERS[counter_index] + .get_or_init(|| { + counter!( + INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL, + "api_operation" => api_operation, + "caller" => caller.as_str(), + "mode" => mode.as_str(), + ) + }) + .increment(1); + } +} + const OPERATION_LABEL: &str = "operation"; const BACKEND_LABEL: &str = "backend"; const SERVER_LABEL: &str = "server"; @@ -1231,6 +1568,61 @@ mod tests { assert_eq!(snapshot.replay_cache_evictions_total, 3); } + #[test] + fn read_version_caller_metric_uses_only_bounded_labels() { + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + with_local_recorder(&recorder, || { + let mut request = crate::s3_http_metrics::S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + crate::s3_http_metrics::observe_s3_http_operation(rustfs_s3_ops::S3Operation::GetObject); + crate::s3_http_metrics::current_s3_http_operation_context() + }) + .expect("S3 request should expose its dispatched operation"); + record_read_version_request_by_caller( + Some(operation), + ReadVersionRequestCaller::SetDiskRead, + ReadVersionRequestMode::Data, + ); + record_read_version_request_by_caller(None, ReadVersionRequestCaller::Peer, ReadVersionRequestMode::MetadataOnly); + }); + + let rows = snapshotter + .snapshot() + .into_vec() + .into_iter() + .filter(|(composite, _, _, _)| composite.key().name() == INTERNODE_READ_VERSION_REQUESTS_BY_CALLER_TOTAL) + .collect::>(); + assert_eq!(rows.len(), 2); + let labels = rows + .iter() + .map(|(composite, _, _, _)| { + composite + .key() + .labels() + .map(|label| (label.key().to_string(), label.value().to_string())) + .collect::>() + }) + .collect::>(); + assert!(labels.iter().any(|labels| { + labels.get("api_operation").map(String::as_str) == Some("s3:GetObject") + && labels.get("caller").map(String::as_str) == Some("set_disk_read") + && labels.get("mode").map(String::as_str) == Some("data") + })); + assert!(labels.iter().any(|labels| { + labels.get("api_operation").map(String::as_str) == Some("unattributed") + && labels.get("mode").map(String::as_str) == Some("metadata_only") + && labels.get("caller").map(String::as_str) == Some("peer") + })); + assert!(labels.iter().all(|labels| { + labels.len() == 3 + && labels + .keys() + .all(|key| key == "api_operation" || key == "caller" || key == "mode") + })); + } + #[test] fn operation_stage_duration_records_low_cardinality_stage_labels() { let recorder = DebuggingRecorder::new(); diff --git a/crates/io-metrics/src/lib.rs b/crates/io-metrics/src/lib.rs index f499c0f02..7fba337bb 100644 --- a/crates/io-metrics/src/lib.rs +++ b/crates/io-metrics/src/lib.rs @@ -49,8 +49,9 @@ #[macro_use] extern crate metrics; +use std::collections::HashMap; use std::sync::{ - Mutex, + LazyLock, Mutex, atomic::{AtomicBool, AtomicU64, Ordering}, }; @@ -61,6 +62,25 @@ use std::sync::{ /// Enabled only through an explicit runtime opt-in. static PUT_STAGE_METRICS_ENABLED: AtomicBool = AtomicBool::new(false); static GET_STAGE_METRICS_ENABLED: AtomicBool = AtomicBool::new(false); +static GET_STAGE_LOCAL_SUMMARY_ENABLED: AtomicBool = AtomicBool::new(false); +static GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: AtomicU64 = AtomicU64::new(64); +static GET_STAGE_LOCAL_SUMMARY_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +type GetStageLocalSummaryKey = (&'static str, &'static str, &'static str, &'static str); + +/// Sampled local aggregate for one bounded GetObject stage label set. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct GetStageLocalSummary { + pub path: &'static str, + pub stage: &'static str, + pub object_class: &'static str, + pub size_bucket: &'static str, + pub sampled_count: u64, + pub duration_nanoseconds: u64, +} + +static GET_STAGE_LOCAL_SUMMARY: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); /// Global switch for all remaining (non GET/PUT-stage) metric emission in this /// crate's free `record_*` functions — I/O scheduler, bytes-pool, zero-copy, @@ -87,6 +107,70 @@ pub fn set_get_stage_metrics_enabled(enabled: bool) { GET_STAGE_METRICS_ENABLED.store(enabled, Ordering::Relaxed); } +pub fn set_get_stage_local_summary_enabled(enabled: bool) { + GET_STAGE_LOCAL_SUMMARY_ENABLED.store(enabled, Ordering::Relaxed); +} + +pub fn set_get_stage_local_summary_sample_rate(sample_rate: u64) { + GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE.store(sample_rate.max(1), Ordering::Relaxed); +} + +pub fn get_stage_local_summary_enabled() -> bool { + GET_STAGE_LOCAL_SUMMARY_ENABLED.load(Ordering::Relaxed) +} + +pub fn get_stage_local_summary_sample_rate() -> u64 { + GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE.load(Ordering::Relaxed) +} + +pub fn take_get_stage_local_summary() -> Vec { + let Ok(mut summary) = GET_STAGE_LOCAL_SUMMARY.lock() else { + return Vec::new(); + }; + let mut rows = summary.drain().map(|(_, row)| row).collect::>(); + rows.sort_unstable_by_key(|row| (row.path, row.stage, row.object_class, row.size_bucket)); + rows +} + +fn record_get_stage_local_summary( + path: &'static str, + stage: &'static str, + object_class: &'static str, + size_bucket: &'static str, + duration_secs: f64, +) { + let sample_rate = get_stage_local_summary_sample_rate(); + if sample_rate > 1 + && !GET_STAGE_LOCAL_SUMMARY_SEQUENCE + .fetch_add(1, Ordering::Relaxed) + .is_multiple_of(sample_rate) + { + return; + } + + let Ok(duration) = std::time::Duration::try_from_secs_f64(duration_secs) else { + return; + }; + let Ok(duration_nanoseconds) = u64::try_from(duration.as_nanos()) else { + return; + }; + + let key = (path, stage, object_class, size_bucket); + let Ok(mut summary) = GET_STAGE_LOCAL_SUMMARY.lock() else { + return; + }; + let row = summary.entry(key).or_insert_with(|| GetStageLocalSummary { + path, + stage, + object_class, + size_bucket, + sampled_count: 0, + duration_nanoseconds: 0, + }); + row.sampled_count = row.sampled_count.saturating_add(1); + row.duration_nanoseconds = row.duration_nanoseconds.saturating_add(duration_nanoseconds); +} + /// Enable or disable general (non GET/PUT-stage) metric emission. /// /// Called once during startup, typically gated by `rustfs_obs::observability_metric_enabled()`. @@ -110,8 +194,210 @@ pub fn put_stage_timer() -> Option { } pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT: &str = "put_object_commit_namespace_lock_wait"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD: &str = "put_object_commit_namespace_lock_held"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME: &str = "put_object_commit_namespace_lock_held_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_PRE_RENAME_PREPARE: &str = + "put_object_commit_namespace_lock_held_pre_rename_prepare"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME_RESULT_TO_TAIL_HANDOFF: &str = + "put_object_commit_namespace_lock_held_rename_result_to_tail_handoff"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_TAIL_HANDOFF: &str = + "put_object_commit_namespace_lock_held_tail_handoff"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_RENAME: &str = "put_object_commit_namespace_lock_held_post_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_GUARD_RELEASE: &str = + "put_object_commit_namespace_lock_held_guard_release"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE: &str = + "put_object_commit_namespace_lock_held_quota_fence_release"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_COMMIT: &str = + "put_object_commit_namespace_lock_held_quota_commit"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT: &str = "put_object_commit_namespace_lock_held_heal_submit"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT: &str = + "put_object_commit_namespace_lock_held_cleanup_receipt"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_METADATA_INVALIDATE: &str = + "put_object_commit_namespace_lock_held_metadata_invalidate"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_OBJECT_INFO_BUILD: &str = + "put_object_commit_namespace_lock_held_object_info_build"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_OBJECT_INFO_TO_RETURN: &str = + "put_object_commit_namespace_lock_held_post_object_info_to_return"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT: &str = "put_object_commit_task_wait"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL: &str = "put_object_commit_future_total"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT: &str = "put_object_commit_operation_select_wait"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL: &str = "put_object_operation_await_total"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT: &str = "put_object_operation_body_to_commit_submit"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN: &str = "put_object_operation_result_return"; +pub const PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY: &str = "put_object_operation_first_poll_delay"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY: &str = + "put_object_commit_namespace_lock_acquired_to_commit_ready"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_PRECONDITION_AND_TIMESTAMP: &str = + "put_object_commit_ready_precondition_and_timestamp"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPECTED_VERSION_LOOKUP: &str = "put_object_commit_ready_expected_version_lookup"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPLICIT_VERSION_LOOKUP: &str = "put_object_commit_ready_explicit_version_lookup"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_RESTORE_VERIFY: &str = "put_object_commit_ready_restore_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_FENCE_VERIFY: &str = "put_object_commit_ready_fence_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_DECOMMISSION_CAPACITY_FENCE: &str = + "put_object_commit_ready_decommission_capacity_fence"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_TRANSACTION_EPOCH_FENCE_READ: &str = + "put_object_commit_ready_transaction_epoch_fence_read"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_BEGIN: &str = "put_object_commit_ready_quota_begin"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_METADATA_ADJUST: &str = "put_object_commit_ready_metadata_adjust"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_RESERVE: &str = "put_object_commit_ready_quota_reserve"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_MUTATION_FENCE_PREPARE: &str = + "put_object_commit_ready_quota_mutation_fence_prepare"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_READY_CONTEXT_PREPARE: &str = "put_object_commit_ready_context_prepare"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH: &str = "put_object_quota_begin_meta_bucket_fast_path"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK: &str = "put_object_quota_begin_metadata_lock"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP: &str = "put_object_quota_begin_cache_lookup"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ: &str = "put_object_quota_begin_config_read"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK: &str = "put_object_quota_begin_lock_lost_check"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE: &str = "put_object_quota_begin_policy_evaluate"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CAPABILITY_PROOF: &str = "put_object_quota_begin_capability_proof"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION: &str = "put_object_quota_begin_snapshot_admission"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP: &str = "put_object_quota_begin_object_store_lookup"; +pub const PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD: &str = "put_object_quota_begin_context_build"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER: &str = "put_object_commit_submit_to_closure_enter"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT: &str = "put_object_commit_closure_enter_to_first_await"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL: &str = "put_object_commit_closure_total"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME: &str = "put_object_commit_before_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_AWAIT_TOTAL: &str = "put_object_commit_pre_rename_await_total"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_QUOTA_MARK_COMMIT_STARTED: &str = + "put_object_commit_pre_rename_quota_mark_commit_started"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_FIRST_FENCE_CHECK: &str = "put_object_commit_pre_rename_first_fence_check"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_RESTORE_VERIFY: &str = "put_object_commit_pre_rename_restore_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TRANSACTION_FENCE_VERIFY: &str = + "put_object_commit_pre_rename_transaction_fence_verify"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SECOND_FENCE_CHECK: &str = "put_object_commit_pre_rename_second_fence_check"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SCANNER_SCOPE_CHECK: &str = "put_object_commit_pre_rename_scanner_scope_check"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TIER_FREE_VERSION_SOURCE_LOOKUP: &str = + "put_object_commit_pre_rename_tier_free_version_source_lookup"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_ASSIGN_RENAME_INDEXES: &str = + "put_object_commit_pre_rename_assign_rename_indexes"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_POST_AWAIT_TO_RENAME: &str = "put_object_commit_pre_rename_post_await_to_rename"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN: &str = "put_object_commit_after_rename_to_return"; pub const PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT: &str = "set_disk_rename_quorum_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_SYNC_FULL_FANOUT_WAIT: &str = "set_disk_rename_sync_full_fanout_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT: &str = "set_disk_rename_early_ack_spawn_to_quorum_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_ATTACH_TAIL_OWNER: &str = "set_disk_rename_early_ack_attach_tail_owner"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_AFTER_QUORUM_WAIT: &str = "set_disk_rename_tail_after_quorum_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT: &str = "set_disk_rename_tail_async_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT: &str = "set_disk_rename_tail_guard_release_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_FINALIZE: &str = "set_disk_rename_tail_finalize"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP: &str = "set_disk_rename_tail_cleanup"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT: &str = "set_disk_rename_tail_cleanup_submit"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP: &str = "set_disk_rename_tail_cleanup_counterfactual_skip"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED: &str = "set_disk_rename_tail_cleanup_defer_enqueued"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_FALLBACK_SYNC: &str = "set_disk_rename_tail_cleanup_defer_fallback_sync"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH: &str = "set_disk_rename_tail_cleanup_defer_queue_push"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH: &str = "set_disk_rename_tail_cleanup_defer_queue_depth"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_defer_queue_depth_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE: &str = "set_disk_rename_tail_cleanup_defer_oldest_age"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_defer_oldest_age_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_TARGETS: &str = "set_disk_rename_tail_cleanup_defer_targets"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_START_DELAY: &str = "set_disk_rename_tail_cleanup_defer_start_delay"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED_LAG: &str = "set_disk_rename_tail_cleanup_defer_completed_lag"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER: &str = "set_disk_rename_tail_cleanup_defer_hold_worker"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY: &str = "set_disk_rename_tail_cleanup_defer_worker_delay"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_PERMIT_WAIT: &str = "set_disk_rename_tail_cleanup_per_disk_permit_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_INFLIGHT: &str = "set_disk_rename_tail_cleanup_per_disk_inflight"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD: &str = "set_disk_rename_tail_cleanup_foreground_yield"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFERRED_FOREGROUND_ACTIVE: &str = + "set_disk_rename_tail_cleanup_deferred_foreground_active"; +pub const PUT_STAGE_SET_DISK_RENAME_FOREGROUND_ACTIVE: &str = "set_disk_rename_foreground_active"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW: &str = "set_disk_rename_tail_cleanup_defer_batch_window"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_SIZE: &str = "set_disk_rename_tail_cleanup_defer_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BURST_BATCH_SIZE: &str = + "set_disk_rename_tail_cleanup_defer_burst_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_ACTIVE: &str = "set_disk_rename_tail_cleanup_defer_worker_active"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_POLL_WAIT: &str = + "set_disk_rename_tail_cleanup_defer_worker_poll_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_IDLE_WAIT: &str = + "set_disk_rename_tail_cleanup_defer_worker_idle_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_JOB_WAIT: &str = + "set_disk_rename_tail_cleanup_defer_worker_job_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED: &str = "set_disk_rename_tail_cleanup_defer_completed"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DURATION: &str = + "set_disk_rename_tail_cleanup_defer_worker_duration"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_RECEIPT: &str = "set_disk_rename_tail_cleanup_target_receipt"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COMMIT_CAPACITY: &str = + "set_disk_rename_tail_cleanup_target_commit_capacity"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_REPORT: &str = "set_disk_rename_tail_cleanup_target_report"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COUNT: &str = "set_disk_rename_tail_cleanup_target_count"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_ZERO_TARGET: &str = "set_disk_rename_tail_cleanup_zero_target"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE: &str = "set_disk_rename_tail_cleanup_tmp_delete"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP: &str = + "set_disk_rename_tail_cleanup_tmp_delete_counterfactual_skip"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DECOMMISSION_GUARD_DROP: &str = + "set_disk_rename_tail_cleanup_decommission_guard_drop"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH: &str = "set_disk_rename_tail_cleanup_lane_queue_depth"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH: &str = "set_disk_rename_tail_cleanup_lane_queue_push"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_lane_queue_depth_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE: &str = "set_disk_rename_tail_cleanup_lane_oldest_age"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE: &str = + "set_disk_rename_tail_cleanup_lane_oldest_age_max_sample"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BATCH_SIZE: &str = "set_disk_rename_tail_cleanup_lane_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BURST_BATCH_SIZE: &str = + "set_disk_rename_tail_cleanup_lane_burst_batch_size"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_ACTIVE: &str = "set_disk_rename_tail_cleanup_lane_worker_active"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_POLL_WAIT: &str = + "set_disk_rename_tail_cleanup_lane_worker_poll_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_IDLE_WAIT: &str = + "set_disk_rename_tail_cleanup_lane_worker_idle_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_JOB_WAIT: &str = "set_disk_rename_tail_cleanup_lane_worker_job_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_SERVICE_TIME: &str = "set_disk_rename_tail_cleanup_lane_service_time"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_REQUEUE: &str = "set_disk_rename_tail_cleanup_lane_requeue"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_FOREGROUND_ACTIVE_HIT: &str = + "set_disk_rename_tail_cleanup_lane_foreground_active_hit"; +pub const PUT_STAGE_SET_DISK_RENAME_TAIL_HEAL_SUBMIT: &str = "set_disk_rename_tail_heal_submit"; pub const PUT_STAGE_SET_DISK_RENAME_DISK_WAIT: &str = "set_disk_rename_disk_wait"; +pub const PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL: &str = "set_disk_rename_disk_wait_local"; +pub const PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE: &str = "set_disk_rename_disk_wait_remote"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE: &str = "set_disk_rename_remote_client_prepare"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC: &str = "set_disk_rename_remote_client_rpc"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT: &str = "set_disk_rename_remote_client_rpc_await"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT: &str = + "set_disk_rename_remote_client_rpc_await_poll_pending_count"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY: &str = + "set_disk_rename_remote_client_rpc_await_first_pending_to_ready"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE: &str = "set_disk_rename_remote_client_request_scope"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL: &str = "set_disk_rename_remote_client_transport_call"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT: &str = + "set_disk_rename_remote_client_transport_call_poll_pending_count"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY: &str = + "set_disk_rename_remote_client_transport_call_first_pending_to_ready"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE: &str = "set_disk_rename_remote_client_replay_response"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER: &str = "set_disk_rename_remote_client_into_inner"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK: &str = "set_disk_rename_remote_client_success_check"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE: &str = "set_disk_rename_remote_client_decode"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS: &str = "set_disk_rename_remote_client_response_process"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS: &str = "set_disk_rename_remote_client_response_headers"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_ENCODE: &str = "set_disk_rename_remote_service_response_encode"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_TRIGGERED: &str = + "put_object_commit_namespace_lock_held_quota_fence_release_triggered"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_SKIPPED: &str = + "put_object_commit_namespace_lock_held_quota_fence_release_skipped"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_TRIGGERED: &str = + "put_object_commit_namespace_lock_held_heal_submit_triggered"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_SKIPPED: &str = + "put_object_commit_namespace_lock_held_heal_submit_skipped"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_TRIGGERED: &str = + "put_object_commit_namespace_lock_held_cleanup_receipt_triggered"; +pub const PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_SKIPPED: &str = + "put_object_commit_namespace_lock_held_cleanup_receipt_skipped"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL: &str = "set_disk_rename_remote_handler_total"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD: &str = "set_disk_rename_remote_handler_overhead"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP: &str = "set_disk_rename_remote_handler_lookup"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT: &str = "set_disk_rename_remote_handler_admit"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME: &str = "set_disk_rename_remote_handler_local_rename"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY: &str = "set_disk_rename_remote_service_entry"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER: &str = "set_disk_rename_remote_service_before_handler"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_HANDLER_AWAIT: &str = "set_disk_rename_remote_service_handler_await"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_AFTER_HANDLER: &str = "set_disk_rename_remote_service_after_handler"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD: &str = "set_disk_rename_remote_service_response_build"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN: &str = "set_disk_rename_remote_service_response_return"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_SERVICE_FUTURE: &str = "set_disk_rename_remote_transport_service_future"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_FIRST_FRAME: &str = "set_disk_rename_remote_transport_first_frame"; +pub const PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_BODY_COMPLETE: &str = "set_disk_rename_remote_transport_body_complete"; pub const PUT_STAGE_SET_DISK_RENAME_FILE_SYNC_PERMIT_WAIT: &str = "set_disk_rename_file_sync_permit_wait"; pub const PUT_STAGE_SET_DISK_RENAME_GLOBAL_FILE_SYNC_PERMIT_WAIT: &str = "set_disk_rename_global_file_sync_permit_wait"; pub const PUT_STAGE_SET_DISK_RENAME_FILE_FDATASYNC: &str = "set_disk_rename_file_fdatasync"; @@ -260,8 +546,8 @@ pub use io_metrics::{ pub use list_objects_metrics::{ LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, LIST_OBJECTS_LOCAL_READ_DIR_OUTCOME_ERROR, LIST_OBJECTS_LOCAL_READ_DIR_OUTCOME_OK, LIST_OBJECTS_SOURCE_WALKER, - ListObjectsGatherObservation, ListObjectsIndexPageObservation, ListObjectsLocalReadDirObservation, init_list_objects_metrics, - record_list_objects_gather, record_list_objects_index_attempt, record_list_objects_index_fallback, + ListObjectsGatherCaller, ListObjectsGatherObservation, ListObjectsIndexPageObservation, ListObjectsLocalReadDirObservation, + init_list_objects_metrics, record_list_objects_gather, record_list_objects_index_attempt, record_list_objects_index_fallback, record_list_objects_index_live_verify_failure, record_list_objects_index_served, record_list_objects_local_read_dir, record_list_objects_merge, }; @@ -460,7 +746,16 @@ pub fn record_get_object_request_start(concurrent_requests: usize) { gauge!("rustfs_io_get_object_concurrent_requests").set(concurrent_requests as f64); } -/// Record GetObject request start without concurrency context. +/// Record one timeout-wrapped GetObject storage operation start. +#[inline(always)] +pub fn record_get_object_timeout_operation_start() { + if !get_stage_metrics_enabled() { + return; + } + counter!("rustfs_io_get_object_timeout_operations_total").increment(1); +} + +/// Record a GetObject request start without concurrency context. #[inline(always)] pub fn record_get_object_request_started() { if !get_stage_metrics_enabled() { @@ -479,6 +774,16 @@ pub fn record_get_object_request_result(status: &str, duration_secs: f64) { histogram!("rustfs_io_get_object_request_duration_seconds", "status" => status.to_string()).record(duration_secs); } +/// Record the result of one timeout-wrapped GetObject storage operation. +#[inline(always)] +pub fn record_get_object_timeout_operation_result(status: &str, duration_secs: f64) { + if !get_stage_metrics_enabled() { + return; + } + counter!("rustfs_io_get_object_timeout_operation_results_total", "status" => status.to_string()).increment(1); + histogram!("rustfs_io_get_object_timeout_operation_duration_seconds", "status" => status.to_string()).record(duration_secs); +} + /// Record PutObject request start. #[inline(always)] pub fn record_put_object_request_start(concurrent_requests: usize) { @@ -760,6 +1065,10 @@ pub fn record_get_object_io_state( /// Record GetObject phase duration for the current read path. #[inline(always)] pub fn record_get_object_stage_duration(path: &'static str, stage: &'static str, duration_secs: f64) { + if get_stage_local_summary_enabled() { + record_get_stage_local_summary(path, stage, "", "", duration_secs); + return; + } if !get_stage_metrics_enabled() { return; } @@ -775,6 +1084,10 @@ pub fn record_get_object_stage_duration_by_size( size_bucket: &'static str, duration_secs: f64, ) { + if get_stage_local_summary_enabled() { + record_get_stage_local_summary(path, stage, object_class, size_bucket, duration_secs); + return; + } if !get_stage_metrics_enabled() { return; } @@ -827,6 +1140,15 @@ pub fn record_get_object_metadata_response(path: &'static str, outcome: &'static counter!("rustfs_io_get_object_metadata_response_total", "path" => path, "outcome" => outcome).increment(1); } +/// Record whether metadata resolution selected an object identity at read quorum. +#[inline(always)] +pub fn record_get_object_metadata_quorum_result(path: &'static str, outcome: &'static str) { + if !get_stage_metrics_enabled() { + return; + } + counter!("rustfs_io_get_object_metadata_quorum_results_total", "path" => path, "outcome" => outcome).increment(1); +} + /// Record one bounded metadata cache decision. #[inline(always)] pub fn record_get_object_metadata_cache_decision(path: &'static str, decision: &'static str, reason: &'static str) { @@ -1668,6 +1990,10 @@ pub fn record_get_object_metadata_phase_duration(duration_secs: f64) { /// Record metadata phase duration with early-stop state label. #[inline(always)] pub fn record_get_object_metadata_phase_duration_with_early_stop(duration_secs: f64, early_stop_active: &'static str) { + if get_stage_local_summary_enabled() { + record_get_stage_local_summary("legacy_duplex", "metadata", early_stop_active, "", duration_secs); + return; + } if !get_stage_metrics_enabled() { return; } @@ -2073,6 +2399,57 @@ pub fn record_put_object_stage_duration_from(stage: &'static str, started_at: Op } } +#[inline(always)] +pub fn record_put_object_stage_count(stage: &'static str, count: u64) { + if !put_stage_metrics_enabled() { + return; + } + counter!("rustfs_s3_put_object_stage_count", "stage" => stage).increment(count); +} + +/// Observe a future's latency and wakeup behavior only while detailed PUT +/// attribution is enabled. The disabled path awaits the original future +/// directly, so normal remote mutation RPCs do not gain a polling wrapper. +pub async fn observe_put_stage_future( + future: F, + duration_stage: &'static str, + pending_count_stage: &'static str, + first_pending_to_ready_stage: &'static str, +) -> F::Output +where + F: std::future::Future, +{ + if !put_stage_metrics_enabled() { + return future.await; + } + + let duration_started = put_stage_timer(); + let mut first_pending_at = None; + let mut pending_count = 0_u64; + let mut future = std::pin::pin!(future); + let output = std::future::poll_fn(|cx| match future.as_mut().poll(cx) { + std::task::Poll::Ready(output) => std::task::Poll::Ready(output), + std::task::Poll::Pending => { + pending_count = pending_count.saturating_add(1); + first_pending_at.get_or_insert_with(std::time::Instant::now); + std::task::Poll::Pending + } + }) + .await; + record_put_object_stage_duration_from(duration_stage, duration_started); + record_put_object_stage_count(pending_count_stage, pending_count); + record_put_object_stage_duration_from(first_pending_to_ready_stage, first_pending_at); + output +} + +#[inline(always)] +pub fn record_put_object_quota_cache_decision(decision: &'static str, reason: &'static str) { + if !put_stage_metrics_enabled() { + return; + } + counter!("rustfs_s3_put_object_quota_cache_total", "decision" => decision, "reason" => reason).increment(1); +} + #[inline(always)] pub fn record_put_object_commit_lock_admission(budget: &'static str, outcome: &'static str) { if !put_stage_metrics_enabled() { @@ -3219,8 +3596,162 @@ mod tests { assert_eq!(PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT, "put_object_commit_namespace_lock_wait"); let stages = [ PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_PRE_RENAME_PREPARE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_RENAME_RESULT_TO_TAIL_HANDOFF, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_TAIL_HANDOFF, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_GUARD_RELEASE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_COMMIT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_METADATA_INVALIDATE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_OBJECT_INFO_BUILD, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_POST_OBJECT_INFO_TO_RETURN, + PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT, + PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL, + PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT, + PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL, + PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT, + PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN, + PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_PRECONDITION_AND_TIMESTAMP, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPECTED_VERSION_LOOKUP, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_EXPLICIT_VERSION_LOOKUP, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_RESTORE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_FENCE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_DECOMMISSION_CAPACITY_FENCE, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_TRANSACTION_EPOCH_FENCE_READ, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_BEGIN, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_METADATA_ADJUST, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_RESERVE, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_QUOTA_MUTATION_FENCE_PREPARE, + PUT_STAGE_PUT_OBJECT_COMMIT_READY_CONTEXT_PREPARE, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_META_BUCKET_FAST_PATH, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_METADATA_LOCK, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CACHE_LOOKUP, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONFIG_READ, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_LOCK_LOST_CHECK, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_POLICY_EVALUATE, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CAPABILITY_PROOF, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_SNAPSHOT_ADMISSION, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_OBJECT_STORE_LOOKUP, + PUT_STAGE_PUT_OBJECT_QUOTA_BEGIN_CONTEXT_BUILD, + PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER, + PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT, + PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL, + PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_AWAIT_TOTAL, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_QUOTA_MARK_COMMIT_STARTED, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_FIRST_FENCE_CHECK, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_RESTORE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TRANSACTION_FENCE_VERIFY, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SECOND_FENCE_CHECK, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_SCANNER_SCOPE_CHECK, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_TIER_FREE_VERSION_SOURCE_LOOKUP, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_ASSIGN_RENAME_INDEXES, + PUT_STAGE_PUT_OBJECT_COMMIT_PRE_RENAME_POST_AWAIT_TO_RENAME, + PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN, PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT, + PUT_STAGE_SET_DISK_RENAME_SYNC_FULL_FANOUT_WAIT, + PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT, + PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_ATTACH_TAIL_OWNER, + PUT_STAGE_SET_DISK_RENAME_TAIL_AFTER_QUORUM_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_GUARD_RELEASE_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_FINALIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_SUBMIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_ENQUEUED, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_FALLBACK_SYNC, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_PUSH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_QUEUE_DEPTH_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_OLDEST_AGE_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_TARGETS, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_START_DELAY, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED_LAG, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DELAY, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_PERMIT_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_PER_DISK_INFLIGHT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_FOREGROUND_YIELD, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFERRED_FOREGROUND_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_FOREGROUND_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_WINDOW, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_BURST_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_POLL_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_IDLE_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_JOB_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_COMPLETED, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DEFER_WORKER_DURATION, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_RECEIPT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COMMIT_CAPACITY, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_REPORT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TARGET_COUNT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_ZERO_TARGET, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_DECOMMISSION_GUARD_DROP, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_PUSH, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_QUEUE_DEPTH_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_OLDEST_AGE_MAX_SAMPLE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_BURST_BATCH_SIZE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_ACTIVE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_POLL_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_IDLE_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_WORKER_JOB_WAIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_SERVICE_TIME, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_REQUEUE, + PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_LANE_FOREGROUND_ACTIVE_HIT, + PUT_STAGE_SET_DISK_RENAME_TAIL_HEAL_SUBMIT, PUT_STAGE_SET_DISK_RENAME_DISK_WAIT, + PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_LOCAL, + PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_ENCODE, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_TRIGGERED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_QUOTA_FENCE_RELEASE_SKIPPED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_TRIGGERED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_HEAL_SUBMIT_SKIPPED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_TRIGGERED, + PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD_CLEANUP_RECEIPT_SKIPPED, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_TOTAL, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_OVERHEAD, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOOKUP, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_ADMIT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_HANDLER_LOCAL_RENAME, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_HANDLER_AWAIT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_AFTER_HANDLER, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD, + PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_SERVICE_FUTURE, + PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_FIRST_FRAME, + PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_BODY_COMPLETE, PUT_STAGE_SET_DISK_RENAME_FILE_SYNC_PERMIT_WAIT, PUT_STAGE_SET_DISK_RENAME_GLOBAL_FILE_SYNC_PERMIT_WAIT, PUT_STAGE_SET_DISK_RENAME_FILE_FDATASYNC, @@ -3233,11 +3764,36 @@ mod tests { let unique = stages.iter().copied().collect::>(); assert_eq!(unique.len(), stages.len()); assert!(stages.iter().all(|stage| { - (stage.starts_with("set_disk_rename_") || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT) + (stage.starts_with("set_disk_rename_") + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_WAIT + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_HELD + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_TASK_WAIT + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_FUTURE_TOTAL + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_OPERATION_SELECT_WAIT + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_AWAIT_TOTAL + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_BODY_TO_COMMIT_SUBMIT + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_RESULT_RETURN + || *stage == PUT_STAGE_PUT_OBJECT_OPERATION_FIRST_POLL_DELAY + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_NAMESPACE_LOCK_ACQUIRED_TO_COMMIT_READY + || stage.starts_with("put_object_commit_ready_") + || stage.starts_with("put_object_quota_begin_") + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_SUBMIT_TO_CLOSURE_ENTER + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_ENTER_TO_FIRST_AWAIT + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_CLOSURE_TOTAL + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_BEFORE_RENAME + || stage.starts_with("put_object_commit_pre_rename_") + || *stage == PUT_STAGE_PUT_OBJECT_COMMIT_AFTER_RENAME_TO_RETURN + || stage.starts_with("put_object_commit_namespace_lock_held_")) && !stage.contains('/') && !stage.contains('{') })); + let count_stages = [ + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT, + PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT, + ]; + assert_eq!(count_stages.iter().copied().collect::>().len(), count_stages.len()); + let recorder = DebuggingRecorder::new(); let snapshotter = recorder.snapshotter(); metrics::with_local_recorder(&recorder, || { @@ -3245,16 +3801,25 @@ mod tests { for stage in stages { record_put_object_stage_duration(stage, 1.0); } + for stage in count_stages { + record_put_object_stage_count(stage, 1); + } set_put_stage_metrics_enabled(true); for stage in stages { record_put_object_stage_duration(stage, 1.0); } + for stage in count_stages { + record_put_object_stage_count(stage, 1); + } set_put_stage_metrics_enabled(false); }); - let recorded = snapshotter - .snapshot() - .into_vec() + let rows = snapshotter.snapshot().into_vec(); + assert_eq!( + counter_total(&rows, "rustfs_s3_put_object_stage_count"), + Some(u64::try_from(count_stages.len()).expect("count stage length fits u64")), + ); + let recorded = rows .into_iter() .filter(|(composite, _, _, _)| { composite.kind() == MetricKind::Histogram && composite.key().name() == "rustfs_s3_put_object_stage_duration_ms" @@ -3349,6 +3914,63 @@ mod tests { ]))); } + #[test] + fn put_stage_future_observer_preserves_outputs_and_counts_pending() { + let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + let runtime = tokio::runtime::Builder::new_current_thread() + .build() + .expect("build current-thread runtime for observer test"); + let duration_stage = PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT; + let count_stage = PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT; + let pending_stage = PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY; + + metrics::with_local_recorder(&recorder, || { + set_put_stage_metrics_enabled(false); + let output = runtime.block_on(observe_put_stage_future( + std::future::ready(42), + duration_stage, + count_stage, + pending_stage, + )); + assert_eq!(output, 42, "disabled observer must preserve the future output"); + }); + assert!(snapshotter.snapshot().into_vec().is_empty(), "disabled observer must emit no metrics"); + + metrics::with_local_recorder(&recorder, || { + set_put_stage_metrics_enabled(true); + let mut pending = 2; + let future = std::future::poll_fn(|cx| { + if pending > 0 { + pending -= 1; + cx.waker().wake_by_ref(); + std::task::Poll::Pending + } else { + std::task::Poll::Ready(Err::<(), _>("original error")) + } + }); + let output = runtime.block_on(observe_put_stage_future(future, duration_stage, count_stage, pending_stage)); + set_put_stage_metrics_enabled(false); + assert_eq!(output, Err("original error"), "enabled observer must preserve errors"); + }); + + let rows = snapshotter.snapshot().into_vec(); + assert_eq!(counter_total(&rows, "rustfs_s3_put_object_stage_count"), Some(2)); + let count_labels = rows + .iter() + .filter(|(composite, _, _, _)| composite.key().name() == "rustfs_s3_put_object_stage_count") + .flat_map(|(composite, _, _, _)| composite.key().labels().map(|label| label.value().to_string())) + .collect::>(); + assert_eq!(count_labels, vec![count_stage.to_string()]); + let durations = rows + .iter() + .filter(|(composite, _, _, _)| composite.key().name() == "rustfs_s3_put_object_stage_duration_ms") + .flat_map(|(composite, _, _, _)| composite.key().labels().map(|label| label.value().to_string())) + .collect::>(); + assert_eq!(durations, HashSet::from([duration_stage.to_string(), pending_stage.to_string()])); + } + #[test] fn put_rename_code_level_metrics_are_static_and_gated() { let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); @@ -3578,6 +4200,35 @@ mod tests { set_put_stage_metrics_enabled(false); } + #[test] + fn get_timeout_operation_and_metadata_quorum_metrics_have_separate_scopes() { + let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + + metrics::with_local_recorder(&recorder, || { + set_get_stage_metrics_enabled(true); + record_get_object_request_start(1); + record_get_object_request_result("ok", 0.01); + record_get_object_timeout_operation_start(); + record_get_object_timeout_operation_result("success", 0.005); + record_get_object_metadata_quorum_result("legacy_duplex", "reached"); + record_get_object_metadata_quorum_result("legacy_duplex", "not_reached"); + set_get_stage_metrics_enabled(false); + }); + + let rows = snapshotter.snapshot().into_vec(); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_requests_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_request_results_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_timeout_operations_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_timeout_operation_results_total"), Some(1)); + assert_eq!(counter_total(&rows, "rustfs_io_get_object_metadata_quorum_results_total"), Some(2)); + assert_eq!( + histogram_samples(&rows, "rustfs_io_get_object_timeout_operation_duration_seconds"), + vec![0.005] + ); + } + #[test] fn test_record_get_object_path_and_stage() { let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner()); diff --git a/crates/io-metrics/src/list_objects_metrics.rs b/crates/io-metrics/src/list_objects_metrics.rs index b5f233557..3c8dd863d 100644 --- a/crates/io-metrics/src/list_objects_metrics.rs +++ b/crates/io-metrics/src/list_objects_metrics.rs @@ -22,7 +22,31 @@ pub const LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED: &str = "limit_reached"; pub const LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED: &str = "input_closed"; pub const LIST_OBJECTS_MERGE_OUTCOME_STARTED: &str = "started"; +/// Bounded caller class for a ListObjects walker gather. +#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] +pub enum ListObjectsGatherCaller { + #[default] + Unattributed, + S3List, + S3ListVersions, + Lifecycle, + Other, +} + +impl ListObjectsGatherCaller { + pub const fn as_str(self) -> &'static str { + match self { + Self::Unattributed => "unattributed", + Self::S3List => "s3_list", + Self::S3ListVersions => "s3_list_versions", + Self::Lifecycle => "lifecycle", + Self::Other => "other", + } + } +} + const LIST_OBJECTS_GATHER_TOTAL: &str = "rustfs_s3_list_objects_gather_total"; +const LIST_OBJECTS_GATHER_CALLER_TOTAL: &str = "rustfs_s3_list_objects_gather_caller_total"; const LIST_OBJECTS_GATHER_DURATION_MS: &str = "rustfs_s3_list_objects_gather_duration_ms"; const LIST_OBJECTS_GATHER_SCANNED_ENTRIES: &str = "rustfs_s3_list_objects_gather_scanned_entries"; const LIST_OBJECTS_GATHER_RETURNED_ENTRIES: &str = "rustfs_s3_list_objects_gather_returned_entries"; @@ -53,6 +77,7 @@ pub const LIST_OBJECTS_LOCAL_READ_DIR_OUTCOME_ERROR: &str = "error"; #[derive(Clone, Copy, Debug, PartialEq)] pub struct ListObjectsGatherObservation { pub source: &'static str, + pub caller: ListObjectsGatherCaller, pub outcome: &'static str, pub limit: i32, pub scanned_entries: usize, @@ -213,6 +238,7 @@ pub fn record_list_objects_gather(observation: ListObjectsGatherObservation) { } else { count_as_f64(observation.scanned_entries) / count_as_f64(observation.returned_entries) }; + let api_operation = crate::s3_http_metrics::current_s3_http_operation_name().unwrap_or("unattributed"); counter!( LIST_OBJECTS_GATHER_TOTAL, @@ -223,6 +249,14 @@ pub fn record_list_objects_gather(observation: ListObjectsGatherObservation) { "has_marker" => bool_label(observation.has_marker), ) .increment(1); + counter!( + LIST_OBJECTS_GATHER_CALLER_TOTAL, + "api_operation" => api_operation, + "caller" => observation.caller.as_str(), + "source" => observation.source, + "outcome" => observation.outcome, + ) + .increment(1); histogram!( LIST_OBJECTS_GATHER_DURATION_MS, "source" => observation.source, @@ -432,6 +466,7 @@ mod tests { let rows = recorded(|| { record_list_objects_gather(ListObjectsGatherObservation { source: LIST_OBJECTS_SOURCE_WALKER, + caller: ListObjectsGatherCaller::Unattributed, outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, limit: 1001, scanned_entries: 42, @@ -444,6 +479,17 @@ mod tests { }); assert_eq!(counter_total(&rows, LIST_OBJECTS_GATHER_TOTAL), Some(1)); + assert_eq!(counter_total(&rows, LIST_OBJECTS_GATHER_CALLER_TOTAL), Some(1)); + let caller_labels = rows + .iter() + .find(|(composite, _, _, _)| composite.key().name() == LIST_OBJECTS_GATHER_CALLER_TOTAL) + .expect("internal caller counter should be emitted") + .0 + .key() + .labels() + .map(|label| (label.key().to_string(), label.value().to_string())) + .collect::>(); + assert_eq!(caller_labels.get("api_operation").map(String::as_str), Some("unattributed")); // Zero returned entries must not divide: the amplification reports the // scan count itself rather than an infinity or a NaN. assert_eq!(histogram_samples(&rows, LIST_OBJECTS_GATHER_SCAN_AMPLIFICATION), vec![42.0]); @@ -452,6 +498,44 @@ mod tests { assert_eq!(histogram_samples(&rows, LIST_OBJECTS_GATHER_DURATION_MS), vec![3.5]); } + #[test] + fn gather_caller_metric_attributes_the_s3_api_operation() { + use rustfs_s3_ops::S3Operation; + + let rows = recorded(|| { + let mut request = crate::s3_http_metrics::S3HttpRequestGuard::new("GET"); + request.in_scope(|| { + crate::s3_http_metrics::observe_s3_http_operation(S3Operation::ListObjectsV2); + record_list_objects_gather(ListObjectsGatherObservation { + source: LIST_OBJECTS_SOURCE_WALKER, + caller: ListObjectsGatherCaller::S3List, + outcome: LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, + limit: 1000, + scanned_entries: 1, + returned_entries: 1, + duration_ms: 1.0, + has_prefix: false, + has_delimiter: false, + has_marker: false, + }); + }); + }); + + assert_eq!(counter_total(&rows, LIST_OBJECTS_GATHER_CALLER_TOTAL), Some(1)); + let labels = rows + .iter() + .find(|(composite, _, _, _)| composite.key().name() == LIST_OBJECTS_GATHER_CALLER_TOTAL) + .expect("caller counter should be emitted") + .0 + .key() + .labels() + .map(|label| (label.key().to_string(), label.value().to_string())) + .collect::>(); + assert_eq!(labels.get("api_operation").map(String::as_str), Some("s3:ListObjectsV2")); + assert_eq!(labels.get("caller").map(String::as_str), Some("s3_list")); + assert_eq!(labels.get("source").map(String::as_str), Some(LIST_OBJECTS_SOURCE_WALKER)); + } + #[test] fn merge_records_a_zero_read_quorum_rather_than_skipping_it() { let rows = recorded(|| record_list_objects_merge(LIST_OBJECTS_SOURCE_WALKER, 4, 0)); diff --git a/crates/io-metrics/src/s3_http_metrics.rs b/crates/io-metrics/src/s3_http_metrics.rs index 7e0cd5b3d..07bc7df3e 100644 --- a/crates/io-metrics/src/s3_http_metrics.rs +++ b/crates/io-metrics/src/s3_http_metrics.rs @@ -18,6 +18,7 @@ use rustfs_s3_ops::S3Operation; use std::cell::Cell; +use std::future::Future; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{LazyLock, OnceLock}; use std::time::{Duration, Instant}; @@ -38,6 +39,81 @@ const UNKNOWN_OPERATION: usize = S3Operation::ALL.len(); static COUNTERS: LazyLock = LazyLock::new(HttpOutcomeCounters::new); static COMPLETION_OBSERVER: OnceLock<(S3HttpCompletionObserverEnabled, S3HttpCompletionObserver)> = OnceLock::new(); +/// A bounded S3 operation captured from the current external-request context. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct S3HttpOperationContext(S3Operation); + +impl S3HttpOperationContext { + /// Return the stable operation index used for bounded metric-handle lookup. + pub fn operation_index(self) -> usize { + self.0.metric_index() + } + + /// Return the stable S3 operation label associated with this context. + pub fn operation_name(self) -> &'static str { + self.0.as_str() + } +} + +/// Return the current S3 operation, when the task is inside a dispatched S3 request. +pub fn current_s3_http_operation_context() -> Option { + #[cfg(feature = "diagnostic-read-version-caller")] + if let Ok(index) = ISOLATED_OPERATION.try_with(Cell::get) { + return S3Operation::ALL.get(index).copied().map(S3HttpOperationContext); + } + CURRENT_OPERATION + .try_with(|current| { + let index = current.get(); + S3Operation::ALL.get(index).copied().map(S3HttpOperationContext) + }) + .ok() + .flatten() +} + +/// Return the current S3 operation label, or `None` for internal/non-S3 work. +pub fn current_s3_http_operation_name() -> Option<&'static str> { + current_s3_http_operation_context().map(S3HttpOperationContext::operation_name) +} + +/// Reapply a captured S3 operation while polling work on a child Tokio task. +pub async fn with_s3_http_operation_context(context: Option, future: F) -> F::Output +where + F: Future, +{ + match context { + Some(context) => { + #[cfg(feature = "diagnostic-read-version-caller")] + return with_isolated_s3_http_operation_context(Some(context), future).await; + #[cfg(not(feature = "diagnostic-read-version-caller"))] + CURRENT_OPERATION.scope(Cell::new(context.operation_index()), future).await + } + None => future.await, + } +} + +/// Restore an item's operation, including explicit absence, without changing +/// the ambient HTTP request guard or borrowing the request that flushed a batch. +#[cfg(feature = "diagnostic-read-version-caller")] +pub async fn with_isolated_s3_http_operation_context(context: Option, future: F) -> F::Output +where + F: Future, +{ + let index = context.map_or(UNKNOWN_OPERATION, S3HttpOperationContext::operation_index); + ISOLATED_OPERATION + .scope(Cell::new(index), async { + match context { + Some(context) => CURRENT_OPERATION.scope(Cell::new(context.operation_index()), future).await, + None => future.await, + } + }) + .await +} + +#[cfg(feature = "diagnostic-read-version-caller")] +tokio::task_local! { + static ISOLATED_OPERATION: Cell; +} + tokio::task_local! { static CURRENT_OPERATION: Cell; } @@ -240,6 +316,25 @@ mod tests { ); } + #[tokio::test] + async fn captured_operation_context_can_be_reapplied_to_child_task() { + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request + .in_scope(|| { + observe_s3_http_operation(S3Operation::GetObject); + current_s3_http_operation_context() + }) + .expect("dispatched S3 operation should be captured"); + + let child_operation = tokio::spawn(with_s3_http_operation_context(Some(operation), async { + current_s3_http_operation_name() + })) + .await + .expect("child task should finish"); + + assert_eq!(child_operation, Some("s3:GetObject")); + } + #[test] fn outcome_counters_distinguish_partial_and_complete_write_failure() { let counters = HttpOutcomeCounters::new(); @@ -279,6 +374,39 @@ mod tests { } } + #[cfg(feature = "diagnostic-read-version-caller")] + #[tokio::test] + async fn isolated_absent_operation_does_not_inherit_the_flush_request() { + let mut request = S3HttpRequestGuard::new("GET"); + let operation = request.in_scope(|| { + observe_s3_http_operation(S3Operation::GetObject); + current_s3_http_operation_context().expect("GetObject operation should be captured") + }); + with_s3_http_operation_context(Some(operation), async { + assert!(S3HttpRequestGuard::is_active()); + with_isolated_s3_http_operation_context(None, async { + assert_eq!(current_s3_http_operation_context(), None); + assert!( + S3HttpRequestGuard::is_active(), + "isolating an item must leave ambient HTTP accounting intact" + ); + tokio::task::yield_now().await; + assert_eq!(current_s3_http_operation_context(), None); + }) + .await; + assert_eq!(current_s3_http_operation_context(), Some(operation)); + }) + .await; + assert_eq!(current_s3_http_operation_context(), None); + with_isolated_s3_http_operation_context(None, async { + assert!( + !S3HttpRequestGuard::is_active(), + "absent operation must not fabricate an HTTP request guard" + ); + }) + .await; + } + #[test] fn request_guard_preserves_operation_across_polls_and_finishes_once() { let totals = || { diff --git a/crates/lifecycle/src/core.rs b/crates/lifecycle/src/core.rs index fbbf35090..138e7cc84 100644 --- a/crates/lifecycle/src/core.rs +++ b/crates/lifecycle/src/core.rs @@ -648,7 +648,11 @@ impl Lifecycle for BucketLifecycleConfiguration { async fn eval(&self, obj: &ObjectOpts) -> Event { // A single-object lookup cannot prove how many newer historical versions // survive. Count-dependent actions wait for the complete-group evaluator. - self.eval_inner(obj, OffsetDateTime::now_utc(), 0).await + let mut event = self.eval_inner(obj, OffsetDateTime::now_utc(), 0).await; + if event.requires_expiry_policy() { + event.expiry_policy = Some(Arc::new(self.clone()).into()); + } + event } async fn predict_expiration(&self, obj: &ObjectOpts) -> Event { @@ -687,6 +691,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }; let predicted_due = predicted.due.unwrap_or(OffsetDateTime::UNIX_EPOCH).unix_timestamp(); let should_replace = event @@ -749,6 +754,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); // Stop after scheduling an expired delete-marker event. break; @@ -764,6 +770,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } } @@ -783,6 +790,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } continue; @@ -812,6 +820,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } } else { @@ -877,6 +886,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }); } } else if let Some(days) = expiration.days { @@ -898,6 +908,7 @@ impl Lifecycle for BucketLifecycleConfiguration { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, }; // MinIO extension: ExpiredObjectAllVersions deletes all versions if rule @@ -935,6 +946,7 @@ impl Lifecycle for BucketLifecycleConfiguration { rule_id: rule.id.clone().unwrap_or_default(), due, storage_class: storage_class.as_str().to_string(), + expiry_policy: None, noncurrent_days: 0, newer_noncurrent_versions: 0, }); @@ -983,6 +995,7 @@ impl Lifecycle for BucketLifecycleConfiguration { newer_noncurrent_versions: retained, due: Some(OffsetDateTime::UNIX_EPOCH), storage_class: "".into(), + expiry_policy: Some(Arc::clone(&self).into()), } } else { Event { @@ -993,6 +1006,7 @@ impl Lifecycle for BucketLifecycleConfiguration { newer_noncurrent_versions: 0, due: Some(OffsetDateTime::UNIX_EPOCH), storage_class: "".into(), + expiry_policy: Some(Arc::clone(&self).into()), } }; } @@ -1284,7 +1298,27 @@ fn ilm_action_priority_rank(action: &IlmAction) -> u8 { } } -#[derive(Debug, Clone)] +/// Immutable policy identity carried from lifecycle evaluation to deletion. +/// Consumers can compare authoritative metadata without exposing the policy's +/// wire representation in storage operation options. Clones share the policy. +#[derive(Clone)] +pub struct ExpiryPolicySnapshot { + configuration: Arc, +} + +impl From> for ExpiryPolicySnapshot { + fn from(configuration: Arc) -> Self { + Self { configuration } + } +} + +impl ExpiryPolicySnapshot { + pub fn matches_configuration(&self, current: Option<&BucketLifecycleConfiguration>) -> bool { + current == Some(self.configuration.as_ref()) + } +} + +#[derive(Clone)] pub struct Event { pub action: IlmAction, pub rule_id: String, @@ -1292,6 +1326,35 @@ pub struct Event { pub noncurrent_days: u32, pub newer_noncurrent_versions: usize, pub storage_class: String, + /// In-memory policy that authorized logical expiration. Execution must + /// compare it with authoritative bucket metadata under the transaction lock. + pub expiry_policy: Option, +} + +impl Event { + pub fn requires_expiry_policy(&self) -> bool { + matches!( + self.action, + IlmAction::DeleteAction + | IlmAction::DeleteVersionAction + | IlmAction::DeleteAllVersionsAction + | IlmAction::DelMarkerDeleteAllVersionsAction + ) + } +} + +impl std::fmt::Debug for Event { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Event") + .field("action", &self.action) + .field("rule_id", &self.rule_id) + .field("due", &self.due) + .field("noncurrent_days", &self.noncurrent_days) + .field("newer_noncurrent_versions", &self.newer_noncurrent_versions) + .field("storage_class", &self.storage_class) + .field("has_expiry_policy", &self.expiry_policy.is_some()) + .finish() + } } impl Default for Event { @@ -1303,6 +1366,7 @@ impl Default for Event { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: "".into(), + expiry_policy: None, } } } diff --git a/crates/lifecycle/src/evaluator.rs b/crates/lifecycle/src/evaluator.rs index 7f9089610..e4e155984 100644 --- a/crates/lifecycle/src/evaluator.rs +++ b/crates/lifecycle/src/evaluator.rs @@ -141,6 +141,11 @@ impl Evaluator { events[i] = event; } } + for event in &mut events { + if event.requires_expiry_policy() { + event.expiry_policy = Some(Arc::clone(&self.policy).into()); + } + } events } @@ -218,6 +223,30 @@ mod tests { use super::*; use rustfs_replication::{ReplicationStatusType, VersionPurgeStatusType}; + #[tokio::test] + async fn expiration_events_retain_the_evaluated_policy() { + let policy = latest_expiration_lifecycle(); + let object = current_object_opts(ReplicationStatusType::Completed); + let event = Evaluator::new(Arc::clone(&policy)) + .eval(std::slice::from_ref(&object)) + .await + .unwrap() + .remove(0); + assert_eq!(event.action, IlmAction::DeleteAction); + assert_eq!(Arc::strong_count(&policy), 2, "the event should share the evaluated policy"); + assert!(event.expiry_policy.as_ref().unwrap().matches_configuration(Some(&policy))); + assert!( + policy + .eval(&object) + .await + .expiry_policy + .as_ref() + .unwrap() + .matches_configuration(Some(&policy)) + ); + assert!(!format!("{event:?}").contains("BucketLifecycleConfiguration")); + } + #[tokio::test] async fn adversarial_restore_expiry_survives_legal_hold() { let mut policy = (*latest_expiration_lifecycle()).clone(); @@ -247,6 +276,7 @@ mod tests { .await .expect("unlocked restored versions should evaluate"); assert_eq!(unlocked.iter().map(|event| event.action).collect::>(), expected); + assert!(unlocked.iter().all(|event| event.expiry_policy.is_none())); for object in &mut objects { object diff --git a/crates/log-analyzer/src/ingest/mod.rs b/crates/log-analyzer/src/ingest/mod.rs index 2ad960479..ba8b97b27 100644 --- a/crates/log-analyzer/src/ingest/mod.rs +++ b/crates/log-analyzer/src/ingest/mod.rs @@ -307,7 +307,13 @@ fn walk_zip_entries( if entry.is_dir() { continue; } - let entry_path = entry.name().to_string(); + let entry_path = match entry.name() { + Ok(name) => name.into_owned(), + Err(_) => { + ctx.skip(format!("{provenance}!"), SkipReason::Unreadable); + continue; + } + }; let child_provenance = format!("{provenance}!{entry_path}"); let child_node = node_from_entry_path(&entry_path).or_else(|| node.clone()); let child_name = leaf_name(&entry_path).to_string(); diff --git a/crates/madmin/src/heal_commands.rs b/crates/madmin/src/heal_commands.rs index b1a22efcd..807d3699b 100644 --- a/crates/madmin/src/heal_commands.rs +++ b/crates/madmin/src/heal_commands.rs @@ -34,6 +34,20 @@ pub struct Infos { /// enum's crate). const DRIVE_STATE_OK: &str = "ok"; +/// Unresolved delete recovery work, independent of ordinary object healing. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct DeleteRecoveryStatus { + pub pending: usize, + pub blocked: usize, + pub failed: usize, +} + +impl DeleteRecoveryStatus { + pub fn has_unresolved(&self) -> bool { + self.pending != 0 || self.blocked != 0 || self.failed != 0 + } +} + #[derive(Clone, Debug, Default, Serialize, Deserialize)] pub struct HealResultItem { /// Only the coordinator's authenticated full scan may set this. Transported @@ -70,6 +84,8 @@ pub struct HealResultItem { pub resolved_version_id: Option<[u8; 16]>, #[serde(rename = "detail")] pub detail: String, + #[serde(default, rename = "deleteRecovery", skip_serializing_if = "Option::is_none")] + pub delete_recovery: Option, #[serde(rename = "parityBlocks")] pub parity_blocks: usize, #[serde(rename = "dataBlocks")] @@ -118,6 +134,38 @@ impl HealResultItem { mod tests { use super::*; + #[test] + fn delete_recovery_status_preserves_partial_results_and_legacy_shape() { + let legacy = serde_json::to_value(HealResultItem::default()).expect("serialize legacy result"); + assert!(legacy.get("deleteRecovery").is_none()); + let decoded: HealResultItem = serde_json::from_value(legacy).expect("decode result without recovery status"); + assert!(decoded.delete_recovery.is_none()); + for status in [ + DeleteRecoveryStatus { + pending: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + blocked: 1, + ..Default::default() + }, + DeleteRecoveryStatus { + failed: 1, + ..Default::default() + }, + ] { + let item = HealResultItem { + delete_recovery: Some(status), + ..Default::default() + }; + let wire = serde_json::to_value(item).expect("serialize recovery status"); + let decoded: HealResultItem = serde_json::from_value(wire).expect("decode recovery status"); + assert_eq!(decoded.delete_recovery, Some(status)); + assert!(status.has_unresolved()); + } + assert!(!DeleteRecoveryStatus::default().has_unresolved()); + } + #[test] fn resolved_heal_version_is_not_wire_proof() { let item = HealResultItem { diff --git a/crates/madmin/src/metrics.rs b/crates/madmin/src/metrics.rs index 1753bde2b..c9b555338 100644 --- a/crates/madmin/src/metrics.rs +++ b/crates/madmin/src/metrics.rs @@ -318,6 +318,19 @@ pub struct ScannerUsageFreshnessSnapshot { pub last_deferred_unix_secs: u64, #[serde(rename = "last_deferred_reason", default)] pub last_deferred_reason: String, + /// Cluster-activity guardrail vetoes, attributed as `@`. + #[serde(rename = "guardrail_veto_total", default)] + pub guardrail_veto_total: u64, + #[serde(rename = "last_guardrail_veto_unix_secs", default)] + pub last_guardrail_veto_unix_secs: u64, + #[serde(rename = "last_guardrail_veto_reason", default)] + pub last_guardrail_veto_reason: String, + #[serde(rename = "disk_io_budget_wait_events", default)] + pub disk_io_budget_wait_events: u64, + #[serde(rename = "disk_io_budget_wait_millis", default)] + pub disk_io_budget_wait_millis: u64, + #[serde(rename = "last_disk_io_budget_wait_unix_secs", default)] + pub last_disk_io_budget_wait_unix_secs: u64, } impl ScannerUsageFreshnessSnapshot { @@ -343,6 +356,20 @@ impl ScannerUsageFreshnessSnapshot { self.last_publication_reason = other.last_publication_reason.clone(); } self.deferred_total = self.deferred_total.saturating_add(other.deferred_total); + self.guardrail_veto_total = self.guardrail_veto_total.saturating_add(other.guardrail_veto_total); + self.disk_io_budget_wait_events = self + .disk_io_budget_wait_events + .saturating_add(other.disk_io_budget_wait_events); + self.disk_io_budget_wait_millis = self + .disk_io_budget_wait_millis + .saturating_add(other.disk_io_budget_wait_millis); + self.last_disk_io_budget_wait_unix_secs = self + .last_disk_io_budget_wait_unix_secs + .max(other.last_disk_io_budget_wait_unix_secs); + if other.last_guardrail_veto_unix_secs > self.last_guardrail_veto_unix_secs { + self.last_guardrail_veto_unix_secs = other.last_guardrail_veto_unix_secs; + self.last_guardrail_veto_reason = other.last_guardrail_veto_reason.clone(); + } if other_deferred_state_at > self_deferred_state_at { self.deferred_pending = other.deferred_pending; } else if other_deferred_state_at == self_deferred_state_at { @@ -431,6 +458,8 @@ pub struct ScannerPacingPressureSnapshot { pub last_cycle_pause_observed: bool, #[serde(rename = "last_cycle_throttle_sleep_ratio", default)] pub last_cycle_throttle_sleep_ratio: f64, + #[serde(rename = "last_cycle_disk_io_budget_wait_ratio", default)] + pub last_cycle_disk_io_budget_wait_ratio: f64, #[serde(rename = "last_cycle_yield_ratio", default)] pub last_cycle_yield_ratio: f64, #[serde(rename = "last_cycle_total_pause_ratio", default)] @@ -446,6 +475,7 @@ impl Default for ScannerPacingPressureSnapshot { last_cycle_budget_limited: false, last_cycle_pause_observed: false, last_cycle_throttle_sleep_ratio: 0.0, + last_cycle_disk_io_budget_wait_ratio: 0.0, last_cycle_yield_ratio: 0.0, last_cycle_total_pause_ratio: 0.0, } @@ -458,6 +488,8 @@ impl ScannerPacingPressureSnapshot { "queued_scans" } else if self.last_cycle_budget_limited { "cycle_budget" + } else if self.last_cycle_disk_io_budget_wait_ratio > 0.0 { + "disk_io_budget" } else if self.last_cycle_pause_observed { "throttle_pause" } else if self.current_active_scans > 0 { @@ -480,6 +512,9 @@ impl ScannerPacingPressureSnapshot { self.last_cycle_throttle_sleep_ratio = self .last_cycle_throttle_sleep_ratio .max(other.last_cycle_throttle_sleep_ratio); + self.last_cycle_disk_io_budget_wait_ratio = self + .last_cycle_disk_io_budget_wait_ratio + .max(other.last_cycle_disk_io_budget_wait_ratio); self.last_cycle_yield_ratio = self.last_cycle_yield_ratio.max(other.last_cycle_yield_ratio); self.last_cycle_total_pause_ratio = self.last_cycle_total_pause_ratio.max(other.last_cycle_total_pause_ratio); self.refresh_primary_pressure(); @@ -629,6 +664,10 @@ pub struct ScannerMetrics { pub current_cycle_throttle_sleep_events: u64, #[serde(rename = "current_cycle_throttle_sleep_duration_seconds", default)] pub current_cycle_throttle_sleep_duration_seconds: f64, + #[serde(rename = "current_cycle_disk_io_budget_wait_events", default)] + pub current_cycle_disk_io_budget_wait_events: u64, + #[serde(rename = "current_cycle_disk_io_budget_wait_duration_seconds", default)] + pub current_cycle_disk_io_budget_wait_duration_seconds: f64, #[serde(rename = "current_cycle_ilm_actions", default)] pub current_cycle_ilm_actions: u64, #[serde(rename = "current_cycle_lifecycle_expiry_actions", default)] @@ -675,6 +714,10 @@ pub struct ScannerMetrics { pub last_cycle_throttle_sleep_events: u64, #[serde(rename = "last_cycle_throttle_sleep_duration_seconds", default)] pub last_cycle_throttle_sleep_duration_seconds: f64, + #[serde(rename = "last_cycle_disk_io_budget_wait_events", default)] + pub last_cycle_disk_io_budget_wait_events: u64, + #[serde(rename = "last_cycle_disk_io_budget_wait_duration_seconds", default)] + pub last_cycle_disk_io_budget_wait_duration_seconds: f64, #[serde(rename = "last_cycle_ilm_actions", default)] pub last_cycle_ilm_actions: u64, #[serde(rename = "last_cycle_heal_objects", default)] @@ -856,6 +899,10 @@ impl ScannerMetrics { .current_cycle_throttle_sleep_events .saturating_add(other.current_cycle_throttle_sleep_events); self.current_cycle_throttle_sleep_duration_seconds += other.current_cycle_throttle_sleep_duration_seconds; + self.current_cycle_disk_io_budget_wait_events = self + .current_cycle_disk_io_budget_wait_events + .saturating_add(other.current_cycle_disk_io_budget_wait_events); + self.current_cycle_disk_io_budget_wait_duration_seconds += other.current_cycle_disk_io_budget_wait_duration_seconds; self.current_cycle_ilm_actions = self.current_cycle_ilm_actions.saturating_add(other.current_cycle_ilm_actions); self.current_cycle_lifecycle_expiry_actions = self .current_cycle_lifecycle_expiry_actions @@ -888,6 +935,10 @@ impl ScannerMetrics { .last_cycle_throttle_sleep_events .saturating_add(other.last_cycle_throttle_sleep_events); self.last_cycle_throttle_sleep_duration_seconds += other.last_cycle_throttle_sleep_duration_seconds; + self.last_cycle_disk_io_budget_wait_events = self + .last_cycle_disk_io_budget_wait_events + .saturating_add(other.last_cycle_disk_io_budget_wait_events); + self.last_cycle_disk_io_budget_wait_duration_seconds += other.last_cycle_disk_io_budget_wait_duration_seconds; self.last_cycle_ilm_actions = self.last_cycle_ilm_actions.saturating_add(other.last_cycle_ilm_actions); self.last_cycle_lifecycle_expiry_actions = self .last_cycle_lifecycle_expiry_actions @@ -1719,6 +1770,26 @@ mod tests { .expect("legacy usage freshness should decode"); assert!(!legacy.deferred_pending); assert_eq!(legacy.deferred_total, 0); + assert_eq!(legacy.guardrail_veto_total, 0); + assert_eq!(legacy.last_guardrail_veto_unix_secs, 0); + assert!(legacy.last_guardrail_veto_reason.is_empty()); + + let mut vetoes = ScannerUsageFreshnessSnapshot::default(); + vetoes.merge(&ScannerUsageFreshnessSnapshot { + guardrail_veto_total: 2, + last_guardrail_veto_unix_secs: 30, + last_guardrail_veto_reason: "publication_blocked@".to_string(), + ..Default::default() + }); + vetoes.merge(&ScannerUsageFreshnessSnapshot { + guardrail_veto_total: 1, + last_guardrail_veto_unix_secs: 20, + last_guardrail_veto_reason: "movement_generation@node-2".to_string(), + ..Default::default() + }); + assert_eq!(vetoes.guardrail_veto_total, 3); + assert_eq!(vetoes.last_guardrail_veto_unix_secs, 30); + assert_eq!(vetoes.last_guardrail_veto_reason, "publication_blocked@"); let mut merged = ScannerUsageFreshnessSnapshot::default(); merged.merge(&ScannerUsageFreshnessSnapshot { diff --git a/crates/obs/README.md b/crates/obs/README.md index a66fc5dea..0a1a9caaf 100644 --- a/crates/obs/README.md +++ b/crates/obs/README.md @@ -127,7 +127,9 @@ All configuration is read from environment variables at startup. `RUSTFS_OBS_TLS_CA_FILE` is read only while OTLP exporters are initialized. Its PEM certificates are added to, not substituted for, the platform trust roots, and TLS chain, expiry, and hostname checks remain enabled. Restart RustFS after rotating the bundle. A missing, unreadable, non-absolute, empty, or invalid bundle fails OTLP initialization. -The export interval is separate from application metric collection. Without interval overrides, node/disk metrics refresh every 60 seconds (`RUSTFS_METRICS_NODE_INTERVAL_SEC`), cluster metrics every 60 seconds (`RUSTFS_METRICS_CLUSTER_INTERVAL_SEC`), and per-bucket metrics every 300 seconds (`RUSTFS_METRICS_BUCKET_INTERVAL_SEC`). See the [collection defaults](src/metrics/config.rs) and [interval configuration](src/metrics/scheduler.rs) for collector-specific and global overrides. +The export interval is separate from application metric collection. Without interval overrides, node/disk metrics refresh every 60 seconds (`RUSTFS_METRICS_NODE_INTERVAL_SEC`), cluster metrics including scanner bucket usage every 60 seconds (`RUSTFS_METRICS_CLUSTER_INTERVAL_SEC`), and the separate bucket collector every 300 seconds (`RUSTFS_METRICS_BUCKET_INTERVAL_SEC`). See the [collection defaults](src/metrics/config.rs) and [interval configuration](src/metrics/scheduler.rs) for collector-specific and global overrides. + +For bucket size and object-count metric names, all-bucket PromQL queries, accounting semantics, and scanner freshness, see [Bucket usage metrics](../../docs/operations/bucket-usage-metrics.md). With OTLP → Collector → Prometheus, the time from a failure to an alert also includes underlying health detection, collection and export work, transport and Collector batching, Prometheus `scrape_interval` and `evaluation_interval`, and any alert rule `for` duration. A shorter scrape interval cannot refresh a value that RustFS has not yet collected and exported. These periodic intervals do not guarantee a fixed detection or alert latency; see the [Prometheus alert rule documentation](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) for the role of `for`. diff --git a/crates/obs/src/global.rs b/crates/obs/src/global.rs index 08f1e83b0..a413a5d01 100644 --- a/crates/obs/src/global.rs +++ b/crates/obs/src/global.rs @@ -260,9 +260,26 @@ mod tests { } #[test] - fn test_grafana_dashboard_deduplicates_cluster_bucket_usage() { - assert!(GRAFANA_DASHBOARD.contains("max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count")); - assert!(GRAFANA_DASHBOARD.contains("max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes")); + fn test_grafana_dashboard_scopes_and_deduplicates_cluster_bucket_usage() { + let dashboard: serde_json::Value = serde_json::from_str(GRAFANA_DASHBOARD).expect("dashboard must be valid JSON"); + for (title, metric) in [ + ("Bucket Objects", "rustfs_cluster_usage_buckets_objects_count"), + ("Bucket Size", "rustfs_cluster_usage_buckets_total_bytes"), + ] { + let panel = dashboard["panels"] + .as_array() + .expect("dashboard panels") + .iter() + .find(|panel| panel["title"] == title) + .expect("bucket usage panel"); + assert_eq!( + panel["targets"][0]["expr"], + format!( + "max by (rustfs_cluster_id, job, bucket) ({metric}{{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"}})" + ), + "{title} must select one cluster and deduplicate its reporting nodes" + ); + } assert!(!GRAFANA_DASHBOARD.contains("sum by (bucket) (rustfs_bucket_api_objects_total")); assert!(!GRAFANA_DASHBOARD.contains("sum by (bucket) (rustfs_bucket_api_usage_bytes")); } diff --git a/crates/obs/src/metrics/storage_snapshot.rs b/crates/obs/src/metrics/storage_snapshot.rs index 8977403e9..d3e1a74ee 100644 --- a/crates/obs/src/metrics/storage_snapshot.rs +++ b/crates/obs/src/metrics/storage_snapshot.rs @@ -408,6 +408,69 @@ mod tests { provider.shutdown().unwrap(); } + #[test] + fn actual_sdk_exports_bucket_size_and_object_count_as_current_gauges() { + use crate::metrics::collectors::{BucketUsageStats, collect_bucket_usage_metrics}; + + let (provider, exporter, mut snapshot) = setup_unscoped(); + let mut buckets = vec![ + BucketUsageStats { + bucket: "client-a".into(), + total_bytes: 4096, + objects_count: 3, + ..Default::default() + }, + BucketUsageStats { + bucket: "client-b".into(), + total_bytes: 8192, + objects_count: 7, + ..Default::default() + }, + BucketUsageStats { + bucket: "empty".into(), + ..Default::default() + }, + ]; + + for expected in [ + vec![("client-a", 4096.0, 3.0), ("client-b", 8192.0, 7.0), ("empty", 0.0, 0.0)], + vec![("client-a", 1024.0, 1.0), ("empty", 0.0, 0.0)], + ] { + let metrics = collect_bucket_usage_metrics(&buckets); + assert!( + metrics.iter().all(|metric| metric.metric_type == MetricType::Gauge), + "bucket inventory values must decrease without counter semantics" + ); + snapshot.replace(metrics).expect("publish current bucket usage"); + let points = flush(&provider, &exporter); + for (name, index) in [ + ("rustfs_cluster_usage_buckets_total_bytes", 0), + ("rustfs_cluster_usage_buckets_objects_count", 1), + ] { + let mut actual = points + .iter() + .filter(|point| point.name == name) + .map(|point| { + assert_eq!(point.scope, "service-test"); + assert_eq!(point.attributes.len(), 1, "usage is cluster-wide, not a per-node count"); + assert_eq!(point.attributes[0].key.as_str(), "bucket"); + (point.attributes[0].value.to_string(), point.value) + }) + .collect::>(); + actual.sort_by(|a, b| a.0.cmp(&b.0)); + let values = expected + .iter() + .map(|(bucket, bytes, objects)| (bucket.to_string(), if index == 0 { *bytes } else { *objects })) + .collect::>(); + assert_eq!(actual, values, "{name} must preserve each bucket's current value"); + } + buckets.remove(1); + buckets[0].total_bytes = 1024; + buckets[0].objects_count = 1; + } + provider.shutdown().expect("shut down bucket usage test provider"); + } + #[test] fn failed_or_stalled_collection_cannot_refresh_last_success_or_export_old_points() { let (provider, exporter, mut snapshot) = setup(); diff --git a/crates/obs/src/telemetry/resource.rs b/crates/obs/src/telemetry/resource.rs index 8a7d003ab..5f9c4aa33 100644 --- a/crates/obs/src/telemetry/resource.rs +++ b/crates/obs/src/telemetry/resource.rs @@ -16,9 +16,9 @@ //! //! A `Resource` describes the entity producing telemetry data. The resource //! built here includes the service name, service version, deployment -//! environment, the stable RustFS node identity, and the local machine IP -//! address so that data can be correlated across services in a distributed -//! system. +//! environment, the stable RustFS node identity, the process ID, and the local +//! machine IP address so telemetry series can be correlated across service +//! restarts and nodes. use crate::config::OtelConfig; use crate::node_identity::{RUSTFS_NODE_ATTRIBUTE, local_node_identity}; @@ -27,7 +27,7 @@ use opentelemetry_sdk::Resource; use opentelemetry_semantic_conventions::{ SCHEMA_URL, attribute::{ - DEPLOYMENT_ENVIRONMENT_NAME, NETWORK_LOCAL_ADDRESS, SERVICE_INSTANCE_ID as OTEL_SERVICE_INSTANCE_ID, + DEPLOYMENT_ENVIRONMENT_NAME, NETWORK_LOCAL_ADDRESS, PROCESS_PID, SERVICE_INSTANCE_ID as OTEL_SERVICE_INSTANCE_ID, SERVICE_VERSION as OTEL_SERVICE_VERSION, }, }; @@ -48,6 +48,8 @@ use std::borrow::Cow; /// - `network.local.address` — the primary local IP of the current host, /// useful as an operational fallback when the stable node name is not yet /// initialized. +/// - `process.pid` — distinguishes telemetry emitted by different processes +/// running on the same node. /// /// All attributes are attached to the resource using the semantic conventions /// schema URL to ensure compatibility with standard OTLP backends. @@ -70,6 +72,7 @@ pub(super) fn build_resource(config: &OtelConfig) -> Resource { KeyValue::new(RUSTFS_NODE_ATTRIBUTE, node_identity.clone()), KeyValue::new(OTEL_SERVICE_INSTANCE_ID, node_identity), KeyValue::new(NETWORK_LOCAL_ADDRESS, local_ip), + KeyValue::new(PROCESS_PID, i64::from(std::process::id())), ], SCHEMA_URL, ) @@ -117,6 +120,12 @@ mod tests { .map(|value| value.to_string()), Some("node1:9000".to_string()) ); + assert_eq!( + resource + .get(&Key::from_static_str(PROCESS_PID)) + .map(|value| value.to_string()), + Some(std::process::id().to_string()) + ); rustfs_common::set_global_local_node_name(&previous).await; } diff --git a/crates/policy/src/policy/function/key_name.rs b/crates/policy/src/policy/function/key_name.rs index 9ead5ca31..807653e02 100644 --- a/crates/policy/src/policy/function/key_name.rs +++ b/crates/policy/src/policy/function/key_name.rs @@ -57,6 +57,7 @@ impl KeyName { KeyName::S3(S3KeyName::S3XAmzContentSha256), KeyName::S3(S3KeyName::S3LocationConstraint), KeyName::S3(S3KeyName::S3VersionId), + KeyName::S3(S3KeyName::S3ObjectCreationOperation), //aws KeyName::Aws(AwsKeyName::AWSReferer), KeyName::Aws(AwsKeyName::AWSSourceIP), @@ -121,12 +122,13 @@ impl KeyName { /// values for these keys. Without that check a caller could satisfy conditions such /// as `aws:userid` or `jwt:groups` simply by sending a header of the same name. /// - /// The `s3:` namespace is mixed: `s3:x-amz-*` keys mirror request headers by - /// design, while the rest (`s3:authType`, `s3:LocationConstraint`, ...) are - /// computed by the server. + /// The `s3:` namespace is mixed: `s3:x-amz-*`, `s3:if-match` and `s3:if-none-match` + /// mirror request headers by design, while the rest (`s3:authType`, + /// `s3:LocationConstraint`, ...) are computed by the server. pub fn is_server_derived(&self) -> bool { match self { KeyName::Aws(_) | KeyName::Jwt(_) | KeyName::Ldap(_) | KeyName::Sts(_) | KeyName::Svc(_) => true, + KeyName::S3(S3KeyName::S3IfMatch | S3KeyName::S3IfNoneMatch) => false, KeyName::S3(_) => !self.name().starts_with("x-amz-"), } } @@ -165,6 +167,9 @@ impl From<&KeyName> for &'static str { #[derive(Clone, EnumString, Debug, IntoStaticStr, Eq, PartialEq, Serialize, Deserialize)] #[serde(try_from = "&str", into = "&str")] pub enum S3KeyName { + #[strum(serialize = "s3:ObjectCreationOperation")] + S3ObjectCreationOperation, + #[strum(serialize = "s3:x-amz-copy-source")] S3XAmzCopySource, @@ -237,6 +242,12 @@ pub enum S3KeyName { #[strum(serialize = "s3:x-amz-grant-write-acp")] S3XAmzGrantWriteAcp, + #[strum(serialize = "s3:if-match")] + S3IfMatch, + + #[strum(serialize = "s3:if-none-match")] + S3IfNoneMatch, + #[strum(serialize = "s3:ExistingObjectTag")] S3ExistingObjectTag, #[strum(serialize = "s3:RequestObjectTagKeys")] @@ -391,6 +402,18 @@ mod tests { use serde::Deserialize; use test_case::test_case; + #[test] + fn object_creation_operation_is_registered_as_server_derived() { + let key = KeyName::try_from("s3:ObjectCreationOperation").expect("AWS object creation condition key should parse"); + assert_eq!(key.name(), "ObjectCreationOperation"); + assert!(key.is_server_derived()); + assert!(KeyName::server_derived_key_names().any(|name| name == "ObjectCreationOperation")); + assert_eq!( + serde_json::to_string(&key).expect("condition key should serialize"), + "\"s3:ObjectCreationOperation\"" + ); + } + #[test_case("s3:x-amz-copy-source", KeyName::S3(S3KeyName::S3XAmzCopySource))] #[test_case("s3:VersionId", KeyName::S3(S3KeyName::S3VersionId) ; "aws_version_id")] #[test_case("s3:versionid", KeyName::S3(S3KeyName::S3VersionId) ; "minio_version_id")] @@ -400,6 +423,8 @@ mod tests { "s3:object-lock-remaining-retention-days", KeyName::S3(S3KeyName::S3ObjectLockRemainingRetentionDays) )] + #[test_case("s3:if-match", KeyName::S3(S3KeyName::S3IfMatch))] + #[test_case("s3:if-none-match", KeyName::S3(S3KeyName::S3IfNoneMatch))] #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))] #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))] #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))] @@ -428,6 +453,8 @@ mod tests { "s3:object-lock-remaining-retention-days", KeyName::S3(S3KeyName::S3ObjectLockRemainingRetentionDays) )] + #[test_case("s3:if-match", KeyName::S3(S3KeyName::S3IfMatch))] + #[test_case("s3:if-none-match", KeyName::S3(S3KeyName::S3IfNoneMatch))] #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))] #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))] #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))] @@ -452,6 +479,8 @@ mod tests { "s3:object-lock-remaining-retention-days", KeyName::S3(S3KeyName::S3ObjectLockRemainingRetentionDays) )] + #[test_case("s3:if-match", KeyName::S3(S3KeyName::S3IfMatch))] + #[test_case("s3:if-none-match", KeyName::S3(S3KeyName::S3IfNoneMatch))] #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))] #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))] #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))] @@ -468,6 +497,14 @@ mod tests { assert_eq!(data, except); } + #[test_case("s3:if-match", "if-match")] + #[test_case("s3:if-none-match", "if-none-match")] + fn conditional_write_key_names_match_request_header_names(key: &str, header: &str) { + let key_name = KeyName::try_from(key).expect("conditional write key must parse"); + assert_eq!(key_name.name(), header); + assert!(!key_name.is_server_derived(), "{key} is filled from a request header"); + } + #[test] fn key_name_from_str_supports_jwt_roles() { assert!(KeyName::try_from("jwt:roles").is_ok()); diff --git a/crates/policy/src/policy/resource.rs b/crates/policy/src/policy/resource.rs index 262b13f4e..6d203accc 100644 --- a/crates/policy/src/policy/resource.rs +++ b/crates/policy/src/policy/resource.rs @@ -38,11 +38,7 @@ impl Serialize for ResourceSet { let mut seq = serializer.serialize_seq(Some(self.0.len()))?; for resource in &self.0 { - let resource_str = match resource { - Resource::S3(value) => format!("{}{}", Resource::S3_PREFIX, value), - Resource::Kms(value) => format!("{}{}", Resource::KMS_PREFIX, value), - }; - seq.serialize_element(&resource_str)?; + seq.serialize_element(resource)?; } seq.end() } @@ -165,6 +161,7 @@ impl<'de> Deserialize<'de> for ResourceSet { #[derive(Hash, Eq, PartialEq, Clone, Debug)] pub enum Resource { + All, S3(String), Kms(String), } @@ -197,6 +194,7 @@ impl Resource { resolver: Option<&dyn PolicyVariableResolver>, ) -> bool { let pattern = match self { + Resource::All => return true, Resource::S3(s) => s.to_owned(), Resource::Kms(s) => s.to_owned(), }; @@ -242,7 +240,9 @@ impl Resource { impl TryFrom<&str> for Resource { type Error = Error; fn try_from(value: &str) -> std::result::Result { - let resource = if let Some(suffix) = value.strip_prefix(Self::S3_PREFIX) { + let resource = if value == "*" { + Resource::All + } else if let Some(suffix) = value.strip_prefix(Self::S3_PREFIX) { Resource::S3(suffix.into()) } else if let Some(suffix) = value.strip_prefix(Self::KMS_PREFIX) { Resource::Kms(suffix.into()) @@ -259,6 +259,7 @@ impl Validator for Resource { type Error = Error; fn is_valid(&self) -> std::result::Result<(), Error> { match self { + Self::All => {} Self::S3(pattern) => { if pattern.is_empty() || pattern.starts_with('/') { return Err(IamError::InvalidResource("s3".into(), pattern.into()).into()); @@ -290,6 +291,7 @@ impl Serialize for Resource { S: serde::Serializer, { match self { + Resource::All => serializer.serialize_str("*"), Resource::S3(s) => serializer.serialize_str(&format!("{}{}", Self::S3_PREFIX, s)), Resource::Kms(s) => serializer.serialize_str(&format!("{}{}", Self::KMS_PREFIX, s)), } @@ -312,6 +314,10 @@ mod tests { use std::collections::HashMap; use test_case::test_case; + #[test_case("*", "mybucket" => true; "all buckets")] + #[test_case("*", "mybucket/myobject" => true; "all objects")] + #[test_case("*", "key/mykey" => true; "all keys")] + #[test_case("*", "" => true; "unscoped action")] #[test_case("arn:aws:s3:::*","mybucket" => true; "1")] #[test_case("arn:aws:s3:::*","mybucket/myobject" => true; "2")] #[test_case("arn:aws:s3:::mybucket*","mybucket" => true; "3")] @@ -346,6 +352,32 @@ mod tests { pollster::block_on(resource.is_match(object, &HashMap::new())) } + #[test] + fn test_wildcard_resource_serialization_round_trip() { + use crate::policy::resource::ResourceSet; + + let resource: Resource = serde_json::from_str(r#""*""#).expect("wildcard resource should parse"); + assert_eq!(serde_json::to_string(&resource).expect("resource should serialize"), r#""*""#); + + for json in [r#""*""#, r#"["*"]"#, r#"["*","*"]"#] { + let set: ResourceSet = serde_json::from_str(json).expect("wildcard resource set should parse"); + assert_eq!(set.len(), 1, "duplicate wildcard resources should be removed"); + let serialized = serde_json::to_string(&set).expect("resource set should serialize"); + assert_eq!(serialized, r#"["*"]"#); + let round_trip: ResourceSet = serde_json::from_str(&serialized).expect("resource set should parse again"); + assert_eq!(round_trip, set); + } + } + + #[test_case(""; "empty resource")] + #[test_case("**"; "multiple wildcards")] + #[test_case("*/*"; "unqualified path")] + #[test_case(" *"; "leading whitespace")] + #[test_case("arn:aws:unknown:::*"; "unknown service")] + fn test_invalid_resource_is_rejected(resource: &str) { + assert!(Resource::try_from(resource).is_err(), "invalid resource must not become a wildcard"); + } + #[test_case("arn:aws:kms:::key/mykey" => true; "key id parses")] #[test_case("arn:aws:kms:::key/*" => true; "key wildcard parses")] #[test_case("arn:aws:kms:::key/app-key.v2" => true; "key id with dot parses")] diff --git a/crates/policy/src/policy/statement.rs b/crates/policy/src/policy/statement.rs index 5b03aca02..960fb0a63 100644 --- a/crates/policy/src/policy/statement.rs +++ b/crates/policy/src/policy/statement.rs @@ -182,15 +182,24 @@ impl Statement { /// left empty. An empty `args.object` means the caller did not scope the request to a /// key, which preserves the legacy match-every-key behaviour. async fn kms_key_scope_matches(&self, args: &Args<'_>, resolver: &VariableResolver) -> bool { + if self.not_resources.iter().any(|resource| matches!(resource, Resource::All)) { + return false; + } + if matches!(args.action, Action::KmsAction(KmsAction::BackupAction | KmsAction::RestoreAction)) - && (!self.resources.is_empty() || !self.not_resources.is_empty()) + && ((!self.resources.is_empty() && !self.resources.iter().any(|resource| matches!(resource, Resource::All))) + || !self.not_resources.is_empty()) { // Global bundle operations require an unscoped Allow, while a Deny // covering any key must still block an operation covering every key. return matches!(self.effect, Effect::Deny); } - let kms_resources: Vec<&Resource> = self.resources.iter().filter(|resource| resource.is_kms()).collect(); + let kms_resources: Vec<&Resource> = self + .resources + .iter() + .filter(|resource| matches!(resource, Resource::All | Resource::Kms(_))) + .collect(); let kms_not_resources: Vec<&Resource> = self.not_resources.iter().filter(|resource| resource.is_kms()).collect(); if kms_resources.len() != self.resources.len() || kms_not_resources.len() != self.not_resources.len() { diff --git a/crates/policy/tests/policy_is_allowed.rs b/crates/policy/tests/policy_is_allowed.rs index 868f25072..d29f36438 100644 --- a/crates/policy/tests/policy_is_allowed.rs +++ b/crates/policy/tests/policy_is_allowed.rs @@ -808,3 +808,215 @@ fn merge_keeps_allow_statements_differing_only_in_not_resource() { "both Allow statements must survive the merge; neither may be dropped as a duplicate" ); } + +#[test] +fn wildcard_deny_only_session_grants_no_storage_access() { + let policy = Policy::parse_config( + br#"{"Statement":[{"Action":["s3:GetObject"],"Effect":"Deny","Resource":"*","Sid":"1"}],"Version":"2012-10-17"}"#, + ) + .expect("deny-only session policy should parse"); + let encoded = serde_json::to_vec(&policy).expect("session policy should serialize into claims"); + let decoded = Policy::parse_config(&encoded).expect("session policy should parse from claims"); + for action in ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"] { + for bucket in ["bucket", "other-bucket"] { + assert!( + !policy_is_allowed( + decoded.clone(), + ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: "data/object".into(), + ..Default::default() + } + ), + "deny-only session must not grant {action} on {bucket}" + ); + } + } +} + +#[test_case("s3:GetObject", "bucket", "object" => true; "matching object action")] +#[test_case("s3:PutObject", "bucket", "object" => false; "ungranted action")] +#[test_case("s3:GetObject", "other-bucket", "object" => true; "other bucket")] +fn wildcard_resource_allow_preserves_action_scope(action: &str, bucket: &str, object: &str) -> bool { + let policy = Policy::parse_config( + br#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:GetObject","Resource":["*"]}]}"#, + ) + .expect("wildcard allow policy should parse"); + policy_is_allowed( + policy, + ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: object.into(), + ..Default::default() + }, + ) +} + +#[test_case("s3:GetObject", "bucket", "object"; "object")] +#[test_case("s3:ListBucket", "bucket", ""; "bucket")] +#[test_case("s3:ListAllMyBuckets", "", ""; "all buckets")] +#[test_case("kms:Decrypt", "", "mykey"; "kms key")] +#[test_case("kms:Backup", "", ""; "kms backup")] +#[test_case("kms:Restore", "", ""; "kms restore")] +fn wildcard_resource_allow_deny_and_exclusion(action: &str, bucket: &str, object: &str) { + let other_resource = if action.starts_with("kms:") { + "arn:aws:kms:::key/other-key" + } else { + "arn:aws:s3:::other-bucket/*" + }; + for (field, resource) in [ + ("Resource", serde_json::json!("*")), + ("Resource", serde_json::json!(["*"])), + ("Resource", serde_json::json!([other_resource, "*"])), + ("NotResource", serde_json::json!("*")), + ("NotResource", serde_json::json!([other_resource, "*"])), + ] { + let mut json = serde_json::json!({ + "Version": "2012-10-17", + "Statement": [{"Effect": "Allow", "Action": action, field: resource}] + }); + let allow = Policy::parse_config(&serde_json::to_vec(&json).expect("test policy should serialize")) + .expect("wildcard policy should parse"); + let args = ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: object.into(), + ..Default::default() + }; + assert_eq!(policy_is_allowed(allow, args), field == "Resource", "{action} with {field}"); + + json["Statement"][0]["Effect"] = serde_json::json!("Deny"); + json["Statement"] + .as_array_mut() + .expect("statements are an array") + .push(serde_json::json!({ + "Effect": "Allow", "Action": action, "Resource": "*" + })); + let deny = Policy::parse_config(&serde_json::to_vec(&json).expect("test policy should serialize")) + .expect("wildcard deny policy should parse"); + assert_eq!( + policy_is_allowed( + deny, + ArgsBuilder { + action: action.into(), + bucket: bucket.into(), + object: object.into(), + ..Default::default() + } + ), + field == "NotResource", + "explicit deny must win unless {field} excludes the resource for {action}" + ); + } +} + +#[test_case(Some("allowed/data/") => true; "matching prefix")] +#[test_case(Some("other/data/") => false; "outside allowed prefix")] +#[test_case(None => false; "missing prefix")] +fn wildcard_resource_preserves_conditions(prefix: Option<&str>) -> bool { + let policy = Policy::parse_config( + br#"{"Version":"2012-10-17","Statement":[{ + "Effect":"Allow","Action":"s3:ListBucket","Resource":"*", + "Condition":{"StringLike":{"s3:prefix":"allowed/*"}} + }]}"#, + ) + .expect("conditional wildcard policy should parse"); + let mut args = ArgsBuilder { + action: "s3:ListBucket".into(), + bucket: "bucket".into(), + ..Default::default() + }; + if let Some(prefix) = prefix { + args.conditions.insert("prefix".into(), vec![prefix.into()]); + } + policy_is_allowed(policy, args) +} + +// Conditional write enforcement (`s3:if-none-match` / `s3:if-match`). Condition values +// carry request headers under their lowercase header name; a missing header leaves +// the key absent, which `Null: true` matches and `StringEquals` never satisfies. + +fn header_conditions(headers: &[(&str, &str)]) -> HashMap> { + headers + .iter() + .map(|(name, value)| ((*name).to_string(), vec![(*value).to_string()])) + .collect() +} + +fn is_allowed_put_object(policy: &Policy, conditions: &HashMap>) -> bool { + let claims: HashMap = HashMap::new(); + let groups: Option> = None; + pollster::block_on(policy.is_allowed(&Args { + account: "Q3AM3UQ867SPQQA43P2F", + groups: &groups, + action: rustfs_policy::policy::action::Action::S3Action(PutObjectAction), + bucket: "mybucket", + conditions, + is_owner: false, + object: "uploads/report.csv", + claims: &claims, + deny_only: false, + })) +} + +fn put_object_policy_with_condition(condition: &str) -> Policy { + let json = format!( + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Action":["s3:PutObject"],"Resource":["arn:aws:s3:::mybucket/*"],"Condition":{condition}}}]}}"# + ); + Policy::parse_config(json.as_bytes()).expect("policy with conditional write key should parse") +} + +#[test_case(r#"{"StringEquals":{"s3:if-none-match":"*"}}"#, &[("if-none-match", "*")] => true ; "if_none_match_string_equals_header_present")] +#[test_case(r#"{"StringEquals":{"s3:if-none-match":"*"}}"#, &[] => false ; "if_none_match_string_equals_header_absent")] +#[test_case(r#"{"StringEquals":{"s3:if-none-match":"*"}}"#, &[("if-match", "\"etag\"")] => false ; "if_none_match_string_equals_other_header")] +#[test_case(r#"{"Null":{"s3:if-none-match":"false"}}"#, &[("if-none-match", "*")] => true ; "if_none_match_not_null_header_present")] +#[test_case(r#"{"Null":{"s3:if-none-match":"false"}}"#, &[] => false ; "if_none_match_not_null_header_absent")] +#[test_case(r#"{"Null":{"s3:if-match":"false"}}"#, &[("if-match", "\"etag\"")] => true ; "if_match_not_null_header_present")] +#[test_case(r#"{"Null":{"s3:if-match":"false"}}"#, &[] => false ; "if_match_not_null_header_absent")] +#[test_case(r#"{"StringEquals":{"s3:if-match":"\"etag\""}}"#, &[("if-match", "\"etag\"")] => true ; "if_match_string_equals_matching_value")] +#[test_case(r#"{"StringEquals":{"s3:if-match":"\"etag\""}}"#, &[("if-match", "\"other\"")] => false ; "if_match_string_equals_different_value")] +fn conditional_write_condition_keys(condition: &str, headers: &[(&str, &str)]) -> bool { + let policy = put_object_policy_with_condition(condition); + is_allowed_put_object(&policy, &header_conditions(headers)) +} + +/// Mirrors the AWS deny-based enforcement example: allow PutObject, but deny it +/// when the `If-Match` header is missing. +#[test_case(&[("if-match", "\"etag\"")] => true ; "header_present")] +#[test_case(&[] => false ; "header_absent")] +fn bucket_policy_denies_put_object_without_if_match(headers: &[(&str, &str)]) -> bool { + let policy: BucketPolicy = serde_json::from_str( + r#"{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": {"AWS": ["*"]}, + "Action": ["s3:PutObject"], + "Resource": ["arn:aws:s3:::mybucket/*"] + }, + { + "Effect": "Deny", + "Principal": {"AWS": ["*"]}, + "Action": ["s3:PutObject"], + "Resource": ["arn:aws:s3:::mybucket/*"], + "Condition": {"Null": {"s3:if-match": "true"}} + } + ] + }"#, + ) + .expect("bucket policy with conditional write key should parse"); + let conditions = header_conditions(headers); + let groups: Option> = None; + pollster::block_on(policy.is_allowed(&BucketPolicyArgs { + account: "Q3AM3UQ867SPQQA43P2F", + groups: &groups, + action: rustfs_policy::policy::action::Action::S3Action(PutObjectAction), + bucket: "mybucket", + conditions: &conditions, + is_owner: false, + object: "uploads/report.csv", + })) +} diff --git a/crates/protos/src/generated/proto_gen/node_service.rs b/crates/protos/src/generated/proto_gen/node_service.rs index f5c12e5ba..1bac44e59 100644 --- a/crates/protos/src/generated/proto_gen/node_service.rs +++ b/crates/protos/src/generated/proto_gen/node_service.rs @@ -785,6 +785,18 @@ pub struct DeleteVersionsRequest { #[prost(bytes = "bytes", tag = "6")] pub opts_bin: ::prost::bytes::Bytes, } +/// Additive, fail-closed conditional batch protocol. Probe requests never mutate disk state. +#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] +pub struct GuardedDeleteVersionsRequest { + #[prost(string, tag = "1")] + pub disk: ::prost::alloc::string::String, + #[prost(string, tag = "2")] + pub volume: ::prost::alloc::string::String, + #[prost(bytes = "bytes", tag = "3")] + pub payload: ::prost::bytes::Bytes, + #[prost(bool, tag = "4")] + pub probe_only: bool, +} #[derive(Clone, PartialEq, ::prost::Message)] pub struct DeleteVersionsResponse { #[prost(bool, tag = "1")] @@ -1494,6 +1506,41 @@ pub struct ReplacementRecoveryStatusResponse { #[prost(enumeration = "ControlPlaneErrorCode", optional, tag = "4")] pub error_code: ::core::option::Option, } +/// Asks a peer how it currently sees the drives served by `host`, so a +/// restarting node can wait for re-admission before publishing readiness. +#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] +pub struct PeerDriveAdmissionRequest { + /// Topology host:port of the drives' owner, as written in the endpoint list. + #[prost(string, tag = "1")] + pub host: ::prost::alloc::string::String, +} +#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] +pub struct PeerDriveAdmission { + #[prost(int32, tag = "1")] + pub pool_index: i32, + #[prost(int32, tag = "2")] + pub set_index: i32, + #[prost(int32, tag = "3")] + pub disk_index: i32, + /// True only when the peer would count this drive towards its own read quorum. + #[prost(bool, tag = "4")] + pub online: bool, + #[prost(string, tag = "5")] + pub runtime_state: ::prost::alloc::string::String, + #[prost(bool, tag = "6")] + pub host_online: bool, +} +#[derive(Clone, PartialEq, ::prost::Message)] +pub struct PeerDriveAdmissionResponse { + #[prost(bool, tag = "1")] + pub success: bool, + #[prost(message, repeated, tag = "2")] + pub drives: ::prost::alloc::vec::Vec, + #[prost(string, optional, tag = "3")] + pub error_info: ::core::option::Option<::prost::alloc::string::String>, + #[prost(enumeration = "ControlPlaneErrorCode", optional, tag = "4")] + pub error_code: ::core::option::Option, +} #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)] pub struct HealControlRequest { #[prost(uint32, tag = "1")] @@ -2135,6 +2182,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "AcquireSnapshotLease")); self.inner.unary(req, path, codec).await } + pub async fn acquire_quota_mutation_fence( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/AcquireQuotaMutationFence"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "AcquireQuotaMutationFence")); + self.inner.unary(req, path, codec).await + } pub async fn renew_snapshot_lease( &mut self, request: impl tonic::IntoRequest, @@ -2165,6 +2227,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "ReleaseSnapshotLease")); self.inner.unary(req, path, codec).await } + pub async fn release_quota_mutation_fence( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/ReleaseQuotaMutationFence"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "ReleaseQuotaMutationFence")); + self.inner.unary(req, path, codec).await + } pub async fn verify_file( &mut self, request: impl tonic::IntoRequest, @@ -2376,6 +2453,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "RenameDataAtIncarnation")); self.inner.unary(req, path, codec).await } + pub async fn rename_data_with_quota_fence( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/RenameDataWithQuotaFence"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "RenameDataWithQuotaFence")); + self.inner.unary(req, path, codec).await + } pub async fn make_volumes( &mut self, request: impl tonic::IntoRequest, @@ -2616,6 +2708,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "DeleteVersions")); self.inner.unary(req, path, codec).await } + pub async fn guarded_delete_versions( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/GuardedDeleteVersions"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "GuardedDeleteVersions")); + self.inner.unary(req, path, codec).await + } pub async fn read_multiple( &mut self, request: impl tonic::IntoRequest, @@ -3278,6 +3385,21 @@ pub mod node_service_client { .insert(GrpcMethod::new("node_service.NodeService", "ReplacementRecoveryStatus")); self.inner.unary(req, path, codec).await } + pub async fn peer_drive_admission( + &mut self, + request: impl tonic::IntoRequest, + ) -> std::result::Result, tonic::Status> { + self.inner + .ready() + .await + .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?; + let codec = tonic_prost::ProstCodec::default(); + let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/PeerDriveAdmission"); + let mut req = request.into_request(); + req.extensions_mut() + .insert(GrpcMethod::new("node_service.NodeService", "PeerDriveAdmission")); + self.inner.unary(req, path, codec).await + } pub async fn get_metacache_listing( &mut self, request: impl tonic::IntoRequest, @@ -3509,6 +3631,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn acquire_quota_mutation_fence( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn renew_snapshot_lease( &self, request: tonic::Request, @@ -3517,6 +3643,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn release_quota_mutation_fence( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn verify_file( &self, request: tonic::Request, @@ -3586,6 +3716,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn rename_data_with_quota_fence( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn make_volumes( &self, request: tonic::Request, @@ -3650,6 +3784,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn guarded_delete_versions( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn read_multiple( &self, request: tonic::Request, @@ -3828,6 +3966,10 @@ pub mod node_service_server { &self, request: tonic::Request, ) -> std::result::Result, tonic::Status>; + async fn peer_drive_admission( + &self, + request: tonic::Request, + ) -> std::result::Result, tonic::Status>; async fn get_metacache_listing( &self, request: tonic::Request, @@ -4335,6 +4477,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/AcquireQuotaMutationFence" => { + #[allow(non_camel_case_types)] + struct AcquireQuotaMutationFenceSvc(pub Arc); + impl tonic::server::UnaryService for AcquireQuotaMutationFenceSvc { + type Response = super::SnapshotLeaseResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::acquire_quota_mutation_fence(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = AcquireQuotaMutationFenceSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/RenewSnapshotLease" => { #[allow(non_camel_case_types)] struct RenewSnapshotLeaseSvc(pub Arc); @@ -4391,6 +4561,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/ReleaseQuotaMutationFence" => { + #[allow(non_camel_case_types)] + struct ReleaseQuotaMutationFenceSvc(pub Arc); + impl tonic::server::UnaryService for ReleaseQuotaMutationFenceSvc { + type Response = super::SnapshotLeaseMutationResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::release_quota_mutation_fence(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = ReleaseQuotaMutationFenceSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/VerifyFile" => { #[allow(non_camel_case_types)] struct VerifyFileSvc(pub Arc); @@ -4786,6 +4984,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/RenameDataWithQuotaFence" => { + #[allow(non_camel_case_types)] + struct RenameDataWithQuotaFenceSvc(pub Arc); + impl tonic::server::UnaryService for RenameDataWithQuotaFenceSvc { + type Response = super::RenameDataResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::rename_data_with_quota_fence(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = RenameDataWithQuotaFenceSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/MakeVolumes" => { #[allow(non_camel_case_types)] struct MakeVolumesSvc(pub Arc); @@ -5234,6 +5460,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/GuardedDeleteVersions" => { + #[allow(non_camel_case_types)] + struct GuardedDeleteVersionsSvc(pub Arc); + impl tonic::server::UnaryService for GuardedDeleteVersionsSvc { + type Response = super::DeleteVersionsResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::guarded_delete_versions(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = GuardedDeleteVersionsSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/ReadMultiple" => { #[allow(non_camel_case_types)] struct ReadMultipleSvc(pub Arc); @@ -6473,6 +6727,34 @@ pub mod node_service_server { }; Box::pin(fut) } + "/node_service.NodeService/PeerDriveAdmission" => { + #[allow(non_camel_case_types)] + struct PeerDriveAdmissionSvc(pub Arc); + impl tonic::server::UnaryService for PeerDriveAdmissionSvc { + type Response = super::PeerDriveAdmissionResponse; + type Future = BoxFuture, tonic::Status>; + fn call(&mut self, request: tonic::Request) -> Self::Future { + let inner = Arc::clone(&self.0); + let fut = async move { ::peer_drive_admission(&inner, request).await }; + Box::pin(fut) + } + } + let accept_compression_encodings = self.accept_compression_encodings; + let send_compression_encodings = self.send_compression_encodings; + let max_decoding_message_size = self.max_decoding_message_size; + let max_encoding_message_size = self.max_encoding_message_size; + let inner = self.inner.clone(); + let fut = async move { + let method = PeerDriveAdmissionSvc(inner); + let codec = tonic_prost::ProstCodec::default(); + let mut grpc = tonic::server::Grpc::new(codec) + .apply_compression_config(accept_compression_encodings, send_compression_encodings) + .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size); + let res = grpc.unary(method, req).await; + Ok(res) + }; + Box::pin(fut) + } "/node_service.NodeService/GetMetacacheListing" => { #[allow(non_camel_case_types)] struct GetMetacacheListingSvc(pub Arc); diff --git a/crates/protos/src/lib.rs b/crates/protos/src/lib.rs index 5b69f697f..847609560 100644 --- a/crates/protos/src/lib.rs +++ b/crates/protos/src/lib.rs @@ -1059,6 +1059,20 @@ pub fn canonical_rename_data_request_body( Ok(body.finish()) } +pub const QUOTA_MUTATION_TOPOLOGY_HEADER: &str = "x-rustfs-quota-topology"; + +/// Binds a quota mutation to the coordinator's complete, ordered storage layout. +/// The inner canonical body retains its operation-specific domain and wire fields. +pub fn canonical_quota_mutation_request_body( + topology_fingerprint: &str, + mutation_body: &[u8], +) -> Result, std::num::TryFromIntError> { + let mut body = CanonicalBodyBuilder::new(b"rustfs-quota-mutation-topology-request-v1\0"); + body.push_str(topology_fingerprint)?; + body.push_bytes(mutation_body)?; + Ok(body.finish()) +} + pub fn canonical_delete_version_request_body( request: &proto_gen::node_service::DeleteVersionRequest, ) -> Result, std::num::TryFromIntError> { @@ -1096,6 +1110,17 @@ pub fn canonical_delete_versions_request_body( Ok(body.finish()) } +pub fn canonical_guarded_delete_versions_request_body( + request: &proto_gen::node_service::GuardedDeleteVersionsRequest, +) -> Result, std::num::TryFromIntError> { + let mut body = CanonicalBodyBuilder::new(b"rustfs-guarded-delete-versions-request-v1\0"); + body.push_str(&request.disk)?; + body.push_str(&request.volume)?; + body.push_bytes(&request.payload)?; + body.push_bool(request.probe_only); + Ok(body.finish()) +} + pub fn canonical_write_metadata_request_body( request: &proto_gen::node_service::WriteMetadataRequest, ) -> Result, std::num::TryFromIntError> { @@ -1344,6 +1369,58 @@ mod disk_mutation_canonical_tests { }; use super::*; + #[test] + fn quota_mutation_envelope_binds_topology_operation_and_field_boundaries() { + let acquire = canonical_snapshot_lease_request_body(&SnapshotLeaseRequest::default()).expect("acquire body"); + let release = + canonical_snapshot_lease_release_request_body(&SnapshotLeaseReleaseRequest::default()).expect("release body"); + let rename = canonical_rename_data_request_body(&RenameDataRequest::default()).expect("rename body"); + let topology = "a".repeat(64); + let envelope = canonical_quota_mutation_request_body(&topology, &acquire).expect("quota envelope"); + assert_ne!(envelope, acquire, "quota and ordinary mutation domains must differ"); + assert_all_distinct(&[ + envelope, + canonical_quota_mutation_request_body(&"b".repeat(64), &acquire).expect("changed topology"), + canonical_quota_mutation_request_body(&topology, &release).expect("release envelope"), + canonical_quota_mutation_request_body(&topology, &rename).expect("rename envelope"), + ]); + assert_ne!( + canonical_quota_mutation_request_body("ab", b"c").expect("first boundary"), + canonical_quota_mutation_request_body("a", b"bc").expect("second boundary"), + "topology and operation bytes must be independently length delimited" + ); + } + + #[test] + fn guarded_delete_digest_binds_probe_and_payload() { + let request = proto_gen::node_service::GuardedDeleteVersionsRequest { + disk: "disk-a".into(), + volume: "bucket-a".into(), + payload: vec![0x80].into(), + probe_only: false, + }; + let original = canonical_guarded_delete_versions_request_body(&request).expect("encode guarded body"); + let mut candidates = Vec::new(); + let mut changed = request.clone(); + changed.disk = "disk-b".into(); + candidates.push(changed); + let mut changed = request.clone(); + changed.volume = "bucket-b".into(); + candidates.push(changed); + let mut changed = request.clone(); + changed.payload = vec![0x81].into(); + candidates.push(changed); + let mut changed = request; + changed.probe_only = true; + candidates.push(changed); + for changed in candidates { + assert_ne!( + canonical_guarded_delete_versions_request_body(&changed).expect("encode changed body"), + original + ); + } + } + #[test] fn delete_recovery_canonical_body_binds_all_wire_fields() { let request = proto_gen::node_service::RecoverDeleteTransactionRequest { diff --git a/crates/protos/src/node.proto b/crates/protos/src/node.proto index fd56c543c..7c82a5e47 100644 --- a/crates/protos/src/node.proto +++ b/crates/protos/src/node.proto @@ -555,6 +555,14 @@ message DeleteVersionsRequest { bytes opts_bin = 6; } +// Additive, fail-closed conditional batch protocol. Probe requests never mutate disk state. +message GuardedDeleteVersionsRequest { + string disk = 1; + string volume = 2; + bytes payload = 3; + bool probe_only = 4; +} + message DeleteVersionsResponse { bool success = 1; repeated string errors = 2; @@ -1046,6 +1054,30 @@ message ReplacementRecoveryStatusResponse { optional ControlPlaneErrorCode error_code = 4; } +// Asks a peer how it currently sees the drives served by `host`, so a +// restarting node can wait for re-admission before publishing readiness. +message PeerDriveAdmissionRequest { + // Topology host:port of the drives' owner, as written in the endpoint list. + string host = 1; +} + +message PeerDriveAdmission { + int32 pool_index = 1; + int32 set_index = 2; + int32 disk_index = 3; + // True only when the peer would count this drive towards its own read quorum. + bool online = 4; + string runtime_state = 5; + bool host_online = 6; +} + +message PeerDriveAdmissionResponse { + bool success = 1; + repeated PeerDriveAdmission drives = 2; + optional string error_info = 3; + optional ControlPlaneErrorCode error_code = 4; +} + message HealControlRequest { uint32 version = 1; string topology_fingerprint = 2; @@ -1235,8 +1267,10 @@ service NodeService { rpc DeleteAtIncarnation(DeleteRequest) returns (DeleteResponse) {}; // auth-policy: body-bound rpc RecoverDeleteTransaction(RecoverDeleteTransactionRequest) returns (RecoverDeleteTransactionResponse) {}; // auth-policy: body-bound rpc AcquireSnapshotLease(SnapshotLeaseRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound + rpc AcquireQuotaMutationFence(SnapshotLeaseRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound rpc RenewSnapshotLease(SnapshotLeaseRenewRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound rpc ReleaseSnapshotLease(SnapshotLeaseReleaseRequest) returns (SnapshotLeaseMutationResponse) {}; // auth-policy: body-bound + rpc ReleaseQuotaMutationFence(SnapshotLeaseReleaseRequest) returns (SnapshotLeaseMutationResponse) {}; // auth-policy: body-bound rpc VerifyFile(VerifyFileRequest) returns (VerifyFileResponse) {}; // auth-policy: read-only rpc ReadParts(ReadPartsRequest) returns (ReadPartsResponse) {}; // auth-policy: read-only rpc CheckParts(CheckPartsRequest) returns (CheckPartsResponse) {}; // auth-policy: read-only @@ -1252,6 +1286,7 @@ service NodeService { rpc WalkDir(WalkDirRequest) returns (stream WalkDirResponse) {}; // auth-policy: streaming rpc RenameData(RenameDataRequest) returns (RenameDataResponse) {}; // auth-policy: body-bound rpc RenameDataAtIncarnation(RenameDataRequest) returns (RenameDataResponse) {}; // auth-policy: body-bound + rpc RenameDataWithQuotaFence(RenameDataRequest) returns (RenameDataResponse) {}; // auth-policy: body-bound rpc MakeVolumes(MakeVolumesRequest) returns (MakeVolumesResponse) {}; // auth-policy: body-bound rpc MakeVolume(MakeVolumeRequest) returns (MakeVolumeResponse) {}; // auth-policy: body-bound rpc ListVolumes(ListVolumesRequest) returns (ListVolumesResponse) {}; // auth-policy: read-only @@ -1268,6 +1303,7 @@ service NodeService { rpc DeleteVersionAtIncarnation(DeleteVersionRequest) returns (DeleteVersionResponse) {}; // auth-policy: body-bound rpc DeleteRetiredMarker(DeleteVersionRequest) returns (DeleteVersionResponse) {}; // auth-policy: body-bound rpc DeleteVersions(DeleteVersionsRequest) returns (DeleteVersionsResponse) {}; // auth-policy: body-bound + rpc GuardedDeleteVersions(GuardedDeleteVersionsRequest) returns (DeleteVersionsResponse) {}; // auth-policy: body-bound rpc ReadMultiple(ReadMultipleRequest) returns (ReadMultipleResponse) {}; // auth-policy: read-only rpc DeleteVolume(DeleteVolumeRequest) returns (DeleteVolumeResponse) {}; // auth-policy: body-bound rpc DiskInfo(DiskInfoRequest) returns (DiskInfoResponse) {}; // auth-policy: read-only @@ -1321,6 +1357,7 @@ service NodeService { rpc ReleaseScannerPublicationLease(ScannerPublicationLeaseReleaseRequest) returns (ScannerPublicationLeaseReleaseResponse) {}; // auth-policy: body-bound rpc BackgroundHealStatus(BackgroundHealStatusRequest) returns (BackgroundHealStatusResponse) {}; // auth-policy: read-only rpc ReplacementRecoveryStatus(ReplacementRecoveryStatusRequest) returns (ReplacementRecoveryStatusResponse) {}; // auth-policy: read-only + rpc PeerDriveAdmission(PeerDriveAdmissionRequest) returns (PeerDriveAdmissionResponse) {}; // auth-policy: read-only rpc GetMetacacheListing(GetMetacacheListingRequest) returns (GetMetacacheListingResponse) {}; // auth-policy: unimplemented rpc UpdateMetacacheListing(UpdateMetacacheListingRequest) returns (UpdateMetacacheListingResponse) {}; // auth-policy: unimplemented rpc ReloadPoolMeta(ReloadPoolMetaRequest) returns (ReloadPoolMetaResponse) {}; // auth-policy: body-bound diff --git a/crates/replication/src/object.rs b/crates/replication/src/object.rs index 5d12f8c5a..c33348694 100644 --- a/crates/replication/src/object.rs +++ b/crates/replication/src/object.rs @@ -120,8 +120,11 @@ pub fn object_lock_put_integrity( /// stored bytes differ from what the source sent — e.g. a target that does not /// decode `aws-chunked` framing stores the frames verbatim and returns their /// ETag. Only a plain single-part MD5 ETag on both sides is decidable; a -/// multipart or opaque (encrypted) ETag, or a withheld replica ETag, returns -/// `false` because no corruption can be concluded from it. +/// multipart ETag, or a withheld replica ETag, returns `false` because no +/// corruption can be concluded from it. An encrypted source's keyed +/// fingerprint is also 32 hex characters and cannot be told apart from an MD5 +/// here, so callers must exempt encrypted sources before asking (the ecstore +/// resyncer does so through its source-encryption classifier). pub fn single_part_replica_etag_mismatch(source_etag: Option<&str>, replica_etag: Option<&str>) -> bool { let Some(source) = source_etag.map(trim_etag) else { return false; diff --git a/crates/s3select-query/Cargo.toml b/crates/s3select-query/Cargo.toml index b786f2197..3fbf7778c 100644 --- a/crates/s3select-query/Cargo.toml +++ b/crates/s3select-query/Cargo.toml @@ -57,6 +57,8 @@ tracing = { workspace = true } [dev-dependencies] rustfs-test-utils = { workspace = true, features = ["put-object-commit-barrier"] } +# Match the streaming inference's field order in Arrow's JSON reference. +serde_json = { workspace = true, features = ["preserve_order"] } [lib] doctest = false diff --git a/crates/s3select-query/src/dispatcher/json_document_schema.rs b/crates/s3select-query/src/dispatcher/json_document_schema.rs index d61d9ff0b..1f8c33c46 100644 --- a/crates/s3select-query/src/dispatcher/json_document_schema.rs +++ b/crates/s3select-query/src/dispatcher/json_document_schema.rs @@ -430,7 +430,7 @@ mod tests { const TEST_SCHEMA_LIMIT: usize = 4 * 1024 * 1024; - fn assert_matches_arrow(input: &[u8]) { + fn assert_matches_arrow(input: &[u8]) -> Schema { let records = input.iter().filter(|byte| **byte == b'\n').count(); let (expected, expected_records) = arrow_infer_json_schema(io::BufReader::new(input), Some(records)) .expect("Arrow should infer the compatibility fixture"); @@ -438,6 +438,33 @@ mod tests { .expect("streaming inference should accept the compatibility fixture"); assert_eq!(actual_records, expected_records); assert_eq!(actual, expected); + actual + } + + #[test] + fn inference_preserves_first_seen_field_order_for_nested_objects() { + let schema = assert_matches_arrow( + br#"{"z":1,"nested":{"z":true,"a":"first"}} +{"a":2,"nested":{"m":3}} +"#, + ); + let expected = Schema::new(vec![ + Field::new("z", DataType::Int64, true), + Field::new( + "nested", + DataType::Struct( + vec![ + Field::new("z", DataType::Boolean, true), + Field::new("a", DataType::Utf8, true), + Field::new("m", DataType::Int64, true), + ] + .into(), + ), + true, + ), + Field::new("a", DataType::Int64, true), + ]); + assert_eq!(schema, expected); } #[test] diff --git a/crates/scanner-metrics/src/metrics.rs b/crates/scanner-metrics/src/metrics.rs index 8467dbd13..cecebf903 100644 --- a/crates/scanner-metrics/src/metrics.rs +++ b/crates/scanner-metrics/src/metrics.rs @@ -919,6 +919,8 @@ pub struct Metrics { current_scan_cycle_yield_duration_millis_start: AtomicU64, current_scan_cycle_throttle_sleep_events_start: AtomicU64, current_scan_cycle_throttle_sleep_duration_millis_start: AtomicU64, + current_scan_cycle_disk_io_budget_wait_events_start: AtomicU64, + current_scan_cycle_disk_io_budget_wait_duration_millis_start: AtomicU64, current_scan_cycle_ilm_actions_start: AtomicU64, current_scan_cycle_lifecycle_expiry_actions_start: AtomicU64, current_scan_cycle_lifecycle_transition_actions_start: AtomicU64, @@ -956,6 +958,8 @@ pub struct Metrics { last_scan_cycle_yield_duration_millis: AtomicU64, last_scan_cycle_throttle_sleep_events: AtomicU64, last_scan_cycle_throttle_sleep_duration_millis: AtomicU64, + last_scan_cycle_disk_io_budget_wait_events: AtomicU64, + last_scan_cycle_disk_io_budget_wait_duration_millis: AtomicU64, last_scan_cycle_ilm_actions: AtomicU64, last_scan_cycle_lifecycle_expiry_actions: AtomicU64, last_scan_cycle_lifecycle_transition_actions: AtomicU64, @@ -971,6 +975,9 @@ pub struct Metrics { partial_scan_cycles_by_source: Vec, scanner_yield_duration_millis: AtomicU64, scanner_throttle_sleep_duration_millis: AtomicU64, + scanner_disk_io_budget_wait_events: AtomicU64, + scanner_disk_io_budget_wait_duration_millis: AtomicU64, + scanner_disk_io_budget_last_wait_unix_secs: AtomicU64, scanner_ilm_actions: AtomicU64, scanner_lifecycle_expiry_actions: AtomicU64, scanner_lifecycle_transition_actions: AtomicU64, @@ -1032,6 +1039,9 @@ pub struct Metrics { scanner_usage_last_publication_unix_secs: AtomicU64, scanner_usage_last_publication_state: Mutex, scanner_usage_last_publication_reason: Mutex, + scanner_guardrail_veto_total: AtomicU64, + scanner_guardrail_last_veto_unix_secs: AtomicU64, + scanner_guardrail_last_veto_reason: Mutex, scanner_usage_last_save_result: AtomicU8, scanner_usage_deferred_pending: AtomicBool, scanner_usage_deferred_total: AtomicU64, @@ -1138,6 +1148,8 @@ pub struct ScanCycleWorkSnapshot { yield_duration_millis: u64, throttle_sleep_events: u64, throttle_sleep_duration_millis: u64, + disk_io_budget_wait_events: u64, + disk_io_budget_wait_duration_millis: u64, ilm_actions: u64, lifecycle_expiry_actions: u64, lifecycle_transition_actions: u64, @@ -1258,6 +1270,8 @@ pub struct ScannerPacingPressureSnapshot { pub last_cycle_budget_limited: bool, pub last_cycle_pause_observed: bool, pub last_cycle_throttle_sleep_ratio: f64, + #[serde(default)] + pub last_cycle_disk_io_budget_wait_ratio: f64, pub last_cycle_yield_ratio: f64, pub last_cycle_total_pause_ratio: f64, } @@ -1271,6 +1285,7 @@ impl Default for ScannerPacingPressureSnapshot { last_cycle_budget_limited: false, last_cycle_pause_observed: false, last_cycle_throttle_sleep_ratio: 0.0, + last_cycle_disk_io_budget_wait_ratio: 0.0, last_cycle_yield_ratio: 0.0, last_cycle_total_pause_ratio: 0.0, } @@ -1359,6 +1374,22 @@ pub struct ScannerUsageFreshnessSnapshot { pub last_deferred_unix_secs: u64, #[serde(default)] pub last_deferred_reason: String, + /// Guardrail vetoes attributed as `@`; a phase-2 peer + /// dirty-usage fallback is reported through the same fields. + #[serde(default)] + pub guardrail_veto_total: u64, + #[serde(default)] + pub last_guardrail_veto_unix_secs: u64, + #[serde(default)] + pub last_guardrail_veto_reason: String, + /// Waits imposed by the per-disk background I/O budget; sustained waits + /// explain slow usage freshness on a budgeted disk. + #[serde(default)] + pub disk_io_budget_wait_events: u64, + #[serde(default)] + pub disk_io_budget_wait_millis: u64, + #[serde(default)] + pub last_disk_io_budget_wait_unix_secs: u64, } #[derive(Clone, Debug, Default, Serialize, Deserialize)] @@ -1427,6 +1458,10 @@ pub struct ScannerMetricsReport { #[serde(default)] pub current_cycle_throttle_sleep_duration_seconds: f64, #[serde(default)] + pub current_cycle_disk_io_budget_wait_events: u64, + #[serde(default)] + pub current_cycle_disk_io_budget_wait_duration_seconds: f64, + #[serde(default)] pub current_cycle_ilm_actions: u64, #[serde(default)] pub current_cycle_lifecycle_expiry_actions: u64, @@ -1466,6 +1501,10 @@ pub struct ScannerMetricsReport { #[serde(default)] pub last_cycle_throttle_sleep_duration_seconds: f64, #[serde(default)] + pub last_cycle_disk_io_budget_wait_events: u64, + #[serde(default)] + pub last_cycle_disk_io_budget_wait_duration_seconds: f64, + #[serde(default)] pub last_cycle_ilm_actions: u64, #[serde(default)] pub last_cycle_lifecycle_expiry_actions: u64, @@ -1627,6 +1666,7 @@ fn scanner_primary_pressure( current_queued_scans: u64, current_active_scans: u64, last_cycle_budget_limited: bool, + last_cycle_disk_io_budget_wait_events: u64, last_cycle_throttle_sleep_events: u64, last_cycle_yield_events: u64, ) -> &'static str { @@ -1634,6 +1674,8 @@ fn scanner_primary_pressure( "queued_scans" } else if last_cycle_budget_limited { "cycle_budget" + } else if last_cycle_disk_io_budget_wait_events > 0 { + "disk_io_budget" } else if last_cycle_throttle_sleep_events > 0 || last_cycle_yield_events > 0 { "throttle_pause" } else if current_active_scans > 0 { @@ -1653,12 +1695,20 @@ fn scanner_pacing_pressure(metrics: &ScannerMetricsReport) -> ScannerPacingPress .max(usize_to_u64_saturated(metrics.active_scan_paths)); let last_cycle_budget_limited = scanner_last_cycle_budget_limited(metrics.last_cycle_result_code, metrics.last_cycle_partial_reason.as_str()); - let last_cycle_pause_observed = metrics.last_cycle_throttle_sleep_events > 0 || metrics.last_cycle_yield_events > 0; + let last_cycle_pause_observed = metrics.last_cycle_throttle_sleep_events > 0 + || metrics.last_cycle_disk_io_budget_wait_events > 0 + || metrics.last_cycle_yield_events > 0; let last_cycle_throttle_sleep_ratio = scanner_ratio(metrics.last_cycle_throttle_sleep_duration_seconds, metrics.last_cycle_duration_seconds); + let last_cycle_disk_io_budget_wait_ratio = scanner_ratio( + metrics.last_cycle_disk_io_budget_wait_duration_seconds, + metrics.last_cycle_duration_seconds, + ); let last_cycle_yield_ratio = scanner_ratio(metrics.last_cycle_yield_duration_seconds, metrics.last_cycle_duration_seconds); let last_cycle_total_pause_ratio = scanner_ratio( - metrics.last_cycle_throttle_sleep_duration_seconds.max(0.0) + metrics.last_cycle_yield_duration_seconds.max(0.0), + metrics.last_cycle_throttle_sleep_duration_seconds.max(0.0) + + metrics.last_cycle_disk_io_budget_wait_duration_seconds.max(0.0) + + metrics.last_cycle_yield_duration_seconds.max(0.0), metrics.last_cycle_duration_seconds, ); @@ -1667,6 +1717,7 @@ fn scanner_pacing_pressure(metrics: &ScannerMetricsReport) -> ScannerPacingPress current_queued_scans, current_active_scans, last_cycle_budget_limited, + metrics.last_cycle_disk_io_budget_wait_events, metrics.last_cycle_throttle_sleep_events, metrics.last_cycle_yield_events, ) @@ -1676,6 +1727,7 @@ fn scanner_pacing_pressure(metrics: &ScannerMetricsReport) -> ScannerPacingPress last_cycle_budget_limited, last_cycle_pause_observed, last_cycle_throttle_sleep_ratio, + last_cycle_disk_io_budget_wait_ratio, last_cycle_yield_ratio, last_cycle_total_pause_ratio, } @@ -1994,6 +2046,8 @@ impl Metrics { current_scan_cycle_yield_duration_millis_start: AtomicU64::new(0), current_scan_cycle_throttle_sleep_events_start: AtomicU64::new(0), current_scan_cycle_throttle_sleep_duration_millis_start: AtomicU64::new(0), + current_scan_cycle_disk_io_budget_wait_events_start: AtomicU64::new(0), + current_scan_cycle_disk_io_budget_wait_duration_millis_start: AtomicU64::new(0), current_scan_cycle_ilm_actions_start: AtomicU64::new(0), current_scan_cycle_lifecycle_expiry_actions_start: AtomicU64::new(0), current_scan_cycle_lifecycle_transition_actions_start: AtomicU64::new(0), @@ -2027,6 +2081,8 @@ impl Metrics { last_scan_cycle_yield_duration_millis: AtomicU64::new(0), last_scan_cycle_throttle_sleep_events: AtomicU64::new(0), last_scan_cycle_throttle_sleep_duration_millis: AtomicU64::new(0), + last_scan_cycle_disk_io_budget_wait_events: AtomicU64::new(0), + last_scan_cycle_disk_io_budget_wait_duration_millis: AtomicU64::new(0), last_scan_cycle_ilm_actions: AtomicU64::new(0), last_scan_cycle_lifecycle_expiry_actions: AtomicU64::new(0), last_scan_cycle_lifecycle_transition_actions: AtomicU64::new(0), @@ -2042,6 +2098,9 @@ impl Metrics { partial_scan_cycles_by_source: ScannerWorkSource::all().iter().map(|_| AtomicU64::new(0)).collect(), scanner_yield_duration_millis: AtomicU64::new(0), scanner_throttle_sleep_duration_millis: AtomicU64::new(0), + scanner_disk_io_budget_wait_events: AtomicU64::new(0), + scanner_disk_io_budget_wait_duration_millis: AtomicU64::new(0), + scanner_disk_io_budget_last_wait_unix_secs: AtomicU64::new(0), scanner_ilm_actions: AtomicU64::new(0), scanner_lifecycle_expiry_actions: AtomicU64::new(0), scanner_lifecycle_transition_actions: AtomicU64::new(0), @@ -2100,6 +2159,9 @@ impl Metrics { scanner_usage_last_publication_unix_secs: AtomicU64::new(0), scanner_usage_last_publication_state: Mutex::new(String::new()), scanner_usage_last_publication_reason: Mutex::new(String::new()), + scanner_guardrail_veto_total: AtomicU64::new(0), + scanner_guardrail_last_veto_unix_secs: AtomicU64::new(0), + scanner_guardrail_last_veto_reason: Mutex::new(String::new()), scanner_usage_last_save_result: AtomicU8::new(ScannerUsageSaveResult::Unknown as u8), scanner_usage_deferred_pending: AtomicBool::new(false), scanner_usage_deferred_total: AtomicU64::new(0), @@ -2262,6 +2324,19 @@ impl Metrics { }); } + /// Record one wait imposed by a per-disk background I/O budget. + pub fn record_scanner_disk_io_budget_wait(&self, duration: Duration) { + self.scanner_disk_io_budget_wait_events.fetch_add(1, Ordering::Relaxed); + let duration_millis = duration_millis_saturated(duration); + let _ = self + .scanner_disk_io_budget_wait_duration_millis + .try_update(Ordering::Relaxed, Ordering::Relaxed, |current| { + Some(current.saturating_add(duration_millis)) + }); + self.scanner_disk_io_budget_last_wait_unix_secs + .store(unix_now_secs(), Ordering::Relaxed); + } + pub fn record_scanner_ilm_action(&self, count: u64) { self.scanner_ilm_actions.fetch_add(count, Ordering::Relaxed); self.record_scanner_source_executed(ScannerWorkSource::Lifecycle, count); @@ -2500,6 +2575,19 @@ impl Metrics { *last_reason = reason; } + /// Record an attributed cluster-activity guardrail veto (`@`) + /// so an operator can tell a rebalance or a PUT tail from a broken probe. + pub fn record_scanner_guardrail_veto(&self, reason: impl Into) { + self.scanner_guardrail_veto_total.fetch_add(1, Ordering::Relaxed); + self.scanner_guardrail_last_veto_unix_secs + .store(unix_now_secs(), Ordering::Relaxed); + let mut last_reason = match self.scanner_guardrail_last_veto_reason.lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + }; + *last_reason = reason.into(); + } + pub fn record_scanner_usage_durable_success(&self) { self.record_scanner_usage_publication("success", ""); self.scanner_usage_last_durable_success_unix_secs @@ -2971,6 +3059,10 @@ impl Metrics { .store(snapshot.throttle_sleep_events, Ordering::Relaxed); self.current_scan_cycle_throttle_sleep_duration_millis_start .store(snapshot.throttle_sleep_duration_millis, Ordering::Relaxed); + self.current_scan_cycle_disk_io_budget_wait_events_start + .store(snapshot.disk_io_budget_wait_events, Ordering::Relaxed); + self.current_scan_cycle_disk_io_budget_wait_duration_millis_start + .store(snapshot.disk_io_budget_wait_duration_millis, Ordering::Relaxed); self.current_scan_cycle_ilm_actions_start .store(snapshot.ilm_actions, Ordering::Relaxed); self.current_scan_cycle_lifecycle_expiry_actions_start @@ -3035,6 +3127,8 @@ impl Metrics { yield_duration_millis: self.scanner_yield_duration_millis.load(Ordering::Relaxed), throttle_sleep_events: self.lifetime(Metric::ThrottleSleep), throttle_sleep_duration_millis: self.scanner_throttle_sleep_duration_millis.load(Ordering::Relaxed), + disk_io_budget_wait_events: self.scanner_disk_io_budget_wait_events.load(Ordering::Relaxed), + disk_io_budget_wait_duration_millis: self.scanner_disk_io_budget_wait_duration_millis.load(Ordering::Relaxed), ilm_actions: self.scanner_ilm_actions.load(Ordering::Relaxed), lifecycle_expiry_actions: self.scanner_lifecycle_expiry_actions.load(Ordering::Relaxed), lifecycle_transition_actions: self.scanner_lifecycle_transition_actions.load(Ordering::Relaxed), @@ -3056,6 +3150,12 @@ impl Metrics { throttle_sleep_duration_millis: self .current_scan_cycle_throttle_sleep_duration_millis_start .load(Ordering::Relaxed), + disk_io_budget_wait_events: self + .current_scan_cycle_disk_io_budget_wait_events_start + .load(Ordering::Relaxed), + disk_io_budget_wait_duration_millis: self + .current_scan_cycle_disk_io_budget_wait_duration_millis_start + .load(Ordering::Relaxed), ilm_actions: self.current_scan_cycle_ilm_actions_start.load(Ordering::Relaxed), lifecycle_expiry_actions: self.current_scan_cycle_lifecycle_expiry_actions_start.load(Ordering::Relaxed), lifecycle_transition_actions: self @@ -3093,6 +3193,12 @@ impl Metrics { throttle_sleep_duration_millis: current .throttle_sleep_duration_millis .saturating_sub(start.throttle_sleep_duration_millis), + disk_io_budget_wait_events: current + .disk_io_budget_wait_events + .saturating_sub(start.disk_io_budget_wait_events), + disk_io_budget_wait_duration_millis: current + .disk_io_budget_wait_duration_millis + .saturating_sub(start.disk_io_budget_wait_duration_millis), ilm_actions: current.ilm_actions.saturating_sub(start.ilm_actions), lifecycle_expiry_actions: current .lifecycle_expiry_actions @@ -3366,6 +3472,10 @@ impl Metrics { .store(work.throttle_sleep_events, Ordering::Relaxed); self.last_scan_cycle_throttle_sleep_duration_millis .store(work.throttle_sleep_duration_millis, Ordering::Relaxed); + self.last_scan_cycle_disk_io_budget_wait_events + .store(work.disk_io_budget_wait_events, Ordering::Relaxed); + self.last_scan_cycle_disk_io_budget_wait_duration_millis + .store(work.disk_io_budget_wait_duration_millis, Ordering::Relaxed); self.last_scan_cycle_ilm_actions.store(work.ilm_actions, Ordering::Relaxed); self.last_scan_cycle_lifecycle_expiry_actions .store(work.lifecycle_expiry_actions, Ordering::Relaxed); @@ -3443,6 +3553,9 @@ impl Metrics { m.current_cycle_yield_duration_seconds = current_work.yield_duration_millis as f64 / 1000.0; m.current_cycle_throttle_sleep_events = current_work.throttle_sleep_events; m.current_cycle_throttle_sleep_duration_seconds = current_work.throttle_sleep_duration_millis as f64 / 1000.0; + m.current_cycle_disk_io_budget_wait_events = current_work.disk_io_budget_wait_events; + m.current_cycle_disk_io_budget_wait_duration_seconds = + current_work.disk_io_budget_wait_duration_millis as f64 / 1000.0; m.current_cycle_ilm_actions = current_work.ilm_actions; m.current_cycle_lifecycle_expiry_actions = current_work.lifecycle_expiry_actions; m.current_cycle_lifecycle_transition_actions = current_work.lifecycle_transition_actions; @@ -3513,6 +3626,11 @@ impl Metrics { m.last_cycle_throttle_sleep_events = self.last_scan_cycle_throttle_sleep_events.load(Ordering::Relaxed); m.last_cycle_throttle_sleep_duration_seconds = self.last_scan_cycle_throttle_sleep_duration_millis.load(Ordering::Relaxed) as f64 / 1000.0; + m.last_cycle_disk_io_budget_wait_events = self.last_scan_cycle_disk_io_budget_wait_events.load(Ordering::Relaxed); + m.last_cycle_disk_io_budget_wait_duration_seconds = self + .last_scan_cycle_disk_io_budget_wait_duration_millis + .load(Ordering::Relaxed) as f64 + / 1000.0; m.last_cycle_ilm_actions = self.last_scan_cycle_ilm_actions.load(Ordering::Relaxed); m.last_cycle_lifecycle_expiry_actions = self.last_scan_cycle_lifecycle_expiry_actions.load(Ordering::Relaxed); m.last_cycle_lifecycle_transition_actions = self.last_scan_cycle_lifecycle_transition_actions.load(Ordering::Relaxed); @@ -3593,6 +3711,15 @@ impl Metrics { Ok(reason) => reason.clone(), Err(poisoned) => poisoned.into_inner().clone(), }, + guardrail_veto_total: self.scanner_guardrail_veto_total.load(Ordering::Relaxed), + disk_io_budget_wait_events: self.scanner_disk_io_budget_wait_events.load(Ordering::Relaxed), + disk_io_budget_wait_millis: self.scanner_disk_io_budget_wait_duration_millis.load(Ordering::Relaxed), + last_disk_io_budget_wait_unix_secs: self.scanner_disk_io_budget_last_wait_unix_secs.load(Ordering::Relaxed), + last_guardrail_veto_unix_secs: self.scanner_guardrail_last_veto_unix_secs.load(Ordering::Relaxed), + last_guardrail_veto_reason: match self.scanner_guardrail_last_veto_reason.lock() { + Ok(reason) => reason.clone(), + Err(poisoned) => poisoned.into_inner().clone(), + }, }; m.throttle_idle_mode_enabled = self.scanner_throttle_idle_mode_enabled.load(Ordering::Relaxed); m.throttle_sleep_factor = self.scanner_throttle_sleep_factor_micros.load(Ordering::Relaxed) as f64 / 1_000_000.0; @@ -4065,6 +4192,32 @@ mod tests { assert_eq!(lifecycle_checked, 4, "ILM-checked versions tracked on the Lifecycle source"); } + #[tokio::test] + async fn report_includes_disk_io_budget_waits_in_pacing_and_freshness() { + let metrics = Metrics::new(); + metrics.record_scanner_disk_io_budget_wait(Duration::from_millis(250)); + metrics.record_scanner_disk_io_budget_wait(Duration::from_millis(750)); + metrics.record_scan_cycle_complete(true, Duration::from_secs(10)); + metrics.record_scan_cycle_work(ScanCycleWorkSnapshot { + disk_io_budget_wait_events: 2, + disk_io_budget_wait_duration_millis: 1_000, + ..Default::default() + }); + + let report = metrics.report().await; + + assert_eq!(report.usage_freshness.disk_io_budget_wait_events, 2); + assert_eq!(report.usage_freshness.disk_io_budget_wait_millis, 1_000); + assert!(report.usage_freshness.last_disk_io_budget_wait_unix_secs > 0); + assert_eq!(report.last_cycle_disk_io_budget_wait_events, 2); + assert_eq!(report.last_cycle_disk_io_budget_wait_duration_seconds, 1.0); + assert_eq!(report.pacing_pressure.primary_pressure, "disk_io_budget"); + assert!(report.pacing_pressure.last_cycle_pause_observed); + assert_eq!(report.pacing_pressure.last_cycle_disk_io_budget_wait_ratio, 0.1); + assert_eq!(report.pacing_pressure.last_cycle_total_pause_ratio, 0.1); + assert_eq!(report.pacing_pressure.last_cycle_throttle_sleep_ratio, 0.0); + } + #[tokio::test] async fn report_derives_scanner_pacing_pressure() { let metrics = Metrics::new(); diff --git a/crates/scanner/Cargo.toml b/crates/scanner/Cargo.toml index 6e92392e4..0205328c6 100644 --- a/crates/scanner/Cargo.toml +++ b/crates/scanner/Cargo.toml @@ -31,6 +31,7 @@ workspace = true [features] default = [] +diagnostic-read-version-caller = ["rustfs-ecstore/diagnostic-read-version-caller"] hotpath = [ "hotpath/hotpath", "hotpath/tokio", diff --git a/crates/scanner/src/data_usage_define.rs b/crates/scanner/src/data_usage_define.rs index 90ac72464..541d3cc58 100644 --- a/crates/scanner/src/data_usage_define.rs +++ b/crates/scanner/src/data_usage_define.rs @@ -56,9 +56,10 @@ pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin"; pub(crate) const DATA_USAGE_CACHE_KEY_FORMAT: u16 = 1; const DATA_USAGE_CACHE_SAVE_RETRIES: u32 = 2; -const DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_MAX: u64 = 5; +const DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_BASE: u64 = 5; const DATA_USAGE_CACHE_BACKUP_SAVE_RETRIES: u32 = 0; const DATA_USAGE_CACHE_SAVE_RETRY_BACKOFF_MAX: Duration = Duration::from_millis(350); +const DATA_USAGE_CACHE_RECONCILE_BACKOFF_MAX: Duration = Duration::from_millis(150); const DATA_USAGE_CACHE_PERSISTENCE_MARGIN: Duration = Duration::from_secs(5); const METRIC_CACHE_SAVE_ATTEMPT_TOTAL: &str = "rustfs_scanner_cache_save_attempt_total"; const METRIC_CACHE_SAVE_TIMEOUT_TOTAL: &str = "rustfs_scanner_cache_save_timeout_total"; @@ -100,19 +101,17 @@ pub(crate) async fn read_config_with_revision( store: Arc, path: &str, ) -> StorageResult<(Option>, DataUsageCacheRevision)> { - match store - .get_object_reader( - RUSTFS_META_BUCKET, - path, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let read = store.get_object_reader(RUSTFS_META_BUCKET, path, None, HeaderMap::new(), &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerRevisionRead, + read, + ); + match read.await { Ok(mut reader) => { let revision = reader .object_info @@ -170,19 +169,17 @@ fn error_chain_has_usage_floor_corruption_signature(error: &(dyn std::error::Err /// Read only the object revision without materializing its body. pub(crate) async fn read_config_revision(store: Arc, path: &str) -> StorageResult { - match store - .get_object_reader( - RUSTFS_META_BUCKET, - path, - None, - HeaderMap::new(), - &ObjectOptions { - no_lock: true, - ..Default::default() - }, - ) - .await - { + let opts = ObjectOptions { + no_lock: true, + ..Default::default() + }; + let read = store.get_object_reader(RUSTFS_META_BUCKET, path, None, HeaderMap::new(), &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let read = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerRevisionRead, + read, + ); + match read.await { Ok(reader) => reader .object_info .etag @@ -518,6 +515,14 @@ impl Serialize for DataUsageScanCoverageReceipt { } } +/// Coverage follows depth-first directory traversal, rather than raw path order. +/// A completed subtree includes its descendants but never an unfinished ancestor. +pub(crate) fn scan_frontier_covers_path(path: &str, frontier: &str) -> bool { + let ancestor = frontier.strip_prefix(path).is_some_and(|suffix| suffix.starts_with('/')); + let descendant = path.strip_prefix(frontier).is_some_and(|suffix| suffix.starts_with('/')); + (path.split('/').cmp(frontier.split('/')).is_le() && !ancestor) || descendant +} + struct CheckpointDigestWriter(Sha256); impl std::io::Write for CheckpointDigestWriter { @@ -895,6 +900,16 @@ impl DataUsageCache { } } + pub(crate) fn has_scan_position(&self) -> bool { + self.info.scan_resume_after.is_some() + || self.info.scan_checkpoint.is_some() + || self.info.scan_raw_enumeration_cursor.is_some() + || self.info.scan_raw_enumeration_page_index.is_some() + || self.info.scan_coverage_receipt.is_some() + } + + /// Resume and reuse under one plan; fixtures that model a single digest. + #[cfg(test)] pub(crate) fn prepare_bucket_checkpoint( &mut self, name: &str, @@ -903,6 +918,34 @@ impl DataUsageCache { source: DataUsageCacheSource, scan_plan_digest: DataUsageScanPlanDigest, identity: DataUsageScanIdentity, + ) -> DataUsageCachePrepareOutcome { + self.prepare_bucket_checkpoint_with_coverage( + name, + next_cycle, + leader_epoch, + source, + scan_plan_digest, + scan_plan_digest, + identity, + ) + } + + /// `scan_plan_digest` is the reuse plan a complete cache is current under; + /// `checkpoint_plan_digest` is the coverage plan a sweep is started and + /// resumed under (`scanner_activity_coverage_digest`). The two differ by + /// the inputs that force a new walk without changing the namespace a + /// walk traverses, so a coordinator restart invalidates finished results + /// but not a sweep in progress. + #[allow(clippy::too_many_arguments)] + pub(crate) fn prepare_bucket_checkpoint_with_coverage( + &mut self, + name: &str, + next_cycle: u64, + leader_epoch: u64, + source: DataUsageCacheSource, + scan_plan_digest: DataUsageScanPlanDigest, + checkpoint_plan_digest: DataUsageScanPlanDigest, + identity: DataUsageScanIdentity, ) -> DataUsageCachePrepareOutcome { if self.info.next_cycle > next_cycle { return DataUsageCachePrepareOutcome::RejectedNewerCycle; @@ -910,6 +953,8 @@ impl DataUsageCache { if self.info.leader_epoch > leader_epoch { return DataUsageCachePrepareOutcome::RejectedNewerLeader; } + let previous_cycle = self.info.next_cycle; + let previous_leader_epoch = self.info.leader_epoch; // A checkpoint is a durable traversal frontier, not a publication // result. A newer leader may adopt it only when the frontier is // independently fenced by the bucket identity and has a validated @@ -938,22 +983,13 @@ impl DataUsageCache { if reusable && self.info.snapshot_complete && self.info.scan_progress.is_none() - && self.info.scan_checkpoint.is_none() - && self.info.scan_raw_enumeration_cursor.is_none() - && self.info.scan_raw_enumeration_page_index.is_none() - && self.info.scan_resume_after.is_none() - && self.info.scan_coverage_receipt.is_none() + && !self.has_scan_position() && self.info.scan_plan_digest == Some(scan_plan_digest) { return self.prepare_for_scan(name, next_cycle, leader_epoch, source, scan_plan_digest, true); } if !reusable { - let had_resume_state = self.info.scan_checkpoint.is_some() - || self.info.scan_resume_after.is_some() - || self.info.scan_coverage_receipt.is_some() - || self.info.scan_raw_enumeration_cursor.is_some() - || self.info.scan_raw_enumeration_page_index.is_some(); - if had_resume_state { + if self.has_scan_position() { let reason = if name == DATA_USAGE_ROOT { "invalid_bucket_scope" } else if !identity.is_valid() { @@ -1007,11 +1043,7 @@ impl DataUsageCache { if self.validated_raw_enumeration_page_index().is_none() { self.info.scan_raw_enumeration_page_index = None; } - let cursor_is_valid = (self.info.scan_checkpoint.is_none() - && self.info.scan_raw_enumeration_cursor.is_none() - && self.info.scan_raw_enumeration_page_index.is_none() - && self.info.scan_resume_after.is_none() - && self.info.scan_coverage_receipt.is_none()) + let cursor_is_valid = !self.has_scan_position() || self.validated_scan_frontier().is_some() || self.info.scan_raw_enumeration_cursor.is_some() || self.info.scan_raw_enumeration_page_index.is_some(); @@ -1039,19 +1071,15 @@ impl DataUsageCache { // so it observes nothing under the plan that finished the previous // sweep. Keeping that plan would mark every sweep of a bucket written // between cycles as mixed, and the bucket could never certify. - let has_position = self.info.scan_resume_after.is_some() - || self.info.scan_checkpoint.is_some() - || self.info.scan_raw_enumeration_cursor.is_some() - || self.info.scan_raw_enumeration_page_index.is_some() - || self.info.scan_coverage_receipt.is_some(); + let has_position = self.has_scan_position(); if let Some(progress) = &mut self.info.scan_progress && has_position { - progress.requested_plan = scan_plan_digest; + progress.requested_plan = checkpoint_plan_digest; } else { self.info.scan_progress = Some(DataUsageScanProgress { - started_plan: scan_plan_digest, - requested_plan: scan_plan_digest, + started_plan: checkpoint_plan_digest, + requested_plan: checkpoint_plan_digest, }); self.info.scan_resume_after = None; self.info.scan_checkpoint = None; @@ -1062,6 +1090,23 @@ impl DataUsageCache { // Old readers do not understand coverage sweeps. An absent plan makes // their existing prepare path rebuild instead of promoting mixed data. self.info.scan_plan_digest = None; + debug!( + target: "rustfs::scanner::data_usage", + event = EVENT_SCANNER_CACHE_LOAD_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_CACHE, + state = "bucket_checkpoint_prepared", + bucket = %name, + pool = source.pool_index, + set = source.set_index, + cycle = next_cycle, + leader_epoch, + previous_cycle, + previous_leader_epoch, + resume_position = self.has_scan_position(), + mixed_coverage = self.info.scan_progress.is_some_and(|progress| progress.started_plan != progress.requested_plan), + "Scanner bucket checkpoint prepared for a coverage sweep" + ); if reusable { DataUsageCachePrepareOutcome::Reused } else { @@ -1070,17 +1115,22 @@ impl DataUsageCache { } fn coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> { - self.coverage_prefix_digest_with_epoch(through, None) + self.coverage_prefix_digest_with_epoch(through, None, 3) + } + + fn v2_coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> { + self.coverage_prefix_digest_with_epoch(through, None, 2) } fn legacy_coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> { - self.coverage_prefix_digest_with_epoch(through, Some(self.info.leader_epoch)) + self.coverage_prefix_digest_with_epoch(through, Some(self.info.leader_epoch), 0) } fn coverage_prefix_digest_with_epoch( &self, through: &str, legacy_leader_epoch: Option, + domain_version: u8, ) -> Result<[u8; 32], serde_json::Error> { let mut writer = CheckpointDigestWriter(Sha256::new()); if let Some(leader_epoch) = legacy_leader_epoch { @@ -1104,10 +1154,12 @@ impl DataUsageCache { // process-local leader epoch is intentionally excluded because it // changes during a safe handoff. The domain version also keeps // old epoch-bound receipts from colliding with this digest. + // Domain 3 permits object-health debt behind the frontier. Older + // readers must reject it because they skip covered failures. serde_json::to_writer( &mut writer, &( - 2u8, + domain_version, &self.info.name, self.info.scan_identity, self.info.source, @@ -1121,6 +1173,11 @@ impl DataUsageCache { .cache .iter() .filter(|(key, _)| { + // Sign exactly the subtrees the reader can reuse, including + // descendants whose raw path order differs from traversal order. + if legacy_leader_epoch.is_none() && domain_version >= 3 { + return scan_frontier_covers_path(key, through); + } let ancestor = through .strip_prefix(key.as_str()) .is_some_and(|suffix| suffix.starts_with('/')); @@ -1156,6 +1213,7 @@ impl DataUsageCache { .is_some_and(|suffix| suffix.starts_with('/')) && self.find(&receipt.through).is_some() && (self.coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest) + || self.v2_coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest) || self.legacy_coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest))) .then_some(receipt.through.as_str()) } diff --git a/crates/scanner/src/data_usage_define/persistence.rs b/crates/scanner/src/data_usage_define/persistence.rs index 00851af64..29d15e4de 100644 --- a/crates/scanner/src/data_usage_define/persistence.rs +++ b/crates/scanner/src/data_usage_define/persistence.rs @@ -370,6 +370,17 @@ impl DataUsageCache { } }; + #[cfg(feature = "diagnostic-read-version-caller")] + let load_fut = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::DataUsageCacheRead, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerCacheLoad, + ); + load_fut.await + }, + ); + match timeout(timeout_duration, load_fut).await { Ok(Ok((Some(cache), revision, _))) => Ok(DataUsageCacheLoadAttempt::Loaded { cache: Box::new(cache), @@ -386,16 +397,36 @@ impl DataUsageCache { crate::runtime_config::scanner_cache_save_timeout() } - pub(crate) fn persistence_timeout() -> Duration { - Self::cache_save_timeout() + pub(crate) fn publication_timeout() -> Duration { + // Aggregate statistics publication must fit its lease window rather + // than inherit the longer checkpoint save/reconciliation allowance. + let timeout_duration = Self::cache_save_timeout(); + timeout_duration .saturating_mul(DATA_USAGE_CACHE_SAVE_RETRIES + 1) .saturating_add(DATA_USAGE_CACHE_SAVE_RETRY_BACKOFF_MAX) - .saturating_add(Self::backup_cache_save_timeout(Self::cache_save_timeout())) + .saturating_add(Self::backup_cache_save_timeout(timeout_duration, Duration::ZERO)) .saturating_add(DATA_USAGE_CACHE_PERSISTENCE_MARGIN) } - fn backup_cache_save_timeout(timeout_duration: Duration) -> Duration { - timeout_duration.min(Duration::from_secs(DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_MAX)) + pub(crate) fn persistence_timeout() -> Duration { + // Each save attempt can be followed by a full read-after-error + // reconciliation attempt, including the best-effort backup. + Self::cache_save_timeout() + .saturating_mul(DATA_USAGE_CACHE_SAVE_RETRIES + DATA_USAGE_CACHE_BACKUP_SAVE_RETRIES + 2) + .saturating_mul(2) + .saturating_add(DATA_USAGE_CACHE_SAVE_RETRY_BACKOFF_MAX) + .saturating_add(DATA_USAGE_CACHE_RECONCILE_BACKOFF_MAX) + .saturating_add(DATA_USAGE_CACHE_PERSISTENCE_MARGIN) + } + + fn backup_cache_save_timeout(timeout_duration: Duration, primary_elapsed: Duration) -> Duration { + // Both copies write and verify the same body. Allow headroom over the + // observed primary cost without exceeding the configured attempt cap; + // fast checkpoints keep the short best-effort budget and zero retries. + primary_elapsed + .saturating_mul(2) + .max(Duration::from_secs(DATA_USAGE_CACHE_BACKUP_SAVE_TIMEOUT_SECS_BASE)) + .min(timeout_duration) } fn record_save_attempt(path_type: &'static str, result: &'static str, duration: Duration) { @@ -617,6 +648,7 @@ impl DataUsageCache { let timeout_duration = Self::cache_save_timeout(); let path = path_join_buf(&[BUCKET_META_PREFIX, name]); + let primary_save_started = Instant::now(); Self::save_path_with_retry( store.clone(), &path, @@ -630,7 +662,7 @@ impl DataUsageCache { let backup_name = format!("{name}.bkp"); let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]); - let backup_timeout_duration = Self::backup_cache_save_timeout(timeout_duration); + let backup_timeout_duration = Self::backup_cache_save_timeout(timeout_duration, primary_save_started.elapsed()); let backup_revision = revisions.and_then(|revisions| revisions.backup.clone()); if revisions.is_some() && backup_revision.is_none() { return Ok(()); @@ -669,6 +701,28 @@ impl DataUsageCache { mod tests { use super::*; + #[test] + fn backup_budget_tracks_cost_within_configured_cap() { + for (cap, cost, expected) in [ + (Duration::ZERO, Duration::ZERO, Duration::ZERO), + (Duration::from_secs(14), Duration::ZERO, Duration::from_secs(5)), + (Duration::from_secs(14), Duration::from_millis(2499), Duration::from_secs(5)), + (Duration::from_secs(14), Duration::from_millis(2500), Duration::from_secs(5)), + (Duration::from_secs(14), Duration::from_millis(2501), Duration::from_millis(5002)), + (Duration::from_secs(14), Duration::from_secs(6), Duration::from_secs(12)), + (Duration::from_secs(14), Duration::from_secs(7), Duration::from_secs(14)), + (Duration::from_secs(14), Duration::from_secs(60), Duration::from_secs(14)), + (Duration::from_secs(2), Duration::from_secs(6), Duration::from_secs(2)), + (Duration::MAX, Duration::MAX, Duration::MAX), + ] { + assert_eq!( + DataUsageCache::backup_cache_save_timeout(cap, cost), + expected, + "cap={cap:?}, cost={cost:?}" + ); + } + } + #[test] fn interrupted_checkpoint_admission_is_not_retried() { let error = StorageError::Io(std::io::Error::new(std::io::ErrorKind::Interrupted, "cancelled while queued")); diff --git a/crates/scanner/src/data_usage_define/tests.rs b/crates/scanner/src/data_usage_define/tests.rs index 5060911d2..fb6176227 100644 --- a/crates/scanner/src/data_usage_define/tests.rs +++ b/crates/scanner/src/data_usage_define/tests.rs @@ -71,6 +71,10 @@ struct BackupFallbackStore { recover_main_revision: AtomicBool, backup_reads: Mutex, puts: Mutex>, + put_delays: [Duration; 2], + read_delays: [Duration; 2], + put_attempts: [AtomicUsize; 2], + main_put_failures: usize, } impl BackupFallbackStore { @@ -82,6 +86,10 @@ impl BackupFallbackStore { recover_main_revision: AtomicBool::new(recover_main_revision), backup_reads: Mutex::new(0), puts: Mutex::new(Vec::new()), + put_delays: [Duration::ZERO; 2], + read_delays: [Duration::ZERO; 2], + put_attempts: std::array::from_fn(|_| AtomicUsize::new(0)), + main_put_failures: 0, } } @@ -93,6 +101,10 @@ impl BackupFallbackStore { recover_main_revision: AtomicBool::new(true), backup_reads: Mutex::new(0), puts: Mutex::new(Vec::new()), + put_delays: [Duration::ZERO; 2], + read_delays: [Duration::ZERO; 2], + put_attempts: std::array::from_fn(|_| AtomicUsize::new(0)), + main_put_failures: 0, } } @@ -113,6 +125,18 @@ impl BackupFallbackStore { enum CacheReadBody { Bytes(Vec), PrefixThenError(Vec), + #[cfg(feature = "diagnostic-read-version-caller")] + Pending, +} + +#[cfg(feature = "diagnostic-read-version-caller")] +struct PendingCacheReader; + +#[cfg(feature = "diagnostic-read-version-caller")] +impl AsyncRead for PendingCacheReader { + fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll> { + Poll::Pending + } } #[derive(Debug)] @@ -138,7 +162,10 @@ impl AsyncRead for PrefixThenErrorReader { struct CacheReadStore { main: CacheReadBody, backup: Option>, + legacy: Option>, puts: AtomicUsize, + #[cfg(feature = "diagnostic-read-version-caller")] + entry_sources: Mutex>>, } #[derive(Debug, Default)] @@ -152,7 +179,10 @@ impl CacheReadStore { Self { main, backup, + legacy: None, puts: AtomicUsize::new(0), + #[cfg(feature = "diagnostic-read-version-caller")] + entry_sources: Mutex::new(Vec::new()), } } @@ -163,6 +193,8 @@ impl CacheReadStore { prefix: Cursor::new(prefix), failed: false, }), + #[cfg(feature = "diagnostic-read-version-caller")] + CacheReadBody::Pending => Box::new(PendingCacheReader), }; ScannerGetObjectReader { stream, @@ -194,6 +226,18 @@ impl ObjectIO for CacheReadStore { _h: Self::HeaderMap, _opts: &Self::ObjectOptions, ) -> StorageResult { + #[cfg(feature = "diagnostic-read-version-caller")] + self.entry_sources + .lock() + .await + .push(crate::storage_api::ecstore_diagnostic::current_read_version_entry_source()); + if bucket == DATA_USAGE_BUCKET.as_str() { + return self + .legacy + .clone() + .map(|data| Self::reader(CacheReadBody::Bytes(data), "legacy-etag")) + .ok_or(Error::FileNotFound); + } if bucket != RUSTFS_META_BUCKET { return Err(Error::FileNotFound); } @@ -326,6 +370,10 @@ impl ObjectIO for BackupFallbackStore { let main_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]); let backup_path = format!("{main_path}.bkp"); + let delay = self.read_delays[usize::from(object.ends_with(".bkp"))]; + if !delay.is_zero() { + sleep(delay).await; + } if object == main_path { let read = self.main_reads.fetch_add(1, Ordering::SeqCst); if read == 0 || !self.recover_main_revision.load(Ordering::SeqCst) { @@ -361,6 +409,15 @@ impl ObjectIO for BackupFallbackStore { .as_ref() .and_then(HTTPPreconditions::if_none_match_value) .map(str::to_owned); + let path_index = usize::from(object.ends_with(".bkp")); + let attempt = self.put_attempts[path_index].fetch_add(1, Ordering::SeqCst); + let delay = self.put_delays[path_index]; + if !delay.is_zero() { + sleep(delay).await; + } + if path_index == 0 && attempt < self.main_put_failures { + return Err(Error::other("ambiguous cache write failure")); + } self.puts.lock().await.push(CachePutRecord { object: object.to_string(), if_match, @@ -400,6 +457,53 @@ fn cache_revisions_map_to_compare_and_swap_preconditions() { assert!(existing.if_none_match_value().is_none()); } +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +async fn diagnostic_revision_readers_keep_scanner_or_earlier_entry() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + let path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]); + let payload = b"scanner revision diagnostic".to_vec(); + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + let store = Arc::new(CacheReadStore::new(CacheReadBody::Bytes(payload.clone()), None)); + let read = async { + let (body, revision) = read_config_with_revision(store.clone(), &path) + .await + .expect("revision and body read"); + assert_eq!(body, Some(payload.clone())); + assert_eq!(revision, DataUsageCacheRevision::Etag("main-etag".to_string())); + assert_eq!( + read_config_revision(store.clone(), &path).await.expect("revision-only read"), + DataUsageCacheRevision::Etag("main-etag".to_string()) + ); + let missing = format!("{path}.missing"); + assert_eq!( + read_config_with_revision(store.clone(), &missing) + .await + .expect("missing body is harmless"), + (None, DataUsageCacheRevision::Missing) + ); + assert_eq!( + read_config_revision(store.clone(), &missing) + .await + .expect("missing revision is harmless"), + DataUsageCacheRevision::Missing + ); + }; + match earlier { + Some(source) => with_read_version_entry(source, read).await, + None => read.await, + } + assert_eq!( + *store.entry_sources.lock().await, + vec![Some(earlier.unwrap_or(ReadVersionEntrySource::ScannerRevisionRead)); 4], + "both readers and missing paths preserve the earliest diagnostic source" + ); + assert_eq!(current_read_version_entry_source(), None, "reader scopes must not leak to the caller"); + } +} + #[tokio::test] async fn backup_cache_load_recovers_main_revision_before_cas_save() { let mut expected = DataUsageCache::default(); @@ -497,6 +601,71 @@ async fn cache_save_reconciles_an_ambiguous_committed_write() { ); } +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +async fn diagnostic_cache_load_attempts_preserve_source_through_fallback_and_timeout() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + + let mut cache = DataUsageCache::default(); + cache.info.name = "diagnostic-cache".to_string(); + let bytes = cache.marshal_msg().expect("cache fixture should serialize"); + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for case in ["primary", "legacy", "missing", "body_error", "timeout"] { + let body = match case { + "body_error" => CacheReadBody::PrefixThenError(vec![0x81, 0xa4, b'n', b'a']), + "timeout" => CacheReadBody::Pending, + _ => CacheReadBody::Bytes(bytes.clone()), + }; + let mut fixture = CacheReadStore::new(body, None); + if case == "legacy" { + fixture.legacy = Some(bytes.clone()); + } + let store = Arc::new(fixture); + let name = if matches!(case, "legacy" | "missing") { + "diagnostic-legacy-cache" + } else { + DATA_USAGE_CACHE_NAME + }; + let timeout = if case == "timeout" { + Duration::from_millis(1) + } else { + Duration::from_secs(1) + }; + let load = DataUsageCache::try_load_inner(store.clone(), name, timeout); + let attempt = match earlier { + Some(source) => with_read_version_entry(source, load).await, + None => load.await, + } + .expect("diagnostic wrapper must preserve load-attempt results"); + match (case, attempt) { + ("primary", DataUsageCacheLoadAttempt::Loaded { cache, revision }) => { + assert_eq!(cache.info.name, "diagnostic-cache"); + assert_eq!(revision, Some(DataUsageCacheRevision::Etag("main-etag".to_string()))); + } + ("legacy", DataUsageCacheLoadAttempt::Loaded { cache, revision }) => { + assert_eq!(cache.info.name, "diagnostic-cache"); + assert_eq!(revision, Some(DataUsageCacheRevision::Missing)); + } + ("missing", DataUsageCacheLoadAttempt::Missing { revision }) => { + assert_eq!(revision, Some(DataUsageCacheRevision::Missing)); + } + ("body_error" | "timeout", DataUsageCacheLoadAttempt::Retryable(_)) => {} + _ => panic!("unexpected load-attempt outcome for {case}"), + } + let calls = if matches!(case, "legacy" | "missing") { 2 } else { 1 }; + assert_eq!( + *store.entry_sources.lock().await, + vec![Some(earlier.unwrap_or(ReadVersionEntrySource::DataUsageCacheRead)); calls], + "primary and legacy reads must retain this attempt's entry source: {case}" + ); + assert_eq!(store.puts.load(Ordering::SeqCst), 0, "load attempts must not save: {case}"); + assert_eq!(current_read_version_entry_source(), None, "scope must restore even after timeout: {case}"); + } + } +} + #[tokio::test] async fn backup_cache_load_fails_closed_without_main_revision_quorum() { let backup = DataUsageCache::default().marshal_msg().expect("serialize backup cache"); @@ -2076,22 +2245,36 @@ fn prepare_bucket_checkpoint_migrates_legacy_epoch_bound_receipt() { ..Default::default() }, ); - let legacy_digest = cache - .legacy_coverage_prefix_digest("bucket/a") - .expect("legacy receipt digest"); - cache.info.scan_coverage_receipt = Some(DataUsageScanCoverageReceipt { - through: "bucket/a".to_string(), - digest: legacy_digest, - }); - assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); - - assert_eq!( - cache.prepare_bucket_checkpoint("bucket", 9, 2, source, TEST_PLAN_DIGEST, identity), - DataUsageCachePrepareOutcome::Reused - ); - assert_eq!(cache.info.leader_epoch, 2); - assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); - assert_ne!(cache.info.scan_coverage_receipt.expect("migrated receipt").digest, legacy_digest); + for legacy_digest in [ + cache + .legacy_coverage_prefix_digest("bucket/a") + .expect("epoch-bound receipt digest"), + cache.v2_coverage_prefix_digest("bucket/a").expect("v2 receipt digest"), + ] { + let mut cache = cache.clone(); + cache.info.scan_coverage_receipt = Some(DataUsageScanCoverageReceipt { + through: "bucket/a".to_string(), + digest: legacy_digest, + }); + assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); + assert_eq!( + cache.prepare_bucket_checkpoint("bucket", 9, 2, source, TEST_PLAN_DIGEST, identity), + DataUsageCachePrepareOutcome::Reused + ); + assert_eq!(cache.info.leader_epoch, 2); + assert_eq!(cache.validated_scan_frontier(), Some("bucket/a")); + let digest = cache.info.scan_coverage_receipt.as_ref().expect("migrated receipt").digest; + assert_ne!(digest, legacy_digest); + assert_ne!(digest, cache.v2_coverage_prefix_digest("bucket/a").expect("old reader v2 digest")); + assert_ne!( + digest, + cache + .legacy_coverage_prefix_digest("bucket/a") + .expect("old reader epoch digest") + ); + cache.cache.get_mut("bucket/a").expect("covered entry").failed_objects = 1; + assert!(cache.validated_scan_frontier().is_none(), "tampered health debt invalidates coverage"); + } } #[test] @@ -2553,6 +2736,88 @@ fn test_cache_path_type_distinguishes_main_and_backup() { assert_eq!(DataUsageCache::cache_path_type("buckets/.usage-cache.bin.bkp"), "backup"); } +#[tokio::test(start_paused = true)] +#[serial_test::serial] +async fn test_save_backup_budget_tracks_primary_cost() { + temp_env::async_with_vars([(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("14"))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + let mut store = BackupFallbackStore::new(b"old backup".to_vec(), true); + store.put_delays = [Duration::from_secs(6); 2]; + let store = Arc::new(store); + let revisions = DataUsageCacheRevisions { + main: DataUsageCacheRevision::Missing, + backup: Some(DataUsageCacheRevision::Etag("backup-etag".to_string())), + }; + DataUsageCache::default() + .save_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME, &revisions) + .await + .expect("a backup with the observed primary cost should complete"); + let puts = store.puts.lock().await; + assert_eq!(puts.len(), 2, "best-effort save success must not hide a stale backup"); + assert!(puts[1].object.ends_with(".bkp")); + assert_eq!(puts[0].if_none_match.as_deref(), Some("*")); + assert_eq!(puts[1].if_match.as_deref(), Some("backup-etag")); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial] +async fn test_save_backup_budget_still_bounds_a_stalled_backup() { + temp_env::async_with_vars([(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("14"))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + let mut store = BackupFallbackStore::new(b"old backup".to_vec(), true); + store.put_delays = [Duration::from_secs(6), Duration::from_secs(60)]; + let store = Arc::new(store); + let started = Instant::now(); + DataUsageCache::default() + .save(store.clone(), DATA_USAGE_CACHE_NAME) + .await + .expect("backup failure remains best effort after a successful main save"); + assert_eq!(store.puts.lock().await.len(), 1, "stalled backup must not commit"); + assert_eq!(*store.backup_reads.lock().await, 1, "one bounded reconciliation, no backup retry"); + assert!(started.elapsed() <= Duration::from_secs(6) + DataUsageCache::cache_save_timeout() + Duration::from_millis(10)); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +#[tokio::test(start_paused = true)] +#[serial_test::serial] +async fn test_checkpoint_budget_covers_retries_and_reconciliation() { + temp_env::async_with_vars([(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("14"))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + let cache = DataUsageCache::default(); + let encoded = rmp_serde::to_vec(&cache).expect("checkpoint should encode"); + for main_put_failures in [2, 3] { + let mut store = BackupFallbackStore::with_recovered_main(b"old backup".to_vec(), encoded.clone()); + store.put_delays = [Duration::from_secs(13), Duration::from_secs(60)]; + store.read_delays = [Duration::from_secs(13); 2]; + store.main_put_failures = main_put_failures; + let store = Arc::new(store); + let revisions = DataUsageCacheRevisions { + main: DataUsageCacheRevision::Missing, + backup: Some(DataUsageCacheRevision::Etag("backup-etag".to_string())), + }; + let started = Instant::now(); + timeout( + DataUsageCache::persistence_timeout(), + cache.save_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME, &revisions), + ) + .await + .expect("the outer checkpoint budget must cover save and reconciliation") + .expect("a successful or reconciled primary must survive a best-effort backup failure"); + assert_eq!(store.put_attempts[0].load(Ordering::SeqCst), 3); + assert_eq!(store.put_attempts[1].load(Ordering::SeqCst), 1, "backup writes must not retry"); + assert_eq!(*store.backup_reads.lock().await, 1, "backup must finish its bounded reconciliation"); + assert!(started.elapsed() < DataUsageCache::persistence_timeout()); + } + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + #[test] fn test_cache_save_timeout_uses_default_when_env_missing() { with_var_unset(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, || { @@ -2561,7 +2826,7 @@ fn test_cache_save_timeout_uses_default_when_env_missing() { DataUsageCache::cache_save_timeout(), Duration::from_secs(rustfs_config::DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS) ); - assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(52_350)); + assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(117_500)); }); crate::runtime_config::refresh_scanner_runtime_config_for_tests(); } @@ -2581,10 +2846,10 @@ fn test_cache_save_timeout_respects_env_and_minimum_bound() { } #[test] -fn test_cache_persistence_timeout_covers_all_save_attempts() { +fn test_cache_persistence_timeout_covers_all_save_and_reconciliation_attempts() { with_var(ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("7"), || { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); - assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(31_350)); + assert_eq!(DataUsageCache::persistence_timeout(), Duration::from_millis(61_500)); }); crate::runtime_config::refresh_scanner_runtime_config_for_tests(); } @@ -2841,3 +3106,39 @@ async fn test_retry_save_op_times_out_and_returns_error_after_retries() { assert!(result.is_err()); assert_eq!(attempts.load(Ordering::SeqCst), (DATA_USAGE_CACHE_SAVE_RETRIES + 1) as usize); } + +#[test] +fn scan_frontier_receipt_binds_nested_debt_outside_raw_lexical_prefix() { + let mut cache = DataUsageCache::default(); + cache.prepare_bucket_checkpoint("bucket", 8, 1, DataUsageCacheSource::new(0, 0), TEST_PLAN_DIGEST, valid_scan_identity()); + cache.replace("bucket", "", DataUsageEntry::default()); + cache.replace("bucket/a", "bucket", DataUsageEntry::default()); + cache.replace( + "bucket/a/z", + "bucket/a", + DataUsageEntry { + failed_objects: 1, + ..Default::default() + }, + ); + cache.replace( + "bucket/a-", + "bucket", + DataUsageEntry { + objects: 1, + size: 1, + ..Default::default() + }, + ); + cache + .seal_scan_frontier(Some("bucket/a-")) + .expect("seal traversal after failed subtree"); + assert_eq!(cache.validated_scan_frontier(), Some("bucket/a-")); + let digest = cache.info.scan_coverage_receipt.as_ref().expect("new receipt").digest; + assert_ne!(digest, cache.v2_coverage_prefix_digest("bucket/a-").expect("old reader digest")); + cache.cache.get_mut("bucket/a/z").expect("nested failure").failed_objects = 0; + assert!( + cache.validated_scan_frontier().is_none(), + "nested debt must be signed even when '/' sorts after '-'" + ); +} diff --git a/crates/scanner/src/lib.rs b/crates/scanner/src/lib.rs index 5a4b57d1e..aa126f168 100644 --- a/crates/scanner/src/lib.rs +++ b/crates/scanner/src/lib.rs @@ -86,15 +86,15 @@ pub use remote_scanner::{ pub use runtime_config::{apply_scanner_runtime_config, scanner_runtime_config_status, validate_scanner_runtime_config}; pub use rustfs_scanner_metrics::last_minute; pub use scanner::{ - SCANNER_RECOVERY_INTENT_ACTION_USAGE_FULL_REBUILD, ScannerCycleRecoveryMarker, ScannerCycleRecoveryStatus, - ScannerCycleScheduleStatus, ScannerPauseBacklogAlertReason, ScannerPauseBacklogPhase, ScannerPauseBacklogReplicaId, - ScannerPauseBacklogReplicaStatus, ScannerPauseBacklogReplicaStatusState, ScannerPauseBacklogStatus, - ScannerPauseBacklogThresholds, ScannerRecoveryIntentAcceptResult, ScannerRecoveryIntentConflict, ScannerRecoveryIntentRecord, - ScannerRecoveryIntentRequest, ScannerUsageStateResetResult, accept_scanner_usage_recovery_intent, - get_scanner_usage_recovery_intent, init_data_scanner, init_scanner_with_recovery, register_scanner_pause_backlog_retirement, - reset_scanner_cycle_recovery, reset_scanner_usage_state_for_full_rebuild, run_scanner_usage_recovery_intent, - scanner_cycle_recovery_status, scanner_cycle_schedule_status, scanner_pause_backlog_status, scanner_recovery_actor_sha256, - scanner_topology_digest, + SCANNER_RECOVERY_INTENT_ACTION_USAGE_FULL_REBUILD, ScannerBitrotStartSource, ScannerBitrotStatus, ScannerCycleRecoveryMarker, + ScannerCycleRecoveryStatus, ScannerCycleScheduleStatus, ScannerPauseBacklogAlertReason, ScannerPauseBacklogPhase, + ScannerPauseBacklogReplicaId, ScannerPauseBacklogReplicaStatus, ScannerPauseBacklogReplicaStatusState, + ScannerPauseBacklogStatus, ScannerPauseBacklogThresholds, ScannerRecoveryIntentAcceptResult, ScannerRecoveryIntentConflict, + ScannerRecoveryIntentRecord, ScannerRecoveryIntentRequest, ScannerUsageStateResetResult, + accept_scanner_usage_recovery_intent, get_scanner_usage_recovery_intent, init_data_scanner, init_scanner_with_recovery, + register_scanner_pause_backlog_retirement, reset_scanner_cycle_recovery, reset_scanner_usage_state_for_full_rebuild, + run_scanner_usage_recovery_intent, scanner_bitrot_status, scanner_cycle_recovery_status, scanner_cycle_schedule_status, + scanner_pause_backlog_status, scanner_recovery_actor_sha256, scanner_topology_digest, }; pub use scanner_io::{ ScannerDirtyUsageAckError, ScannerDirtyUsageBucket, ScannerDirtyUsageClearObserver, ScannerDirtyUsageMutationObserver, @@ -103,12 +103,15 @@ pub use scanner_io::{ acknowledge_scoped_dirty_usage, clear_dirty_usage_bucket, encode_durable_dirty_usage_producer_replay_record, record_dirty_usage_bucket, record_dirty_usage_bucket_from_producer, record_dirty_usage_bucket_from_producers, record_dirty_usage_object, record_dirty_usage_object_from_producer, record_scanner_maintenance_change, - replay_durable_dirty_usage_producer_record, scanner_activity_epoch, scanner_dirty_usage_bucket_generation, - scanner_dirty_usage_snapshot, scanner_dirty_usage_state, scanner_maintenance_generation, - set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, + record_status_class_mutation, replay_durable_dirty_usage_producer_record, scanner_activity_epoch, + scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state, + scanner_maintenance_generation, set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, }; pub use segment_invalidation::SegmentInvalidationProducerIdentity; -pub use sleeper::{DynamicSleeper, SCANNER_IDLE_MODE, SCANNER_SLEEPER}; +pub use sleeper::{ + DynamicSleeper, SCANNER_DISK_IO_BUDGET_MAX_WAIT, SCANNER_IDLE_MODE, SCANNER_SLEEPER, ScannerDiskIoBudgetStatus, + ScannerDiskMedia, scanner_disk_io_budget_status, +}; use std::sync::atomic::{AtomicU64, Ordering}; pub use storage_api::ScannerReplicationConfig as ReplicationConfig; pub use storage_api::scan::{ @@ -734,17 +737,23 @@ where S: ScannerObjectIO, { let mut reader = ScannerPutObjReader::from_vec(data); - api.put_object( - RUSTFS_META_BUCKET, - file, - &mut reader, - &ScannerObjectOptions { - max_parity: true, - http_preconditions: Some(preconditions), - ..Default::default() + let opts = ScannerObjectOptions { + max_parity: true, + http_preconditions: Some(preconditions), + ..Default::default() + }; + let write = api.put_object(RUSTFS_META_BUCKET, file, &mut reader, &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let write = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerConfigWrite, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerConfigWrite, + ); + write.await }, - ) - .await + ); + write.await } pub(crate) async fn save_config_with_publication_admission_for_epoch( @@ -859,19 +868,25 @@ where fence.to_string(), ); } - api.put_object( - RUSTFS_META_BUCKET, - file, - &mut reader, - &ScannerObjectOptions { - max_parity: true, - http_preconditions: Some(preconditions), - scanner_publication_commit_scope, - user_defined, - ..Default::default() + let opts = ScannerObjectOptions { + max_parity: true, + http_preconditions: Some(preconditions), + scanner_publication_commit_scope, + user_defined, + ..Default::default() + }; + let write = api.put_object(RUSTFS_META_BUCKET, file, &mut reader, &opts); + #[cfg(feature = "diagnostic-read-version-caller")] + let write = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerConfigWrite, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerConfigWrite, + ); + write.await }, - ) - .await + ); + write.await } pub(crate) async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> std::result::Result<(), DiskError> { diff --git a/crates/scanner/src/raw_page_index.rs b/crates/scanner/src/raw_page_index.rs index 023f9f585..dc8d75123 100644 --- a/crates/scanner/src/raw_page_index.rs +++ b/crates/scanner/src/raw_page_index.rs @@ -310,7 +310,8 @@ impl RawEnumerationPageIndex { return Err(RawEnumerationPageIndexError::StaleGeneration); } let entries_start = inner.validated_committed_entries()?.len(); - inner.commit_building_page(entries_start) + inner.commit_building_page(entries_start)?; + inner.pages.last().cloned().ok_or(RawEnumerationPageIndexError::EmptyCommit) } pub fn page(&self, page_index: u64, expected_digest: [u8; 32]) -> Result<&RawEnumerationPage, RawEnumerationPageIndexError> { @@ -347,8 +348,9 @@ impl RawEnumerationPageIndex { impl RawEnumerationPageIndexInner { // Callers validate the committed prefix at the restore/API boundary. The - // live writer maintains its entry count without revisiting that prefix. - fn commit_building_page(&mut self, entries_start: usize) -> Result { + // live writer maintains its entry count without revisiting that prefix, + // and the committed page moves into `pages` without a copy. + fn commit_building_page(&mut self, entries_start: usize) -> Result<(), RawEnumerationPageIndexError> { let Some(building) = self.building.as_ref() else { return Err(RawEnumerationPageIndexError::EmptyCommit); }; @@ -365,9 +367,9 @@ impl RawEnumerationPageIndexInner { }; let page = RawEnumerationPage::new(&self.parent, building); self.complete = page.terminal; - self.pages.push(page.clone()); + self.pages.push(page); self.generation = self.generation.saturating_add(1); - Ok(page) + Ok(()) } fn status(&self) -> RawEnumerationPageOwnerStatus { diff --git a/crates/scanner/src/raw_page_index/writer.rs b/crates/scanner/src/raw_page_index/writer.rs index 9cd7f04da..a6f93738d 100644 --- a/crates/scanner/src/raw_page_index/writer.rs +++ b/crates/scanner/src/raw_page_index/writer.rs @@ -112,8 +112,10 @@ impl RawEnumerationPageWriter { terminal: false, }); self.indexed.insert(entry.clone()); - building.entries.push(entry); - building.entries.sort(); + // The building page is normalized (strictly increasing) on every + // path that reaches here, so a positional insert keeps it so. + let position = building.entries.binary_search(&entry).unwrap_or_else(|position| position); + building.entries.insert(position, entry); building.terminal = false; self.inner.generation = self.inner.generation.saturating_add(1); } diff --git a/crates/scanner/src/remote_scanner.rs b/crates/scanner/src/remote_scanner.rs index 61036e1c8..932e73045 100644 --- a/crates/scanner/src/remote_scanner.rs +++ b/crates/scanner/src/remote_scanner.rs @@ -318,3 +318,60 @@ fn try_admit_remote_scanner_key(disk_key: String) -> Result, +} + +#[derive(Default)] +struct LocalDiskAdmissionState { + admission: Option, + holders: usize, +} + +/// One walker's stake in a [`LocalDiskAdmissionShare`]. +pub(crate) struct LocalDiskAdmissionHold { + share: Arc, +} + +impl LocalDiskAdmissionShare { + pub(crate) fn new() -> Arc { + Arc::new(Self { + state: Mutex::new(LocalDiskAdmissionState::default()), + }) + } + + pub(crate) fn acquire(self: &Arc, disk: &Disk) -> Result { + self.acquire_with(|| try_admit_remote_scanner(disk)) + } + + fn acquire_with( + self: &Arc, + admit: impl FnOnce() -> Result, + ) -> Result { + let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + if state.admission.is_none() { + state.admission = Some(admit()?); + } + state.holders = state.holders.saturating_add(1); + Ok(LocalDiskAdmissionHold { share: Arc::clone(self) }) + } +} + +impl Drop for LocalDiskAdmissionHold { + fn drop(&mut self) { + let mut state = self.share.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner); + state.holders = state.holders.saturating_sub(1); + if state.holders == 0 { + state.admission = None; + } + } +} diff --git a/crates/scanner/src/remote_scanner/stream.rs b/crates/scanner/src/remote_scanner/stream.rs index 8dc15dad6..df250f36b 100644 --- a/crates/scanner/src/remote_scanner/stream.rs +++ b/crates/scanner/src/remote_scanner/stream.rs @@ -18,7 +18,8 @@ use crate::scanner_budget::{ScannerCycleBudget, ScannerCycleBudgetConfig}; use crate::scanner_io::{ DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult, ScannerDiskScanOptions, ScannerDiskScanOutcome, ScannerIODisk, acquire_scanner_cache_locks, cache_root_entry_info, - current_cache_root_or_prepare_with_generation, persist_scanner_checkpoint, scanner_set_disk_inventory, + current_cache_root_or_prepare_with_generation, log_saved_partial_bucket_cache, persist_scanner_checkpoint, + scanner_set_disk_inventory, }; use crate::storage_api::owner::NS_SCANNER_PROTOCOL_VERSION; use crate::{ @@ -681,7 +682,9 @@ async fn scan_and_persist_local_bucket( .ok_or_else(|| RemoteScannerServerError::worker("remote namespace scanner object layer is unavailable"))?; validate_remote_scanner_request_fence_with_store(next_cycle, leader_epoch, store.clone()) .await - .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner leader fence is stale: {err}")))?; + .map_err(|err| { + RemoteScannerServerError::worker(format!("remote namespace scanner leader fence verification failed: {err}")) + })?; let source = cache_source_for_disk(disk.as_ref()) .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner source is unavailable: {err}")))?; let set = store @@ -737,7 +740,14 @@ async fn scan_and_persist_local_bucket( scan_mode, ) .await - .ok(), + .map_err(|err| { + RemoteScannerServerError::retry_bucket(format!( + "remote namespace scanner checkpoint identity unavailable during admission: {err}" + )) + })?, + // One digest travels on the wire: a remote sweep resumes under + // the reuse plan it was requested with. + checkpoint_plan_digest: None, }, ); match scan_state { @@ -886,7 +896,7 @@ async fn scan_and_persist_local_bucket( Err(err) => err, }; return Err(RemoteScannerServerError::worker(format!( - "remote namespace scanner leader fence changed during bucket scan: {}", + "remote namespace scanner leader fence verification failed during bucket scan: {}", err ))); } @@ -906,8 +916,11 @@ async fn scan_and_persist_local_bucket( scan_mode, ) .await - .ok() - != Some(expected) + .map_err(|err| { + RemoteScannerServerError::retry_bucket(format!( + "remote namespace scanner checkpoint identity unavailable after scanning: {err}" + )) + })? != Some(expected) { return Err(RemoteScannerServerError::retry_bucket( "remote scanner checkpoint identity changed during scanning", @@ -923,7 +936,9 @@ async fn scan_and_persist_local_bucket( phase_changed.notify_one(); validate_remote_scanner_request_fence_with_store(next_cycle, leader_epoch, store.clone()) .await - .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner leader fence changed: {err}")))?; + .map_err(|err| { + RemoteScannerServerError::worker(format!("remote namespace scanner leader fence verification failed: {err}")) + })?; let done_save = Metrics::time(Metric::SaveUsage); // Each physical main/backup PUT must still prove the epoch captured before // the scan. A movement transition that starts and ends during the scan @@ -943,7 +958,9 @@ async fn scan_and_persist_local_bucket( } validate_remote_scanner_request_fence_with_store(next_cycle, leader_epoch, store) .await - .map_err(|err| RemoteScannerServerError::worker(format!("remote namespace scanner leader fence changed: {err}")))?; + .map_err(|err| { + RemoteScannerServerError::worker(format!("remote namespace scanner leader fence verification failed: {err}")) + })?; if guard.is_lock_lost() { return Err(RemoteScannerServerError::worker( "remote namespace scanner cache lock was lost during persistence", @@ -951,6 +968,9 @@ async fn scan_and_persist_local_bucket( } if let Some(partial_result) = partial_result { + if matches!(partial_result, RemoteScannerFrameResult::Partial) { + log_saved_partial_bucket_cache(&cache, &cache_name); + } return Ok(partial_result); } if cache.info.source != Some(source) || !cache.info.snapshot_complete { diff --git a/crates/scanner/src/remote_scanner/stream/tests.rs b/crates/scanner/src/remote_scanner/stream/tests.rs index 7b5e2df0b..06f2e6328 100644 --- a/crates/scanner/src/remote_scanner/stream/tests.rs +++ b/crates/scanner/src/remote_scanner/stream/tests.rs @@ -785,6 +785,50 @@ fn admission_rejects_busy_disk_before_replay_state_changes() { assert!(try_admit_remote_scanner_key(key).is_ok()); } +#[test] +fn local_disk_admission_share_holds_one_permit_until_the_last_walker_leaves() { + let key = format!("test-disk-{}", Uuid::new_v4()); + let share = LocalDiskAdmissionShare::new(); + let first = share + .acquire_with(|| try_admit_remote_scanner_key(key.clone())) + .expect("first walker acquires the disk permit"); + let second = share + .acquire_with(|| try_admit_remote_scanner_key(key.clone())) + .expect("second walker shares the held permit"); + assert!( + matches!(try_admit_remote_scanner_key(key.clone()), Err(ScannerError::RemoteDiskBusy)), + "a remote request sees one busy disk while any walker is active" + ); + drop(first); + assert!( + matches!(try_admit_remote_scanner_key(key.clone()), Err(ScannerError::RemoteDiskBusy)), + "the permit is released by the last holder, not the first" + ); + drop(second); + assert!(try_admit_remote_scanner_key(key).is_ok(), "no walker left: the disk is free again"); +} + +#[test] +fn local_disk_admission_share_retries_after_a_rejected_admission() { + let key = format!("test-disk-{}", Uuid::new_v4()); + let share = LocalDiskAdmissionShare::new(); + let remote = try_admit_remote_scanner_key(key.clone()).expect("remote request occupies the disk"); + assert!( + matches!( + share.acquire_with(|| try_admit_remote_scanner_key(key.clone())), + Err(ScannerError::RemoteDiskBusy) + ), + "walkers do not pre-empt an active remote session" + ); + drop(remote); + let hold = share + .acquire_with(|| try_admit_remote_scanner_key(key.clone())) + .expect("a rejected attempt leaves the share empty for a later retry"); + assert!(matches!(try_admit_remote_scanner_key(key.clone()), Err(ScannerError::RemoteDiskBusy))); + drop(hold); + assert!(try_admit_remote_scanner_key(key).is_ok()); +} + #[test] fn replay_cache_requires_contiguous_sequence_per_disk_session() { let session_id = Uuid::new_v4(); diff --git a/crates/scanner/src/runtime_config.rs b/crates/scanner/src/runtime_config.rs index cdace6a8f..2551ca578 100644 --- a/crates/scanner/src/runtime_config.rs +++ b/crates/scanner/src/runtime_config.rs @@ -14,23 +14,25 @@ use crate::resolve_scanner_server_config; use crate::scanner_budget::ScannerCycleBudgetConfig; -use crate::sleeper::{SCANNER_SLEEPER, scanner_default_speed}; +use crate::sleeper::{SCANNER_SLEEPER, ScannerDiskIoBudgetConfig, ScannerIoBudgetLimit, scanner_default_speed}; use rustfs_config::{ DEFAULT_DELIMITER, DEFAULT_HEAL_BITROT_CYCLE_SECS, DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS, DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE, DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES, DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, - DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, - DEFAULT_SCANNER_SPEED, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, ENV_SCANNER_ALERT_EXCESS_FOLDERS, - ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, ENV_SCANNER_ALERT_EXCESS_VERSIONS, ENV_SCANNER_BITROT_CYCLE_SECS, - ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DIRECTORIES, - ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, ENV_SCANNER_IDLE_MODE, + DEFAULT_SCANNER_DISK_IO_BUDGET, DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, + DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED, DEFAULT_SCANNER_WALKERS_PER_DISK, + DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, ENV_SCANNER_ALERT_EXCESS_FOLDERS, ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, + ENV_SCANNER_ALERT_EXCESS_VERSIONS, ENV_SCANNER_BITROT_CYCLE_SECS, ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, + ENV_SCANNER_CYCLE_MAX_DIRECTORIES, ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, + ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, ENV_SCANNER_DISK_IO_BUDGET_IOPS, ENV_SCANNER_IDLE_MODE, ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, - ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, - SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, - SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, - SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, + ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_WALKERS_PER_DISK, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, + HEAL_SUB_SYS, MAX_SCANNER_WALKERS_PER_DISK, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, + SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, + SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, + SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, SCANNER_DISK_IO_BUDGET_IOPS, SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, SCANNER_SUB_SYS, - SCANNER_YIELD_EVERY_N_OBJECTS, ScannerSpeed, + SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, ScannerSpeed, server_config::{Config as ServerConfig, KVS}, }; use serde::Serialize; @@ -53,6 +55,8 @@ const ENV_SCANNER_START_DELAY_SECS_DEPRECATED: &str = "RUSTFS_DATA_SCANNER_START const NO_DEFAULT_CYCLE_OVERRIDE: u64 = 0; const MAX_SCANNER_DELAY_FACTOR: f64 = 10_000.0; const SCANNER_DELAY_RANGE_REASON: &str = "expected scanner delay between 0 and 10000"; +const SCANNER_WALKERS_PER_DISK_RANGE_REASON: &str = "expected walkers per disk between 1 and 16"; +const SCANNER_DISK_IO_BUDGET_REASON: &str = "expected auto, off, or a non-negative integer rate"; static SCANNER_DEFAULT_CYCLE_SECS: AtomicU64 = AtomicU64::new(NO_DEFAULT_CYCLE_OVERRIDE); static SCANNER_RUNTIME_CONFIG_GENERATION: AtomicU64 = AtomicU64::new(0); @@ -96,6 +100,12 @@ pub(crate) struct ScannerRuntimeConfig { pub(crate) max_concurrent_set_scans_source: ScannerRuntimeConfigSource, pub(crate) max_concurrent_disk_scans: usize, pub(crate) max_concurrent_disk_scans_source: ScannerRuntimeConfigSource, + pub(crate) walkers_per_disk: usize, + pub(crate) walkers_per_disk_source: ScannerRuntimeConfigSource, + pub(crate) disk_io_budget_iops: ScannerIoBudgetLimit, + pub(crate) disk_io_budget_iops_source: ScannerRuntimeConfigSource, + pub(crate) disk_io_budget_bytes_per_sec: ScannerIoBudgetLimit, + pub(crate) disk_io_budget_bytes_per_sec_source: ScannerRuntimeConfigSource, pub(crate) yield_every_n_objects: u64, pub(crate) yield_every_n_objects_source: ScannerRuntimeConfigSource, pub(crate) alert_excess_versions: u64, @@ -138,6 +148,12 @@ impl Default for ScannerRuntimeConfig { max_concurrent_set_scans_source: ScannerRuntimeConfigSource::Default, max_concurrent_disk_scans: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, max_concurrent_disk_scans_source: ScannerRuntimeConfigSource::Default, + walkers_per_disk: DEFAULT_SCANNER_WALKERS_PER_DISK, + walkers_per_disk_source: ScannerRuntimeConfigSource::Default, + disk_io_budget_iops: ScannerIoBudgetLimit::Auto, + disk_io_budget_iops_source: ScannerRuntimeConfigSource::Default, + disk_io_budget_bytes_per_sec: ScannerIoBudgetLimit::Auto, + disk_io_budget_bytes_per_sec_source: ScannerRuntimeConfigSource::Default, yield_every_n_objects: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, yield_every_n_objects_source: ScannerRuntimeConfigSource::Default, alert_excess_versions: DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS, @@ -181,6 +197,10 @@ pub struct ScannerRuntimeConfigStatus { pub cache_save_timeout_seconds: ScannerRuntimeConfigValue, pub max_concurrent_set_scans: ScannerRuntimeConfigValue, pub max_concurrent_disk_scans: ScannerRuntimeConfigValue, + pub walkers_per_disk: ScannerRuntimeConfigValue, + /// `auto`, `off`, or an explicit rate, as configured. + pub disk_io_budget_iops: ScannerRuntimeConfigValue, + pub disk_io_budget_bytes_per_sec: ScannerRuntimeConfigValue, pub yield_every_n_objects: ScannerRuntimeConfigValue, pub alert_excess_versions: ScannerRuntimeConfigValue, pub alert_excess_version_size: ScannerRuntimeConfigValue, @@ -274,6 +294,24 @@ fn parse_config_speed(value: String) -> Result Result { + ScannerIoBudgetLimit::parse(&value).ok_or_else(|| invalid_value(key, value, SCANNER_DISK_IO_BUDGET_REASON)) +} + +fn lookup_io_budget( + kvs: Option<&KVS>, + key: &'static str, + env_key: &'static str, +) -> Result<(ScannerIoBudgetLimit, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { + if let Some(value) = rustfs_utils::get_env_opt_str(env_key) { + return parse_config_io_budget(env_key, value).map(|limit| (limit, ScannerRuntimeConfigSource::Env)); + } + if let Some(value) = config_value(kvs, key, DEFAULT_SCANNER_DISK_IO_BUDGET) { + return parse_config_io_budget(key, value).map(|limit| (limit, ScannerRuntimeConfigSource::Config)); + } + Ok((ScannerIoBudgetLimit::Auto, ScannerRuntimeConfigSource::Default)) +} + fn parse_config_delay(key: &'static str, value: String) -> Result { let parsed = value .parse::() @@ -392,6 +430,15 @@ fn validate_persisted_scanner_runtime_config(config: &ServerConfig) -> Result<() validate_optional_config_u64(scanner_kvs, SCANNER_CACHE_SAVE_TIMEOUT, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS)?; validate_optional_config_usize(scanner_kvs, SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS)?; validate_optional_config_usize(scanner_kvs, SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS)?; + if let Some(value) = config_value(scanner_kvs, SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_WALKERS_PER_DISK) { + walkers_per_disk_from_count(SCANNER_WALKERS_PER_DISK, parse_config_usize(SCANNER_WALKERS_PER_DISK, value)?)?; + } + if let Some(value) = config_value(scanner_kvs, SCANNER_DISK_IO_BUDGET_IOPS, DEFAULT_SCANNER_DISK_IO_BUDGET) { + parse_config_io_budget(SCANNER_DISK_IO_BUDGET_IOPS, value)?; + } + if let Some(value) = config_value(scanner_kvs, SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, DEFAULT_SCANNER_DISK_IO_BUDGET) { + parse_config_io_budget(SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, value)?; + } validate_optional_config_u64(scanner_kvs, SCANNER_YIELD_EVERY_N_OBJECTS, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS)?; validate_optional_config_u64(scanner_kvs, SCANNER_ALERT_EXCESS_VERSIONS, DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS)?; validate_optional_config_u64(scanner_kvs, SCANNER_ALERT_EXCESS_VERSION_SIZE, DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE)?; @@ -484,6 +531,39 @@ fn cycle_duration_from_secs(key: &'static str, secs: u64) -> Result Result { + if (1..=MAX_SCANNER_WALKERS_PER_DISK).contains(&count) { + Ok(count) + } else { + Err(invalid_value(key, count.to_string(), SCANNER_WALKERS_PER_DISK_RANGE_REASON)) + } +} + +fn lookup_walkers_per_disk(kvs: Option<&KVS>) -> Result<(usize, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { + match rustfs_utils::get_env_parse_outcome::(ENV_SCANNER_WALKERS_PER_DISK) { + rustfs_utils::EnvParseOutcome::Parsed(count) => { + return walkers_per_disk_from_count(ENV_SCANNER_WALKERS_PER_DISK, count) + .map(|count| (count, ScannerRuntimeConfigSource::Env)); + } + rustfs_utils::EnvParseOutcome::Invalid => { + return Err(invalid_value( + ENV_SCANNER_WALKERS_PER_DISK, + "", + "expected unsigned integer count", + )); + } + rustfs_utils::EnvParseOutcome::Absent => {} + } + + if let Some(value) = config_value(kvs, SCANNER_WALKERS_PER_DISK, DEFAULT_SCANNER_WALKERS_PER_DISK) { + let count = parse_config_usize(SCANNER_WALKERS_PER_DISK, value)?; + return walkers_per_disk_from_count(SCANNER_WALKERS_PER_DISK, count) + .map(|count| (count, ScannerRuntimeConfigSource::Config)); + } + + Ok((DEFAULT_SCANNER_WALKERS_PER_DISK, ScannerRuntimeConfigSource::Default)) +} + fn lookup_start_delay(kvs: Option<&KVS>) -> Result<(Option, ScannerRuntimeConfigSource), ScannerRuntimeConfigError> { let aliases = [ENV_SCANNER_START_DELAY_SECS_DEPRECATED]; if let Some(secs) = rustfs_utils::get_env_opt_u64_with_aliases(ENV_SCANNER_START_DELAY_SECS, &aliases) { @@ -639,6 +719,14 @@ pub(crate) fn lookup_scanner_runtime_config( ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, )?; + let (walkers_per_disk, walkers_per_disk_source) = lookup_walkers_per_disk(scanner_kvs)?; + let (disk_io_budget_iops, disk_io_budget_iops_source) = + lookup_io_budget(scanner_kvs, SCANNER_DISK_IO_BUDGET_IOPS, ENV_SCANNER_DISK_IO_BUDGET_IOPS)?; + let (disk_io_budget_bytes_per_sec, disk_io_budget_bytes_per_sec_source) = lookup_io_budget( + scanner_kvs, + SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, + ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, + )?; let (yield_every_n_objects, yield_every_n_objects_source) = lookup_u64( scanner_kvs, SCANNER_YIELD_EVERY_N_OBJECTS, @@ -689,6 +777,12 @@ pub(crate) fn lookup_scanner_runtime_config( max_concurrent_set_scans_source, max_concurrent_disk_scans, max_concurrent_disk_scans_source, + walkers_per_disk, + walkers_per_disk_source, + disk_io_budget_iops, + disk_io_budget_iops_source, + disk_io_budget_bytes_per_sec, + disk_io_budget_bytes_per_sec_source, yield_every_n_objects, yield_every_n_objects_source, alert_excess_versions, @@ -702,6 +796,11 @@ pub(crate) fn lookup_scanner_runtime_config( fn apply_resolved_runtime_config(config: ScannerRuntimeConfig) { SCANNER_SLEEPER.update_from_runtime_config(config.delay, config.max_wait, config.idle_mode, config.yield_every_n_objects); + SCANNER_SLEEPER.update_disk_io_budget(ScannerDiskIoBudgetConfig { + iops: config.disk_io_budget_iops, + bytes_per_sec: config.disk_io_budget_bytes_per_sec, + speed: config.speed, + }); if let Ok(mut guard) = SCANNER_RUNTIME_CONFIG.write() { *guard = config; } @@ -825,6 +924,18 @@ pub fn scanner_runtime_config_status() -> ScannerRuntimeConfigStatus { value: config.max_concurrent_disk_scans, source: config.max_concurrent_disk_scans_source, }, + walkers_per_disk: ScannerRuntimeConfigValue { + value: config.walkers_per_disk, + source: config.walkers_per_disk_source, + }, + disk_io_budget_iops: ScannerRuntimeConfigValue { + value: config.disk_io_budget_iops.to_string(), + source: config.disk_io_budget_iops_source, + }, + disk_io_budget_bytes_per_sec: ScannerRuntimeConfigValue { + value: config.disk_io_budget_bytes_per_sec.to_string(), + source: config.disk_io_budget_bytes_per_sec_source, + }, yield_every_n_objects: ScannerRuntimeConfigValue { value: config.yield_every_n_objects, source: config.yield_every_n_objects_source, @@ -872,6 +983,10 @@ pub(crate) fn scanner_max_concurrent_disk_scans_configured() -> usize { current_scanner_runtime_config().max_concurrent_disk_scans } +pub(crate) fn scanner_walkers_per_disk_configured() -> usize { + current_scanner_runtime_config().walkers_per_disk +} + pub(crate) fn scanner_alert_excess_versions() -> u64 { current_scanner_runtime_config().alert_excess_versions } @@ -888,13 +1003,16 @@ pub(crate) fn scanner_alert_excess_folders() -> u64 { mod tests { use super::{ScannerRuntimeConfigSource, lookup_scanner_runtime_config, validate_scanner_runtime_config}; use crate::init_ecstore_config_for_scanner_tests; + use crate::sleeper::ScannerIoBudgetLimit; use rustfs_config::server_config::{Config as ServerConfig, KVS}; use rustfs_config::{ DEFAULT_DELIMITER, DEFAULT_HEAL_BITROT_CYCLE_SECS, ENV_SCANNER_BITROT_CYCLE_SECS, ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, - ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, SCANNER_BITROT_CYCLE, - SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, - SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed, + ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, ENV_SCANNER_DISK_IO_BUDGET_IOPS, ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, + ENV_SCANNER_WALKERS_PER_DISK, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, + SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, + SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, SCANNER_DISK_IO_BUDGET_IOPS, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, + SCANNER_WALKERS_PER_DISK, ScannerSpeed, }; use serial_test::serial; use std::collections::HashMap; @@ -1150,6 +1268,117 @@ mod tests { }); } + #[test] + #[serial] + fn scanner_runtime_config_walkers_per_disk_defaults_to_one_walker() { + let config = server_config_with_scanner(&[]); + + with_var_unset(ENV_SCANNER_WALKERS_PER_DISK, || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + + assert_eq!(resolved.walkers_per_disk, 1); + assert_eq!(resolved.walkers_per_disk_source, ScannerRuntimeConfigSource::Default); + }); + } + + #[test] + #[serial] + fn scanner_runtime_config_disk_io_budget_defaults_to_auto_and_parses_overrides() { + let config = server_config_with_scanner(&[ + (SCANNER_DISK_IO_BUDGET_IOPS, "250"), + (SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, "off"), + ]); + + with_var_unset(ENV_SCANNER_DISK_IO_BUDGET_IOPS, || { + with_var_unset(ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, || { + let resolved = + lookup_scanner_runtime_config(Some(&server_config_with_scanner(&[]))).expect("default budget config"); + assert_eq!(resolved.disk_io_budget_iops, ScannerIoBudgetLimit::Auto); + assert_eq!(resolved.disk_io_budget_iops_source, ScannerRuntimeConfigSource::Default); + assert_eq!(resolved.disk_io_budget_bytes_per_sec, ScannerIoBudgetLimit::Auto); + assert_eq!(resolved.disk_io_budget_bytes_per_sec_source, ScannerRuntimeConfigSource::Default); + + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("persisted budget config"); + assert_eq!(resolved.disk_io_budget_iops, ScannerIoBudgetLimit::Explicit(250)); + assert_eq!(resolved.disk_io_budget_iops_source, ScannerRuntimeConfigSource::Config); + assert_eq!(resolved.disk_io_budget_bytes_per_sec, ScannerIoBudgetLimit::Unlimited); + assert_eq!(resolved.disk_io_budget_bytes_per_sec_source, ScannerRuntimeConfigSource::Config); + + let invalid = server_config_with_scanner(&[(SCANNER_DISK_IO_BUDGET_IOPS, "fast")]); + let err = validate_scanner_runtime_config(&invalid).expect_err("a non-numeric budget must be rejected"); + assert!(err.to_string().contains(SCANNER_DISK_IO_BUDGET_IOPS), "{err}"); + assert!(lookup_scanner_runtime_config(Some(&invalid)).is_err()); + }); + }); + + with_var(ENV_SCANNER_DISK_IO_BUDGET_IOPS, Some("0"), || { + with_var(ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, Some("auto"), || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("env budget config"); + assert_eq!(resolved.disk_io_budget_iops, ScannerIoBudgetLimit::Explicit(0)); + assert_eq!(resolved.disk_io_budget_iops_source, ScannerRuntimeConfigSource::Env); + assert_eq!(resolved.disk_io_budget_bytes_per_sec, ScannerIoBudgetLimit::Auto); + assert_eq!(resolved.disk_io_budget_bytes_per_sec_source, ScannerRuntimeConfigSource::Env); + + super::apply_resolved_runtime_config(resolved); + let status = super::scanner_runtime_config_status(); + assert_eq!(status.disk_io_budget_iops.value, "0"); + assert_eq!(status.disk_io_budget_iops.source, ScannerRuntimeConfigSource::Env); + assert_eq!(status.disk_io_budget_bytes_per_sec.value, "auto"); + assert_eq!(status.disk_io_budget_bytes_per_sec.source, ScannerRuntimeConfigSource::Env); + }); + }); + with_var(ENV_SCANNER_DISK_IO_BUDGET_IOPS, Some("-5"), || { + assert!(lookup_scanner_runtime_config(Some(&config)).is_err()); + }); + super::refresh_scanner_runtime_config_for_tests(); + } + + #[test] + #[serial] + fn scanner_runtime_config_walkers_per_disk_accepts_persisted_range_and_rejects_zero_or_excess() { + with_var_unset(ENV_SCANNER_WALKERS_PER_DISK, || { + for (value, expected) in [("1", 1), ("4", 4), ("16", 16)] { + let config = server_config_with_scanner(&[(SCANNER_WALKERS_PER_DISK, value)]); + validate_scanner_runtime_config(&config).expect("walkers per disk inside the range should validate"); + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + assert_eq!(resolved.walkers_per_disk, expected, "persisted {value}"); + let expected_source = if expected == 1 { + ScannerRuntimeConfigSource::Default + } else { + ScannerRuntimeConfigSource::Config + }; + assert_eq!(resolved.walkers_per_disk_source, expected_source, "persisted {value}"); + } + for value in ["0", "17", "-1", "many"] { + let config = server_config_with_scanner(&[(SCANNER_WALKERS_PER_DISK, value)]); + let err = validate_scanner_runtime_config(&config).expect_err("walkers per disk outside the range must fail"); + assert!(err.to_string().contains("walkers_per_disk"), "persisted {value}: {err}"); + assert!( + lookup_scanner_runtime_config(Some(&config)).is_err(), + "persisted {value} must not resolve into a runtime config" + ); + } + }); + } + + #[test] + #[serial] + fn scanner_runtime_config_walkers_per_disk_env_overrides_and_is_range_checked() { + let config = server_config_with_scanner(&[(SCANNER_WALKERS_PER_DISK, "2")]); + + with_var(ENV_SCANNER_WALKERS_PER_DISK, Some("3"), || { + let resolved = lookup_scanner_runtime_config(Some(&config)).expect("scanner runtime config"); + assert_eq!(resolved.walkers_per_disk, 3); + assert_eq!(resolved.walkers_per_disk_source, ScannerRuntimeConfigSource::Env); + }); + for value in ["0", "17", "x"] { + with_var(ENV_SCANNER_WALKERS_PER_DISK, Some(value), || { + let err = lookup_scanner_runtime_config(Some(&config)).expect_err("env walkers per disk is range checked"); + assert!(err.to_string().contains(ENV_SCANNER_WALKERS_PER_DISK), "env {value}: {err}"); + }); + } + } + #[test] fn scanner_runtime_config_validation_rejects_non_default_target() { let config = server_config_with_scanner_target("analytics", &[(SCANNER_SPEED, "slow")]); diff --git a/crates/scanner/src/scanner.rs b/crates/scanner/src/scanner.rs index 3624be128..27cf0b372 100644 --- a/crates/scanner/src/scanner.rs +++ b/crates/scanner/src/scanner.rs @@ -18,7 +18,10 @@ use std::sync::Mutex as StdMutex; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, LazyLock, RwLock}; -use self::heal_info::{BackgroundHealInfoReadStatus, read_background_heal_info_with_epoch, save_background_heal_info_for_epoch}; +use self::heal_info::{ + BackgroundHealInfoReadStatus, ScannerBitrotStart, read_background_heal_info_with_epoch, record_scanner_bitrot_status, + resolve_background_heal_info_for_cycle, save_background_heal_info_for_epoch, scanner_bitrot_start_for_cycle, +}; use crate::data_usage_define::{ BACKGROUND_HEAL_INFO_PATH, DATA_USAGE_BLOOM_NAME_PATH, DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, DATA_USAGE_RECOVERY_PATH, DataUsageCache, DataUsageCacheRevision, LEGACY_DATA_USAGE_OBJ_NAME_PATH, read_config_revision, @@ -254,9 +257,8 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2; /// unavailable peer cannot drive a tight retry loop. const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5); const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60); -/// A transient backend outage remains self-healing after the short retry -/// budget is exhausted, but the probe is intentionally sparse until storage -/// recovers or an operator reset wakes the scanner. +/// Probe cadence while data movement pauses the scanner. Recovery retries use +/// the bounded backoff in `cycle_state` instead of a flat pause. const SCANNER_CYCLE_RECOVERY_PAUSED_INTERVAL: Duration = Duration::from_secs(5 * 60); /// Permanent recovery states still get a sparse status probe so a reset that /// races the wait registration cannot leave the scanner asleep forever. @@ -1094,25 +1096,19 @@ where "Scanner runtime iteration failed" ); } - let recovery_status = scanner_cycle_recovery_status(); - if recovery_status.retryable { - recovery_retry_count = recovery_retry_count.saturating_add(1); - let _ = record_scanner_cycle_recovery_retry(recovery_retry_count); + // A retryable recovery observation never becomes terminal: the + // retry is recorded with a bounded backoff and the observation + // itself names the next attempt (rustfs/rustfs#8297, #8670). + let recovery_retry = if scanner_cycle_recovery_status().retryable { + let retry = record_scanner_cycle_recovery_retry(recovery_retry_count.saturating_add(1), scanner_cycle_interval()); + recovery_retry_count = retry.attempt; + Some(retry.delay) } else { recovery_retry_count = 0; - } + None + }; let recovery_status = scanner_cycle_recovery_status(); - if recovery_status.state == "paused" { - transient_backoff.record_retryable_cycle(false); - tokio::select! { - _ = ctx_clone.cancelled() => break, - _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {}, - _ = tokio::time::sleep(SCANNER_CYCLE_RECOVERY_PAUSED_INTERVAL) => {}, - } - recovery_retry_count = 0; - continue; - } if !recovery_status.retryable && matches!(recovery_status.state.as_str(), "blocked" | "recovery-required" | "cleanup-pending") { @@ -1125,7 +1121,10 @@ where continue; } - let retry_delay = if recovery_status.retryable || run_result.is_err() { + let retry_delay = if let Some(delay) = recovery_retry { + transient_backoff.record_retryable_cycle(true); + delay + } else if run_result.is_err() { transient_backoff.record_retryable_cycle(true); transient_backoff .retry_interval(scanner_cycle_interval()) @@ -1157,10 +1156,17 @@ impl ScannerMaintenanceFeatures { self.lifecycle || self.replication || self.inspection_failed } - fn requires_full_scan(self, observed_generation: Option, current_generation: u64, wake: ScannerCycleWakeReason) -> bool { + fn requires_full_scan( + self, + observed_generation: Option, + current_generation: u64, + wake: ScannerCycleWakeReason, + superseded_retry: bool, + ) -> bool { self.needs_regular_cycle() || observed_generation != Some(current_generation) - || !matches!(wake, ScannerCycleWakeReason::DirtyUsage | ScannerCycleWakeReason::ClusterActivity) + || !(matches!(wake, ScannerCycleWakeReason::DirtyUsage | ScannerCycleWakeReason::ClusterActivity) + || (wake == ScannerCycleWakeReason::Timer && superseded_retry)) } } @@ -1191,16 +1197,6 @@ impl ScannerUsageBootstrapRebuild { enabled && !self.pending } - fn requires_full_scan( - self, - maintenance_features: ScannerMaintenanceFeatures, - observed_generation: Option, - current_generation: u64, - wake: ScannerCycleWakeReason, - ) -> bool { - self.pending || maintenance_features.requires_full_scan(observed_generation, current_generation, wake) - } - fn record_cycle(&mut self, outcome: ScannerCycleOutcome) -> bool { if matches!( outcome, @@ -1469,12 +1465,7 @@ impl ScannerBitrotPolicy { } } -fn get_cycle_scan_mode( - current_cycle: u64, - bitrot_start_cycle: u64, - bitrot_start_time: Option>, - policy: ScannerBitrotPolicy, -) -> HealScanMode { +fn get_cycle_scan_mode(current_cycle: u64, start: ScannerBitrotStart, policy: ScannerBitrotPolicy) -> HealScanMode { let Some(bitrot_cycle) = policy.cycle else { return HealScanMode::Normal; }; @@ -1487,18 +1478,23 @@ fn get_cycle_scan_mode( return HealScanMode::Deep; } - if current_cycle.saturating_sub(bitrot_start_cycle) < deep_window_cycles { - return HealScanMode::Deep; - } - - let Some(bitrot_start_time) = bitrot_start_time else { - return HealScanMode::Deep; + // Only a recorded Deep window keeps the selection window open. A missing + // record used to be read as a window that started at cycle 0, which ran + // Deep for `deep_window_cycles` cycles whenever the start could not be + // read back (rustfs/rustfs#8015); an in-memory anchor only counts the + // cadence. + let started = match start { + ScannerBitrotStart::None => return HealScanMode::Normal, + ScannerBitrotStart::Window { cycle, started } => { + if current_cycle.saturating_sub(cycle) < deep_window_cycles { + return HealScanMode::Deep; + } + started + } + ScannerBitrotStart::Anchor(anchored) => anchored, }; - let elapsed = Utc::now() - .signed_duration_since(bitrot_start_time) - .to_std() - .unwrap_or(Duration::ZERO); + let elapsed = Utc::now().signed_duration_since(started).to_std().unwrap_or(Duration::ZERO); if elapsed >= bitrot_cycle { HealScanMode::Deep } else { @@ -1597,7 +1593,7 @@ fn get_lock_acquire_timeout() -> Duration { } fn data_usage_persist_timeout() -> Duration { - DataUsageCache::persistence_timeout() + DataUsageCache::publication_timeout() } #[cfg(not(test))] @@ -1802,20 +1798,18 @@ where } BackgroundHealInfoReadStatus::Loaded | BackgroundHealInfoReadStatus::Missing => {} } - let mut background_heal_info = background_heal_read.info; let background_heal_epoch = background_heal_read.expected_epoch; - let bitrot_policy = ScannerBitrotPolicy::new( - configured_bitrot_cycle, - !storeapi.setup_is_erasure_sd().await, - heal_object_select_prob() as u64, - ); + let deep_scan_supported = !storeapi.setup_is_erasure_sd().await; + let bitrot_policy = ScannerBitrotPolicy::new(configured_bitrot_cycle, deep_scan_supported, heal_object_select_prob() as u64); + // The persisted record wins when it carries a start; the in-memory record + // covers a save that was skipped or lost since this process recorded it. + let (mut background_heal_info, mut bitrot_start_source) = resolve_background_heal_info_for_cycle(&background_heal_read); + let bitrot_start = scanner_bitrot_start_for_cycle(&background_heal_info, Utc::now()); + if matches!(bitrot_start, ScannerBitrotStart::Anchor(_)) { + bitrot_start_source = ScannerBitrotStartSource::Memory; + } - let scan_mode = get_cycle_scan_mode( - cycle_info.current, - background_heal_info.bitrot_start_cycle, - background_heal_info.bitrot_start_time, - bitrot_policy, - ); + let scan_mode = get_cycle_scan_mode(cycle_info.current, bitrot_start, bitrot_policy); info!( target: "rustfs::scanner", event = EVENT_SCANNER_CYCLE_STATE, @@ -1823,6 +1817,7 @@ where subsystem = LOG_SUBSYSTEM_RUNTIME, cycle = cycle_info.current, scan_mode = ?scan_mode, + bitrot_start_source = bitrot_start_source.as_str(), state = "started", "Scanner cycle started" ); @@ -1835,8 +1830,19 @@ where bitrot_policy, ) { background_heal_info = new_heal_info.clone(); + if new_heal_info.bitrot_start_time != background_heal_read.info.bitrot_start_time { + // A start recorded by this cycle is only in memory until a later + // read returns it from storage. + bitrot_start_source = ScannerBitrotStartSource::Memory; + } save_background_heal_info_for_epoch(storeapi.clone(), new_heal_info, background_heal_epoch).await; } + record_scanner_bitrot_status( + scanner_bitrot_start_for_cycle(&background_heal_info, Utc::now()), + bitrot_start_source, + deep_scan_supported, + scan_mode, + ); let cycle_start = std::time::Instant::now(); // Baseline reads are part of the same publication proof as the eventual @@ -1907,9 +1913,15 @@ where service_cohort: scheduling.service_cohort, #[cfg(test)] resolved_scope_observer: None, + #[cfg(test)] + post_walk_gate: None, }, ) .await; + let publication_plan = scan_result + .as_ref() + .map(ScannerCyclePublicationPlan::for_result) + .unwrap_or(ScannerCyclePublicationPlan::ProgressOnly); let publication_defer_reason = match &scan_result { Ok(result) if result @@ -1924,7 +1936,7 @@ where // fence remains enforced by the usage store and the cycle is advanced // as partial without acknowledging dirty usage. Ok(result) - if scanner_cycle_uses_observational_publication(result.status, result.has_observational_snapshot()) + if publication_plan.is_observational() && matches!( result.status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) @@ -1933,18 +1945,11 @@ where None } Ok(result) => { - final_data_usage_publication_defer_reason( - storeapi.as_ref(), - result.status, - scanner_cycle_uses_observational_publication(result.status, result.has_observational_snapshot()), - ) - .await + final_data_usage_publication_defer_reason(storeapi.as_ref(), result.status, publication_plan.is_observational()).await } Err(_) => Some(ScannerCycleDeferReason::ActivityBaselineUnavailable), }; - let observational_only = scan_result - .as_ref() - .is_ok_and(|result| scanner_cycle_uses_observational_publication(result.status, result.has_observational_snapshot())); + let observational_only = publication_plan.is_observational(); let publication_epoch = scan_result.as_ref().ok().and_then(ScannerCycleResult::publication_epoch); let remote_publication_lease_targets = if publication_defer_reason.is_none() { scan_result @@ -1959,16 +1964,12 @@ where let remote_lease_defer_reason = if remote_publication_lease_targets.is_empty() { None } else if let Some(notification_system) = storeapi.scanner_notification_system() { - let publication_proof_ctx = cycle_budget.token(); - let lease_result = await_scanner_publication_proof( - &publication_proof_ctx, - cycle_info.current, - "lease_acquire", - || { + let lease_result = publication_plan + .await_leases(ctx, &cycle_budget, cycle_info.current, |observational_lease| { let targets = remote_publication_lease_targets.clone(); let notification_system = Arc::clone(¬ification_system); async move { - if observational_only { + if observational_lease { notification_system .acquire_observational_scanner_publication_leases(targets) .await @@ -1976,10 +1977,8 @@ where notification_system.acquire_scanner_publication_leases(targets).await } } - }, - |err| scanner_publication_lease_error_is_retryable(&err.to_string()), - ) - .await; + }) + .await; match lease_result { ScannerPublicationProofWait::Ready(grants) => { remote_publication_leases = Some((notification_system, grants)); @@ -1999,34 +1998,40 @@ where // it is never inserted into FileInfo metadata. Keep the representation // bounded and require one authenticated token per remote target so a // partial grant can never silently fall back to an unfenced rename. - let remote_lease_fence = remote_publication_leases.as_ref().and_then(|(_, grants)| { - if grants.len() != remote_publication_lease_targets.len() || grants.len() > SCANNER_PUBLICATION_LEASE_FENCE_MAX_ENTRIES { - return None; - } - let mut fence = BTreeMap::new(); - for grant in grants { - if grant.host.is_empty() || grant.host.len() > 1024 { + let remote_lease_fence = remote_publication_leases + .as_ref() + .filter(|_| publication_plan.has_usage_update()) + .and_then(|(_, grants)| { + if grants.len() != remote_publication_lease_targets.len() + || grants.len() > SCANNER_PUBLICATION_LEASE_FENCE_MAX_ENTRIES + { return None; } - if fence.insert(grant.host.clone(), grant.lease.token.to_string()).is_some() { + let mut fence = BTreeMap::new(); + for grant in grants { + if grant.host.is_empty() || grant.host.len() > 1024 { + return None; + } + if fence.insert(grant.host.clone(), grant.lease.token.to_string()).is_some() { + return None; + } + } + if remote_publication_lease_targets + .iter() + .any(|(host, _, _)| !fence.contains_key(host)) + { return None; } - } - if remote_publication_lease_targets - .iter() - .any(|(host, _, _)| !fence.contains_key(host)) - { - return None; - } - serde_json::to_string(&fence) - .ok() - .filter(|encoded| encoded.len() <= SCANNER_PUBLICATION_LEASE_FENCE_MAX_BYTES) - }); - let remote_lease_fence_defer_reason = (remote_publication_lease_fence_targets_are_required( - remote_publication_lease_targets.len(), - remote_publication_leases.is_some(), - remote_lease_fence.is_some(), - )) + serde_json::to_string(&fence) + .ok() + .filter(|encoded| encoded.len() <= SCANNER_PUBLICATION_LEASE_FENCE_MAX_BYTES) + }); + let remote_lease_fence_defer_reason = (publication_plan.has_usage_update() + && remote_publication_lease_fence_targets_are_required( + remote_publication_lease_targets.len(), + remote_publication_leases.is_some(), + remote_lease_fence.is_some(), + )) .then_some(ScannerCycleDeferReason::ActivityBaselineUnavailable); let publication_defer_reason = publication_defer_reason .or(remote_lease_defer_reason) @@ -2039,7 +2044,10 @@ where && let Ok(result) = &scan_result && result.status == ScannerCycleStatus::Complete { - scanner_post_lease_activity_defer_reason(result.activity_digest(), probe_scanner_activity(storeapi.as_ref(), true).await) + scanner_post_lease_activity_defer_reason( + result.activity_baseline(), + probe_scanner_activity(storeapi.as_ref(), true).await, + ) } else { None }; @@ -2049,6 +2057,7 @@ where let budget_elapsed = cycle_budget.budget_elapsed() && !ctx.is_cancelled(); let remote_lease_probe = remote_publication_leases .as_ref() + .filter(|_| publication_plan.has_usage_update()) .map(|(notification_system, grants)| (Arc::clone(notification_system), grants.clone())); let remote_lease_release_safe = Arc::new(AtomicBool::new(true)); let mut usage_publication_result = match publication_defer_reason { @@ -2056,6 +2065,10 @@ where drop(receiver); DataUsagePublicationResult::from(DataUsagePersistOutcome::Deferred(reason)) } + None if !publication_plan.has_usage_update() => { + drop(receiver); + DataUsagePublicationResult::from(DataUsagePersistOutcome::NoUpdate) + } None => { // ScannerIO emits its complete or observational update only after // all set workers finish. Persist after the final activity fence; @@ -2064,7 +2077,6 @@ where let ctx_clone = ctx.clone(); let route_probe_store = storeapi.clone(); let route_probe_observational_only = observational_only; - let remote_lease_fence = remote_lease_fence.clone(); let remote_lease_release_safe_for_task = Arc::clone(&remote_lease_release_safe); let remote_lease_tokens = remote_publication_leases .as_ref() @@ -2366,15 +2378,19 @@ where } usage_publication_result.restrict_outcome(usage_persist_outcome); - let partial_cause = if scan_cycle_result.has_observational_snapshot() + let partial_cause = if scan_cycle_result.has_partial_bucket_progress() && !scan_cycle_result.has_usage_update() { + "saved_bucket_progress" + } else if scan_cycle_result.has_usage_update() && matches!( scan_cycle_result.status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) - ) { + ) + { "activity_unverified_observation" } else { "incomplete_coverage" }; + let pending_dirty_usage_buckets = scan_cycle_result.pending_dirty_usage_buckets(); let (completion_outcome, scanner_pending_maintenance_work, remote_dirty_usage_acknowledgements) = finalize_scanner_cycle_result(scan_cycle_result, usage_publication_result); let remote_dirty_usage_pending = if remote_dirty_usage_acknowledgements.is_empty() { @@ -2551,8 +2567,9 @@ where global_metrics().clear_current_scan_mode(); retain_recent_cycle_completions(&mut cycle_info.cycle_completed); - if !persist_scanner_cycle_state_for_epoch(ctx, storeapi.clone(), cycle_info, cycle_revision, leader_epoch, publication_epoch) + if persist_scanner_cycle_state_for_epoch(ctx, storeapi.clone(), cycle_info, cycle_revision, leader_epoch, publication_epoch) .await + != ScannerCycleStatePersistOutcome::Saved { if let Some(expected_epoch) = publication_epoch && scanner_publication_admission_for_epoch(storeapi.clone(), expected_epoch) @@ -2586,6 +2603,7 @@ where state = "completed", duration = ?now.elapsed(), cycles_total = cycle_info.cycle_completed.len(), + pending_dirty_buckets = pending_dirty_usage_buckets, "Scanner cycle completed" ); @@ -2743,6 +2761,7 @@ where Err(e) => { match classify_scanner_leader_lock_failure(&e) { ScannerLeaderLockFailure::Contended => { + clear_scanner_transient_cycle_recovery_status(); record_scanner_cycle_schedule_role("follower"); record_scanner_leader_lock_state("contended"); global_metrics().record_scanner_leader_liveness("contended", false, "").await; @@ -2848,6 +2867,7 @@ where observe_scanner_activity(&storeapi, distributed, &mut scanner_activity_seen).await; } + let mut rebuild_from_quarantine = false; let (mut cycle_info, mut leader_epoch, mut cycle_revision) = match load_scanner_cycle_state_for_startup(storeapi.clone()).await { ScannerCycleStateStartup::Ready { @@ -2855,6 +2875,13 @@ where leader_epoch, revision, } => (cycle, leader_epoch, revision), + ScannerCycleStateStartup::RebuildFromQuarantine { revision } => { + // The usage floor raises the cycle below and the leadership + // claim raises the epoch; the quarantined revision is the CAS + // precondition so a concurrent writer cannot be overwritten. + rebuild_from_quarantine = true; + (CurrentCycle::default(), 0, revision) + } ScannerCycleStateStartup::Blocked => { global_metrics().set_cycle(None).await; finish_scanner_leader_iteration(false, "stopped", String::new()).await; @@ -2988,6 +3015,9 @@ where finish_scanner_leader_iteration(false, "epoch_claim_failed", "leadership epoch claim failed".to_string()).await; return Ok(()); } + if rebuild_from_quarantine { + record_scanner_cycle_state_rebuilt_from_quarantine(cycle_info.next, leader_epoch); + } if usage_floor_startup == PersistedUsageFloorStartup::RecoveredLegacyIncompleteFence && let Err(err) = complete_legacy_incomplete_usage_floor_recovery(storeapi.clone(), leader_epoch).await { @@ -3017,7 +3047,9 @@ where if !ctx.is_cancelled() && matches!( initial_pause_backlog_attempt, - ScannerPauseBacklogAttemptDecision::Untracked | ScannerPauseBacklogAttemptDecision::Tracked(_) + ScannerPauseBacklogAttemptDecision::Untracked + | ScannerPauseBacklogAttemptDecision::Tracked(_) + | ScannerPauseBacklogAttemptDecision::Degraded ) { // Preserve previous behavior: run one cycle immediately after lock acquisition. @@ -3084,9 +3116,12 @@ where } }; finish_scanner_pause_backlog_cycle(&mut pause_backlog, &storeapi, initial_pause_backlog_attempt, initial_outcome).await; - if initial_outcome == ScannerCycleOutcome::StatePersistenceFailed { + if initial_outcome == ScannerCycleOutcome::StatePersistenceFailed + && let Err(reason) = + reload_cycle_state_under_same_epoch(&storeapi, &mut cycle_info, &mut cycle_revision, leader_epoch).await + { global_metrics().set_cycle(None).await; - let error = "scanner cycle state persistence failed; retrying from durable state".to_string(); + let error = format!("scanner cycle state persistence failed; {reason}; retrying from durable state"); finish_scanner_leader_iteration(false, "state_persist_failed", error.clone()).await; return Err(ScannerError::Other(error)); } @@ -3094,7 +3129,10 @@ where clean_idle_backoff.reset(); } superseded_backoff.record_retryable_cycle(initial_outcome == ScannerCycleOutcome::Superseded); - deferred_backoff.record_retryable_cycle(matches!(initial_outcome, ScannerCycleOutcome::Deferred(_))); + deferred_backoff.record_retryable_cycle(matches!( + initial_outcome, + ScannerCycleOutcome::Deferred(_) | ScannerCycleOutcome::StatePersistenceFailed + )); dirty_usage_generation_seen = dirty_generation_before_cycle; if guard.is_lock_lost() { record_scanner_leader_lock_lost("Scanner leader lock lost during the initial cycle").await; @@ -3184,7 +3222,8 @@ where )); let mut wait_plan = scanner_cycle_wait_plan(&runtime_config, clean_idle_backoff, backoff_enabled, randomized_cycle_delay_for); - let superseded_retry_interval = scanner_superseded_retry_interval(superseded_backoff, &runtime_config); + let superseded_retry_interval = + scanner_superseded_retry_interval(superseded_backoff, &runtime_config, maintenance_features); let deferred_retry_interval = deferred_backoff.retry_interval(runtime_config.cycle_interval); let mut convergence_retry_interval = superseded_retry_interval.or(deferred_retry_interval); if let Some(retry_interval) = convergence_retry_interval { @@ -3341,10 +3380,7 @@ where } else { pause_backlog.begin_attempt(scanner_pause_backlog_now()).await }; - if matches!( - pause_backlog_attempt, - ScannerPauseBacklogAttemptDecision::RateLimited | ScannerPauseBacklogAttemptDecision::PersistenceUnavailable - ) { + if pause_backlog_attempt == ScannerPauseBacklogAttemptDecision::RateLimited { continue; } let usage_bootstrap_pending_before_cycle = usage_bootstrap_rebuild.pending(); @@ -3362,11 +3398,13 @@ where leader_epoch, cycle_budget.clone(), ScannerCycleScheduling { - requires_full_scan: usage_bootstrap_rebuild.requires_full_scan( - maintenance_features, + // Cache reuse is admitted by the persisted coverage proof in ScannerIO. + // Bootstrap remains pending until authoritative publication succeeds. + requires_full_scan: maintenance_features.requires_full_scan( maintenance_generation_seen, scanner_maintenance_generation(), wake_reason, + superseded_retry_interval.is_some(), ), service_cohort: Some(service_cohort.clone()), }, @@ -3407,9 +3445,12 @@ where } }; finish_scanner_pause_backlog_cycle(&mut pause_backlog, &storeapi, pause_backlog_attempt, outcome).await; - if outcome == ScannerCycleOutcome::StatePersistenceFailed { + if outcome == ScannerCycleOutcome::StatePersistenceFailed + && let Err(reason) = + reload_cycle_state_under_same_epoch(&storeapi, &mut cycle_info, &mut cycle_revision, leader_epoch).await + { global_metrics().set_cycle(None).await; - let error = "scanner cycle state persistence failed; retrying from durable state".to_string(); + let error = format!("scanner cycle state persistence failed; {reason}; retrying from durable state"); finish_scanner_leader_iteration(false, "state_persist_failed", error.clone()).await; return Err(ScannerError::Other(error)); } @@ -3417,7 +3458,10 @@ where clean_idle_backoff.reset(); } superseded_backoff.record_retryable_cycle(outcome == ScannerCycleOutcome::Superseded); - deferred_backoff.record_retryable_cycle(matches!(outcome, ScannerCycleOutcome::Deferred(_))); + deferred_backoff.record_retryable_cycle(matches!( + outcome, + ScannerCycleOutcome::Deferred(_) | ScannerCycleOutcome::StatePersistenceFailed + )); dirty_usage_generation_seen = dirty_generation_before_cycle; if guard.is_lock_lost() { record_scanner_leader_lock_lost("Scanner leader lock lost during a scanner cycle").await; @@ -3510,6 +3554,48 @@ where Ok(()) } +/// Re-read durable cycle state after a save that could not be confirmed. +/// The leader keeps its epoch only while storage still names that epoch; a +/// newer owner or an unreadable primary ends the iteration so the next claim +/// re-fences (rustfs/rustfs#8141: a non-fence failure must not change the +/// `(cycle, epoch)` fence peers validate). +async fn reload_cycle_state_under_same_epoch( + storeapi: &Arc, + cycle_info: &mut CurrentCycle, + cycle_revision: &mut DataUsageCacheRevision, + leader_epoch: u64, +) -> Result<(), String> +where + S: ScannerObjectIO, +{ + let (persisted, revision) = read_config_with_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) + .await + .map_err(|err| format!("durable cycle state could not be read: {err}"))?; + let persisted = persisted.ok_or_else(|| "durable cycle state is missing".to_string())?; + let (persisted_cycle, persisted_epoch) = + decode_scanner_cycle_state(&persisted).map_err(|err| format!("durable cycle state is invalid: {err}"))?; + if persisted_epoch != leader_epoch { + return Err(format!( + "durable cycle state is owned by leadership epoch {persisted_epoch}, this leader holds {leader_epoch}" + )); + } + warn!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = "state_persist_reloaded", + leader_epoch, + durable_next_cycle = persisted_cycle.next, + in_memory_next_cycle = cycle_info.next, + "Scanner cycle state reloaded under the same leadership epoch after an unconfirmed save" + ); + *cycle_info = persisted_cycle; + *cycle_revision = revision; + Ok(()) +} + struct ScannerScanModeGuard; impl ScannerScanModeGuard { @@ -3559,6 +3645,13 @@ where } } +/// The local publication fence. Only storage data movement defers a +/// publication: an active or terminal decommission, a rebalance, or an +/// exhausted movement sequence. An ordinary PUT whose rename fanout is still +/// pending is not a fence; its bucket is shaded through the dirty-usage +/// generation and certified per bucket, so a sustained write load cannot +/// hold the authoritative snapshot back (rustfs/rustfs#8494, +/// rustfs/rustfs#8305). async fn scanner_local_publication_defer_reason(storeapi: &S, observational_only: bool) -> Option where S: ScannerStorage, @@ -3571,49 +3664,124 @@ where if !blocked { return None; } - // Pending namespace commits invalidate this publication attempt, but only - // storage movement creates durable, rate-limited catch-up debt. - if storeapi.scanner_data_movement_publication_blocked().await { + // Both barriers describe storage movement. The authoritative barrier also + // holds terminal decommission entries an operator has not cleared, which + // is the same durable, rate-limited catch-up debt. + debug!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + stage = "local_barrier", + blocker = "data_movement", + "Scanner usage publication deferred" + ); + Some(ScannerCycleDeferReason::DataMovement) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ScannerCyclePublicationPlan { + ProgressOnly, + Observational, + Authoritative, +} + +impl ScannerCyclePublicationPlan { + fn for_result(result: &ScannerCycleResult) -> Self { + match (result.has_usage_update(), result.status) { + (false, _) => Self::ProgressOnly, + ( + true, + ScannerCycleStatus::Superseded + | ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) + | ScannerCycleStatus::Incomplete, + ) => Self::Observational, + (true, _) => Self::Authoritative, + } + } + + fn has_usage_update(self) -> bool { + self != Self::ProgressOnly + } + + fn is_observational(self) -> bool { + self == Self::Observational + } + + fn uses_observational_lease(self) -> bool { + self != Self::Authoritative + } + + async fn await_leases( + self, + parent: &CancellationToken, + budget: &ScannerCycleBudget, + cycle: u64, + mut acquire: F, + ) -> ScannerPublicationProofWait + where + E: std::fmt::Display, + F: FnMut(bool) -> Fut, + Fut: Future>, + { + // Exhausting the walk budget must not cancel the movement proof needed + // to save partial progress. No usage update means namespace stability + // is unnecessary, but peer session/generation and movement checks stay. + let proof_ctx = if self == Self::ProgressOnly { + parent.clone() + } else { + budget.token() + }; debug!( target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, + event = EVENT_SCANNER_CYCLE_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_RUNTIME, - stage = "local_barrier", - blocker = "data_movement", - "Scanner usage publication deferred" + state = "publication_proof_started", + cycle, + stage = "lease_acquire", + publication_kind = match self { + Self::ProgressOnly => "progress_only", + Self::Observational => "observational", + Self::Authoritative => "authoritative", + }, + "Scanner is acquiring peer publication proof" ); - Some(ScannerCycleDeferReason::DataMovement) - } else { - debug!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - stage = "local_barrier", - blocker = "pending_namespace_commit", - "Scanner usage publication deferred" + let proof = await_scanner_publication_proof( + &proof_ctx, + cycle, + "lease_acquire", + || acquire(self.uses_observational_lease()), + |err| scanner_publication_lease_error_is_retryable(&err.to_string()), ); - Some(ScannerCycleDeferReason::ActivityBaselineUnavailable) + if self == Self::ProgressOnly { + // Object/directory budgets can expire with no runtime deadline. + // Bound the control-plane drain independently of the walk budget. + tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, proof) + .await + .unwrap_or(ScannerPublicationProofWait::Cancelled) + } else { + proof.await + } } } -fn scanner_cycle_uses_observational_publication(status: ScannerCycleStatus, has_observational_candidate: bool) -> bool { - has_observational_candidate - && matches!( - status, - ScannerCycleStatus::Superseded | ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) - ) -} - +/// Re-prove the guardrail after every peer granted a publication lease. The +/// comparison uses the guardrail fields (`scanner_activity_guardrail_veto`), +/// not the full digest: local generation movement in this window is the next +/// cycle's work and its buckets stay in the dirty map. A peer whose dirty +/// usage moved keeps deferring in this phase, as it does during the walk. fn scanner_post_lease_activity_defer_reason( - expected_digest: Option<[u8; 32]>, + baseline: Option<&ScannerActivitySnapshot>, activity: Result, ) -> Option { match activity { Ok(snapshot) if scanner_activity_allows_usage_publication(&snapshot) - && expected_digest == Some(scanner_activity_snapshot_digest(&snapshot)) => + && baseline.is_some_and(|baseline| { + scanner_activity_guardrail_veto(baseline, &snapshot).is_none() + && scanner_peer_dirty_usage_moved(baseline, &snapshot).is_none() + }) => { None } @@ -3621,8 +3789,13 @@ fn scanner_post_lease_activity_defer_reason( let blocker = match &observed { Err(_) => "probe_failed", Ok(snapshot) if !scanner_activity_allows_usage_publication(snapshot) => "publication_blocked", - Ok(_) if expected_digest.is_none() => "baseline_missing", - Ok(_) => "activity_changed", + Ok(_) if baseline.is_none() => "baseline_missing", + Ok(snapshot) => baseline + .and_then(|baseline| { + scanner_activity_guardrail_veto(baseline, snapshot) + .or_else(|| scanner_peer_dirty_usage_moved(baseline, snapshot)) + }) + .map_or("activity_changed", |veto| veto.field), }; debug!( target: "rustfs::scanner", @@ -3802,7 +3975,17 @@ fn scanner_cycle_completion_outcome_for_result( ) -> ScannerCycleOutcome { let has_dirty_usage = scan_cycle_result.has_dirty_usage_to_acknowledge(); let has_failed_dirty_usage = scan_cycle_result.has_failed_dirty_usage(); - if scan_cycle_result.has_observational_snapshot() + if scan_cycle_result.status == ScannerCycleStatus::Incomplete + && scan_cycle_result.has_partial_bucket_progress() + && !scan_cycle_result.has_usage_update() + && !has_failed_dirty_usage + && usage_persist_outcome == DataUsagePersistOutcome::NoUpdate + { + // No root was submitted: retain dirty usage and advance only the + // fenced partial-cycle state. A fresh sweep still has to certify usage. + return ScannerCycleOutcome::Partial; + } + if scan_cycle_result.has_usage_update() && matches!( scan_cycle_result.status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) @@ -3901,10 +4084,11 @@ use usage_store::*; pub(crate) use activity::scanner_node_activity_for_tests; pub use activity::scanner_topology_digest; pub(crate) use activity::{ - ScannerActivitySnapshot, ScannerDirtyUsageAcknowledgement, ScannerDirtyUsageAcknowledgementKind, probe_scanner_activity, - scanner_activity_allows_usage_publication, scanner_activity_dirty_usage_state_for_host, + LOCAL_SCANNER_ACTIVITY_NODE, ScannerActivityGuardrailVeto, ScannerActivitySnapshot, ScannerDirtyUsageAcknowledgement, + ScannerDirtyUsageAcknowledgementKind, probe_scanner_activity, scanner_activity_allows_usage_publication, + scanner_activity_coverage_digest, scanner_activity_dirty_usage_state_for_host, scanner_activity_guardrail_veto, scanner_activity_publication_lease_targets, scanner_activity_snapshot_digest, scanner_activity_structural_digest, - scanner_dirty_usage_acknowledgements, + scanner_dirty_usage_acknowledgements, scanner_peer_dirty_usage_moved, }; pub(crate) use activity::{ScannerCycleOutcome, scanner_cycle_outcome_with_pending_maintenance}; pub use backlog::{ @@ -3923,7 +4107,10 @@ pub use cycle_state::{ pub(crate) use cycle_state::{ current_scanner_leader_epoch, decode_persisted_scanner_cycle_fence, load_scanner_cycle_state_for_startup, }; -pub use heal_info::{BackgroundHealInfo, read_background_heal_info, save_background_heal_info}; +pub use heal_info::{ + BackgroundHealInfo, ScannerBitrotStartSource, ScannerBitrotStatus, read_background_heal_info, save_background_heal_info, + scanner_bitrot_status, +}; pub use usage_store::store_data_usage_in_backend; #[cfg(test)] diff --git a/crates/scanner/src/scanner/activity.rs b/crates/scanner/src/scanner/activity.rs index 2b63d115a..2375a930e 100644 --- a/crates/scanner/src/scanner/activity.rs +++ b/crates/scanner/src/scanner/activity.rs @@ -152,17 +152,19 @@ impl ScannerRetryBackoff { pub(super) fn scanner_superseded_retry_interval( backoff: ScannerRetryBackoff, runtime_config: &ScannerRuntimeConfig, + maintenance_features: ScannerMaintenanceFeatures, ) -> Option { - let retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?; - if runtime_config.cycle_interval_source == ScannerRuntimeConfigSource::Default { - return Some(retry_interval); + let mut retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?; + // Preserve an operator-selected duty-cycle floor during convergence. + if runtime_config.cycle_interval_source != ScannerRuntimeConfigSource::Default { + retry_interval = retry_interval.max(runtime_config.cycle_interval); } - - // An explicit cycle is an operator-selected duty-cycle floor. A - // superseded snapshot keeps its dirty work pending, but retrying that work - // sooner than the configured cadence would turn continuous writes into a - // repeated full-walk loop despite the override. - Some(retry_interval.max(runtime_config.cycle_interval)) + // A complete but non-authoritative walk must not postpone lifecycle or + // replication evaluation beyond its ordinary maintenance cadence. + if maintenance_features.needs_regular_cycle() { + retry_interval = retry_interval.min(runtime_config.cycle_interval.max(Duration::from_secs(1))); + } + Some(retry_interval) } const SCANNER_PUBLICATION_PROOF_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30); @@ -260,8 +262,9 @@ where stage, attempt = consecutive_failures, retry_delay = ?retry_delay, + pending_duration = ?started_at.elapsed(), error = %err, - "Scanner retained a completed scan while publication proof is unavailable" + "Scanner is waiting for publication proof" ); } else { debug!( @@ -466,7 +469,7 @@ pub(super) struct ScannerMovementWaitContext<'a, G, F> { pub(super) recovery_wake: Option<&'a Notify>, } -pub(super) const LOCAL_SCANNER_ACTIVITY_NODE: &str = ""; +pub(crate) const LOCAL_SCANNER_ACTIVITY_NODE: &str = ""; #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct ScannerNodeActivity { @@ -1030,15 +1033,25 @@ pub(crate) fn scanner_activity_snapshot_digest(snapshot: &ScannerActivitySnapsho hasher.update(activity.dirty_usage_generation.to_be_bytes()); hasher.update([u8::from(activity.dirty_usage_pending)]); hasher.update(activity.movement_generation.to_be_bytes()); - hasher.update([u8::from(activity.publication_blocked)]); } hasher.finalize().into() } -/// Hash the activity inputs that make an existing scanner cache unsafe to -/// reuse. Regular namespace writes and dirty-usage generations are omitted: -/// their affected buckets are tracked separately and may be refreshed from a -/// complete authoritative cache baseline. +/// The reuse identity: hash the activity inputs that make an existing +/// scanner result unsafe to reuse without walking again. Besides the +/// coverage identity below it binds each peer's `instance_id` and +/// `maintenance_generation`: a restarted node may have lost the dirty hints +/// of its last journal window and a changed lifecycle or replication rule +/// needs a maintenance pass, so both invalidate the authoritative baseline +/// and the set caches and force the next cycle to walk every bucket. With +/// the bucket list it is the plan under which set caches and the +/// scoped-scan baseline are reused (`scanner_bucket_plan_digest`). +/// +/// Regular namespace writes and dirty-usage generations are omitted: their +/// buckets are tracked separately and rescanned from a complete baseline. +/// `publication_blocked` is omitted from every identity: a movement pause +/// vetoes the cycle it moves in (`scanner_activity_guardrail_veto`) and +/// describes nothing about the heap. pub(crate) fn scanner_activity_structural_digest(snapshot: &ScannerActivitySnapshot) -> [u8; 32] { let mut hasher = Sha256::new(); hasher.update(u64::try_from(snapshot.len()).unwrap_or(u64::MAX).to_be_bytes()); @@ -1054,11 +1067,150 @@ pub(crate) fn scanner_activity_structural_digest(snapshot: &ScannerActivitySnaps hasher.update(activity.topology_digest); hasher.update([u8::from(activity.data_movement_active)]); hasher.update(activity.movement_generation.to_be_bytes()); - hasher.update([u8::from(activity.publication_blocked)]); } hasher.finalize().into() } +/// The coverage identity: hash the activity inputs under which a bucket +/// sweep's traversal stays one traversal, so a checkpoint resumed in a later +/// cycle is not mixed coverage. It binds the peer set, the protocol, the +/// topology and the data-movement state. A coordinator restart, a +/// maintenance publication or a publication pause changes none of the +/// namespace a sweep walks, so `instance_id`, `maintenance_generation` and +/// `publication_blocked` are left out; hashing them made every resumed sweep +/// restart as mixed coverage after such an event (rustfs/rustfs#8494). The +/// local node's namespace and dirty generations are left out too: the +/// bucket's own dirty generation joins the checkpoint plan in +/// `scanner_bucket_cache_digest`, which is what certification compares. A +/// remote peer's generations stay in, because its dirty buckets are still +/// acknowledged per cycle rather than per bucket, and a write there between +/// the two halves of a sweep would otherwise be acknowledged without having +/// been walked. +pub(crate) fn scanner_activity_coverage_digest(snapshot: &ScannerActivitySnapshot) -> [u8; 32] { + let mut hasher = Sha256::new(); + hasher.update(u64::try_from(snapshot.len()).unwrap_or(u64::MAX).to_be_bytes()); + for (host, activity) in snapshot { + let host_bytes = host.as_bytes(); + hasher.update(u64::try_from(host_bytes.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(host_bytes); + hasher.update(activity.protocol_version.to_be_bytes()); + hasher.update(activity.topology_digest); + hasher.update([u8::from(activity.data_movement_active)]); + hasher.update(activity.movement_generation.to_be_bytes()); + if host.as_str() != LOCAL_SCANNER_ACTIVITY_NODE { + hasher.update(activity.namespace_generation.to_be_bytes()); + hasher.update(activity.dirty_usage_generation.to_be_bytes()); + hasher.update([u8::from(activity.dirty_usage_pending)]); + } + } + hasher.finalize().into() +} + +/// An attributed veto of the cluster activity guardrail: the named field on +/// the named host moved between the pre-walk and post-walk probes. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct ScannerActivityGuardrailVeto { + pub(crate) host: String, + pub(crate) field: &'static str, +} + +impl ScannerActivityGuardrailVeto { + pub(crate) fn new(host: &str, field: &'static str) -> Self { + Self { + host: host.to_string(), + field, + } + } + + /// Operator-facing label, `@`. + pub(crate) fn reason(&self) -> String { + format!("{}@{}", self.field, self.host) + } +} + +/// `ScannerNodeActivity` fields whose movement means the heap the probe +/// describes is not the heap that was walked. Each is a guardrail veto on its +/// own; `peer_set` covers a host that appeared or disappeared. +#[cfg(test)] +pub(crate) const SCANNER_ACTIVITY_GUARDRAIL_VETO_FIELDS: [&str; 8] = [ + "peer_set", + "instance_id", + "protocol_version", + "topology_digest", + "maintenance_generation", + "data_movement_active", + "movement_generation", + "publication_blocked", +]; + +/// `ScannerNodeActivity` fields that only shade buckets: they are consumed +/// per bucket by dirty-usage certification and never veto on their own. +#[cfg(test)] +pub(crate) const SCANNER_ACTIVITY_SHADE_FIELDS: [&str; 3] = + ["namespace_generation", "dirty_usage_generation", "dirty_usage_pending"]; + +/// Compare the guardrail fields of two activity probes and name the first +/// field that moved. `maintenance_generation` is a veto because no cycle today +/// distinguishes whether it published maintenance results. The veto fields +/// are compared one by one; which of them also change the reuse or the +/// coverage identity is decided by `scanner_activity_structural_digest` and +/// `scanner_activity_coverage_digest`, not here. +pub(crate) fn scanner_activity_guardrail_veto( + before: &ScannerActivitySnapshot, + after: &ScannerActivitySnapshot, +) -> Option { + for (host, before_activity) in before { + let Some(after_activity) = after.get(host) else { + return Some(ScannerActivityGuardrailVeto::new(host, "peer_set")); + }; + let moved = if before_activity.instance_id != after_activity.instance_id { + "instance_id" + } else if before_activity.protocol_version != after_activity.protocol_version { + "protocol_version" + } else if before_activity.topology_digest != after_activity.topology_digest { + "topology_digest" + } else if before_activity.maintenance_generation != after_activity.maintenance_generation { + "maintenance_generation" + } else if before_activity.data_movement_active != after_activity.data_movement_active { + "data_movement_active" + } else if before_activity.movement_generation != after_activity.movement_generation { + "movement_generation" + } else if before_activity.publication_blocked != after_activity.publication_blocked { + "publication_blocked" + } else { + continue; + }; + return Some(ScannerActivityGuardrailVeto::new(host, moved)); + } + after + .keys() + .find(|host| !before.contains_key(*host)) + .map(|host| ScannerActivityGuardrailVeto::new(host, "peer_set")) +} + +/// Name the first remote peer whose dirty-usage state moved between the two +/// probes. The local node's dirty state is certified per bucket, but a peer's +/// moved buckets cannot be named without a post-walk scoped snapshot, so this +/// phase keeps the whole-cycle fallback for peers; phase 5 lifts it. +pub(crate) fn scanner_peer_dirty_usage_moved( + before: &ScannerActivitySnapshot, + after: &ScannerActivitySnapshot, +) -> Option { + before + .iter() + .filter(|(host, _)| host.as_str() != LOCAL_SCANNER_ACTIVITY_NODE) + .find_map(|(host, before_activity)| { + let after_activity = after.get(host)?; + if before_activity.dirty_usage_generation != after_activity.dirty_usage_generation { + Some(ScannerActivityGuardrailVeto::new(host, "dirty_usage_generation")) + } else if before_activity.dirty_usage_pending != after_activity.dirty_usage_pending { + Some(ScannerActivityGuardrailVeto::new(host, "dirty_usage_pending")) + } else { + None + } + }) +} + pub(crate) fn scanner_activity_allows_usage_publication(snapshot: &ScannerActivitySnapshot) -> bool { !snapshot.is_empty() && snapshot.values().all(|activity| { diff --git a/crates/scanner/src/scanner/backlog.rs b/crates/scanner/src/scanner/backlog.rs index 8bc63669a..e63bd77dd 100644 --- a/crates/scanner/src/scanner/backlog.rs +++ b/crates/scanner/src/scanner/backlog.rs @@ -679,7 +679,9 @@ pub(super) enum ScannerPauseBacklogAttemptDecision { Untracked, RateLimited, Tracked(u64), - PersistenceUnavailable, + /// The ledger is not durable. The cycle still runs, untracked, behind the + /// data-movement fence; the claim is retried on later observations. + Degraded, } #[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize)] @@ -1866,12 +1868,15 @@ where Ok(controller) } + /// Run without a durable ledger (rustfs/rustfs#8251). Cycles are not + /// skipped: publication stays fenced by data movement, and the claim is + /// retried by `observe` once the refresh interval has passed. pub(super) fn unavailable(storeapi: Arc, error: String, now: u64) -> Self { - set_runtime_error(Some(error)); + set_runtime_error(Some(degraded_runtime_error(error))); let loaded = LoadedScannerPauseBacklog { ledger: ScannerPauseBacklogLedger::default(), durable: false, - persistence_state: "unavailable".to_string(), + persistence_state: SCANNER_PAUSE_BACKLOG_PERSISTENCE_DEGRADED.to_string(), replicas: Vec::new(), replica_count: 0, healthy_replicas: 0, @@ -1892,7 +1897,9 @@ where pub(super) fn scheduling_delay(&self, now: u64, usage_bootstrap_rebuild_pending: bool) -> Option { if self.persistence_disabled { - return Some(Duration::from_secs(self.persistence_retry_at_unix_secs.saturating_sub(now))); + // A degraded ledger follows the ordinary cadence; the claim retry + // rides on each observation instead of forcing a probe schedule. + return None; } if usage_bootstrap_rebuild_pending && !self.loaded.ledger.blocks_usage_bootstrap_rebuild() { return None; @@ -1932,7 +1939,7 @@ where begin: fn(&mut ScannerPauseBacklogLedger, u64) -> ScannerPauseBacklogAttemptDecision, ) -> ScannerPauseBacklogAttemptDecision { if self.persistence_disabled { - return ScannerPauseBacklogAttemptDecision::PersistenceUnavailable; + return ScannerPauseBacklogAttemptDecision::Degraded; } let mut candidate = self.loaded.ledger.clone(); let decision = begin(&mut candidate, now); @@ -1942,7 +1949,7 @@ where } if let Err(err) = prepare_scanner_pause_backlog_persist(&mut candidate, now) { self.disable_persistence(err, now); - return ScannerPauseBacklogAttemptDecision::PersistenceUnavailable; + return ScannerPauseBacklogAttemptDecision::Degraded; } match persist_scanner_pause_backlog(self.storeapi.clone(), &self.loaded, candidate).await { Ok(loaded) => { @@ -1953,7 +1960,7 @@ where } Err(err) => { self.disable_persistence(err, now); - ScannerPauseBacklogAttemptDecision::PersistenceUnavailable + ScannerPauseBacklogAttemptDecision::Degraded } } } @@ -2008,7 +2015,8 @@ where fn disable_persistence(&mut self, error: String, now: u64) { self.persistence_disabled = true; self.persistence_retry_at_unix_secs = now.saturating_add(SCANNER_PAUSE_REFRESH_INTERVAL_SECONDS); - set_runtime_error(Some(error)); + self.loaded.persistence_state = SCANNER_PAUSE_BACKLOG_PERSISTENCE_DEGRADED.to_string(); + set_runtime_error(Some(degraded_runtime_error(error))); self.record_status(now); } @@ -2027,7 +2035,7 @@ where } Err(error) => { self.persistence_retry_at_unix_secs = now.saturating_add(SCANNER_PAUSE_REFRESH_INTERVAL_SECONDS); - set_runtime_error(Some(error)); + set_runtime_error(Some(degraded_runtime_error(error))); self.record_status(now); false } @@ -2048,6 +2056,14 @@ fn prepare_scanner_pause_backlog_persist(ledger: &mut ScannerPauseBacklogLedger, ledger.validate() } +const SCANNER_PAUSE_BACKLOG_PERSISTENCE_DEGRADED: &str = "degraded"; + +fn degraded_runtime_error(error: String) -> String { + format!( + "{error}; scanner cycles continue without a durable pause ledger behind the data-movement fence and the ledger claim is retried at most every {SCANNER_PAUSE_REFRESH_INTERVAL_SECONDS} seconds" + ) +} + fn set_runtime_error(error: Option) { *SCANNER_PAUSE_BACKLOG_RUNTIME_ERROR .write() @@ -2068,6 +2084,9 @@ pub async fn scanner_pause_backlog_status(storeapi: Arc) -> ScannerPaus match select_scanner_pause_backlog_replicas(replicas) { Ok(loaded) => { let mut status = loaded.status(now, runtime_error()); + if status.error.is_some() { + status.persistence_state = SCANNER_PAUSE_BACKLOG_PERSISTENCE_DEGRADED.to_string(); + } if status.error.is_some() || !status.durable || status.stale_or_unavailable_replicas > 0 @@ -2078,12 +2097,18 @@ pub async fn scanner_pause_backlog_status(storeapi: Arc) -> ScannerPaus status } Err(failure) => { - let error = runtime_error().map_or(failure.to_string(), |runtime| format!("{failure}; {runtime}")); + let runtime = runtime_error(); + let persistence_state = if runtime.is_some() { + SCANNER_PAUSE_BACKLOG_PERSISTENCE_DEGRADED + } else { + "unavailable" + }; + let error = runtime.map_or(failure.to_string(), |runtime| format!("{failure}; {runtime}")); let replica_diagnostics = scanner_pause_backlog_replica_diagnostics(failure.replicas); status_from_ledger( &ScannerPauseBacklogLedger::default(), now, - "unavailable".to_string(), + persistence_state.to_string(), false, replica_count, 0, diff --git a/crates/scanner/src/scanner/cycle_state.rs b/crates/scanner/src/scanner/cycle_state.rs index 720302608..fe0e87748 100644 --- a/crates/scanner/src/scanner/cycle_state.rs +++ b/crates/scanner/src/scanner/cycle_state.rs @@ -26,6 +26,18 @@ use tokio::io::AsyncReadExt as _; const SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION: u16 = 1; const MAX_SCANNER_CYCLE_STATE_BYTES: u64 = 1024 * 1024; pub(super) const MAX_SCANNER_CYCLE_RECOVERY_RETRIES: u32 = 5; +/// Longest automatic wait between two leader iterations while a retryable +/// recovery observation persists. It replaces the former terminal pause: a +/// fault that outlives the short retry budget keeps retrying at this cadence. +pub(super) const SCANNER_CYCLE_RECOVERY_BACKOFF_CAP: Duration = Duration::from_secs(10 * 60); +const SCANNER_CYCLE_RECOVERY_BACKOFF_BASE: Duration = Duration::from_secs(5); +/// A retryable observation older than this restarts its retry budget so the +/// visible count and the backoff schedule do not grow without bound. +const SCANNER_CYCLE_RECOVERY_RETRY_WINDOW_SECS: u64 = 24 * 60 * 60; +const SCANNER_CYCLE_RECOVERY_STATE_BACKOFF: &str = "backoff"; +const SCANNER_CYCLE_RECOVERY_STATE_REBUILDING: &str = "rebuilding"; +const SCANNER_CYCLE_RECOVERY_NEXT_ATTEMPT_MARKER: &str = "; next automatic attempt at unix "; +const SCANNER_CYCLE_RECOVERY_REBUILT_FROM_QUARANTINE: &str = "rebuilt_from_quarantine"; const METRIC_SCANNER_CYCLE_RECOVERY_REQUIRED: &str = "rustfs_scanner_cycle_recovery_required"; const METRIC_SCANNER_CYCLE_RECOVERY_RETRY_COUNT: &str = "rustfs_scanner_cycle_recovery_retry_count"; const USAGE_FLOOR_LOAD_FAILED: &str = "usage_floor_load_failed"; @@ -219,6 +231,11 @@ pub struct ScannerCycleRecoveryStatus { pub max_retries: u32, /// Whether the scanner may retry this state automatically. pub retryable: bool, + /// Unix time of the next automatic attempt while `retryable` is set. + pub next_attempt_at_unix_secs: Option, + /// Unix time at which this node rebuilt a quarantined cycle state through + /// its fenced leadership claim; the quarantine marker stays as evidence. + pub rebuilt_from_quarantine_at_unix_secs: Option, pub reason: Option, } @@ -287,6 +304,12 @@ static SCANNER_CYCLE_RECOVERY_STATUS: LazyLock ScannerCycleRecoveryStatus { SCANNER_CYCLE_RECOVERY_STATUS @@ -317,7 +340,8 @@ pub(super) fn publish_scanner_cleanup_status(status: ScannerCycleRecoveryStatus, let recovery_required = if matches!( status.state.as_str(), "blocked" - | "paused" + | SCANNER_CYCLE_RECOVERY_STATE_BACKOFF + | SCANNER_CYCLE_RECOVERY_STATE_REBUILDING | "recovery-required" | "cleanup-pending" | "usage_floor_load_failed" @@ -353,14 +377,28 @@ fn publish_scanner_cleanup_failure(reason: String, expected: u64) { let _ = publish_scanner_cleanup_status(status, Some(expected)); } +pub(super) fn clear_scanner_transient_cycle_recovery_status() { + // Snapshot the version before reading status so any concurrent publish rejects the CAS below. + let version = scanner_cleanup_status_version(); + let status = scanner_cycle_recovery_status(); + // A follower can discard its failed leadership observation. Durable recovery + // markers are revalidated before any later leadership claim. + if status.classification.is_none() && matches!(status.state.as_str(), "transient" | SCANNER_CYCLE_RECOVERY_STATE_BACKOFF) { + let _ = publish_scanner_cleanup_status(recovery_status("healthy", None, false), Some(version)); + } +} + pub(super) fn record_scanner_usage_floor_failure(reason: String) { let previous = scanner_cycle_recovery_status(); let same_failure = previous.classification.as_deref() == Some(USAGE_FLOOR_LOAD_FAILED); let now = unix_now_secs(); - let (first_detected_at_unix_secs, retry_count) = if same_failure { - (previous.first_detected_at_unix_secs.or(Some(now)), previous.retry_count) + // Only the first detection survives across attempts. The retry budget is + // owned by the supervisor; carrying it here made every later attempt + // exhaust the budget immediately (rustfs/rustfs#8670). + let first_detected_at_unix_secs = if same_failure { + previous.first_detected_at_unix_secs.or(Some(now)) } else { - (Some(now), 0) + Some(now) }; set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { path: DATA_USAGE_OBJ_NAME_PATH.clone(), @@ -368,7 +406,7 @@ pub(super) fn record_scanner_usage_floor_failure(reason: String) { classification: Some(USAGE_FLOOR_LOAD_FAILED.to_string()), first_detected_at_unix_secs, last_attempt_at_unix_secs: Some(now), - retry_count, + retry_count: 0, max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, retryable: true, reason: Some(reason), @@ -459,24 +497,99 @@ pub(super) fn clear_legacy_incomplete_usage_floor_recovery_status() { } } -pub(super) fn record_scanner_cycle_recovery_retry(attempt: u32) -> bool { +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct ScannerCycleRecoveryRetry { + /// Effective attempt number the supervisor continues counting from. + pub(super) attempt: u32, + /// Wait before the next automatic attempt. + pub(super) delay: Duration, +} + +/// Bounded exponential backoff for retryable recovery observations: it starts +/// at the transient retry base (or the configured cycle interval when that is +/// shorter) and never exceeds [`SCANNER_CYCLE_RECOVERY_BACKOFF_CAP`]. +pub(super) fn scanner_cycle_recovery_backoff_delay(attempt: u32, cycle_interval: Duration) -> Duration { + let exponent = attempt.saturating_sub(1).min(31); + let multiplier = 1u32.checked_shl(exponent).unwrap_or(u32::MAX); + let base = cycle_interval.clamp(Duration::from_secs(1), SCANNER_CYCLE_RECOVERY_BACKOFF_BASE); + base.saturating_mul(multiplier).min(SCANNER_CYCLE_RECOVERY_BACKOFF_CAP) +} + +fn reason_with_next_attempt(reason: Option, next_attempt_at_unix_secs: u64, delay: Duration) -> String { + let reason = reason.unwrap_or_default(); + let base = reason + .split(SCANNER_CYCLE_RECOVERY_NEXT_ATTEMPT_MARKER) + .next() + .unwrap_or_default(); + format!( + "{base}{SCANNER_CYCLE_RECOVERY_NEXT_ATTEMPT_MARKER}{next_attempt_at_unix_secs} (in {}s)", + delay.as_secs() + ) +} + +/// Record one automatic retry of the current retryable observation and return +/// the schedule for the next attempt. The observation never becomes terminal: +/// once the short retry budget is spent the state reads `backoff` and keeps +/// retrying at a capped cadence, and the reason names the next attempt time. +pub(super) fn record_scanner_cycle_recovery_retry(attempt: u32, cycle_interval: Duration) -> ScannerCycleRecoveryRetry { + let now = unix_now_secs(); let mut status = scanner_cycle_recovery_status(); if status.classification.as_deref() == Some(CACHE_CYCLE_AHEAD) { - return true; + return ScannerCycleRecoveryRetry { + attempt, + delay: scanner_cycle_recovery_backoff_delay(attempt, cycle_interval), + }; } - status.retry_count = status.retry_count.max(u64::from(attempt)); - status.last_attempt_at_unix_secs = Some(unix_now_secs()); - if status.max_retries != 0 && status.retry_count >= u64::from(status.max_retries) { - status.state = "paused".to_string(); - status.retryable = false; - status.reason = Some("scanner cycle recovery retry budget reached; sparse backend probes continue".to_string()); - set_scanner_cycle_recovery_status(status); - false + let window_started_at = if attempt <= 1 { + SCANNER_CYCLE_RECOVERY_WINDOW_STARTED_AT.store(now, Ordering::Relaxed); + now } else { - status.retryable = true; - set_scanner_cycle_recovery_status(status); - true + match SCANNER_CYCLE_RECOVERY_WINDOW_STARTED_AT.load(Ordering::Relaxed) { + 0 => { + SCANNER_CYCLE_RECOVERY_WINDOW_STARTED_AT.store(now, Ordering::Relaxed); + now + } + started => started, + } + }; + let first_detected_at_unix_secs = status + .first_detected_at_unix_secs + .map_or(window_started_at, |first| first.min(window_started_at)); + if now.saturating_sub(first_detected_at_unix_secs) >= SCANNER_CYCLE_RECOVERY_RETRY_WINDOW_SECS { + SCANNER_CYCLE_RECOVERY_WINDOW_STARTED_AT.store(now, Ordering::Relaxed); + status.first_detected_at_unix_secs = Some(now); + status.retry_count = 1; + } else { + status.first_detected_at_unix_secs = Some(first_detected_at_unix_secs); + status.retry_count = status.retry_count.max(u64::from(attempt)); } + let attempt = u32::try_from(status.retry_count).unwrap_or(u32::MAX); + let delay = scanner_cycle_recovery_backoff_delay(attempt, cycle_interval); + let next_attempt_at_unix_secs = now.saturating_add(delay.as_secs()); + let budget_spent = status.max_retries != 0 && status.retry_count >= u64::from(status.max_retries); + if budget_spent && status.state != SCANNER_CYCLE_RECOVERY_STATE_BACKOFF { + warn!( + target: "rustfs::scanner", + event = EVENT_SCANNER_CYCLE_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + state = SCANNER_CYCLE_RECOVERY_STATE_BACKOFF, + previous_state = %status.state, + classification = status.classification.as_deref().unwrap_or_default(), + retry_count = status.retry_count, + next_attempt_at_unix_secs, + "Scanner cycle recovery retry budget reached; continuing with bounded backoff" + ); + } + if budget_spent { + status.state = SCANNER_CYCLE_RECOVERY_STATE_BACKOFF.to_string(); + } + status.retryable = true; + status.last_attempt_at_unix_secs = Some(now); + status.next_attempt_at_unix_secs = Some(next_attempt_at_unix_secs); + status.reason = Some(reason_with_next_attempt(status.reason.take(), next_attempt_at_unix_secs, delay)); + set_scanner_cycle_recovery_status(status); + ScannerCycleRecoveryRetry { attempt, delay } } fn unix_now_secs() -> u64 { @@ -1005,6 +1118,12 @@ pub(crate) enum ScannerCycleStateStartup { leader_epoch: u64, revision: DataUsageCacheRevision, }, + /// The primary at `revision` is quarantined as corrupt. The caller starts + /// from an uninitialized cycle, raises it to the usage floor, and lets the + /// leadership claim CAS-replace the quarantined revision. + RebuildFromQuarantine { + revision: DataUsageCacheRevision, + }, Blocked, Transient(ScannerError), } @@ -1029,7 +1148,7 @@ enum CycleStateBodyReadError { Backend(#[source] EcstoreError), } -fn recovery_status_from_marker(marker: &ScannerCycleRecoveryMarker, state: &str) -> ScannerCycleRecoveryStatus { +fn recovery_status_from_marker(marker: &ScannerCycleRecoveryMarker, state: &str, retryable: bool) -> ScannerCycleRecoveryStatus { ScannerCycleRecoveryStatus { path: marker.path.clone(), quarantine_path: Some(marker.quarantine_path.clone()), @@ -1042,11 +1161,81 @@ fn recovery_status_from_marker(marker: &ScannerCycleRecoveryMarker, state: &str) last_attempt_at_unix_secs: Some(marker.last_attempt_at_unix_secs), retry_count: marker.retry_count, max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, - retryable: false, + retryable, reason: Some(marker.reason.clone()), + ..Default::default() } } +/// Only a counter this reader classified as corrupt is rebuilt automatically. +/// A future-schema counter belongs to a newer binary, and an operator reset in +/// progress (`cleanup-pending`) owns the primary until it finishes. +fn marker_schedules_automatic_rebuild(marker: &ScannerCycleRecoveryMarker) -> bool { + marker.state == "blocked" && marker.classification == "corrupt" +} + +/// The durable quarantine marker is the evidence; the rebuild itself goes +/// through the ordinary fenced leadership claim, which CAS-replaces the +/// quarantined revision and raises the epoch above the usage floor. +fn schedule_cycle_state_rebuild( + marker: &ScannerCycleRecoveryMarker, + revision: DataUsageCacheRevision, +) -> ScannerCycleStateStartup { + let mut status = recovery_status_from_marker(marker, SCANNER_CYCLE_RECOVERY_STATE_REBUILDING, true); + status.reason = Some(format!( + "{}; rebuilding the cycle counter from the usage floor under a new leadership epoch; quarantine evidence is retained at {}", + marker.reason, marker.quarantine_path + )); + set_scanner_cycle_recovery_status(status); + info!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %marker.path, + state = "quarantine_rebuild_scheduled", + primary_revision = %marker.primary_revision, + quarantine_path = %marker.quarantine_path, + "Scanner cycle state rebuild scheduled from quarantine" + ); + ScannerCycleStateStartup::RebuildFromQuarantine { revision } +} + +/// Publish the durable outcome of an automatic rebuild once the leadership +/// claim persisted the new counter. The marker is intentionally left behind. +pub(super) fn record_scanner_cycle_state_rebuilt_from_quarantine(next_cycle: u64, leader_epoch: u64) { + let previous = scanner_cycle_recovery_status(); + let now = unix_now_secs(); + set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "healthy".to_string(), + generation: Some(next_cycle), + leader_epoch: Some(leader_epoch), + first_detected_at_unix_secs: previous.first_detected_at_unix_secs, + last_attempt_at_unix_secs: Some(now), + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + rebuilt_from_quarantine_at_unix_secs: Some(now), + reason: Some(format!( + "cycle state rebuilt from quarantine; evidence retained at {}", + DATA_USAGE_BLOOM_RECOVERY_PATH.as_str() + )), + ..Default::default() + }); + info!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = SCANNER_CYCLE_RECOVERY_REBUILT_FROM_QUARANTINE, + next_cycle, + leader_epoch, + quarantine_path = %&*DATA_USAGE_BLOOM_RECOVERY_PATH, + "Scanner cycle state rebuilt from quarantine" + ); +} + fn marker_matches_revision(marker: &ScannerCycleRecoveryMarker, revision: &DataUsageCacheRevision) -> bool { matches!(revision, DataUsageCacheRevision::Etag(etag) if marker.primary_revision == *etag) } @@ -1379,6 +1568,7 @@ async fn quarantine_invalid_cycle_state_with_reason( max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, retryable: true, reason: Some(reason.to_string()), + ..Default::default() }; set_scanner_cycle_recovery_status(base_status); let Some(expected_epoch) = scanner_publication_epoch(storeapi.clone()).await else { @@ -1394,10 +1584,21 @@ async fn quarantine_invalid_cycle_state_with_reason( match persist_cycle_recovery_marker(storeapi, revision, generation, leader_epoch, classification, reason, expected_epoch) .await { - Ok(marker) => set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, "blocked")), + Ok(marker) => { + let automatic_rebuild = marker_schedules_automatic_rebuild(&marker); + let mut status = recovery_status_from_marker(&marker, "blocked", automatic_rebuild); + if automatic_rebuild { + status.reason = Some(format!( + "{}; the next leader iteration rebuilds the cycle counter from the usage floor", + marker.reason + )); + } + set_scanner_cycle_recovery_status(status); + } Err(CycleRecoveryMarkerReadError::Backend(_)) => { - // Keep the poison object untouched and retry marker creation with the - // bounded startup backoff; recovery-required never becomes healthy. + // Keep the poison object untouched and retry marker creation with + // the bounded recovery backoff; the quarantine marker must be + // durable before the counter is rebuilt. return ScannerCycleStateStartup::Transient(ScannerError::Other( "failed to persist scanner cycle recovery marker".to_string(), )); @@ -1461,6 +1662,23 @@ async fn mark_cycle_recovery_cleanup_pending( pub(crate) async fn load_scanner_cycle_state_for_startup( storeapi: Arc, +) -> ScannerCycleStateStartup { + let load = load_scanner_cycle_state_inner(storeapi); + #[cfg(feature = "diagnostic-read-version-caller")] + let load = crate::storage_api::ecstore_diagnostic::with_read_version_entry( + crate::storage_api::ecstore_diagnostic::ReadVersionEntrySource::ScannerCycleStateRead, + async { + crate::storage_api::ecstore_diagnostic::observe_read_version_context_boundary( + crate::storage_api::ecstore_diagnostic::ReadVersionContextBoundary::ScannerCycleStateLoad, + ); + load.await + }, + ); + load.await +} + +async fn load_scanner_cycle_state_inner( + storeapi: Arc, ) -> ScannerCycleStateStartup { let marker = match read_cycle_recovery_marker_bytes(storeapi.clone()).await { Ok((None, _)) => None, @@ -1536,12 +1754,15 @@ pub(crate) async fn load_scanner_cycle_state_for_startup( | EcstoreError::ConfigNotFound, ) => { if let Some((marker, _)) = marker { + if marker_schedules_automatic_rebuild(&marker) { + return schedule_cycle_state_rebuild(&marker, DataUsageCacheRevision::Missing); + } let state = if marker.state == "cleanup-pending" { "cleanup-pending" } else { "recovery-required" }; - set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, state)); + set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, state, false)); return ScannerCycleStateStartup::Blocked; } set_scanner_cycle_recovery_status(recovery_status("healthy", None, false)); @@ -1585,12 +1806,15 @@ pub(crate) async fn load_scanner_cycle_state_for_startup( .as_ref() .filter(|(marker, _)| marker.state == "cleanup-pending" || marker_matches_revision(marker, &revision)) { + if marker_schedules_automatic_rebuild(marker) { + return schedule_cycle_state_rebuild(marker, revision); + } let state = if marker.state == "cleanup-pending" { "cleanup-pending" } else { "blocked" }; - set_scanner_cycle_recovery_status(recovery_status_from_marker(marker, state)); + set_scanner_cycle_recovery_status(recovery_status_from_marker(marker, state, false)); return ScannerCycleStateStartup::Blocked; } let data = match read_cycle_state_body(&mut reader).await { @@ -1658,7 +1882,7 @@ pub(super) async fn resume_scanner_cycle_cleanup(ctx: CancellationToken, storeap } }; let mut observation = - publish_scanner_cleanup_status(recovery_status_from_marker(&marker, &marker.state), Some(status_version)); + publish_scanner_cleanup_status(recovery_status_from_marker(&marker, &marker.state, false), Some(status_version)); if marker.state != "cleanup-pending" { return Ok(()); } @@ -1753,7 +1977,7 @@ pub(super) async fn reset_scanner_cycle_recovery_for_intent( .ok_or_else(|| { ScannerError::Other("scanner cleanup recovery requires an unchanged cleanup-pending marker".to_string()) })?; - publish_status(recovery_status_from_marker(&marker, "cleanup-pending")); + publish_status(recovery_status_from_marker(&marker, "cleanup-pending", false)); } let Some(marker_data) = marker_data else { // A delete may commit before its reply is lost. Confirm both durable @@ -1857,7 +2081,7 @@ pub(super) async fn reset_scanner_cycle_recovery_for_intent( let (cleanup_marker, cleanup_marker_revision) = mark_cycle_recovery_cleanup_pending(storeapi.clone(), marker.clone(), &marker_revision, reset_epoch, &owns_reset) .await?; - publish_status(recovery_status_from_marker(&cleanup_marker, "cleanup-pending")); + publish_status(recovery_status_from_marker(&cleanup_marker, "cleanup-pending", false)); let usage_floor = persisted_usage_floor(storeapi.clone()).await?; let fence_epoch = primary_epoch .max(usage_floor.leader_epoch) @@ -2057,6 +2281,7 @@ pub(super) async fn reset_scanner_cycle_recovery_for_intent( max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, retryable: false, reason: Some("cycle state rebuilt but usage epoch fencing failed".to_string()), + ..Default::default() }); return Err(err); } @@ -2085,6 +2310,7 @@ pub(super) async fn reset_scanner_cycle_recovery_for_intent( max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, retryable: false, reason: Some("rebuilt scanner cycle state changed before marker cleanup".to_string()), + ..Default::default() }); return Err(ScannerError::Other( "rebuilt scanner cycle state changed before recovery marker cleanup".to_string(), @@ -2598,11 +2824,13 @@ pub async fn reset_scanner_usage_state_for_full_rebuild( }; let (cycle, cycle_epoch, cycle_revision) = read_cycle_state_for_usage_reset(storeapi.clone()).await?; let slots = read_usage_state_reset_slots(storeapi.clone()).await?; + // Corrupt usage slots without any cycle state carry no epoch to honor: + // the operator asked for a full rebuild, so start from the empty baseline + // exactly as a missing floor would (rustfs/rustfs#8670). Quota keeps + // failing closed for quota-bearing buckets until the rebuilt baseline + // publishes, which is the existing bootstrap-pending contract. let usage_floor = match usage_state_reset_floor(&slots)? { ScannerUsageResetFloor::Trusted(floor) => floor, - ScannerUsageResetFloor::Corrupt if matches!(cycle_revision, DataUsageCacheRevision::Missing) => { - return Err(ScannerError::Other("scanner usage reset has no trusted cycle or usage floor".to_string())); - } ScannerUsageResetFloor::Missing | ScannerUsageResetFloor::Corrupt => PersistedUsageFloor { next_cycle: cycle.next, leader_epoch: cycle_epoch, @@ -3772,6 +4000,19 @@ pub(super) async fn persist_scanner_cycle_state( leader_epoch: u64, ) -> bool { persist_scanner_cycle_state_for_epoch(ctx, storeapi, cycle_info, revision, leader_epoch, None).await + == ScannerCycleStatePersistOutcome::Saved +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum ScannerCycleStatePersistOutcome { + /// Durable cycle state carries this leader's epoch and progress. + Saved, + /// Durable state is owned by a newer leadership epoch; this leader yields + /// and must not retry under its own epoch. + Fenced, + /// The save could not be confirmed and no newer owner was observed. The + /// caller may reload durable state and continue under the same epoch. + Unconfirmed, } pub(super) async fn persist_scanner_cycle_state_for_epoch( @@ -3781,7 +4022,8 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( revision: &mut DataUsageCacheRevision, leader_epoch: u64, expected_publication_epoch: Option, -) -> bool { +) -> ScannerCycleStatePersistOutcome { + use ScannerCycleStatePersistOutcome::{Fenced, Saved, Unconfirmed}; let buf = match encode_scanner_cycle_state(cycle_info, leader_epoch) { Ok(buf) => buf, Err(e) => { @@ -3795,7 +4037,7 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( error = %e, "Scanner state encoding failed" ); - return false; + return Unconfirmed; } }; @@ -3811,16 +4053,16 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( retry, "Scanner state persistence cancelled by the leader fence" ); - return false; + return Unconfirmed; } #[cfg(test)] notify_scanner_cycle_state_persist_test_hook(leader_epoch); let Some(read_epoch) = scanner_publication_epoch(storeapi.clone()).await else { - return false; + return Unconfirmed; }; if expected_publication_epoch.is_some_and(|expected| expected != read_epoch) { - return false; + return Unconfirmed; } let save_result = { let Some(_publication_admission) = scanner_publication_admission_for_epoch(storeapi.clone(), read_epoch).await else { @@ -3833,7 +4075,7 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( state = "publication_admission_unavailable", "Scanner state persistence skipped without movement admission" ); - return false; + return Unconfirmed; }; let save = save_config_with_preconditions( storeapi.clone(), @@ -3859,7 +4101,7 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( }; result }; - match save_result { + let precondition_failed = match save_result { Ok(object_info) => { let Some(etag) = object_info.etag.filter(|etag| !etag.is_empty()) else { error!( @@ -3871,7 +4113,7 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( state = "missing_revision", "Scanner state save returned no ETag" ); - return false; + return Unconfirmed; }; *revision = DataUsageCacheRevision::Etag(etag); if ctx.is_cancelled() { @@ -3885,14 +4127,14 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( retry, "Scanner state save completed after the leader fence was cancelled" ); - return false; + return Unconfirmed; } if let Some(expected_epoch) = expected_publication_epoch && scanner_publication_admission_for_epoch(storeapi.clone(), expected_epoch) .await .is_none() { - return false; + return Unconfirmed; } debug!( target: "rustfs::scanner", @@ -3903,153 +4145,158 @@ pub(super) async fn persist_scanner_cycle_state_for_epoch( state = "saved", "Scanner state saved" ); - return true; - } - Err(EcstoreError::PreconditionFailed) => { - let (persisted, persisted_revision) = - match read_config_with_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()).await { - Ok(result) => result, - Err(e) => { - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "conflict_reload_failed", - error = %e, - "Scanner state conflict reconciliation failed" - ); - return false; - } - }; - *revision = persisted_revision; - if ctx.is_cancelled() { - debug!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "cancelled_after_conflict", - retry, - "Scanner state conflict reconciliation cancelled by the leader fence" - ); - return false; - } - - if let Some(persisted) = persisted { - if persisted.len() < 8 { - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "conflict_state_invalid", - length = persisted.len(), - "Scanner state conflict winner is truncated" - ); - return false; - } - - let (persisted_cycle, persisted_epoch) = match decode_scanner_cycle_state(&persisted) { - Ok(state) => state, - Err(e) => { - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "conflict_state_decode_failed", - error = %e, - "Scanner state conflict winner could not be decoded" - ); - return false; - } - }; - if persisted_epoch != leader_epoch { - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "leader_epoch_fenced", - expected_epoch = leader_epoch, - persisted_epoch, - "Scanner state save rejected by a newer leadership epoch" - ); - return false; - } - - if persisted_cycle.next >= cycle_info.next { - if let Some(expected_epoch) = expected_publication_epoch - && scanner_publication_admission_for_epoch(storeapi.clone(), expected_epoch) - .await - .is_none() - { - return false; - } - *cycle_info = persisted_cycle; - debug!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "conflict_reconciled", - retry, - "Scanner state adopted the current persisted cycle" - ); - return true; - } - } - - if retry < SCANNER_PERSIST_CAS_RETRIES { - debug!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "conflict_retry", - retry = retry + 1, - "Scanner state CAS conflict will be retried" - ); - continue; - } - - error!( - target: "rustfs::scanner", - event = EVENT_SCANNER_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_RUNTIME, - path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "conflict_retries_exhausted", - retries = SCANNER_PERSIST_CAS_RETRIES, - "Scanner state CAS conflict retries exhausted" - ); - return false; + return Saved; } + Err(EcstoreError::PreconditionFailed) => true, Err(e) => { + // A timeout or backend error does not change ownership. The + // write may still have committed, so reconcile against the + // durable revision before retrying under the same epoch + // (rustfs/rustfs#8141). + warn!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = "save_failed", + retry, + error = %e, + "Scanner state save failed; reconciling durable state before retrying" + ); + false + } + }; + + let (persisted, persisted_revision) = + match read_config_with_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()).await { + Ok(result) => result, + Err(e) => { + error!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = "conflict_reload_failed", + error = %e, + "Scanner state conflict reconciliation failed" + ); + return Unconfirmed; + } + }; + *revision = persisted_revision; + if ctx.is_cancelled() { + debug!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = "cancelled_after_conflict", + retry, + "Scanner state conflict reconciliation cancelled by the leader fence" + ); + return Unconfirmed; + } + + if let Some(persisted) = persisted { + if persisted.len() < 8 { error!( target: "rustfs::scanner", event = EVENT_SCANNER_PERSIST_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_RUNTIME, path = %&*DATA_USAGE_BLOOM_NAME_PATH, - state = "failed", - error = %e, - "Scanner state persistence failed" + state = "conflict_state_invalid", + length = persisted.len(), + "Scanner state conflict winner is truncated" ); - return false; + return Unconfirmed; + } + + let (persisted_cycle, persisted_epoch) = match decode_scanner_cycle_state(&persisted) { + Ok(state) => state, + Err(e) => { + error!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = "conflict_state_decode_failed", + error = %e, + "Scanner state conflict winner could not be decoded" + ); + return Unconfirmed; + } + }; + if persisted_epoch != leader_epoch { + error!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = "leader_epoch_fenced", + expected_epoch = leader_epoch, + persisted_epoch, + "Scanner state save rejected by a newer leadership epoch" + ); + return Fenced; + } + + if persisted_cycle.next >= cycle_info.next { + if let Some(expected_epoch) = expected_publication_epoch + && scanner_publication_admission_for_epoch(storeapi.clone(), expected_epoch) + .await + .is_none() + { + return Unconfirmed; + } + *cycle_info = persisted_cycle; + debug!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = if precondition_failed { "conflict_reconciled" } else { "save_confirmed" }, + retry, + "Scanner state adopted the current persisted cycle" + ); + return Saved; } } + + if retry < SCANNER_PERSIST_CAS_RETRIES { + debug!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = if precondition_failed { "conflict_retry" } else { "save_retry" }, + retry = retry + 1, + "Scanner state save will be retried under the same epoch" + ); + continue; + } + + error!( + target: "rustfs::scanner", + event = EVENT_SCANNER_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + path = %&*DATA_USAGE_BLOOM_NAME_PATH, + state = if precondition_failed { "conflict_retries_exhausted" } else { "save_retries_exhausted" }, + retries = SCANNER_PERSIST_CAS_RETRIES, + "Scanner state save retries exhausted" + ); + return Unconfirmed; } - false + Unconfirmed } #[cfg(test)] @@ -4102,7 +4349,8 @@ pub(super) async fn finalize_partial_scan_cycle_for_epoch( leader_epoch, expected_publication_epoch, ) - .await; + .await + == ScannerCycleStatePersistOutcome::Saved; if !persisted && let Some(expected_epoch) = expected_publication_epoch && scanner_publication_admission_for_epoch(storeapi, expected_epoch) @@ -4176,7 +4424,8 @@ pub(super) async fn persist_required_scanner_cycle_floor_for_epoch( leader_epoch, options.expected_publication_epoch, ) - .await; + .await + == ScannerCycleStatePersistOutcome::Saved; if !persisted && let Some(expected_epoch) = options.expected_publication_epoch && scanner_publication_admission_for_epoch(storeapi, expected_epoch) diff --git a/crates/scanner/src/scanner/heal_info.rs b/crates/scanner/src/scanner/heal_info.rs index 5f898d44a..9e91e9245 100644 --- a/crates/scanner/src/scanner/heal_info.rs +++ b/crates/scanner/src/scanner/heal_info.rs @@ -13,6 +13,8 @@ // limitations under the License. /// The background-heal info object persisted between scanner cycles. use super::*; +use std::sync::LazyLock as StdLazyLock; +use std::sync::RwLock as StdRwLock; /// Background healing information #[derive(Clone, Debug, Default, Serialize, Deserialize)] @@ -136,6 +138,10 @@ pub(super) async fn save_background_heal_info_for_epoch( ) where S: ScannerStorage, { + // Remember the record before any durability check: a skipped or failed + // save must not erase the cadence start this process decided on. + remember_background_heal_info(&info); + // Serialize to JSON let data = match serde_json::to_vec(&info) { Ok(data) => data, @@ -186,3 +192,190 @@ pub(super) async fn save_background_heal_info_for_epoch( ); } } + +/// Where the bitrot cadence start used by a cycle came from. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ScannerBitrotStartSource { + /// No cycle has recorded a start yet. + None, + /// This process recorded the start; storage has not returned it yet. + Memory, + /// The start was read back from the persisted background-heal object. + Persisted, +} + +impl ScannerBitrotStartSource { + pub fn as_str(self) -> &'static str { + match self { + Self::None => "none", + Self::Memory => "memory", + Self::Persisted => "persisted", + } + } +} + +/// The bitrot cadence reference a cycle decides its scan mode from. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum ScannerBitrotStart { + /// Nothing recorded and no anchor; a cycle never decides from this. + None, + /// A Deep selection window opened at `cycle` / `started`, from the + /// persisted or the remembered record. + Window { cycle: u64, started: DateTime }, + /// No Deep window is recorded anywhere; the cadence counts from the first + /// cycle this process ran without one. Never persisted, so it opens no + /// selection window and old readers see no new state. + Anchor(DateTime), +} + +impl ScannerBitrotStart { + pub(super) fn kind(self) -> &'static str { + match self { + Self::None => "none", + Self::Window { .. } => "deep_window", + Self::Anchor(_) => "anchor", + } + } + + fn cycle(self) -> u64 { + match self { + Self::Window { cycle, .. } => cycle, + Self::None | Self::Anchor(_) => 0, + } + } + + fn started(self) -> Option> { + match self { + Self::None => None, + Self::Window { started, .. } => Some(started), + Self::Anchor(anchored) => Some(anchored), + } + } +} + +/// How the scanner decided the scan mode of the running or latest cycle. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +pub struct ScannerBitrotStatus { + /// `false` on an erasure single-disk deployment, where scanner object + /// heal sampling is unsupported and Deep is never selected. + pub deep_scan_supported: bool, + pub bitrot_start_source: ScannerBitrotStartSource, + /// `deep_window` when a recorded Deep window start is in effect, + /// `anchor` when only this process's in-memory cadence anchor exists. + pub bitrot_start_kind: &'static str, + pub bitrot_start_cycle: u64, + pub bitrot_start_unix_secs: u64, + pub current_scan_mode: &'static str, +} + +impl Default for ScannerBitrotStatus { + fn default() -> Self { + Self { + deep_scan_supported: false, + bitrot_start_source: ScannerBitrotStartSource::None, + bitrot_start_kind: ScannerBitrotStart::None.kind(), + bitrot_start_cycle: 0, + bitrot_start_unix_secs: 0, + current_scan_mode: HealScanMode::Unknown.as_str(), + } + } +} + +/// Process-lifetime copy of the last background-heal record this node +/// decided on, so a skipped or lost save does not erase the cadence start. +static REMEMBERED_BACKGROUND_HEAL_INFO: StdRwLock> = StdRwLock::new(None); + +static SCANNER_BITROT_STATUS: StdLazyLock> = + StdLazyLock::new(|| StdRwLock::new(ScannerBitrotStatus::default())); + +/// When this process first ran a cycle without any recorded Deep window. +static SCANNER_BITROT_ANCHOR: StdRwLock>> = StdRwLock::new(None); + +fn bitrot_cadence_anchor(now: DateTime) -> DateTime { + let mut anchor = SCANNER_BITROT_ANCHOR.write().unwrap_or_else(|poisoned| poisoned.into_inner()); + *anchor.get_or_insert(now) +} + +#[cfg(test)] +pub(super) fn reset_bitrot_cadence_anchor_for_test() { + *SCANNER_BITROT_ANCHOR.write().unwrap_or_else(|poisoned| poisoned.into_inner()) = None; +} + +/// The cadence reference for a cycle: the recorded Deep window start when the +/// record carries one, otherwise this process's in-memory anchor. +pub(super) fn scanner_bitrot_start_for_cycle(info: &BackgroundHealInfo, now: DateTime) -> ScannerBitrotStart { + match info.bitrot_start_time { + Some(started) => ScannerBitrotStart::Window { + cycle: info.bitrot_start_cycle, + started, + }, + None => ScannerBitrotStart::Anchor(bitrot_cadence_anchor(now)), + } +} + +pub(super) fn remember_background_heal_info(info: &BackgroundHealInfo) { + *REMEMBERED_BACKGROUND_HEAL_INFO + .write() + .unwrap_or_else(|poisoned| poisoned.into_inner()) = Some(info.clone()); +} + +fn remembered_background_heal_info() -> Option { + REMEMBERED_BACKGROUND_HEAL_INFO + .read() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .clone() +} + +/// Pick the record whose cadence start is newest. Ties go to storage so a +/// start that was persisted successfully reports as `persisted`; a record +/// without a start cannot win over one that has it. +pub(super) fn resolve_background_heal_info( + persisted: Option<&BackgroundHealInfo>, + remembered: Option<&BackgroundHealInfo>, +) -> (BackgroundHealInfo, ScannerBitrotStartSource) { + let persisted_start = persisted.filter(|info| info.bitrot_start_time.is_some()); + let remembered_start = remembered.filter(|info| info.bitrot_start_time.is_some()); + match (persisted_start, remembered_start) { + (Some(stored), Some(memory)) if memory.bitrot_start_time > stored.bitrot_start_time => { + (memory.clone(), ScannerBitrotStartSource::Memory) + } + (Some(stored), _) => (stored.clone(), ScannerBitrotStartSource::Persisted), + (None, Some(memory)) => (memory.clone(), ScannerBitrotStartSource::Memory), + (None, None) => (persisted.cloned().unwrap_or_default(), ScannerBitrotStartSource::None), + } +} + +/// Resolve the record a cycle decides from: the persisted read when it was +/// loaded, otherwise what this process remembered. +pub(super) fn resolve_background_heal_info_for_cycle( + read: &BackgroundHealInfoRead, +) -> (BackgroundHealInfo, ScannerBitrotStartSource) { + let persisted = (read.status == BackgroundHealInfoReadStatus::Loaded).then_some(&read.info); + resolve_background_heal_info(persisted, remembered_background_heal_info().as_ref()) +} + +pub(super) fn record_scanner_bitrot_status( + start: ScannerBitrotStart, + source: ScannerBitrotStartSource, + deep_scan_supported: bool, + scan_mode: HealScanMode, +) { + let status = ScannerBitrotStatus { + deep_scan_supported, + bitrot_start_source: source, + bitrot_start_kind: start.kind(), + bitrot_start_cycle: start.cycle(), + bitrot_start_unix_secs: start + .started() + .and_then(|started| u64::try_from(started.timestamp()).ok()) + .unwrap_or(0), + current_scan_mode: scan_mode.as_str(), + }; + *SCANNER_BITROT_STATUS.write().unwrap_or_else(|poisoned| poisoned.into_inner()) = status; +} + +/// Deep cadence state of the running or latest cycle on this node. +pub fn scanner_bitrot_status() -> ScannerBitrotStatus { + *SCANNER_BITROT_STATUS.read().unwrap_or_else(|poisoned| poisoned.into_inner()) +} diff --git a/crates/scanner/src/scanner/tests.rs b/crates/scanner/src/scanner/tests.rs index 4c3c9879c..7b298aa4d 100644 --- a/crates/scanner/src/scanner/tests.rs +++ b/crates/scanner/src/scanner/tests.rs @@ -12,10 +12,15 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::heal_info::{classify_background_heal_read_error, decode_background_heal_info}; +use super::heal_info::{ + ScannerBitrotStart, classify_background_heal_read_error, decode_background_heal_info, record_scanner_bitrot_status, + reset_bitrot_cadence_anchor_for_test, resolve_background_heal_info, scanner_bitrot_start_for_cycle, +}; use super::*; use crate::EcstoreResult; -use crate::storage_api::owner::{WriteCommitGuard, WriteCompletion, ecstore_hold_namespace_commit}; +use crate::storage_api::owner::{ + WriteCommitGuard, WriteCompletion, ecstore_hold_namespace_commit, ecstore_namespace_commits_pending, +}; use crate::storage_api::scan::{BucketOperations as _, ObjectIO as _}; use crate::{ DATA_USAGE_BLOOM_RECOVERY_PATH, DATA_USAGE_CACHE_KEY_FORMAT, DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, @@ -37,7 +42,9 @@ use tokio::time::{Duration, advance}; const TEST_DEFAULT_SCANNER_CYCLE_SECS: u64 = 24 * 60 * 60; +mod cycle_budget_publication; pub(super) mod cycle_persist_failure; +mod gate_self_heal; mod quota_reset_preservation; mod recovery_control; @@ -138,6 +145,11 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein let endpoint_pools = EndpointServerPools::from(pools); let instance_ctx = Arc::new(InstanceContext::new()); instance_ctx.set_endpoints(endpoint_pools.clone()); + if pool_count == 1 && sets_per_pool == 1 && drives_per_set == 1 { + instance_ctx + .update_erasure_type(crate::storage_api::EcstoreSetupType::ErasureSD) + .await; + } if preinitialize_disks { init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone()) .await @@ -883,6 +895,12 @@ impl Drop for ScannerDefaultCycleGuard { #[derive(Debug, Default)] struct MemoryConfigStore { + #[cfg(feature = "diagnostic-read-version-caller")] + entry_sources: Mutex>>, + #[cfg(feature = "diagnostic-read-version-caller")] + write_entry_sources: Mutex>>, + #[cfg(feature = "diagnostic-read-version-caller")] + pending_diagnostic_write: AtomicBool, objects: Mutex>>, revisions: Mutex>, insert_after_gets: Mutex>>, @@ -890,6 +908,8 @@ struct MemoryConfigStore { delayed_gets: Mutex>, non_regular_objects: Mutex>, fail_put_number: Mutex>, + /// Fail every put whose number is at least the stored value. + fail_puts_from: Mutex>, object_not_found_put_number: Mutex>, error_after_commit_put_number: Mutex>, interleaving_puts: Mutex)>>, @@ -937,6 +957,11 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore { _h: http::HeaderMap, _opts: &ObjectOptions, ) -> EcstoreResult { + #[cfg(feature = "diagnostic-read-version-caller")] + self.entry_sources + .lock() + .await + .push(crate::storage_api::ecstore_diagnostic::current_read_version_entry_source()); let key = memory_config_key(bucket, object); if let Some(error) = self.read_errors.lock().await.get(&key).cloned() { return Err(error); @@ -983,6 +1008,16 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore { data: &mut PutObjReader, opts: &ObjectOptions, ) -> EcstoreResult { + #[cfg(feature = "diagnostic-read-version-caller")] + { + self.write_entry_sources + .lock() + .await + .push(crate::storage_api::ecstore_diagnostic::current_read_version_entry_source()); + if self.pending_diagnostic_write.load(Ordering::Relaxed) { + std::future::pending::<()>().await; + } + } let mut buf = Vec::new(); data.stream.read_to_end(&mut buf).await?; let key = memory_config_key(bucket, object); @@ -993,7 +1028,14 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore { *put_count }; - if self.fail_put_number.lock().await.get(&key) == Some(&put_count) { + if self.fail_put_number.lock().await.get(&key) == Some(&put_count) + || self + .fail_puts_from + .lock() + .await + .get(&key) + .is_some_and(|from| put_count >= *from) + { return Err(EcstoreError::other("injected put failure")); } if self.object_not_found_put_number.lock().await.get(&key) == Some(&put_count) { @@ -1461,9 +1503,25 @@ async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() { #[tokio::test] #[serial] -async fn coordinator_persists_only_an_observation_during_pending_put() { +async fn coordinator_publishes_authority_during_pending_put() { + for drives_per_set in [1, 4] { + assert_bootstrap_authority_during_pending_put(drives_per_set).await; + } +} + +/// A PUT whose rename fanout is still pending when the walk finishes used to +/// turn the first cycle into an observation and keep authority bootstrap +/// pending until a quiet retry (rustfs/rustfs#8494). The tail is ordinary +/// namespace activity: the cycle publishes authority, discharges bootstrap, +/// and the object the tail commits afterwards is the next cycle's work. +async fn assert_bootstrap_authority_during_pending_put(drives_per_set: usize) { crate::scanner_io::clear_dirty_usage_buckets_for_tests(); - let (_temp_dir, store) = setup_scanner_cycle_store().await; + let temp_dir = tempfile::tempdir().expect("bootstrap directory should be created"); + let store = setup_scanner_cycle_store_at_path_with_layout(temp_dir.path(), false, 1, 1, drives_per_set).await; + initialize_usage_baseline_bootstrap(store.clone()) + .await + .expect("cold startup should persist its bootstrap identity"); + let mut bootstrap = ScannerUsageBootstrapRebuild::from_startup(PersistedUsageFloorStartup::BootstrapPending); let mut pause_backlog = ScannerPauseBacklogController::claim(store.clone(), scanner_pause_backlog_now()) .await .expect("scanner pause backlog should be available"); @@ -1488,16 +1546,34 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { .await .expect("fixture object should finish its rename fanout"); } + let cold_bucket = "scanner-coordinator-cold"; + store + .make_bucket(cold_bucket, &crate::storage_api::scan::MakeBucketOptions::default()) + .await + .expect("cold bucket should be created"); + let mut cold_reader = PutObjReader::from_vec(b"cold".to_vec()); + store.pools[0].disk_set[0] + .put_object( + cold_bucket, + "object", + &mut cold_reader, + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("cold object should persist before scanning"); + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); crate::scanner_io::record_dirty_usage_bucket(&bucket); - let dirty_before = crate::scanner_io::dirty_usage_buckets_for_tests(); let baseline = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) .await .expect("fixture usage baseline should be readable"); let pending = ecstore_hold_namespace_commit(store.as_ref()); assert_eq!( scanner_local_publication_defer_reason(store.as_ref(), false).await, - Some(ScannerCycleDeferReason::ActivityBaselineUnavailable), - "an ordinary namespace commit must not be classified as data movement" + None, + "an ordinary namespace commit is neither data movement nor a publication fence" ); assert_eq!( scanner_local_publication_defer_reason(store.as_ref(), true).await, @@ -1530,37 +1606,50 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { ) .await .expect("the coordinator must finish its namespace walk while a PUT is pending"); - assert_eq!(budget.progress().0, 1, "the coordinator must reach actual object traversal"); + assert_eq!(budget.progress().0, 2, "the coordinator must reach actual object traversal"); assert!( matches!( outcome, - ScannerCycleOutcome::Superseded | ScannerCycleOutcome::Partial | ScannerCycleOutcome::CompletedWithPendingMaintenance + ScannerCycleOutcome::Completed | ScannerCycleOutcome::CompletedWithPendingMaintenance ), - "a pending PUT may store an observation without publishing authority: {outcome:?}" + "a pending PUT tail must not withhold authority from a complete walk: {outcome:?}" ); finish_scanner_pause_backlog_cycle(&mut pause_backlog, &store, pause_backlog_attempt, outcome).await; let pause_backlog_status = scanner_pause_backlog_status(store.clone()).await; assert_eq!(pause_backlog_status.phase, ScannerPauseBacklogPhase::Idle); assert!(!pause_backlog_status.pending_full_scan); assert_eq!(pause_backlog_status.catch_up_attempts, 0); - assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + assert!( + !crate::scanner_io::dirty_usage_buckets_for_tests().contains_key(&bucket), + "a certified bucket leaves the dirty map" + ); assert!( matches!(revision, DataUsageCacheRevision::Etag(_)), - "the cycle-state floor should advance after its durable observation" + "the cycle-state floor should advance after its durable publication" ); - assert_eq!( - read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) - .await - .expect("the prior authoritative usage must remain readable"), - baseline, - "the pending candidate must not replace the authoritative baseline" - ); - let observed = read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + let published = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) .await - .expect("the pending cycle should persist its non-authoritative observation"); - let observed: DataUsageInfo = serde_json::from_slice(&observed).expect("observed usage should decode"); - assert_eq!(observed.usage_snapshot_converged, Some(false)); + .expect("the authoritative usage should be persisted"); + assert_ne!(published, baseline, "the complete cycle replaces the bootstrap baseline"); + let published: DataUsageInfo = serde_json::from_slice(&published).expect("published usage should decode"); + assert_eq!(published.usage_snapshot_converged, Some(true)); + assert!(!published.usage_snapshot_observational); + assert!(published.is_complete_bucket_usage_snapshot()); + assert_eq!(published.objects_total_count, 2); + assert!(bootstrap.record_cycle(outcome)); + assert!( + !bootstrap.pending(), + "a complete cycle under a pending PUT tail discharges authority bootstrap" + ); + crate::storage_api::ecstore_invalidate_data_usage_snapshot_cache().await; + let displayed = crate::storage_api::scanner_io::load_admin_data_usage_from_backend_cached(store.clone()) + .await + .expect("admin should load the durable authoritative snapshot"); + assert_eq!(displayed.objects_total_count, 2); + assert_eq!(displayed.usage_snapshot_converged, Some(true)); + // The tail commits a larger body after the walk. Its bucket is shaded + // for the next cycle exactly as the production mutation observer does. let committed_body = b"committed-after-walk"; let mut reader = PutObjReader::from_vec(committed_body.to_vec()); { @@ -1578,7 +1667,7 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { .await .expect("the pending tail must change the physical object before it drains"); } - assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + crate::scanner_io::record_dirty_usage_bucket(&bucket); drop(pending); let retry_budget = ScannerCycleBudget::new_with_progress_tracking(&ctx, ScannerCycleBudgetConfig::default()); let retry_cycle = cycle_info.next; @@ -1592,22 +1681,35 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { 1, Arc::clone(&retry_budget), ScannerCycleScheduling { - requires_full_scan: true, + requires_full_scan: ScannerMaintenanceFeatures::default().requires_full_scan( + Some(scanner_maintenance_generation()), + scanner_maintenance_generation(), + ScannerCycleWakeReason::Timer, + true, + ), service_cohort: None, }, ), ) .await .expect("the same cycle must converge after the pending PUT drains"); + // Without a recorded Deep window both layouts run Normal (an erasure set + // used to open a Deep window here and revisit every bucket), so the + // retry reuses the cold inventory and only walks the dirty bucket. assert_eq!( retry_budget.progress().0, 1, - "the same-cycle retry must not reuse the pre-tail bucket cache" + "Normal retries reuse cold inventory (drives_per_set={drives_per_set})" ); assert!(matches!( outcome, ScannerCycleOutcome::Completed | ScannerCycleOutcome::CompletedWithPendingMaintenance )); + assert!( + !bootstrap.record_cycle(outcome), + "authority bootstrap was already discharged by the first cycle" + ); + assert!(!bootstrap.pending()); assert_eq!(cycle_info.next, retry_cycle + 1); assert!(!crate::scanner_io::dirty_usage_buckets_for_tests().contains_key(&bucket)); let usage = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) @@ -1616,10 +1718,10 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { let usage: DataUsageInfo = serde_json::from_slice(&usage).expect("the persisted usage should decode"); assert_eq!(usage.usage_snapshot_converged, Some(true)); assert_eq!(usage.scanner_cycle, Some(retry_cycle)); - assert_eq!(usage.objects_total_count, 1); + assert_eq!(usage.objects_total_count, 2); assert_eq!( usage.objects_total_size, - u64::try_from(committed_body.len()).expect("fixture body length") + u64::try_from(committed_body.len() + 4).expect("fixture total body length") ); let bucket_usage = usage .buckets_usage @@ -1633,28 +1735,61 @@ async fn coordinator_persists_only_an_observation_during_pending_put() { #[tokio::test] #[serial] -async fn complete_cycle_keeps_strict_publication_fence_with_observational_flag() { +async fn complete_cycle_keeps_movement_fence_with_observational_flag() { let (_temp_dir, store) = setup_scanner_cycle_store().await; let pending = ecstore_hold_namespace_commit(store.as_ref()); - let complete_uses_observation = scanner_cycle_uses_observational_publication(ScannerCycleStatus::Complete, true); + let uses_observation = |status| { + ScannerCyclePublicationPlan::for_result(&ScannerCycleResult::new(status, None).with_usage_update_emitted(true)) + .is_observational() + }; + let complete_uses_observation = uses_observation(ScannerCycleStatus::Complete); assert!(!complete_uses_observation, "a converged Complete snapshot is authoritative"); assert_eq!( scanner_local_publication_defer_reason(store.as_ref(), complete_uses_observation).await, - Some(ScannerCycleDeferReason::ActivityBaselineUnavailable), - "a Complete cycle must remain fenced while an ordinary namespace commit is pending" + None, + "an ordinary namespace commit shades its bucket; it is not a publication fence" + ); + drop(pending); + + // A terminal decommission entry is storage movement the operator must + // clear: it fences the authoritative and the observational route alike. + store.pool_meta.write().await.pools[0].decommission = Some(crate::storage_api::owner::EcstorePoolDecommissionInfo { + failed: true, + ..Default::default() + }); + for observational_only in [false, true] { + assert_eq!( + scanner_local_publication_defer_reason(store.as_ref(), observational_only).await, + Some(ScannerCycleDeferReason::DataMovement), + "observational_only={observational_only}" + ); + } + store.pool_meta.write().await.pools[0].decommission = None; + + assert!(uses_observation(ScannerCycleStatus::Superseded)); + assert!(uses_observation(ScannerCycleStatus::Deferred( + ScannerCycleDeferReason::ActivityBaselineUnavailable + ))); + assert!(!uses_observation(ScannerCycleStatus::Deferred(ScannerCycleDeferReason::DataMovement))); +} + +#[tokio::test] +#[serial] +async fn incomplete_observation_uses_observational_lease_during_pending_namespace_commit() { + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let pending = ecstore_hold_namespace_commit(store.as_ref()); + let result = ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None); + assert!(!ScannerCyclePublicationPlan::for_result(&result).is_observational()); + let uses_observation = ScannerCyclePublicationPlan::for_result(&result.with_usage_update_emitted(true)).is_observational(); + + assert!(uses_observation); + assert_eq!( + scanner_local_publication_defer_reason(store.as_ref(), uses_observation).await, + None, + "an incomplete cycle's observation must not be blocked by an ordinary namespace commit" ); - assert!(scanner_cycle_uses_observational_publication(ScannerCycleStatus::Superseded, true)); - assert!(scanner_cycle_uses_observational_publication( - ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), - true - )); - assert!(!scanner_cycle_uses_observational_publication(ScannerCycleStatus::Incomplete, true)); - assert!(!scanner_cycle_uses_observational_publication( - ScannerCycleStatus::Deferred(ScannerCycleDeferReason::DataMovement), - true - )); drop(pending); } @@ -1838,6 +1973,140 @@ fn scanner_startup_fails_closed_on_nonempty_corrupt_cycle_state() { assert!(encode_scanner_cycle_state(&exhausted, 7).is_err()); } +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +#[serial] +async fn diagnostic_scanner_config_writes_preserve_origin_for_both_helpers_and_cancellation() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for shared in [false, true] { + for case in ["success", "error", "cancelled"] { + let store = Arc::new(MemoryConfigStore::default()); + let path = "buckets/diagnostic/write-state"; + let key = memory_config_key(RUSTFS_META_BUCKET, path); + if case == "error" { + store.fail_put_number.lock().await.insert(key.clone(), 1); + } + store.pending_diagnostic_write.store(case == "cancelled", Ordering::Relaxed); + let write = async { + let operation = async { + let preconditions = crate::HTTPPreconditions { + if_none_match: Some("*".to_string()), + ..Default::default() + }; + if shared { + crate::save_config_shared_with_preconditions_and_lease_fence_and_scope( + store.clone(), + path, + bytes::Bytes::from_static(b"state"), + None, + preconditions, + None, + None, + ) + .await + } else { + crate::save_config_with_preconditions(store.clone(), path, b"state".to_vec(), preconditions).await + } + }; + let result = tokio::time::timeout(Duration::from_secs(1), operation).await; + match case { + "success" => assert!(result.expect("write completes").is_ok()), + "error" => assert!(result.expect("error completes").is_err()), + _ => assert!(result.is_err(), "pending write is cancelled"), + } + }; + match earlier { + Some(source) => with_read_version_entry(source, write).await, + None => write.await, + } + assert_eq!( + *store.write_entry_sources.lock().await, + vec![Some(earlier.unwrap_or(ReadVersionEntrySource::ScannerConfigWrite))] + ); + assert_eq!(current_read_version_entry_source(), None, "scope restores: {case}"); + let stored = store.objects.lock().await.get(&key).cloned(); + assert_eq!(stored, (case == "success").then(|| b"state".to_vec()), "write result preserved: {case}"); + } + } + } +} + +#[cfg(feature = "diagnostic-read-version-caller")] +#[tokio::test] +#[serial] +async fn diagnostic_cycle_state_load_preserves_entry_for_results_and_cancellation() { + use crate::storage_api::ecstore_diagnostic::{ + ReadVersionEntrySource, current_read_version_entry_source, with_read_version_entry, + }; + + for earlier in [None, Some(ReadVersionEntrySource::TierConfigReload)] { + for case in ["missing", "valid", "error", "cancelled"] { + let store = Arc::new(MemoryConfigStore::default()); + let marker = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + let state = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + if case == "valid" { + let cycle = CurrentCycle { + next: 9, + ..Default::default() + }; + store + .objects + .lock() + .await + .insert(state.clone(), encode_scanner_cycle_state(&cycle, 3).expect("cycle fixture")); + store.revisions.lock().await.insert(state, 7); + } else if case == "error" { + store + .read_errors + .lock() + .await + .insert(marker.clone(), EcstoreError::ErasureReadQuorum); + } else if case == "cancelled" { + store.delayed_gets.lock().await.insert(marker, Duration::from_secs(1)); + } + let load = async { + if case == "cancelled" { + assert!( + tokio::time::timeout(Duration::from_millis(1), load_scanner_cycle_state_for_startup(store.clone())) + .await + .is_err() + ); + return; + } + let result = load_scanner_cycle_state_for_startup(store.clone()).await; + match (case, result) { + ("missing", ScannerCycleStateStartup::Ready { .. }) => {} + ( + "valid", + ScannerCycleStateStartup::Ready { + cycle, leader_epoch: 3, .. + }, + ) => assert_eq!(cycle.next, 9), + ("error", ScannerCycleStateStartup::Transient(_)) => {} + _ => panic!("unexpected cycle load outcome: {case}"), + } + }; + match earlier { + Some(source) => with_read_version_entry(source, load).await, + None => load.await, + } + let observed = store.entry_sources.lock().await; + assert!(!observed.is_empty(), "load must enter a reader: {case}"); + assert!( + observed + .iter() + .all(|source| *source == Some(earlier.unwrap_or(ReadVersionEntrySource::ScannerCycleStateRead))), + "all readers preserve the original source: {case}" + ); + assert_eq!(current_read_version_entry_source(), None, "scope restores for all outcomes: {case}"); + } + } +} + #[tokio::test] #[serial] async fn corrupt_cycle_state_is_quarantined_once() { @@ -1864,11 +2133,19 @@ async fn corrupt_cycle_state_is_quarantined_once() { assert_eq!(marker.quarantine_path, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); assert_eq!(marker.classification, "corrupt"); - // A second startup sees the matching marker before consuming the poison body. + // A second startup sees the durable marker before consuming the poison + // body and schedules the automatic rebuild instead of blocking again. assert!(matches!( load_scanner_cycle_state_for_startup(store.clone()).await, - ScannerCycleStateStartup::Blocked + ScannerCycleStateStartup::RebuildFromQuarantine { + revision: DataUsageCacheRevision::Etag(ref revision) + } if revision == "memory-7" )); + assert_eq!( + store.objects.lock().await.get(&marker_key), + Some(&marker_data), + "scheduling the rebuild must not rewrite the marker" + ); // Replacing the primary object advances its revision; the stale marker must // not quarantine the newer, valid state. @@ -1941,8 +2218,24 @@ async fn concurrent_leaders_cannot_quarantine_newer_cycle_state() { load_scanner_cycle_state_for_startup(store.clone()), load_scanner_cycle_state_for_startup(store.clone()), ); - assert!(matches!(first, ScannerCycleStateStartup::Blocked)); - assert!(matches!(second, ScannerCycleStateStartup::Blocked)); + // Whichever contender publishes the marker blocks; the other either races + // the marker write (blocked or transient) or sees the durable marker and + // schedules the rebuild. Neither may quarantine a different revision. + for outcome in [&first, &second] { + assert!( + matches!( + outcome, + ScannerCycleStateStartup::Blocked + | ScannerCycleStateStartup::Transient(_) + | ScannerCycleStateStartup::RebuildFromQuarantine { .. } + ), + "{outcome:?}" + ); + } + assert!( + matches!(first, ScannerCycleStateStartup::Blocked) || matches!(second, ScannerCycleStateStartup::Blocked), + "one contender must publish the marker: {first:?} / {second:?}" + ); let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); let marker_data = store @@ -3469,7 +3762,7 @@ async fn scanner_usage_floor_recovery_retry_budget_uses_marker_epoch_identity() persisted_usage_floor_for_startup(store.clone(), true) .await .expect("legacy empty primary should enter recovery"); - assert!(record_scanner_cycle_recovery_retry(3)); + assert_eq!(record_scanner_cycle_recovery_retry(3, Duration::from_secs(1)).attempt, 3); let first_detected = scanner_cycle_recovery_status().first_detected_at_unix_secs; let primary = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) @@ -3632,10 +3925,22 @@ fn scanner_usage_floor_failure_is_exposed_and_cleared() { assert_eq!(blocked.reason.as_deref(), Some("persisted usage floor is invalid")); let first_detected = blocked.first_detected_at_unix_secs; - assert!(record_scanner_cycle_recovery_retry(2)); + let retry = record_scanner_cycle_recovery_retry(2, Duration::from_secs(1)); + assert_eq!(retry.attempt, 2); + assert_eq!(retry.delay, Duration::from_secs(2)); + let scheduled = scanner_cycle_recovery_status(); + assert_eq!(scheduled.retry_count, 2); + assert!(scheduled.next_attempt_at_unix_secs.is_some()); + assert!( + scheduled + .reason + .as_deref() + .is_some_and(|reason| reason.starts_with("persisted usage floor is invalid; next automatic attempt at unix ")) + ); record_scanner_usage_floor_failure("persisted usage floor remains invalid".to_string()); let retried = scanner_cycle_recovery_status(); - assert_eq!(retried.retry_count, 2); + assert_eq!(retried.retry_count, 0, "the supervisor owns the retry budget across attempts"); + assert!(retried.next_attempt_at_unix_secs.is_none()); assert_eq!(retried.first_detected_at_unix_secs, first_detected); clear_scanner_usage_floor_failure(); @@ -3657,7 +3962,7 @@ fn scanner_usage_floor_recovery_stays_retryable_until_claim_cleanup() { assert_eq!(pending.quarantine_path.as_deref(), Some(DATA_USAGE_RECOVERY_PATH.as_str())); let first_detected = pending.first_detected_at_unix_secs; - assert!(record_scanner_cycle_recovery_retry(3)); + assert_eq!(record_scanner_cycle_recovery_retry(3, Duration::from_secs(1)).attempt, 3); record_legacy_incomplete_usage_floor_recovery_pending(7); let retried = scanner_cycle_recovery_status(); assert_eq!(retried.retry_count, 3); @@ -3689,7 +3994,7 @@ fn scanner_cache_cycle_ahead_is_visible_until_a_later_scan_clears_it() { assert_eq!(observed_again.retry_count, 0); assert_eq!(observed_again.first_detected_at_unix_secs, first_detected); - assert!(record_scanner_cycle_recovery_retry(4)); + assert_eq!(record_scanner_cycle_recovery_retry(4, Duration::from_secs(1)).attempt, 4); assert_eq!(scanner_cycle_recovery_status().retry_count, 0); record_scanner_cache_cycle_recovery_attempt(); @@ -4627,7 +4932,8 @@ async fn test_finalize_partial_scan_cycle_reports_persist_failure() { let store = Arc::new(MemoryConfigStore::default()); let ctx = CancellationToken::new(); let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); - store.fail_put_number.lock().await.insert(key, 1); + // Every attempt of the bounded in-save retry fails. + store.fail_puts_from.lock().await.insert(key, 1); let mut revision = DataUsageCacheRevision::Missing; let mut cycle_info = CurrentCycle { current: 12, @@ -5790,30 +6096,47 @@ async fn scanner_usage_state_reset_cancels_during_publication_admission() { } } +/// Corrupt slots without any cycle state are the rustfs/rustfs#8670 shape. The +/// full rebuild starts from the empty baseline exactly as a missing floor +/// does: the result is a bootstrap-pending marker, never an authoritative zero +/// floor, so quota-bearing buckets keep failing closed until a complete +/// publication. A valid backup still supplies the recovery floor instead. #[tokio::test] #[serial] -async fn scanner_usage_state_reset_rejects_corruption_without_a_trusted_floor() { +async fn scanner_usage_state_reset_rebuilds_corruption_without_a_trusted_floor_from_empty_baseline() { let (_temp_dir, store) = setup_scanner_cycle_store_with_usage_baseline(false).await; save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), b"{corrupt".to_vec()) .await .expect("corrupt primary should persist"); - let before = read_config_with_revision(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + let result = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone()) .await - .expect("evidence should load"); - let err = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone()) + .expect("corrupt slots without a cycle state start from the empty baseline"); + assert_eq!(result.leader_epoch, 1); + assert_eq!(result.next_cycle, 0); + assert_eq!(result.usage_state, "bootstrap-pending"); + let primary = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) .await - .expect_err("corruption must not become a zero floor"); - assert!(err.to_string().contains("no trusted cycle or usage floor"), "{err}"); - assert_eq!( - read_config_with_revision(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .expect("reset should replace the corrupt primary with a bootstrap marker"); + let primary: DataUsageInfo = serde_json::from_slice(&primary).expect("bootstrap marker should decode"); + assert!(data_usage_info_is_bootstrap_pending(&primary), "corruption must not become a zero floor"); + assert!(!data_usage_info_has_persisted_baseline_identity(&primary)); + let (cycle, leader_epoch) = decode_scanner_cycle_state( + &read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()) .await - .expect("evidence should remain"), - before - ); - assert!(matches!( - read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()).await, - Err(EcstoreError::ConfigNotFound) - )); + .expect("reset should fence a fresh cycle state"), + ) + .expect("fenced cycle state should decode"); + assert_eq!((cycle.next, leader_epoch), (0, 1)); + let (floor, startup) = persisted_usage_floor_for_startup(store.clone(), true) + .await + .expect("startup should accept the bootstrap marker"); + assert_eq!(startup, PersistedUsageFloorStartup::BootstrapPending); + assert_eq!(floor.leader_epoch, 1); + + let (_temp_dir, store) = setup_scanner_cycle_store_with_usage_baseline(false).await; + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), b"{corrupt".to_vec()) + .await + .expect("corrupt primary should persist"); let mut backup = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH), 0); backup.scanner_epoch = Some(7); backup.scanner_cycle = Some(40); @@ -5865,10 +6188,12 @@ async fn scanner_usage_state_reset_rejects_replaced_intent_and_newer_cleanup_slo assert_eq!(read_config(store, &path).await.expect("newer snapshot should remain"), bytes); } +/// Valid JSON without a baseline identity proves no floor either. The full +/// rebuild treats it like corruption: it starts from the empty baseline with +/// a bootstrap-pending marker instead of promoting the untrusted values. #[tokio::test] #[serial] -async fn scanner_usage_state_reset_rejects_decodable_untrusted_floor() { - let (_temp_dir, store) = setup_scanner_cycle_store_with_usage_baseline(false).await; +async fn scanner_usage_state_reset_rebuilds_decodable_untrusted_floor_from_empty_baseline() { let invalid_identity = DataUsageInfo { usage_snapshot_complete: true, buckets_count: 1, @@ -5876,6 +6201,7 @@ async fn scanner_usage_state_reset_rejects_decodable_untrusted_floor() { ..Default::default() }; for usage in [DataUsageInfo::default(), invalid_identity] { + let (_temp_dir, store) = setup_scanner_cycle_store_with_usage_baseline(false).await; save_config( store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), @@ -5883,23 +6209,22 @@ async fn scanner_usage_state_reset_rejects_decodable_untrusted_floor() { ) .await .expect("untrusted primary should persist"); - let before = read_config_with_revision(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + let result = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone()) .await - .expect("primary should load"); - let err = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone()) + .expect("an untrusted floor without a cycle state starts from the empty baseline"); + assert_eq!((result.leader_epoch, result.next_cycle), (1, 0)); + assert_eq!(result.usage_state, "bootstrap-pending"); + let primary = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) .await - .expect_err("valid JSON alone cannot prove a usage floor"); - assert!(err.to_string().contains("no trusted cycle or usage floor"), "{err}"); - assert_eq!( - read_config_with_revision(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) - .await - .expect("evidence should remain"), - before - ); - assert!(matches!( - read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()).await, - Err(EcstoreError::ConfigNotFound) - )); + .expect("reset should replace the untrusted primary with a bootstrap marker"); + let primary: DataUsageInfo = serde_json::from_slice(&primary).expect("bootstrap marker should decode"); + assert!(data_usage_info_is_bootstrap_pending(&primary), "valid JSON alone must not become a floor"); + assert_eq!(primary.buckets_count, 0, "untrusted values must not be promoted"); + let (floor, startup) = persisted_usage_floor_for_startup(store.clone(), true) + .await + .expect("startup should accept the bootstrap marker"); + assert_eq!(startup, PersistedUsageFloorStartup::BootstrapPending); + assert_eq!(floor.leader_epoch, 1); } } @@ -6380,6 +6705,7 @@ async fn test_observational_usage_defers_when_authoritative_baseline_is_missing( let (sender, receiver) = mpsc::channel(1); let mut observation = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(20)), 1); observation.usage_snapshot_converged = Some(false); + observation.usage_snapshot_observational = true; sender.send(observation).await.expect("observation should enqueue"); drop(sender); @@ -6428,6 +6754,7 @@ async fn test_observational_usage_uses_fenced_backup_when_v2_primary_has_no_iden let (sender, receiver) = mpsc::channel(1); let mut observation = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(20)), 1); observation.usage_snapshot_converged = Some(false); + observation.usage_snapshot_observational = true; sender.send(observation).await.expect("observation should enqueue"); drop(sender); @@ -6469,6 +6796,7 @@ async fn test_observational_usage_uses_bootstrap_pending_primary_as_baseline() { let (sender, receiver) = mpsc::channel(1); let mut observation = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(20)), 1); observation.usage_snapshot_converged = Some(false); + observation.usage_snapshot_observational = true; sender.send(observation).await.expect("observation should enqueue"); drop(sender); @@ -7351,6 +7679,7 @@ async fn test_store_data_usage_in_backend_preserves_superseded_status() { scanner_epoch: Some(7), scanner_cycle: Some(11), usage_snapshot_converged: Some(false), + usage_snapshot_observational: true, ..complete_usage_with_bucket_count(None, 1) }) .await @@ -7378,6 +7707,56 @@ async fn test_store_data_usage_in_backend_preserves_superseded_status() { ); } +#[tokio::test] +async fn test_store_data_usage_in_backend_publishes_pending_bucket_certification_as_authoritative() { + let store = Arc::new(MemoryConfigStore::default()); + let authoritative_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str()); + let observed_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()); + let authoritative = DataUsageInfo { + scanner_epoch: Some(7), + scanner_cycle: Some(10), + usage_snapshot_converged: Some(true), + ..complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH), 1) + }; + store.objects.lock().await.insert( + authoritative_key.clone(), + serde_json::to_vec(&authoritative).expect("authoritative snapshot should encode"), + ); + store.revisions.lock().await.insert(authoritative_key.clone(), 1); + + let (sender, receiver) = mpsc::channel(1); + sender + .send(DataUsageInfo { + last_update: Some(std::time::SystemTime::UNIX_EPOCH + Duration::from_secs(1)), + scanner_epoch: Some(7), + scanner_cycle: Some(11), + usage_snapshot_converged: Some(false), + usage_snapshot_pending_buckets: vec!["bucket-0".to_string()], + usage_snapshot_pending_bucket_count: 1, + ..complete_usage_with_bucket_count(None, 1) + }) + .await + .expect("certified usage snapshot should enqueue"); + drop(sender); + + let outcome = store_data_usage_in_backend_with_outcome(CancellationToken::new(), store.clone(), receiver).await; + assert_eq!(outcome, DataUsagePersistOutcome::Saved); + let objects = store.objects.lock().await; + assert!( + !objects.contains_key(&observed_key), + "pending buckets do not make a snapshot observational" + ); + let saved = serde_json::from_slice::(objects.get(&authoritative_key).expect("authoritative snapshot")) + .expect("persisted usage snapshot should decode"); + assert_eq!(saved.scanner_cycle, Some(11)); + assert_eq!(saved.usage_snapshot_converged, Some(false)); + assert_eq!(saved.usage_snapshot_pending_buckets, vec!["bucket-0".to_string()]); + assert_eq!(saved.usage_snapshot_pending_bucket_count, 1); + assert!(!saved.usage_snapshot_observational); + assert_eq!(saved.usage_snapshot_authoritative_baseline, None); + assert!(data_usage_info_has_persisted_baseline_identity(&saved)); +} + #[tokio::test] #[serial] async fn test_store_data_usage_in_backend_removes_observed_after_authoritative_save() { @@ -7401,6 +7780,7 @@ async fn test_store_data_usage_in_backend_removes_observed_after_authoritative_s scanner_epoch: Some(7), scanner_cycle: Some(11), usage_snapshot_converged: Some(false), + usage_snapshot_observational: true, ..complete_usage_with_bucket_count(None, 1) }) .await @@ -7816,7 +8196,7 @@ fn finalizing_post_scan_observation_advances_partially_without_dirty_ack() { ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), Some(dirty_snapshot), ) - .with_observational_snapshot_published(true); + .with_usage_update_emitted(true); let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(observed, DataUsagePersistOutcome::Saved.into()); @@ -8437,7 +8817,7 @@ fn superseded_retry_preserves_explicit_cycle_cadence() { ..Default::default() }; assert_eq!( - scanner_superseded_retry_interval(backoff, &default_config), + scanner_superseded_retry_interval(backoff, &default_config, ScannerMaintenanceFeatures::default()), Some(SCANNER_RETRY_BASE_INTERVAL), "the default adaptive cadence must retain fast convergence" ); @@ -8453,14 +8833,124 @@ fn superseded_retry_preserves_explicit_cycle_cadence() { ..Default::default() }; + for features in [ + ScannerMaintenanceFeatures::default(), + ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }, + ] { + assert_eq!( + scanner_superseded_retry_interval(backoff, &runtime_config, features), + Some(runtime_config.cycle_interval), + "{source:?} cycle cadence must not be shortened by convergence retries" + ); + } + } +} + +#[test] +fn superseded_retry_does_not_delay_required_maintenance() { + let features = ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }; + assert!(features.needs_regular_cycle()); + let config = ScannerRuntimeConfig { + cycle_interval: Duration::from_secs(2), + cycle_interval_source: ScannerRuntimeConfigSource::Env, + ..Default::default() + }; + let mut backoff = ScannerRetryBackoff::default(); + for consecutive_cycles in 1..=12 { + backoff.record_retryable_cycle(true); assert_eq!( - scanner_superseded_retry_interval(backoff, &runtime_config), - Some(runtime_config.cycle_interval), - "{source:?} cycle cadence must not be shortened by convergence retries" + scanner_superseded_retry_interval(backoff, &config, features), + Some(config.cycle_interval), + "superseded cycle {consecutive_cycles} must preserve regular maintenance" ); } } +#[test] +fn superseded_retry_only_caps_regular_maintenance() { + let config = ScannerRuntimeConfig { + cycle_interval: Duration::from_secs(60), + ..Default::default() + }; + let mut backoff = ScannerRetryBackoff::default(); + for _ in 0..6 { + backoff.record_retryable_cycle(true); + } + assert_eq!( + scanner_superseded_retry_interval(backoff, &config, ScannerMaintenanceFeatures::default()), + Some(Duration::from_secs(160)), + ); + for features in [ + ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }, + ScannerMaintenanceFeatures { + replication: true, + ..Default::default() + }, + ScannerMaintenanceFeatures { + inspection_failed: true, + ..Default::default() + }, + ] { + assert_eq!(scanner_superseded_retry_interval(backoff, &config, features), Some(config.cycle_interval)); + assert_eq!(scanner_superseded_retry_interval(ScannerRetryBackoff::default(), &config, features), None); + } +} + +#[tokio::test(start_paused = true)] +#[serial] +async fn superseded_maintenance_wait_keeps_convergence_fencing() { + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + let config = ScannerRuntimeConfig { + cycle_interval: Duration::from_secs(60), + ..Default::default() + }; + let features = ScannerMaintenanceFeatures { + lifecycle: true, + ..Default::default() + }; + let mut backoff = ScannerRetryBackoff::default(); + for _ in 0..12 { + backoff.record_retryable_cycle(true); + } + let interval = scanner_superseded_retry_interval(backoff, &config, features); + assert_eq!(interval, Some(config.cycle_interval)); + let generations = ScannerCycleObservedGenerations::for_wait( + &config, + interval, + crate::scanner_io::dirty_usage_generation(), + crate::runtime_config::scanner_runtime_config_generation(), + crate::scanner_io::scanner_maintenance_generation(), + ); + assert!(generations.dirty_usage.is_none()); + assert!(generations.defer_cluster_activity); + let ctx = CancellationToken::new(); + let mut seen = Some(BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))])); + let changed = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 8, 3))]); + let started = tokio::time::Instant::now(); + let reason = wait_for_next_scanner_cycle_with_activity( + &ctx, + interval.expect("maintenance must retain its convergence interval"), + Some(Duration::from_secs(1)), + &mut seen, + generations, + || false, + || std::future::ready(Ok(changed.clone())), + ) + .await; + assert_eq!(reason, ScannerCycleWakeReason::Timer); + assert_eq!(started.elapsed(), config.cycle_interval); + assert_eq!(seen, Some(changed)); +} + #[test] fn publication_proof_retry_backoff_reaches_its_short_cap() { for (failures, expected) in [(1, 5), (2, 10), (3, 20), (4, 30), (20, 30)] { @@ -8995,18 +9485,10 @@ fn usage_bootstrap_rebuild_bypasses_clean_idle_until_authoritative_cycle() { cycle_interval: Duration::from_secs(60), ..Default::default() }; - let features = ScannerMaintenanceFeatures::default(); - let generation = Some(scanner_maintenance_generation()); let mut clean_idle_backoff = ScannerCleanIdleBackoff { interval_multiplier: 8 }; let mut rebuild = ScannerUsageBootstrapRebuild::from_startup(PersistedUsageFloorStartup::BootstrapPending); assert!(!rebuild.clean_idle_backoff_enabled(true)); - assert!(rebuild.requires_full_scan( - features, - generation, - scanner_maintenance_generation(), - ScannerCycleWakeReason::DirtyUsage, - )); let plan = rebuild.wait_plan(scanner_cycle_wait_plan(&config, clean_idle_backoff, true, std::convert::identity), None); assert_eq!(plan.delay, Duration::ZERO); @@ -9040,6 +9522,40 @@ fn usage_bootstrap_rebuild_bypasses_clean_idle_until_authoritative_cycle() { ); } +#[test] +fn superseded_retry_preserves_full_maintenance_boundaries() { + let features = ScannerMaintenanceFeatures::default(); + for (wake, ordinary_full, retry_full) in [ + (ScannerCycleWakeReason::Timer, true, false), + (ScannerCycleWakeReason::DirtyUsage, false, false), + (ScannerCycleWakeReason::ClusterActivity, false, false), + (ScannerCycleWakeReason::ClusterMaintenance, true, true), + (ScannerCycleWakeReason::MovementGeneration, true, true), + (ScannerCycleWakeReason::Recovery, true, true), + ] { + assert_eq!(features.requires_full_scan(Some(7), 7, wake, false), ordinary_full, "{wake:?}"); + assert_eq!(features.requires_full_scan(Some(7), 7, wake, true), retry_full, "{wake:?} retry"); + assert!(features.requires_full_scan(None, 7, wake, true)); + assert!(features.requires_full_scan(Some(6), 7, wake, true)); + for unsafe_features in [ + ScannerMaintenanceFeatures { + lifecycle: true, + ..features + }, + ScannerMaintenanceFeatures { + replication: true, + ..features + }, + ScannerMaintenanceFeatures { + inspection_failed: true, + ..features + }, + ] { + assert!(unsafe_features.requires_full_scan(Some(7), 7, wake, true)); + } + } +} + #[tokio::test] #[serial] async fn scoped_scan_explicit_bitrot_keeps_dirty_planning_without_idle_backoff() { @@ -9068,12 +9584,12 @@ async fn scoped_scan_explicit_bitrot_keeps_dirty_planning_without_idle_backoff() (ScannerCycleWakeReason::ClusterMaintenance, true), ] { assert_eq!( - features.requires_full_scan(generation, scanner_maintenance_generation(), wake), + features.requires_full_scan(generation, scanner_maintenance_generation(), wake, false), full, "{wake:?}" ); } - assert!(features.requires_full_scan(None, scanner_maintenance_generation(), ScannerCycleWakeReason::DirtyUsage)); + assert!(features.requires_full_scan(None, scanner_maintenance_generation(), ScannerCycleWakeReason::DirtyUsage, false)); for unsafe_features in [ ScannerMaintenanceFeatures { lifecycle: true, @@ -9091,18 +9607,25 @@ async fn scoped_scan_explicit_bitrot_keeps_dirty_planning_without_idle_backoff() assert!(unsafe_features.requires_full_scan( generation, scanner_maintenance_generation(), - ScannerCycleWakeReason::DirtyUsage + ScannerCycleWakeReason::DirtyUsage, + false )); } crate::scanner_io::record_scanner_maintenance_change("maintenance-proof-change"); - assert!(features.requires_full_scan(generation, scanner_maintenance_generation(), ScannerCycleWakeReason::DirtyUsage)); + assert!(features.requires_full_scan( + generation, + scanner_maintenance_generation(), + ScannerCycleWakeReason::DirtyUsage, + false + )); let (refreshed, refreshed_generation) = detect_stable_scanner_maintenance_features(&ctx, &store) .await .expect("changed maintenance generation should be inspected"); assert!(!refreshed.requires_full_scan( Some(refreshed_generation), scanner_maintenance_generation(), - ScannerCycleWakeReason::DirtyUsage + ScannerCycleWakeReason::DirtyUsage, + false )); crate::scanner_io::clear_dirty_usage_buckets_for_tests(); }) @@ -9630,6 +10153,7 @@ fn scoped_scan_remote_dirty_coverage_invalidates_local_bucket_current() { require_source: true, tier_registry_generation: None, checkpoint_identity: None, + checkpoint_plan_digest: None, }, ), crate::scanner_io::DataUsageCacheScanState::Prepared { @@ -9640,24 +10164,47 @@ fn scoped_scan_remote_dirty_coverage_invalidates_local_bucket_current() { } #[test] -fn post_lease_activity_proof_rejects_a_put_tail_that_finished_before_lease_acquisition() { - let before = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]); - let expected_digest = Some(scanner_activity_snapshot_digest(&before)); - assert_eq!(scanner_post_lease_activity_defer_reason(expected_digest, Ok(before.clone())), None); +fn post_lease_activity_proof_rejects_a_peer_whose_dirty_usage_moved_before_lease_acquisition() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3)), + ]); + assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(before.clone())), None); + // Local generation movement in the lease window is the next cycle's + // work: its buckets stay in the dirty map and the walk remains certified. + let mut local_shaded = before.clone(); + let local = local_shaded + .get_mut(LOCAL_SCANNER_ACTIVITY_NODE) + .expect("local node should be present"); + local.namespace_generation += 1; + local.dirty_usage_generation += 1; + local.dirty_usage_pending = true; + assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(local_shaded)), None); + + // A peer PUT tail that drained before the lease leaves the movement + // generation accepted by the lease unchanged, but the peer's dirty state + // moved and this phase cannot name its buckets. let mut after = before.clone(); - after - .get_mut("node-2") - .expect("writer should be present") - .namespace_generation += 1; + let writer = after.get_mut("node-2").expect("writer should be present"); + writer.namespace_generation += 1; + writer.dirty_usage_generation += 1; assert_eq!( before["node-2"].movement_generation, after["node-2"].movement_generation, "the existing movement-only lease remains valid after a PUT tail drains" ); assert!(scanner_activity_allows_usage_publication(&after)); - let reason = scanner_post_lease_activity_defer_reason(expected_digest, Ok(after)); + let reason = scanner_post_lease_activity_defer_reason(Some(&before), Ok(after)); assert_eq!(reason, Some(ScannerCycleDeferReason::ActivityBaselineUnavailable)); + // A guardrail field moving in the window still defers. + let mut moved = before.clone(); + moved.get_mut("node-2").expect("peer should be present").movement_generation += 1; + assert_eq!( + scanner_post_lease_activity_defer_reason(Some(&before), Ok(moved)), + Some(ScannerCycleDeferReason::ActivityBaselineUnavailable) + ); + let result = ScannerCycleResult::new(ScannerCycleStatus::Complete, None).with_remote_dirty_usage_acknowledgements(vec![ ScannerDirtyUsageAcknowledgement { host: "node-2".to_string(), @@ -9740,15 +10287,13 @@ fn remote_lease_validation_failure_preserves_movement_defer_for_remote_fence_los #[test] fn post_lease_activity_proof_requires_a_complete_matching_baseline() { let before = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]); - let digest = scanner_activity_snapshot_digest(&before); let mut blocked = before.clone(); blocked.get_mut("node-2").expect("peer should be present").publication_blocked = true; - let blocked_digest = scanner_activity_snapshot_digest(&blocked); for (expected, observed) in [ - (None, Ok(before)), - (Some(digest), Err("peer is unavailable".to_string())), - (Some(digest), Ok(BTreeMap::new())), - (Some(blocked_digest), Ok(blocked)), + (None, Ok(before.clone())), + (Some(&before), Err("peer is unavailable".to_string())), + (Some(&before), Ok(BTreeMap::new())), + (Some(&blocked), Ok(blocked.clone())), ] { assert_eq!( scanner_post_lease_activity_defer_reason(expected, observed), @@ -9833,7 +10378,7 @@ fn scanner_activity_structural_digest_ignores_regular_bucket_writes() { } #[test] -fn scanner_activity_structural_digest_fences_restart_and_maintenance() { +fn scanner_activity_structural_digest_fences_restart_and_maintenance_but_not_publication_pauses() { let baseline = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]); let mut restarted = baseline.clone(); restarted.get_mut("node-2").expect("node should exist").instance_id = "epoch-b".to_string(); @@ -9842,7 +10387,12 @@ fn scanner_activity_structural_digest_fences_restart_and_maintenance() { .get_mut("node-2") .expect("node should exist") .maintenance_generation = 4; + let mut paused = baseline.clone(); + paused.get_mut("node-2").expect("node should exist").publication_blocked = true; + // A restarted peer may have lost its last dirty-hint window and a changed + // maintenance rule needs a maintenance pass, so both invalidate every + // reusable result (the baseline and the set caches). assert_ne!( scanner_activity_structural_digest(&baseline), scanner_activity_structural_digest(&restarted) @@ -9851,8 +10401,145 @@ fn scanner_activity_structural_digest_fences_restart_and_maintenance() { scanner_activity_structural_digest(&baseline), scanner_activity_structural_digest(&maintained) ); + // A publication pause describes nothing about the heap: it vetoes the + // cycle it moves in and is part of no identity. + assert_eq!(scanner_activity_structural_digest(&baseline), scanner_activity_structural_digest(&paused)); + assert_eq!(scanner_activity_snapshot_digest(&baseline), scanner_activity_snapshot_digest(&paused)); + assert_eq!(scanner_activity_coverage_digest(&baseline), scanner_activity_coverage_digest(&paused)); + assert_eq!( + scanner_activity_guardrail_veto(&baseline, &paused).map(|veto| veto.field), + Some("publication_blocked") + ); } +#[test] +fn scanner_activity_coverage_digest_survives_restart_and_maintenance_but_fences_the_heap() { + let baseline = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("process-1", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote-a", 7, 3)), + ]); + type Mutation = (&'static str, Box); + + // A coordinator restart, a maintenance publication, a publication pause + // or the local node's own generations do not change the namespace a + // sweep traverses, so a checkpoint resumes under the same plan; each of + // the first three is still a guardrail veto on its own. + let same_coverage: Vec = vec![ + ("instance_id", Box::new(|activity| activity.instance_id = "restarted".to_string())), + ("maintenance_generation", Box::new(|activity| activity.maintenance_generation += 1)), + ("publication_blocked", Box::new(|activity| activity.publication_blocked = true)), + ]; + for host in [LOCAL_SCANNER_ACTIVITY_NODE, "node-2"] { + for (field, mutate) in &same_coverage { + let mut changed = baseline.clone(); + mutate(changed.get_mut(host).expect("host should be present")); + assert_eq!( + scanner_activity_coverage_digest(&baseline), + scanner_activity_coverage_digest(&changed), + "{field} on {host} must not change the coverage identity" + ); + assert_eq!( + scanner_activity_guardrail_veto(&baseline, &changed).map(|veto| veto.field), + Some(*field), + "{field} on {host} still vetoes the cycle it moved in" + ); + } + } + let local_generations: Vec = vec![ + ("namespace_generation", Box::new(|activity| activity.namespace_generation += 1)), + ("dirty_usage_generation", Box::new(|activity| activity.dirty_usage_generation += 1)), + ("dirty_usage_pending", Box::new(|activity| activity.dirty_usage_pending = true)), + ]; + for (field, mutate) in &local_generations { + let mut changed = baseline.clone(); + mutate(changed.get_mut(LOCAL_SCANNER_ACTIVITY_NODE).expect("local activity")); + assert_eq!( + scanner_activity_coverage_digest(&baseline), + scanner_activity_coverage_digest(&changed), + "local {field} is certified per bucket through the bucket's own dirty generation" + ); + // A peer's generations stay in: its dirty buckets are acknowledged + // per cycle, so a write there between two halves of a sweep must + // restart the sweep. + let mut peer_changed = baseline.clone(); + mutate(peer_changed.get_mut("node-2").expect("peer activity")); + assert_ne!( + scanner_activity_coverage_digest(&baseline), + scanner_activity_coverage_digest(&peer_changed), + "remote {field} must change the coverage identity" + ); + } + let heap: Vec = vec![ + ("protocol_version", Box::new(|activity| activity.protocol_version += 1)), + ("topology_digest", Box::new(|activity| activity.topology_digest = [4; 32])), + ("data_movement_active", Box::new(|activity| activity.data_movement_active = true)), + ("movement_generation", Box::new(|activity| activity.movement_generation += 1)), + ]; + for host in [LOCAL_SCANNER_ACTIVITY_NODE, "node-2"] { + for (field, mutate) in &heap { + let mut changed = baseline.clone(); + mutate(changed.get_mut(host).expect("host should be present")); + assert_ne!( + scanner_activity_coverage_digest(&baseline), + scanner_activity_coverage_digest(&changed), + "{field} on {host} must change the coverage identity" + ); + assert_ne!( + scanner_activity_structural_digest(&baseline), + scanner_activity_structural_digest(&changed), + "{field} on {host} must change the reuse identity" + ); + } + } + let mut peer_lost = baseline.clone(); + peer_lost.remove("node-2"); + assert_ne!(scanner_activity_coverage_digest(&baseline), scanner_activity_coverage_digest(&peer_lost)); +} + +#[test] +fn bucket_checkpoint_plan_survives_a_coordinator_restart() { + let bucket_info = |name: &str| crate::storage_api::scanner_io::BucketInfo { + name: name.to_string(), + created: None, + deleted: None, + versioning: false, + object_locking: false, + }; + let buckets = vec![bucket_info("alpha"), bucket_info("beta")]; + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("process-1", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote-a", 7, 3)), + ]); + let mut restarted = before.clone(); + let local = restarted + .get_mut(LOCAL_SCANNER_ACTIVITY_NODE) + .expect("local activity should exist"); + local.instance_id = "process-2".to_string(); + local.namespace_generation = 0; + local.maintenance_generation = 0; + local.dirty_usage_generation = 0; + local.dirty_usage_pending = false; + local.publication_blocked = true; + + // The plan a bucket checkpoint was started under must be the plan the + // restarted coordinator requests, or every resumed sweep restarts as + // mixed coverage and the bucket can never certify (rustfs/rustfs#8494). + // The reuse plan does change: finished results are walked again. + let coverage_plan = + |snapshot| crate::scanner_io::scanner_bucket_plan_digest(&buckets, scanner_activity_coverage_digest(snapshot)); + let reuse_plan = + |snapshot| crate::scanner_io::scanner_bucket_plan_digest(&buckets, scanner_activity_structural_digest(snapshot)); + assert_eq!(coverage_plan(&before), coverage_plan(&restarted)); + assert_ne!(reuse_plan(&before), reuse_plan(&restarted)); + + let mut extra_bucket = buckets.clone(); + extra_bucket.push(bucket_info("gamma")); + assert_ne!( + coverage_plan(&before), + crate::scanner_io::scanner_bucket_plan_digest(&extra_bucket, scanner_activity_coverage_digest(&before)), + "the bucket list stays part of the plan" + ); +} #[test] fn scanner_dirty_usage_acknowledgements_exclude_local_and_clean_nodes() { let snapshot = BTreeMap::from([ @@ -9981,6 +10668,132 @@ fn remote_restart_is_distinct_from_deferred_cluster_activity() { assert!(scanner_activity_observed_work(ScannerActivityObservation::RemoteRestarted)); } +type ActivityFieldMutations = Vec<(&'static str, Box)>; + +#[test] +fn guardrail_veto_is_attributed_to_the_field_and_host_that_moved() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote", 7, 3)), + ]); + assert_eq!(scanner_activity_guardrail_veto(&before, &before), None); + + let cases: ActivityFieldMutations = vec![ + ("instance_id", Box::new(|activity| activity.instance_id = "restarted".to_string())), + ("protocol_version", Box::new(|activity| activity.protocol_version += 1)), + ("topology_digest", Box::new(|activity| activity.topology_digest = [4; 32])), + ("maintenance_generation", Box::new(|activity| activity.maintenance_generation += 1)), + ("data_movement_active", Box::new(|activity| activity.data_movement_active = true)), + ("movement_generation", Box::new(|activity| activity.movement_generation += 1)), + ("publication_blocked", Box::new(|activity| activity.publication_blocked = true)), + ]; + for host in [LOCAL_SCANNER_ACTIVITY_NODE, "node-2"] { + for (field, mutate) in &cases { + let mut after = before.clone(); + mutate(after.get_mut(host).expect("host should be present")); + let veto = scanner_activity_guardrail_veto(&before, &after).unwrap_or_else(|| panic!("{field} on {host} must veto")); + assert_eq!(veto.field, *field, "{host}"); + assert_eq!(veto.host, host); + assert_eq!(veto.reason(), format!("{field}@{host}")); + assert!(SCANNER_ACTIVITY_GUARDRAIL_VETO_FIELDS.contains(field)); + } + } + + let mut peer_lost = before.clone(); + peer_lost.remove("node-2"); + assert_eq!( + scanner_activity_guardrail_veto(&before, &peer_lost), + Some(ScannerActivityGuardrailVeto::new("node-2", "peer_set")) + ); + let mut peer_added = before.clone(); + peer_added.insert("node-3".to_string(), scanner_node_activity("third", 1, 1)); + assert_eq!( + scanner_activity_guardrail_veto(&before, &peer_added), + Some(ScannerActivityGuardrailVeto::new("node-3", "peer_set")) + ); +} + +#[test] +fn generation_movement_shades_buckets_without_a_guardrail_veto() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote", 7, 3)), + ]); + let shade: ActivityFieldMutations = vec![ + ("namespace_generation", Box::new(|activity| activity.namespace_generation += 1)), + ("dirty_usage_generation", Box::new(|activity| activity.dirty_usage_generation += 1)), + ("dirty_usage_pending", Box::new(|activity| activity.dirty_usage_pending = true)), + ]; + for host in [LOCAL_SCANNER_ACTIVITY_NODE, "node-2"] { + for (field, mutate) in &shade { + let mut after = before.clone(); + mutate(after.get_mut(host).expect("host should be present")); + assert_ne!(after, before); + assert_eq!(scanner_activity_guardrail_veto(&before, &after), None, "{field} on {host} must not veto"); + assert!(SCANNER_ACTIVITY_SHADE_FIELDS.contains(field)); + assert_eq!( + scanner_activity_structural_digest(&before), + scanner_activity_structural_digest(&after), + "{field} must stay outside the structural digest" + ); + } + } + + // Everything shaded at once on every host is still not a veto. + let mut all_shaded = before.clone(); + for activity in all_shaded.values_mut() { + activity.namespace_generation += 3; + activity.dirty_usage_generation += 3; + activity.dirty_usage_pending = true; + } + assert_eq!(scanner_activity_guardrail_veto(&before, &all_shaded), None); +} + +#[test] +fn peer_dirty_usage_movement_is_reported_for_remote_hosts_only() { + let before = BTreeMap::from([ + (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)), + ("node-2".to_string(), scanner_node_activity("remote", 7, 3)), + ]); + assert_eq!(scanner_peer_dirty_usage_moved(&before, &before), None); + + let mut local_moved = before.clone(); + let local = local_moved + .get_mut(LOCAL_SCANNER_ACTIVITY_NODE) + .expect("local activity should exist"); + local.dirty_usage_generation += 1; + local.dirty_usage_pending = true; + local.namespace_generation += 1; + assert_eq!(scanner_peer_dirty_usage_moved(&before, &local_moved), None); + + let mut remote_namespace_only = before.clone(); + remote_namespace_only + .get_mut("node-2") + .expect("remote activity should exist") + .namespace_generation += 1; + assert_eq!(scanner_peer_dirty_usage_moved(&before, &remote_namespace_only), None); + + let mut remote_generation = before.clone(); + remote_generation + .get_mut("node-2") + .expect("remote activity should exist") + .dirty_usage_generation += 1; + assert_eq!( + scanner_peer_dirty_usage_moved(&before, &remote_generation), + Some(ScannerActivityGuardrailVeto::new("node-2", "dirty_usage_generation")) + ); + + let mut remote_pending = before.clone(); + remote_pending + .get_mut("node-2") + .expect("remote activity should exist") + .dirty_usage_pending = true; + assert_eq!( + scanner_peer_dirty_usage_moved(&before, &remote_pending), + Some(ScannerActivityGuardrailVeto::new("node-2", "dirty_usage_pending")) + ); +} + #[test] fn local_maintenance_wakeup_releases_a_remote_maintenance_block() { let blocked = scanner_activity_backoff_blocked_after_wake(false, ScannerCycleWakeReason::ClusterMaintenance); @@ -10289,7 +11102,11 @@ async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() { #[serial] fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() { with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || { - let mode = get_cycle_scan_mode(10, 0, Some(Utc::now()), ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1024)); + let start = ScannerBitrotStart::Window { + cycle: 0, + started: Utc::now(), + }; + let mode = get_cycle_scan_mode(10, start, ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1024)); assert_eq!(mode, HealScanMode::Deep); }); } @@ -10301,12 +11118,17 @@ fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() { let recent = Utc::now() - chrono::Duration::minutes(30); let old = Utc::now() - chrono::Duration::hours(2); + let recent = ScannerBitrotStart::Window { + cycle: 0, + started: recent, + }; + let old = ScannerBitrotStart::Window { cycle: 0, started: old }; assert_eq!( - get_cycle_scan_mode(2048, 0, Some(recent), ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1024)), + get_cycle_scan_mode(2048, recent, ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1024)), HealScanMode::Normal ); assert_eq!( - get_cycle_scan_mode(2048, 0, Some(old), ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1024)), + get_cycle_scan_mode(2048, old, ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1024)), HealScanMode::Deep ); }); @@ -10317,22 +11139,243 @@ fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() { fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() { with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("off"), || { assert_eq!( - get_cycle_scan_mode(1, 0, None, ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1)), + get_cycle_scan_mode(1, ScannerBitrotStart::None, ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1)), + HealScanMode::Normal + ); + assert_eq!( + get_cycle_scan_mode( + 1, + ScannerBitrotStart::Anchor(Utc::now() - chrono::Duration::days(400)), + ScannerBitrotPolicy::new(bitrot_scan_cycle(), true, 1) + ), HealScanMode::Normal ); }); } +#[test] +fn scan_mode_without_recorded_bitrot_start_stays_normal_until_the_cycle_elapses() { + // Regression for rustfs/backlog#2905 (rustfs/rustfs#8015): a cycle that + // finds no recorded bitrot start (missing heal object, or a save that + // keeps failing) must not open a Deep window. Three consecutive cycles + // with the default `BackgroundHealInfo` used to run Deep through the + // selection window because `bitrot_start_cycle` defaults to 0. + let policy = ScannerBitrotPolicy::new(Some(Duration::from_secs(3600)), true, 1024); + let modes: Vec<_> = (1..=3) + .map(|cycle| get_cycle_scan_mode(cycle, ScannerBitrotStart::None, policy)) + .collect(); + assert_eq!(modes, vec![HealScanMode::Normal; 3]); + + // An in-memory anchor counts the cadence without opening a selection + // window: Normal until the cycle elapses, Deep afterwards. + let anchored = ScannerBitrotStart::Anchor(Utc::now()); + let modes: Vec<_> = (1..=3).map(|cycle| get_cycle_scan_mode(cycle, anchored, policy)).collect(); + assert_eq!(modes, vec![HealScanMode::Normal; 3]); + assert_eq!( + get_cycle_scan_mode(4, ScannerBitrotStart::Anchor(Utc::now() - chrono::Duration::hours(2)), policy), + HealScanMode::Deep + ); + + // A recorded Deep window keeps its selection window, and an elapsed + // cadence still triggers the next window. + let started = Utc::now(); + let window = ScannerBitrotStart::Window { cycle: 1, started }; + assert_eq!(get_cycle_scan_mode(1, window, policy), HealScanMode::Deep); + assert_eq!( + get_cycle_scan_mode( + 2048, + ScannerBitrotStart::Window { + cycle: 1, + started: started - chrono::Duration::hours(2), + }, + policy + ), + HealScanMode::Deep + ); + assert_eq!(get_cycle_scan_mode(2048, window, policy), HealScanMode::Normal); +} + +#[test] +#[serial] +fn bitrot_cadence_anchor_is_remembered_in_memory_without_a_persisted_write() { + reset_bitrot_cadence_anchor_for_test(); + let now = Utc::now(); + let policy = ScannerBitrotPolicy::new(Some(Duration::from_secs(3600)), true, 1024); + + // A Normal cycle without a recorded window writes nothing to storage ... + assert!( + background_heal_info_for_scan_start(BackgroundHealInfo::default(), 3, HealScanMode::Normal, now, policy).is_none(), + "a Normal cycle must not persist a start that would open a Deep window" + ); + + // ... but the process anchors the cadence at its first such cycle and + // keeps that anchor for later cycles. + let first = scanner_bitrot_start_for_cycle(&BackgroundHealInfo::default(), now); + assert_eq!(first, ScannerBitrotStart::Anchor(now)); + let later = scanner_bitrot_start_for_cycle(&BackgroundHealInfo::default(), now + chrono::Duration::hours(2)); + assert_eq!(later, ScannerBitrotStart::Anchor(now)); + assert_eq!(get_cycle_scan_mode(4, later, policy), HealScanMode::Normal); + + // A recorded window start takes precedence over the anchor. + let recorded = BackgroundHealInfo { + bitrot_start_time: Some(now), + bitrot_start_cycle: 7, + current_scan_mode: HealScanMode::Normal, + }; + assert_eq!( + scanner_bitrot_start_for_cycle(&recorded, now), + ScannerBitrotStart::Window { cycle: 7, started: now } + ); + + // Once the cadence elapses on the anchor, the Deep cycle records the + // window start for persistence (and for the in-memory record). + let opened = background_heal_info_for_scan_start(BackgroundHealInfo::default(), 9, HealScanMode::Deep, now, policy) + .expect("opening a Deep window records its start"); + assert_eq!(opened.bitrot_start_cycle, 9); + assert_eq!(opened.bitrot_start_time, Some(now)); + assert_eq!(opened.current_scan_mode, HealScanMode::Deep); + reset_bitrot_cadence_anchor_for_test(); +} + +#[test] +#[serial] +fn bitrot_start_memory_record_outlives_a_skipped_persisted_save() { + reset_bitrot_cadence_anchor_for_test(); + let now = Utc::now(); + let policy = ScannerBitrotPolicy::new(Some(Duration::from_secs(3600)), true, 1024); + let recorded = BackgroundHealInfo { + bitrot_start_time: Some(now), + bitrot_start_cycle: 4, + current_scan_mode: HealScanMode::Normal, + }; + + // Storage never returned a start (missing object, or a save skipped + // without movement admission): the in-memory record keeps the cadence. + let (info, source) = resolve_background_heal_info(None, Some(&recorded)); + assert_eq!(source, ScannerBitrotStartSource::Memory); + assert_eq!(info.bitrot_start_cycle, 4); + let start = scanner_bitrot_start_for_cycle(&info, now); + assert_eq!(start, ScannerBitrotStart::Window { cycle: 4, started: now }); + assert_eq!( + get_cycle_scan_mode(5, start, policy), + HealScanMode::Deep, + "a remembered start keeps the Deep selection window it opened" + ); + assert_eq!( + get_cycle_scan_mode(2048, start, policy), + HealScanMode::Normal, + "and stays Normal after the window until the cadence elapses" + ); + + // A persisted start wins ties and older in-memory records. + let stored = BackgroundHealInfo { + bitrot_start_time: Some(now), + bitrot_start_cycle: 4, + current_scan_mode: HealScanMode::Deep, + }; + let (info, source) = resolve_background_heal_info(Some(&stored), Some(&recorded)); + assert_eq!(source, ScannerBitrotStartSource::Persisted); + assert_eq!(info.current_scan_mode, HealScanMode::Deep); + + // A newer in-memory start (a Deep window opened after the last + // successful save) wins over the stale persisted record. + let newer = BackgroundHealInfo { + bitrot_start_time: Some(now + chrono::Duration::seconds(1)), + bitrot_start_cycle: 9, + current_scan_mode: HealScanMode::Deep, + }; + let (info, source) = resolve_background_heal_info(Some(&stored), Some(&newer)); + assert_eq!(source, ScannerBitrotStartSource::Memory); + assert_eq!(info.bitrot_start_cycle, 9); + + // Nothing recorded anywhere: the persisted record is used as-is, the + // source says so, and the cycle runs Normal instead of opening a window. + let legacy = BackgroundHealInfo { + bitrot_start_time: None, + bitrot_start_cycle: 0, + current_scan_mode: HealScanMode::Deep, + }; + let (info, source) = resolve_background_heal_info(Some(&legacy), None); + assert_eq!(source, ScannerBitrotStartSource::None); + assert_eq!(info.current_scan_mode, HealScanMode::Deep); + let start = scanner_bitrot_start_for_cycle(&info, now); + assert_eq!(start, ScannerBitrotStart::Anchor(now)); + assert_eq!(get_cycle_scan_mode(1, start, policy), HealScanMode::Normal); + reset_bitrot_cadence_anchor_for_test(); +} + +#[test] +#[serial] +fn scan_mode_status_reports_bitrot_start_source_and_topology_support() { + let now = Utc::now(); + let info = BackgroundHealInfo { + bitrot_start_time: Some(now), + bitrot_start_cycle: 2, + current_scan_mode: HealScanMode::Normal, + }; + record_scanner_bitrot_status( + scanner_bitrot_start_for_cycle(&info, now), + ScannerBitrotStartSource::Memory, + false, + HealScanMode::Normal, + ); + + let status = scanner_bitrot_status(); + assert!(!status.deep_scan_supported); + assert_eq!(status.bitrot_start_source, ScannerBitrotStartSource::Memory); + assert_eq!(status.bitrot_start_source.as_str(), "memory"); + assert_eq!(status.bitrot_start_kind, "deep_window"); + assert_eq!(status.bitrot_start_cycle, 2); + assert_eq!( + status.bitrot_start_unix_secs, + u64::try_from(now.timestamp()).expect("current time is after the epoch") + ); + assert_eq!(status.current_scan_mode, "normal"); + let encoded = serde_json::to_value(status).expect("bitrot status serializes"); + assert_eq!(encoded["bitrot_start_source"], "memory"); + assert_eq!(encoded["deep_scan_supported"], false); + + record_scanner_bitrot_status(ScannerBitrotStart::None, ScannerBitrotStartSource::None, true, HealScanMode::Deep); + let status = scanner_bitrot_status(); + assert!(status.deep_scan_supported); + assert_eq!(status.bitrot_start_source, ScannerBitrotStartSource::None); + assert_eq!(status.bitrot_start_kind, "none"); + assert_eq!(status.bitrot_start_unix_secs, 0); + assert_eq!(status.current_scan_mode, "deep"); + + record_scanner_bitrot_status( + ScannerBitrotStart::Anchor(now), + ScannerBitrotStartSource::Memory, + true, + HealScanMode::Normal, + ); + let status = scanner_bitrot_status(); + assert_eq!(status.bitrot_start_kind, "anchor"); + assert_eq!(status.bitrot_start_cycle, 0); + assert_eq!( + status.bitrot_start_unix_secs, + u64::try_from(now.timestamp()).expect("current time is after the epoch") + ); +} + #[test] fn test_erasure_sd_does_not_enter_deep_scan_mode() { let started = Utc::now(); let cycle = Some(Duration::from_secs(3600)); let unsupported_policy = ScannerBitrotPolicy::new(cycle, false, 1024); - assert_eq!(get_cycle_scan_mode(10, 10, Some(started), unsupported_policy), HealScanMode::Normal); - assert_eq!(get_cycle_scan_mode(11, 10, None, unsupported_policy), HealScanMode::Normal); + let window = ScannerBitrotStart::Window { cycle: 10, started }; + assert_eq!(get_cycle_scan_mode(10, window, unsupported_policy), HealScanMode::Normal); assert_eq!( - get_cycle_scan_mode(10, 10, None, ScannerBitrotPolicy::new(Some(Duration::ZERO), false, 1024)), + get_cycle_scan_mode(11, ScannerBitrotStart::None, unsupported_policy), + HealScanMode::Normal + ); + assert_eq!( + get_cycle_scan_mode(11, ScannerBitrotStart::Anchor(started), unsupported_policy), + HealScanMode::Normal + ); + assert_eq!( + get_cycle_scan_mode(10, ScannerBitrotStart::None, ScannerBitrotPolicy::new(Some(Duration::ZERO), false, 1024)), HealScanMode::Normal ); diff --git a/crates/scanner/src/scanner/tests/cycle_budget_publication.rs b/crates/scanner/src/scanner/tests/cycle_budget_publication.rs new file mode 100644 index 000000000..a0eb09034 --- /dev/null +++ b/crates/scanner/src/scanner/tests/cycle_budget_publication.rs @@ -0,0 +1,534 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use rustfs_utils::path::path_join_buf; + +#[tokio::test] +#[serial] +async fn exhausted_budget_still_proves_peer_movement_without_namespace_authority() { + let (_temp_dir, store) = setup_scanner_cycle_store_with_pool_count(true, 2).await; + let pending = ecstore_hold_namespace_commit(store.as_ref()); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + budget.cancel_for_runtime(); + let generation = store.scanner_data_movement_generation(); + let result = ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_remote_publication_lease_targets(vec![( + "peer".into(), + "session".into(), + generation, + )]); + let plan = ScannerCyclePublicationPlan::for_result(&result); + let calls = AtomicUsize::new(0); + + let proof = plan + .await_leases(&ctx, &budget, 352, |observational| { + calls.fetch_add(1, Ordering::Relaxed); + store.acquire_scanner_publication_lease_with_purpose(generation, Duration::from_secs(60), observational) + }) + .await; + let ScannerPublicationProofWait::Ready((token, admitted_generation)) = proof else { + panic!("a stopped walk must still prove peer movement while the leader context is alive"); + }; + assert_eq!(calls.load(Ordering::Relaxed), 1); + assert!(!ctx.is_cancelled()); + store + .validate_scanner_publication_lease_with_purpose(token, admitted_generation, true) + .await + .expect("progress must retain a movement-only lease"); + assert!( + store + .validate_scanner_publication_lease_with_purpose(token, admitted_generation, false) + .await + .is_err(), + "the lease must not authorize an authoritative snapshot" + ); + assert!(store.release_scanner_publication_lease(token).await); + + let stale = plan + .await_leases(&ctx, &budget, 352, |observational| { + store.acquire_scanner_publication_lease_with_purpose(generation + 1, Duration::from_secs(60), observational) + }) + .await; + assert!(matches!(stale, ScannerPublicationProofWait::Rejected(_))); + drop(pending); + + store.pool_meta.write().await.pools[0].decommission = Some(crate::storage_api::owner::EcstorePoolDecommissionInfo { + failed: true, + ..Default::default() + }); + assert!(store.scanner_data_movement_pause_status().await.paused); + let current_generation = store.scanner_data_movement_generation(); + let moving = plan + .await_leases(&ctx, &budget, 352, |observational| { + store.acquire_scanner_publication_lease_with_purpose(current_generation, Duration::from_secs(60), observational) + }) + .await; + assert!( + matches!(moving, ScannerPublicationProofWait::Rejected(error) if error.to_string().contains("blocked by data movement")) + ); +} + +#[tokio::test] +async fn emitted_updates_keep_budget_cancellation_and_publication_purpose() { + for (status, expected_plan) in [ + (ScannerCycleStatus::Complete, ScannerCyclePublicationPlan::Authoritative), + (ScannerCycleStatus::Incomplete, ScannerCyclePublicationPlan::Observational), + (ScannerCycleStatus::Superseded, ScannerCyclePublicationPlan::Observational), + ( + ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), + ScannerCyclePublicationPlan::Observational, + ), + ( + ScannerCycleStatus::Deferred(ScannerCycleDeferReason::DataMovement), + ScannerCyclePublicationPlan::Authoritative, + ), + ] { + let result = ScannerCycleResult::new(status, None).with_usage_update_emitted(true); + let plan = ScannerCyclePublicationPlan::for_result(&result); + assert_eq!(plan, expected_plan); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let ready = plan + .await_leases(&ctx, &budget, 352, |observational| async move { Ok::<_, String>(observational) }) + .await; + assert!(matches!(ready, ScannerPublicationProofWait::Ready(observational) if observational == plan.is_observational())); + budget.cancel_for_runtime(); + let calls = AtomicUsize::new(0); + let cancelled = plan + .await_leases(&ctx, &budget, 352, |_| { + calls.fetch_add(1, Ordering::Relaxed); + async { Ok::<(), String>(()) } + }) + .await; + assert!(matches!(cancelled, ScannerPublicationProofWait::Cancelled)); + assert_eq!(calls.load(Ordering::Relaxed), 0); + } +} + +#[tokio::test] +async fn progress_proof_honors_parent_cancellation() { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + ctx.cancel(); + let calls = AtomicUsize::new(0); + let proof = ScannerCyclePublicationPlan::ProgressOnly + .await_leases(&ctx, &budget, 352, |_| { + calls.fetch_add(1, Ordering::Relaxed); + async { Ok::<_, String>(()) } + }) + .await; + assert!(matches!(proof, ScannerPublicationProofWait::Cancelled)); + assert_eq!(calls.load(Ordering::Relaxed), 0); +} + +#[tokio::test(start_paused = true)] +async fn progress_proof_is_bounded_without_a_runtime_budget() { + for stalled in [false, true] { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new( + &ctx, + ScannerCycleBudgetConfig { + max_objects: Some(1), + ..Default::default() + }, + ); + budget.record_object_scanned(); + assert!(budget.budget_elapsed()); + let calls = AtomicUsize::new(0); + let started = Instant::now(); + let proof = ScannerCyclePublicationPlan::ProgressOnly + .await_leases(&ctx, &budget, 352, |_| { + calls.fetch_add(1, Ordering::Relaxed); + async move { + if stalled { + std::future::pending::<()>().await; + } + Err::<(), _>("scanner publication lease capacity is exhausted") + } + }) + .await; + assert!(matches!(proof, ScannerPublicationProofWait::Cancelled)); + assert!(calls.load(Ordering::Relaxed) > 0); + assert_eq!(started.elapsed(), SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT); + assert!(!ctx.is_cancelled()); + } +} + +#[tokio::test] +#[serial] +async fn coordinator_persists_budget_progress_without_usage_or_dirty_acknowledgement() { + assert_coordinator_partial_progress(false, None, false).await; +} + +#[tokio::test] +#[serial] +async fn coordinator_preserves_checkpoint_when_bucket_identity_is_unavailable() { + assert_coordinator_partial_progress(false, None, true).await; +} + +#[tokio::test] +#[serial] +async fn coordinator_persists_mixed_plan_tail_without_usage_or_dirty_acknowledgement() { + assert_coordinator_partial_progress(true, None, false).await; +} + +#[tokio::test] +#[serial] +async fn coordinator_partial_progress_requires_successful_cycle_state_save() { + for after_commit in [false, true] { + assert_coordinator_partial_progress(true, Some(after_commit), false).await; + } +} + +async fn assert_coordinator_partial_progress(resume_mixed_plan: bool, fail_cycle_save: Option, block_identity_read: bool) { + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + let (_temp_dir, store) = setup_scanner_cycle_store().await; + let bucket = "scanner-budget-progress"; + store + .make_bucket(bucket, &crate::storage_api::scan::MakeBucketOptions::default()) + .await + .expect("fixture bucket"); + for object in ["first", "second", "third"] { + store.pools[0].disk_set[0] + .put_object( + bucket, + object, + &mut PutObjReader::from_vec(b"value".to_vec()), + &ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }, + ) + .await + .expect("fixture object"); + } + crate::scanner_io::record_dirty_usage_bucket(bucket); + let dirty_before = crate::scanner_io::dirty_usage_buckets_for_tests(); + let baseline = read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("usage baseline"); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking( + &ctx, + ScannerCycleBudgetConfig { + max_objects: Some(1), + ..Default::default() + }, + ); + let mut cycle = CurrentCycle { + next: 352, + ..Default::default() + }; + let mut revision = DataUsageCacheRevision::Missing; + let outcome = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + Arc::clone(&budget), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert!(matches!(outcome, ScannerCycleOutcome::Partial)); + assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Objects)); + assert!(budget.progress().0 > 0); + assert!(budget.cycle_state_persisted()); + assert_eq!(cycle.next, 353); + assert!(cycle.cycle_completed.is_empty()); + assert!(matches!(revision, DataUsageCacheRevision::Etag(_))); + let state = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("durable cycle progress"); + let (persisted, epoch) = decode_scanner_cycle_state(&state).expect("valid cycle state"); + assert_eq!(persisted.next, 353); + assert_eq!(epoch, 1); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("unchanged root"), + baseline + ); + assert!( + read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + .await + .is_err() + ); + assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + + if block_identity_read { + let set = store.pools[0].disk_set[0].clone(); + let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]); + let mut checkpoint = DataUsageCache::default(); + let revisions = checkpoint + .load_with_revisions(set.clone(), &cache_name) + .await + .expect("load the saved checkpoint before blocking its identity read"); + assert!(checkpoint.info.scan_identity.is_some()); + assert!(checkpoint.has_scan_position()); + let before = serde_json::to_value(&checkpoint).expect("serialize the existing checkpoint"); + let identity_key = path_join_buf(&[crate::BUCKET_META_PREFIX, bucket, ".bucket-incarnation"]); + let lock = store + .new_ns_lock(RUSTFS_META_BUCKET, &identity_key) + .await + .expect("bucket identity object lock"); + let guard = lock + .get_write_lock(Duration::from_secs(30)) + .await + .expect("hold the identity sidecar unavailable to scanner reads"); + let blocked_budget = ScannerCycleBudget::new_with_progress_tracking( + &ctx, + ScannerCycleBudgetConfig { + max_objects: Some(1), + ..Default::default() + }, + ); + let blocked_outcome = tokio::time::timeout( + Duration::from_secs(60), + run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + Arc::clone(&blocked_budget), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ), + ) + .await + .expect("blocked identity reads must finish within the existing acquisition bounds"); + assert_eq!( + blocked_outcome, + ScannerCycleOutcome::Failed, + "no verified bucket work must not publish a cycle" + ); + assert_eq!(blocked_budget.progress().0, 0, "unverified identity must not start a legacy rebuild"); + let after_revisions = checkpoint + .load_with_revisions(set, &cache_name) + .await + .expect("reload the retained checkpoint while only the identity object is locked"); + assert_eq!(after_revisions, revisions, "unavailable identity must not overwrite saved progress"); + assert_eq!(serde_json::to_value(&checkpoint).expect("serialize retained checkpoint"), before); + assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + drop(guard); + + let recovery_budget = ScannerCycleBudget::new_with_progress_tracking( + &ctx, + ScannerCycleBudgetConfig { + max_objects: Some(1), + ..Default::default() + }, + ); + let recovered = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + Arc::clone(&recovery_budget), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert_eq!( + recovered, + ScannerCycleOutcome::Partial, + "the same checkpoint must resume after identity reads recover" + ); + assert_eq!(recovery_budget.progress().0, 1); + checkpoint + .load(store.pools[0].disk_set[0].clone(), &cache_name) + .await + .expect("reload resumed progress"); + assert!(checkpoint.info.scan_identity.is_some()); + assert!(checkpoint.has_scan_position()); + assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + } + + if resume_mixed_plan { + let set = store.pools[0].disk_set[0].clone(); + let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]); + let mut checkpoint = DataUsageCache::default(); + let revisions = checkpoint + .load_with_revisions(set.clone(), &cache_name) + .await + .expect("load the bounded sweep's durable checkpoint"); + assert!( + checkpoint.info.scan_checkpoint.is_some() + || checkpoint.info.scan_resume_after.is_some() + || checkpoint.info.scan_raw_enumeration_cursor.is_some() + || checkpoint.info.scan_raw_enumeration_page_index.is_some() + || checkpoint.info.scan_coverage_receipt.is_some(), + "bounded sweep must retain a durable position: {:?}", + checkpoint.info + ); + let progress = checkpoint.info.scan_progress.as_mut().expect("bounded sweep has a plan"); + // A restarted process requests a new plan while retaining the old prefix. + progress.started_plan.0[0] ^= 1; + let epoch = checkpoint.info.scan_identity.expect("checkpoint identity").publication_epoch; + checkpoint + .save_with_revisions_for_epoch(set.clone(), &cache_name, &revisions, epoch) + .await + .expect("persist a prefix from a different process plan"); + + let tail_budget = ScannerCycleBudget::new_with_progress_tracking(&ctx, ScannerCycleBudgetConfig::default()); + // A committed-then-failed write is confirmed from durable state by the + // save itself, so only an uncommitted fault that outlives every bounded + // retry makes the cycle-state save fail. + let failure_guard = fail_cycle_save.map(|after_commit| { + if after_commit { + super::cycle_persist_failure::fail_next_save(1, true) + } else { + super::cycle_persist_failure::fail_saves(1, false, SCANNER_PERSIST_CAS_RETRIES + 1) + } + }); + let tail_outcome = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + Arc::clone(&tail_budget), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert!(!tail_budget.budget_elapsed(), "this is not the budget-exhaustion fast path"); + checkpoint.load(set, &cache_name).await.expect("reload the saved mixed tail"); + assert!(!checkpoint.info.snapshot_complete); + assert!(checkpoint.info.scan_checkpoint.is_none()); + assert!(checkpoint.info.scan_resume_after.is_none()); + let progress = checkpoint + .info + .scan_progress + .expect("a fresh verification sweep is still required"); + assert_eq!(progress.started_plan, progress.requested_plan); + assert_eq!( + read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("root retained"), + baseline + ); + assert!( + read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str()) + .await + .is_err() + ); + assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty_before); + if let Some(after_commit) = fail_cycle_save { + assert!(super::cycle_persist_failure::take_failure(1).is_none(), "injected fault must have fired"); + let state = read_config(store, &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("last durable cycle"); + let durable_next = decode_scanner_cycle_state(&state) + .expect("cycle after failed acknowledgement") + .0 + .next; + if after_commit { + // The write committed before the error surfaced; the save + // confirms it from durable state under the same epoch. + assert_eq!(tail_outcome, ScannerCycleOutcome::Partial); + assert!(tail_budget.cycle_state_persisted()); + assert_eq!(durable_next, 354); + } else { + // Every bounded retry failed: progress stays unacknowledged + // and the durable cycle does not move. + assert_eq!(tail_outcome, ScannerCycleOutcome::StatePersistenceFailed); + assert!(!tail_budget.cycle_state_persisted()); + assert_eq!(durable_next, 353); + } + drop(failure_guard); + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + return; + } + assert_eq!( + tail_outcome, + ScannerCycleOutcome::Partial, + "saved recovery progress is not a usage-save failure" + ); + assert_eq!(cycle.next, 354); + assert!(cycle.cycle_completed.is_empty()); + assert!(tail_budget.cycle_state_persisted()); + let state = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("durable tail cycle"); + assert_eq!(decode_scanner_cycle_state(&state).expect("tail cycle state").0.next, 354); + + let verification = run_data_scanner_cycle_with_budget( + &ctx, + &store, + &mut cycle, + &mut revision, + 1, + ScannerCycleBudget::new_with_progress_tracking(&ctx, ScannerCycleBudgetConfig::default()), + ScannerCycleScheduling { + requires_full_scan: true, + service_cohort: None, + }, + ) + .await; + assert_eq!(verification, ScannerCycleOutcome::Completed, "only the fresh sweep certifies usage"); + let usage: DataUsageInfo = serde_json::from_slice( + &read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str()) + .await + .expect("verified root"), + ) + .expect("verified usage JSON"); + assert_eq!((usage.objects_total_count, usage.objects_total_size), (3, 15)); + assert!(!crate::scanner_io::dirty_usage_buckets_pending()); + } + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); +} + +#[test] +fn partial_progress_does_not_mask_missing_updates_or_persistence_failure() { + for (persisted, expected) in [ + (DataUsagePersistOutcome::NoUpdate, ScannerCycleOutcome::Partial), + (DataUsagePersistOutcome::Failed, ScannerCycleOutcome::Failed), + (DataUsagePersistOutcome::Current, ScannerCycleOutcome::Failed), + ( + DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::DataMovement), + ScannerCycleOutcome::Deferred(ScannerCycleDeferReason::DataMovement), + ), + ] { + let result = ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_partial_bucket_progress(true); + assert_eq!(scanner_cycle_completion_outcome_for_result(&result, persisted), expected); + } + for result in [ + ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None), + ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None) + .with_partial_bucket_progress(true) + .with_usage_update_emitted(true), + ScannerCycleResult::new(ScannerCycleStatus::Complete, None).with_partial_bucket_progress(true), + ] { + assert_eq!( + scanner_cycle_completion_outcome_for_result(&result, DataUsagePersistOutcome::NoUpdate), + ScannerCycleOutcome::Failed, + "partial progress cannot stand in for an expected root update: {result:?}" + ); + } +} diff --git a/crates/scanner/src/scanner/tests/cycle_persist_failure.rs b/crates/scanner/src/scanner/tests/cycle_persist_failure.rs index 14345f101..40ce5c11a 100644 --- a/crates/scanner/src/scanner/tests/cycle_persist_failure.rs +++ b/crates/scanner/src/scanner/tests/cycle_persist_failure.rs @@ -18,11 +18,31 @@ struct Failure { epoch: u64, successful_saves: usize, after_commit: bool, + /// Consecutive write attempts that fail before the fault clears. + remaining: usize, } static FAILURE: StdMutex> = StdMutex::new(None); -struct FailureGuard; +pub(super) struct FailureGuard; + +pub(super) fn fail_next_save(epoch: u64, after_commit: bool) -> FailureGuard { + fail_saves(epoch, after_commit, 1) +} + +/// Fail `remaining` consecutive write attempts under `epoch`; a count above +/// the bounded in-save retries makes the whole save fail. +pub(super) fn fail_saves(epoch: u64, after_commit: bool, remaining: usize) -> FailureGuard { + let mut slot = FAILURE.lock().expect("failure injection lock"); + assert!(slot.is_none(), "state failure injection must not replace another test's fault"); + *slot = Some(Failure { + epoch, + successful_saves: 0, + after_commit, + remaining, + }); + FailureGuard +} impl Drop for FailureGuard { fn drop(&mut self) { @@ -37,15 +57,24 @@ pub(in crate::scanner) fn take_failure(epoch: u64) -> Option { failure.successful_saves -= 1; return None; } + if failure.remaining > 1 { + failure.remaining -= 1; + return Some(failure.after_commit); + } slot.take().map(|failure| failure.after_commit) } -async fn assert_leader_reloads_after_failed_save(successful_saves: usize, after_commit: bool) { +/// rustfs/rustfs#8141: a cycle-state save that fails without a newer owner +/// must keep the current epoch. A transient fault is retried inside the save; +/// an exhausted save reloads durable state under the same epoch. Either way +/// the leader keeps dispatching with the fence peers already validate. +async fn assert_leader_keeps_epoch_after_failed_save(successful_saves: usize, after_commit: bool, remaining: usize) { temp_env::async_with_vars([(ENV_SCANNER_CYCLE, Some("1")), (ENV_SCANNER_START_DELAY_SECS, Some("0"))], async { crate::runtime_config::refresh_scanner_runtime_config_for_tests(); crate::scanner_io::clear_dirty_usage_buckets_for_tests(); let (_temp_dir, store) = setup_scanner_cycle_store().await; let ctx = CancellationToken::new(); + let _cancel_on_drop = ctx.clone().drop_guard(); let mut cycle = CurrentCycle { next: 12, ..Default::default() @@ -56,68 +85,48 @@ async fn assert_leader_reloads_after_failed_save(successful_saves: usize, after_ epoch: 43, successful_saves, after_commit, + remaining, }); let failure_guard = FailureGuard; - let result = tokio::time::timeout(Duration::from_secs(20), run_data_scanner(ctx.clone(), store.clone())) - .await - .expect("a failed cycle-state save must stop this leader iteration"); - assert!( - result - .expect_err("state persistence failure must be retryable") - .to_string() - .contains("retrying from durable state") - ); - assert!( - FAILURE.lock().expect("failure injection lock").is_none(), - "the injected failure must fire" - ); - drop(failure_guard); - assert!(!ctx.is_cancelled(), "recovery must not stop the scanner supervisor"); - assert!(!global_metrics().report().await.current_cycle_active); - - let (durable, epoch) = decode_scanner_cycle_state( - &read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) - .await - .expect("durable cycle state"), - ) - .expect("valid durable cycle state"); - assert_eq!(epoch, 43); - let expected_next = 12 + u64::try_from(successful_saves).expect("test count fits") + u64::from(after_commit); - assert_eq!(durable.next, expected_next); - - let resumed_ctx = ctx.child_token(); - let _cancel_on_drop = resumed_ctx.clone().drop_guard(); - let mut resumed = tokio::spawn({ - let ctx = resumed_ctx.clone(); + let mut leader = tokio::spawn({ + let ctx = ctx.clone(); let store = store.clone(); async move { run_data_scanner(ctx, store).await } }); - tokio::time::timeout(Duration::from_secs(20), async { + // Progress persisted before the fault plus the faulted cycle plus one + // more cycle proves the leader kept going under the same epoch. + let expected_next = 12 + u64::try_from(successful_saves).expect("test count fits") + 2; + let (cycle, epoch) = tokio::time::timeout(Duration::from_secs(30), async { loop { tokio::select! { - result = &mut resumed => panic!("resumed scanner stopped before making progress: {result:?}"), + result = &mut leader => panic!("a failed save must not end the leader iteration: {result:?}"), _ = tokio::time::sleep(Duration::from_millis(10)) => {} } + let fired = FAILURE.lock().expect("failure injection lock").is_none(); let state = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) .await - .expect("resumed state"); - let (cycle, epoch) = decode_scanner_cycle_state(&state).expect("resumed cycle state"); - if epoch == 44 && cycle.next > expected_next && !global_metrics().report().await.current_cycle_active { - crate::remote_scanner::validate_remote_scanner_request_fence_with_store(cycle.next, epoch, store.clone()) - .await - .expect("the recovered cycle must pass the peer fence"); - break; + .expect("durable cycle state"); + let (cycle, epoch) = decode_scanner_cycle_state(&state).expect("durable cycle state should decode"); + if fired && cycle.next >= expected_next { + break (cycle, epoch); } } }) .await - .expect("the same process must resume durable scanner progress"); - resumed_ctx.cancel(); - resumed + .expect("the leader must keep persisting progress after the injected save failure"); + drop(failure_guard); + assert_eq!(epoch, 43, "a non-fence save failure must not change the leadership epoch"); + crate::remote_scanner::validate_remote_scanner_request_fence_with_store(cycle.next, epoch, store.clone()) .await - .expect("resumed scanner task should not panic") - .expect("resumed scanner should stop cleanly"); + .expect("the durable cycle must keep passing the peer fence"); + assert!(!global_metrics().report().await.current_cycle_active || cycle.next >= expected_next); + + ctx.cancel(); + leader + .await + .expect("leader task should not panic") + .expect("leader should stop cleanly"); global_metrics().set_cycle(None).await; crate::scanner_io::clear_dirty_usage_buckets_for_tests(); }) @@ -127,24 +136,30 @@ async fn assert_leader_reloads_after_failed_save(successful_saves: usize, after_ #[tokio::test] #[serial] -async fn initial_cycle_save_failure_reloads_durable_state() { - assert_leader_reloads_after_failed_save(0, false).await; +async fn initial_cycle_save_failure_keeps_epoch() { + assert_leader_keeps_epoch_after_failed_save(0, false, 1).await; } #[tokio::test] #[serial] -async fn later_cycle_save_failure_reloads_durable_state() { - assert_leader_reloads_after_failed_save(1, false).await; +async fn later_cycle_save_failure_keeps_epoch() { + assert_leader_keeps_epoch_after_failed_save(1, false, 1).await; } #[tokio::test] #[serial] -async fn initial_cycle_post_commit_failure_reloads_durable_state() { - assert_leader_reloads_after_failed_save(0, true).await; +async fn initial_cycle_post_commit_failure_keeps_epoch() { + assert_leader_keeps_epoch_after_failed_save(0, true, 1).await; } #[tokio::test] #[serial] -async fn later_cycle_post_commit_failure_reloads_durable_state() { - assert_leader_reloads_after_failed_save(1, true).await; +async fn later_cycle_post_commit_failure_keeps_epoch() { + assert_leader_keeps_epoch_after_failed_save(1, true, 1).await; +} + +#[tokio::test] +#[serial] +async fn exhausted_save_retries_reload_durable_state_under_the_same_epoch() { + assert_leader_keeps_epoch_after_failed_save(0, false, 3).await; } diff --git a/crates/scanner/src/scanner/tests/gate_self_heal.rs b/crates/scanner/src/scanner/tests/gate_self_heal.rs new file mode 100644 index 000000000..75fb812fe --- /dev/null +++ b/crates/scanner/src/scanner/tests/gate_self_heal.rs @@ -0,0 +1,860 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Startup-gate self-healing (rustfs/backlog#2901): every fail-closed scanner +//! state must have an automatic exit with a bounded dwell time, without +//! deleting quarantine evidence or relaxing the leadership, CAS, movement, or +//! usage-floor fences. + +use super::*; +use crate::storage_api::owner::{EcstoreRebalStatus, EcstoreRebalanceInfo, EcstoreRebalanceMeta, EcstoreRebalanceStats}; + +const SUPERVISOR_ENV: [(&str, Option<&str>); 3] = [ + ("RUSTFS_LOCK_ACQUIRE_TIMEOUT", Some("1")), + (ENV_SCANNER_START_DELAY_SECS, Some("0")), + (ENV_SCANNER_CYCLE, Some("1")), +]; +const WAIT_BUDGET: Duration = Duration::from_secs(30); + +fn reset_recovery_status() { + publish_scanner_cleanup_status( + ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()), + state: "healthy".to_string(), + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + ..Default::default() + }, + None, + ) + .expect("test must be able to reset the local recovery observation"); +} + +async fn wait_for_recovery_attempt(attempt: u32) -> ScannerCycleRecoveryStatus { + tokio::time::timeout(WAIT_BUDGET, async { + loop { + let status = scanner_cycle_recovery_status(); + if status.retry_count >= u64::from(attempt) { + return status; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("recovery attempt {attempt} was never recorded: {:?}", scanner_cycle_recovery_status())) +} + +async fn wait_for_recovery_state(state: &str) -> ScannerCycleRecoveryStatus { + tokio::time::timeout(WAIT_BUDGET, async { + loop { + let status = scanner_cycle_recovery_status(); + if status.state == state { + return status; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("recovery never reached {state}: {:?}", scanner_cycle_recovery_status())) +} + +async fn persisted_cycle(store: &Arc) -> Option<(CurrentCycle, u64)> { + let data = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH).await.ok()?; + decode_scanner_cycle_state(&data).ok() +} + +async fn wait_for_persisted_cycle(store: &Arc, accept: impl Fn(&CurrentCycle, u64) -> bool) -> (CurrentCycle, u64) { + tokio::time::timeout(WAIT_BUDGET, async { + loop { + if let Some((cycle, epoch)) = persisted_cycle(store).await + && accept(&cycle, epoch) + { + return (cycle, epoch); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .unwrap_or_else(|_| { + panic!( + "persisted cycle state never reached the expected shape: {:?}", + scanner_cycle_recovery_status() + ) + }) +} + +async fn stop_supervisor(ctx: CancellationToken) { + ctx.cancel(); + tokio::time::timeout(Duration::from_secs(15), async { + while crate::scanner_runtime_initialized() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("cancelled scanner supervisor must stop"); +} + +fn valid_usage_floor(epoch: u64, cycle: u64) -> Vec { + serde_json::to_vec(&DataUsageInfo { + scanner_epoch: Some(epoch), + scanner_cycle: Some(cycle), + ..complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH), 0) + }) + .expect("usage floor should encode") +} + +fn started_rebalance() -> EcstoreRebalanceMeta { + EcstoreRebalanceMeta { + id: "gate-self-heal".to_string(), + pool_stats: vec![EcstoreRebalanceStats { + participating: true, + info: EcstoreRebalanceInfo { + start_time: Some(time::OffsetDateTime::now_utc()), + status: EcstoreRebalStatus::Started, + ..Default::default() + }, + ..Default::default() + }], + ..Default::default() + } +} + +/// rustfs/rustfs#8670: a usage floor that is present but untrusted must not +/// drive the supervisor into a terminal `paused, retryable=false` observation. +/// Every attempt stays retryable with a bounded backoff, and repairing the +/// floor recovers the scanner without a process restart. +#[tokio::test] +#[serial] +async fn usage_floor_load_failure_backs_off_and_recovers_without_restart() { + temp_env::async_with_vars(SUPERVISOR_ENV, async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + reset_recovery_status(); + let (_dir, store) = setup_scanner_cycle_store().await; + // Present but without a baseline identity: the shape #8670 reported, + // whose error points the operator at the usage-state reset. + save_config( + store.clone(), + DATA_USAGE_OBJ_NAME_PATH.as_str(), + serde_json::to_vec(&DataUsageInfo::default()).expect("identity-less usage should encode"), + ) + .await + .expect("untrusted usage floor should persist"); + assert!(!crate::scanner_runtime_initialized(), "test must own the scanner runtime"); + let ctx = CancellationToken::new(); + let _cancel_on_drop = ctx.clone().drop_guard(); + init_data_scanner_with_storage(ctx.clone(), store.clone()).await; + + let attempts = MAX_SCANNER_CYCLE_RECOVERY_RETRIES + 2; + let mut first_detected = None; + for attempt in 1..=attempts { + let status = wait_for_recovery_attempt(attempt).await; + assert_eq!(status.classification.as_deref(), Some("usage_floor_load_failed"), "attempt {attempt}"); + assert!(status.retryable, "attempt {attempt} must stay retryable: {status:?}"); + assert_ne!(status.state, "paused", "attempt {attempt} must not report a terminal pause"); + if attempt >= MAX_SCANNER_CYCLE_RECOVERY_RETRIES { + assert_eq!(status.state, "backoff", "attempt {attempt}: {status:?}"); + } else { + assert_eq!(status.state, "usage_floor_load_failed", "attempt {attempt}: {status:?}"); + } + assert!( + status + .reason + .as_deref() + .is_some_and(|reason| reason.contains("usage-state/reset") && reason.contains("next automatic attempt")), + "the operator hint and the next attempt must both be visible: {status:?}" + ); + assert!(status.next_attempt_at_unix_secs.is_some(), "attempt {attempt}: {status:?}"); + match first_detected { + None => first_detected = status.first_detected_at_unix_secs, + Some(first) => assert_eq!(status.first_detected_at_unix_secs, Some(first), "attempt {attempt}"), + } + if attempt == attempts { + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), valid_usage_floor(0, 0)) + .await + .expect("repaired usage floor should persist"); + } + notify_scanner_cycle_recovery_wake(); + } + + let healthy = wait_for_recovery_state("healthy").await; + assert_eq!(healthy.retry_count, 0, "a successful attempt clears the retry budget: {healthy:?}"); + assert!(!healthy.retryable); + wait_for_persisted_cycle(&store, |cycle, _| cycle.next >= 1).await; + assert_eq!(scanner_cycle_schedule_status().execution_role, "leader"); + stop_supervisor(ctx).await; + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +async fn seed_transient_cycle_read_failure() -> ScannerCycleRecoveryStatus { + let store = Arc::new(MemoryConfigStore::default()); + store.read_errors.lock().await.insert( + memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()), + EcstoreError::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), DATA_USAGE_BLOOM_NAME_PATH.clone()), + ); + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Transient(_) + )); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.state, "transient"); + assert!(status.retryable); + status +} + +/// rustfs/rustfs#8297: a peer outage longer than the short retry budget +/// (five attempts, about 75 s, inside a 180 s planned restart) must leave the +/// observation retryable with a bounded backoff instead of a terminal pause. +#[tokio::test] +#[serial] +async fn transient_cycle_state_outage_exhausting_the_budget_stays_retryable() { + { + reset_recovery_status(); + for attempt in 1..=MAX_SCANNER_CYCLE_RECOVERY_RETRIES + 3 { + seed_transient_cycle_read_failure().await; + let retry = record_scanner_cycle_recovery_retry(attempt, Duration::from_secs(1)); + assert_eq!(retry.attempt, attempt); + assert!(retry.delay <= SCANNER_CYCLE_RECOVERY_BACKOFF_CAP); + let status = scanner_cycle_recovery_status(); + assert!(status.retryable, "attempt {attempt} must stay retryable: {status:?}"); + assert!(status.next_attempt_at_unix_secs.is_some(), "attempt {attempt}: {status:?}"); + assert_ne!(status.state, "paused", "attempt {attempt} must not report a terminal pause"); + assert_eq!(status.retry_count, u64::from(attempt)); + if attempt >= MAX_SCANNER_CYCLE_RECOVERY_RETRIES { + assert_eq!(status.state, "backoff", "attempt {attempt}: {status:?}"); + } else { + assert_eq!(status.state, "transient", "attempt {attempt}: {status:?}"); + } + } + // The peer returns: a clean load clears the observation entirely. + let store = Arc::new(MemoryConfigStore::default()); + assert!(matches!( + load_scanner_cycle_state_for_startup(store).await, + ScannerCycleStateStartup::Ready { .. } + )); + let healthy = scanner_cycle_recovery_status(); + assert_eq!(healthy.state, "healthy"); + assert_eq!(healthy.retry_count, 0); + } +} + +async fn poison_pause_backlog_replica(store: &Arc) { + Arc::clone(store) + .save_scanner_pause_backlog_replica(0, 0, b"{not-json".to_vec(), crate::HTTPPreconditions::default()) + .await + .expect("poisoned pause backlog replica should persist"); + assert!( + ScannerPauseBacklogController::claim(store.clone(), scanner_pause_backlog_now()) + .await + .is_err(), + "an invalid replica must keep the pause backlog unclaimable" + ); +} + +/// rustfs/rustfs#8251: a pause backlog that can never be claimed must not +/// skip every cycle. The leader runs degraded, the status says so, and the +/// movement fence still decides whether a cycle may start. +#[tokio::test] +#[serial] +async fn unclaimable_pause_backlog_runs_cycles_degraded_behind_the_movement_fence() { + temp_env::async_with_vars(SUPERVISOR_ENV, async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + reset_recovery_status(); + let (_dir, store) = setup_scanner_cycle_store().await; + poison_pause_backlog_replica(&store).await; + *store.rebalance_meta.write().await = Some(started_rebalance()); + assert!(store.scanner_data_movement_pause_status().await.paused); + + let ctx = CancellationToken::new(); + let _cancel_on_drop = ctx.clone().drop_guard(); + let mut leader = tokio::spawn({ + let ctx = ctx.clone(); + let store = store.clone(); + async move { run_data_scanner(ctx, store).await } + }); + // Movement is active: a degraded ledger must not admit a cycle. + tokio::time::sleep(Duration::from_secs(2)).await; + assert!( + persisted_cycle(&store).await.is_none(), + "no cycle may run while data movement pauses the scanner" + ); + assert!(!global_metrics().report().await.current_cycle_active); + assert!(!leader.is_finished(), "a paused leader waits for movement to end"); + + // Movement ends: the degraded leader runs cycles immediately. + *store.rebalance_meta.write().await = None; + store.scanner_data_movement_changed().notify_one(); + let (cycle, epoch) = tokio::time::timeout(WAIT_BUDGET, async { + loop { + tokio::select! { + result = &mut leader => panic!("degraded leader iteration stopped early: {result:?}"), + _ = tokio::time::sleep(Duration::from_millis(10)) => {} + } + if let Some((cycle, epoch)) = persisted_cycle(&store).await + && cycle.next >= 1 + { + break (cycle, epoch); + } + } + }) + .await + .expect("a degraded pause backlog must not prevent scanner cycles"); + assert!(epoch >= 1, "the degraded cycle still ran under a claimed epoch: {cycle:?}"); + + let backlog = scanner_pause_backlog_status(store.clone()).await; + assert_eq!(backlog.persistence_state, "degraded", "{backlog:?}"); + assert!(!backlog.durable); + assert!( + backlog + .error + .as_deref() + .is_some_and(|error| error.contains("scanner pause backlog")) + ); + assert!(backlog.alerting); + + ctx.cancel(); + leader + .await + .expect("degraded leader task should not panic") + .expect("degraded leader should stop cleanly"); + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +/// A corrupt `.bloomcycle.bin` is a 24-byte counter. After the quarantine +/// marker is durable, the next leader iteration rebuilds the counter from the +/// usage floor through the ordinary fenced leadership claim. The marker stays +/// behind as evidence and the supported operator reset can still remove it. +#[tokio::test] +#[serial] +async fn quarantined_corrupt_cycle_state_is_rebuilt_on_the_next_iteration() { + temp_env::async_with_vars(SUPERVISOR_ENV, async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + reset_recovery_status(); + let (_dir, store) = setup_scanner_cycle_store().await; + save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), valid_usage_floor(3, 41)) + .await + .expect("usage floor should persist"); + save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0xff, 0x00, 0x01]) + .await + .expect("corrupt cycle state should persist"); + assert!(!crate::scanner_runtime_initialized(), "test must own the scanner runtime"); + let ctx = CancellationToken::new(); + let _cancel_on_drop = ctx.clone().drop_guard(); + init_data_scanner_with_storage(ctx.clone(), store.clone()).await; + + let marker = tokio::time::timeout(WAIT_BUDGET, async { + loop { + if let Ok(marker) = read_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await { + break marker; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("the first iteration must quarantine the corrupt cycle state"); + let marker: ScannerCycleRecoveryMarker = serde_json::from_slice(&marker).expect("quarantine marker should decode"); + assert_eq!(marker.classification, "corrupt"); + assert_eq!(marker.state, "blocked"); + let quarantined = tokio::time::timeout(WAIT_BUDGET, async { + loop { + let status = scanner_cycle_recovery_status(); + if status.primary_revision.as_deref() == Some(marker.primary_revision.as_str()) { + break status; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("quarantine must be visible in the local recovery status"); + assert!(quarantined.retryable, "a corrupt counter must schedule its own rebuild: {quarantined:?}"); + notify_scanner_cycle_recovery_wake(); + + let (rebuilt, epoch) = wait_for_persisted_cycle(&store, |_, epoch| epoch >= 1).await; + assert!(rebuilt.next >= 42, "rebuilt counter must honor the usage floor cycle: {rebuilt:?}"); + assert!(epoch >= 4, "rebuilt epoch must exceed the usage floor epoch: {epoch}"); + let healthy = wait_for_recovery_state("healthy").await; + assert!(!healthy.retryable); + assert!( + healthy.rebuilt_from_quarantine_at_unix_secs.is_some(), + "the rebuild must be recorded in the recovery status: {healthy:?}" + ); + assert_eq!(healthy.leader_epoch, Some(epoch)); + let retained = read_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()) + .await + .expect("quarantine evidence must be retained after the automatic rebuild"); + let retained: ScannerCycleRecoveryMarker = serde_json::from_slice(&retained).expect("retained marker should decode"); + assert_eq!( + retained.primary_revision, marker.primary_revision, + "evidence must still name the corrupt revision" + ); + stop_supervisor(ctx).await; + + // A restart with the stale marker starts normally, and the supported + // operator reset is still the way to remove the evidence. + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Ready { .. } + )); + reset_scanner_cycle_recovery(CancellationToken::new(), store.clone()) + .await + .expect("operator reset clears a stale quarantine marker"); + assert!(matches!( + read_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await, + Err(EcstoreError::ConfigNotFound) + )); + global_metrics().set_cycle(None).await; + crate::scanner_io::clear_dirty_usage_buckets_for_tests(); + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + +/// A future-schema counter was written by a newer binary. An older reader +/// must keep failing closed instead of rebuilding over it. +#[tokio::test] +#[serial] +async fn future_schema_cycle_state_stays_blocked_after_quarantine() { + reset_recovery_status(); + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + let mut future = 17_u64.to_le_bytes().to_vec(); + future.extend_from_slice(b"RSCYC999"); + future.extend_from_slice(&4_u64.to_le_bytes()); + store.objects.lock().await.insert(state_key.clone(), future); + store.revisions.lock().await.insert(state_key.clone(), 13); + + for load in 0..2 { + assert!( + matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Blocked + ), + "load {load}" + ); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.classification.as_deref(), Some("future_schema")); + assert!(!status.retryable, "load {load}: {status:?}"); + assert_eq!(status.state, "blocked", "load {load}: {status:?}"); + } + assert!(store.objects.lock().await.contains_key(&state_key), "future state must not be rewritten"); +} + +/// rustfs/rustfs#8670: every usage slot is corrupt and the cycle state is +/// missing. The full-rebuild reset starts from an empty baseline instead of +/// refusing with "no trusted cycle or usage floor". +#[tokio::test] +#[serial] +async fn full_rebuild_reset_starts_from_empty_baseline_without_trusted_floor_or_cycle() { + reset_recovery_status(); + let (_dir, store) = setup_scanner_cycle_store_with_usage_baseline(false).await; + for path in [ + DATA_USAGE_OBJ_NAME_PATH.as_str().to_string(), + format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()), + LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str().to_string(), + ] { + save_config(store.clone(), &path, b"not-json".to_vec()) + .await + .expect("corrupt usage slot should persist"); + } + assert!(matches!( + read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH).await, + Err(EcstoreError::ConfigNotFound) + )); + assert!( + persisted_usage_floor_for_startup(store.clone(), true).await.is_err(), + "corrupt usage slots must keep startup fail-closed until the operator resets" + ); + + let result = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone()) + .await + .expect("full-rebuild reset must recover a cluster whose floor and cycle are both untrusted"); + assert_eq!(result.leader_epoch, 1); + assert_eq!(result.next_cycle, 0); + assert_eq!(result.usage_state, "bootstrap-pending"); + + let (cycle, epoch) = persisted_cycle(&store).await.expect("reset must fence a fresh cycle state"); + assert_eq!((cycle.next, epoch), (0, 1)); + let (floor, startup) = persisted_usage_floor_for_startup(store.clone(), true) + .await + .expect("startup must accept the rebuilt bootstrap marker"); + assert_eq!(startup, PersistedUsageFloorStartup::BootstrapPending); + assert_eq!(floor.leader_epoch, 1); + assert_eq!(scanner_cycle_recovery_status().state, "healthy"); +} + +async fn seeded_memory_cycle(epoch: u64) -> (Arc, CurrentCycle, DataUsageCacheRevision) { + let store = Arc::new(MemoryConfigStore::default()); + let ctx = CancellationToken::new(); + let mut cycle = CurrentCycle { + next: 12, + ..Default::default() + }; + let mut revision = DataUsageCacheRevision::Missing; + assert!(persist_scanner_cycle_state(&ctx, store.clone(), &mut cycle, &mut revision, epoch).await); + (store, cycle, revision) +} + +async fn put_count(store: &MemoryConfigStore) -> usize { + store + .put_counts + .lock() + .await + .get(&memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str())) + .copied() + .unwrap_or(0) +} + +/// rustfs/rustfs#8141: a transient write failure is retried under the same +/// epoch inside the same save, so the leader never has to reclaim and fence +/// in-flight remote scans with a new epoch. +#[tokio::test] +async fn cycle_state_save_retries_a_transient_write_failure_under_the_same_epoch() { + let (store, mut cycle, mut revision) = seeded_memory_cycle(7).await; + let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.fail_put_number.lock().await.insert(key, 2); + cycle.next = 13; + + let ctx = CancellationToken::new(); + assert!( + persist_scanner_cycle_state(&ctx, store.clone(), &mut cycle, &mut revision, 7).await, + "one failed write must be retried under the same epoch" + ); + assert_eq!(put_count(&store).await, 3); + let (durable, epoch) = decode_scanner_cycle_state( + &read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("durable cycle state"), + ) + .expect("durable cycle state should decode"); + assert_eq!((durable.next, epoch), (13, 7)); + assert_eq!(revision, DataUsageCacheRevision::Etag("memory-2".to_string())); +} + +/// A write that committed but reported an error is confirmed by reading the +/// durable state back; no second write and no epoch change are needed. +#[tokio::test] +async fn cycle_state_save_confirms_a_post_commit_failure_from_durable_state() { + let (store, mut cycle, mut revision) = seeded_memory_cycle(7).await; + let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.error_after_commit_put_number.lock().await.insert(key, 2); + cycle.next = 13; + + let ctx = CancellationToken::new(); + assert!( + persist_scanner_cycle_state(&ctx, store.clone(), &mut cycle, &mut revision, 7).await, + "a committed write must be confirmed from durable state" + ); + assert_eq!(put_count(&store).await, 2, "confirmation must not rewrite the committed state"); + assert_eq!(cycle.next, 13); + assert_eq!(revision, DataUsageCacheRevision::Etag("memory-2".to_string())); +} + +/// A CAS conflict that reveals a newer leadership epoch is a real fence: the +/// save yields immediately and never retries against the newer owner. +#[tokio::test] +async fn cycle_state_save_yields_to_a_newer_epoch_without_retrying() { + let (store, mut cycle, mut revision) = seeded_memory_cycle(7).await; + let mut newer = cycle.clone(); + newer.next = 20; + let mut newer_revision = revision.clone(); + let ctx = CancellationToken::new(); + assert!(persist_scanner_cycle_state(&ctx, store.clone(), &mut newer, &mut newer_revision, 8).await); + let puts_before = put_count(&store).await; + cycle.next = 13; + + assert!( + !persist_scanner_cycle_state(&ctx, store.clone(), &mut cycle, &mut revision, 7).await, + "a newer epoch owns the cycle state" + ); + assert_eq!(put_count(&store).await, puts_before + 1, "a fenced save must not retry"); + let (durable, epoch) = decode_scanner_cycle_state( + &read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH) + .await + .expect("durable cycle state"), + ) + .expect("durable cycle state should decode"); + assert_eq!((durable.next, epoch), (20, 8)); +} + +/// The retry schedule is bounded: it grows from the transient base (or the +/// shorter configured interval) and never exceeds the ten-minute cap. +#[test] +fn recovery_backoff_delay_is_capped() { + assert_eq!(scanner_cycle_recovery_backoff_delay(1, Duration::from_secs(60)), Duration::from_secs(5)); + assert_eq!(scanner_cycle_recovery_backoff_delay(2, Duration::from_secs(60)), Duration::from_secs(10)); + assert_eq!(scanner_cycle_recovery_backoff_delay(5, Duration::from_secs(60)), Duration::from_secs(80)); + assert_eq!( + scanner_cycle_recovery_backoff_delay(8, Duration::from_secs(60)), + SCANNER_CYCLE_RECOVERY_BACKOFF_CAP + ); + assert_eq!( + scanner_cycle_recovery_backoff_delay(u32::MAX, Duration::from_secs(60)), + SCANNER_CYCLE_RECOVERY_BACKOFF_CAP + ); + assert_eq!(scanner_cycle_recovery_backoff_delay(1, Duration::from_secs(1)), Duration::from_secs(1)); + assert_eq!(scanner_cycle_recovery_backoff_delay(0, Duration::ZERO), Duration::from_secs(1)); +} + +/// An observation older than the daily window restarts its retry budget so +/// the visible count and the backoff do not grow without bound. +#[test] +#[serial] +fn recovery_retry_budget_restarts_after_the_daily_window() { + reset_recovery_status(); + let stale_first_detected = unix_now_for_test().saturating_sub(24 * 60 * 60 + 1); + publish_scanner_cleanup_status( + ScannerCycleRecoveryStatus { + path: DATA_USAGE_BLOOM_NAME_PATH.clone(), + state: "transient".to_string(), + first_detected_at_unix_secs: Some(stale_first_detected), + retry_count: 9, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable: true, + reason: Some("cycle state could not be inspected".to_string()), + ..Default::default() + }, + None, + ) + .expect("seed stale observation"); + + let retry = record_scanner_cycle_recovery_retry(10, Duration::from_secs(60)); + assert_eq!(retry.attempt, 1, "the budget restarts from the first attempt"); + assert_eq!(retry.delay, Duration::from_secs(5)); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.retry_count, 1); + assert_eq!(status.state, "transient"); + assert!(status.retryable); + assert!( + status + .first_detected_at_unix_secs + .is_some_and(|first| first > stale_first_detected) + ); + reset_recovery_status(); +} + +fn unix_now_for_test() -> u64 { + u64::try_from(Utc::now().timestamp()).expect("current time fits") +} + +/// Unit view of the automatic rebuild: the first load quarantines and leaves +/// the observation retryable, the second load hands the quarantined revision +/// to the leadership claim instead of blocking. +#[tokio::test] +#[serial] +async fn corrupt_cycle_state_schedules_a_rebuild_after_its_marker_is_durable() { + reset_recovery_status(); + let store = Arc::new(MemoryConfigStore::default()); + let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()); + store.objects.lock().await.insert(state_key.clone(), vec![1]); + store.revisions.lock().await.insert(state_key.clone(), 9); + + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Blocked + )); + let quarantined = scanner_cycle_recovery_status(); + assert_eq!(quarantined.state, "blocked"); + assert_eq!(quarantined.classification.as_deref(), Some("corrupt")); + assert!(quarantined.retryable, "{quarantined:?}"); + assert!( + quarantined + .reason + .as_deref() + .is_some_and(|reason| reason.contains("next leader iteration rebuilds")), + "{quarantined:?}" + ); + let marker = store + .objects + .lock() + .await + .get(&marker_key) + .cloned() + .expect("quarantine marker must be durable before any rebuild"); + + let ScannerCycleStateStartup::RebuildFromQuarantine { revision } = load_scanner_cycle_state_for_startup(store.clone()).await + else { + panic!( + "a durable corrupt quarantine must schedule a rebuild: {:?}", + scanner_cycle_recovery_status() + ); + }; + assert_eq!(revision, DataUsageCacheRevision::Etag("memory-9".to_string())); + let rebuilding = scanner_cycle_recovery_status(); + assert_eq!(rebuilding.state, "rebuilding"); + assert!(rebuilding.retryable); + assert_eq!(rebuilding.primary_revision.as_deref(), Some("memory-9")); + assert_eq!( + store.objects.lock().await.get(&marker_key), + Some(&marker), + "scheduling must not touch the evidence" + ); + assert_eq!( + store.objects.lock().await.get(&state_key), + Some(&vec![1]), + "scheduling must not touch the primary" + ); + + // The claim CAS-replaces the quarantined revision under a higher epoch. + seed_usage_snapshot_for_leadership_claim(&store).await; + let mut cycle = CurrentCycle::default(); + let mut revision = revision; + let mut leader_epoch = 0; + assert!( + claim_scanner_leadership( + &CancellationToken::new(), + store.clone(), + &mut cycle, + &mut revision, + &mut leader_epoch, + false, + ScannerCycleResetPolicy::None, + ) + .await + ); + assert_eq!(leader_epoch, 1); + let (durable, epoch) = decode_scanner_cycle_state(store.objects.lock().await.get(&state_key).expect("rebuilt primary")) + .expect("rebuilt primary should decode"); + assert_eq!((durable.next, epoch), (0, 1)); + assert_eq!( + store.objects.lock().await.get(&marker_key), + Some(&marker), + "the rebuild keeps the evidence" + ); + assert!(matches!( + load_scanner_cycle_state_for_startup(store.clone()).await, + ScannerCycleStateStartup::Ready { leader_epoch: 1, .. } + )); + reset_recovery_status(); +} + +fn backlog_observation(now: u64, paused: bool, movement_generation: u64) -> ScannerPauseBacklogObservation { + ScannerPauseBacklogObservation { + now_unix_secs: now, + paused, + movement_generation, + movement_work_items: 0, + pause_started_at_unix_secs: 0, + dirty_usage_buckets: 0, + discovered_expiry_items: 0, + discovered_transition_items: 0, + } +} + +/// A degraded controller admits cycles, keeps the ordinary cadence, and once +/// the claim succeeds again the durable ledger resumes its catch-up phase. +#[tokio::test] +#[serial] +async fn degraded_pause_backlog_resumes_durable_catch_up_after_the_claim_recovers() { + let (_dir, store) = setup_scanner_cycle_store().await; + let now = scanner_pause_backlog_now(); + let mut durable = ScannerPauseBacklogController::claim(store.clone(), now) + .await + .expect("fresh ledger claim"); + durable.observe(backlog_observation(now.saturating_add(1), true, 7)).await; + durable.observe(backlog_observation(now.saturating_add(2), false, 7)).await; + let persisted = scanner_pause_backlog_status(store.clone()).await; + assert_eq!(persisted.phase, ScannerPauseBacklogPhase::CatchingUp); + assert!(persisted.pending_full_scan); + drop(durable); + + let degraded_at = now.saturating_add(10); + let mut controller = + ScannerPauseBacklogController::unavailable(store.clone(), "simulated claim failure".to_string(), degraded_at); + assert_eq!(controller.begin_attempt(degraded_at).await, ScannerPauseBacklogAttemptDecision::Degraded); + assert_eq!( + controller.begin_usage_bootstrap_rebuild_attempt(degraded_at).await, + ScannerPauseBacklogAttemptDecision::Degraded + ); + assert_eq!( + controller.scheduling_delay(degraded_at, false), + None, + "degraded mode keeps the ordinary cadence" + ); + let status = scanner_pause_backlog_status(store.clone()).await; + assert_eq!(status.persistence_state, "degraded"); + assert!( + status + .error + .as_deref() + .is_some_and(|error| error.contains("simulated claim failure")) + ); + + // Before the refresh interval the observation does not re-claim. + controller + .observe(backlog_observation(degraded_at.saturating_add(1), false, 7)) + .await; + assert_eq!( + controller.begin_attempt(degraded_at.saturating_add(1)).await, + ScannerPauseBacklogAttemptDecision::Degraded + ); + + // After it, the claim succeeds and the persisted catch-up phase is back. + let recovered_at = degraded_at.saturating_add(5 * 60); + controller.observe(backlog_observation(recovered_at, false, 7)).await; + assert!(matches!( + controller.begin_attempt(recovered_at).await, + ScannerPauseBacklogAttemptDecision::Tracked(_) + )); + let recovered = scanner_pause_backlog_status(store.clone()).await; + assert_eq!(recovered.persistence_state, "healthy", "{recovered:?}"); + assert!(recovered.durable); + assert_eq!(recovered.phase, ScannerPauseBacklogPhase::CatchingUp); + assert!(recovered.pending_full_scan, "catch-up debt survives the degraded window"); + assert!(recovered.error.is_none()); +} + +/// Exhausting the bounded in-save retries is reported as unconfirmed, not as +/// a fence: the caller reloads durable state under the same epoch. +#[tokio::test] +async fn cycle_state_save_reports_exhausted_retries_as_unconfirmed() { + let (store, mut cycle, mut revision) = seeded_memory_cycle(7).await; + let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()); + store.fail_puts_from.lock().await.insert(key, 2); + cycle.next = 13; + let ctx = CancellationToken::new(); + assert_eq!( + persist_scanner_cycle_state_for_epoch(&ctx, store.clone(), &mut cycle, &mut revision, 7, None).await, + ScannerCycleStatePersistOutcome::Unconfirmed + ); + assert_eq!(put_count(&store).await, 1 + SCANNER_PERSIST_CAS_RETRIES + 1); + assert_eq!( + revision, + DataUsageCacheRevision::Etag("memory-1".to_string()), + "the durable revision is kept" + ); + + let fenced_store = Arc::new(MemoryConfigStore::default()); + let mut newer = CurrentCycle { + next: 20, + ..Default::default() + }; + let mut newer_revision = DataUsageCacheRevision::Missing; + assert!(persist_scanner_cycle_state(&ctx, fenced_store.clone(), &mut newer, &mut newer_revision, 8).await); + let mut stale_revision = DataUsageCacheRevision::Etag("memory-0".to_string()); + assert_eq!( + persist_scanner_cycle_state_for_epoch(&ctx, fenced_store.clone(), &mut cycle, &mut stale_revision, 7, None).await, + ScannerCycleStatePersistOutcome::Fenced + ); +} diff --git a/crates/scanner/src/scanner/tests/recovery_control.rs b/crates/scanner/src/scanner/tests/recovery_control.rs index 61ac96bc5..5f8f806b5 100644 --- a/crates/scanner/src/scanner/tests/recovery_control.rs +++ b/crates/scanner/src/scanner/tests/recovery_control.rs @@ -81,6 +81,164 @@ async fn persisted_state(store: &Arc) -> Vec<(Option>, DataUsag state } +async fn seed_transient_cycle_read_failure() -> ScannerCycleRecoveryStatus { + let store = Arc::new(MemoryConfigStore::default()); + let error = EcstoreError::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), DATA_USAGE_BLOOM_NAME_PATH.clone()); + let expected_error = format!("Scanner error: failed to inspect scanner cycle state: {error}"); + store + .read_errors + .lock() + .await + .insert(memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()), error); + let ScannerCycleStateStartup::Transient(error) = load_scanner_cycle_state_for_startup(store).await else { + panic!("a quorum outage must produce transient cycle recovery"); + }; + assert_eq!(error.to_string(), expected_error); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.state, "transient"); + assert!(status.retryable); + assert_eq!(status.classification, None); + status +} + +#[tokio::test] +#[serial] +async fn issue_8297_contended_follower_clears_only_transient_cycle_recovery() { + temp_env::async_with_vars([("RUSTFS_LOCK_ACQUIRE_TIMEOUT", Some("1"))], async { + let (_dir, store) = setup_scanner_cycle_store().await; + seed_cleanup(&store, "cleanup-pending").await; + let before = persisted_state(&store).await; + let lock = store + .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock") + .await + .expect("leader lock"); + let guard = lock + .get_write_lock_quiet(Duration::from_secs(1)) + .await + .expect("hold live leader"); + let transient = seed_transient_cycle_read_failure().await; + let retry = record_scanner_cycle_recovery_retry(MAX_SCANNER_CYCLE_RECOVERY_RETRIES, Duration::from_secs(1)); + assert_eq!(retry.attempt, MAX_SCANNER_CYCLE_RECOVERY_RETRIES); + let backoff = scanner_cycle_recovery_status(); + assert_eq!(backoff.state, "backoff"); + assert!(backoff.retryable, "budget exhaustion must stay retryable: {backoff:?}"); + assert!(backoff.next_attempt_at_unix_secs.is_some()); + let mut cases = vec![(transient, true), (backoff, true)]; + for (state, classification, retryable) in [ + ("recovery-required", None, false), + ("blocked", None, false), + ("blocked", Some("corrupt"), false), + ("cleanup-pending", None, true), + ("cleanup-pending", Some("corrupt"), true), + ("usage_floor_load_failed", Some("usage_floor_load_failed"), true), + ("usage_floor_recovery_pending", Some("legacy_empty_usage_floor"), true), + ("backoff", Some("usage_floor_load_failed"), true), + ("cache_cycle_ahead", Some("cache_cycle_ahead"), true), + ] { + cases.push(( + ScannerCycleRecoveryStatus { + state: state.to_string(), + classification: classification.map(str::to_string), + retry_count: 5, + max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES, + retryable, + reason: Some("preserve recovery state".to_string()), + ..Default::default() + }, + false, + )); + } + for (status, clear) in cases { + let expected = serde_json::to_value(&status).expect("expected recovery status"); + publish_scanner_cleanup_status(status, None).expect("seed local recovery status"); + tokio::time::timeout( + Duration::from_secs(30), + run_data_scanner_with_maintenance_state( + CancellationToken::new(), + store.clone(), + ScannerMaintenanceFeatures::default(), + None, + ), + ) + .await + .expect("contended scanner iteration must finish") + .expect("lock contention is a normal follower outcome"); + assert_eq!(scanner_cycle_schedule_status().execution_role, "follower"); + let actual = scanner_cycle_recovery_status(); + if clear { + assert_eq!(actual.state, "healthy", "stale follower recovery: {expected}"); + assert_eq!(actual.retry_count, 0); + assert!(!actual.retryable); + } else { + assert_eq!(serde_json::to_value(actual).expect("actual recovery status"), expected); + } + assert!(!guard.is_lock_lost(), "follower must not revoke the leader"); + assert_eq!(persisted_state(&store).await, before, "follower must not rewrite durable recovery state"); + } + }) + .await; +} + +#[tokio::test] +#[serial] +async fn issue_8297_supervisor_recovers_paused_follower_without_restart() { + temp_env::async_with_vars( + [ + ("RUSTFS_LOCK_ACQUIRE_TIMEOUT", Some("1")), + (ENV_SCANNER_START_DELAY_SECS, Some("0")), + (ENV_SCANNER_CYCLE, Some("1")), + ], + async { + let (_dir, store) = setup_scanner_cycle_store().await; + seed_cleanup(&store, "cleanup-pending").await; + let before = persisted_state(&store).await; + let lock = store + .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock") + .await + .expect("leader lock"); + let guard = lock + .get_write_lock_quiet(Duration::from_secs(1)) + .await + .expect("hold live leader"); + for attempt in 1..=MAX_SCANNER_CYCLE_RECOVERY_RETRIES { + seed_transient_cycle_read_failure().await; + assert_eq!(record_scanner_cycle_recovery_retry(attempt, Duration::from_secs(1)).attempt, attempt); + } + let backoff = scanner_cycle_recovery_status(); + assert_eq!(backoff.state, "backoff"); + assert!(backoff.retryable); + assert!(!crate::scanner_runtime_initialized(), "test must own the scanner runtime"); + let ctx = CancellationToken::new(); + let _cancel_on_drop = ctx.clone().drop_guard(); + init_data_scanner_with_storage(ctx.clone(), store.clone()).await; + assert!(crate::scanner_runtime_initialized()); + let recovered = tokio::time::timeout(Duration::from_secs(30), async { + while scanner_cycle_recovery_status().state != "healthy" { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await; + ctx.cancel(); + tokio::time::timeout(Duration::from_secs(15), async { + while crate::scanner_runtime_initialized() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("cancelled scanner supervisor must stop"); + recovered.expect("follower must recover without waiting out the backoff or restarting the process"); + assert_eq!(scanner_cycle_schedule_status().execution_role, "follower"); + let status = scanner_cycle_recovery_status(); + assert_eq!(status.retry_count, 0); + assert!(!status.retryable); + assert!(!guard.is_lock_lost(), "supervisor must not revoke the leader"); + assert_eq!(persisted_state(&store).await, before, "supervisor must leave durable recovery untouched"); + }, + ) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); +} + async fn run_disabled_startup(ctx: CancellationToken, store: Arc) { let initialized_before = crate::scanner_runtime_initialized(); let cleanup = init_scanner_with_recovery(ctx, store, false).await; diff --git a/crates/scanner/src/scanner/tests/scoped_ack_publication.rs b/crates/scanner/src/scanner/tests/scoped_ack_publication.rs index 472e595d7..2914c985a 100644 --- a/crates/scanner/src/scanner/tests/scoped_ack_publication.rs +++ b/crates/scanner/src/scanner/tests/scoped_ack_publication.rs @@ -14,7 +14,7 @@ const PROOF_CYCLE: u64 = 11; async fn settle_namespace_commits(store: &ECStore) { tokio::time::timeout(Duration::from_secs(30), async { - while store.scanner_data_usage_publication_blocked().await { + while ecstore_namespace_commits_pending(store) { tokio::time::sleep(Duration::from_millis(1)).await; } }) @@ -48,6 +48,7 @@ async fn complete_candidate(store: &Arc, cycle: u64) -> (crate::scanner requires_full_scan: true, service_cohort: None, resolved_scope_observer: None, + post_walk_gate: None, }, ) .await @@ -242,6 +243,7 @@ async fn scoped_ack_publication_observed_candidate_reuse_requires_a_new_root_pro // Seed a complete but unconverged observation from real scanner coverage; // the production writer attaches its authoritative baseline identity. observed_candidate.usage_snapshot_converged = Some(false); + observed_candidate.usage_snapshot_observational = true; let observation = publish_candidate(&store, &prior_scan, observed_candidate, None).await; let (outcome, proof) = observation.into_parts(); assert_eq!(outcome, DataUsagePersistOutcome::Saved); @@ -269,6 +271,7 @@ async fn scoped_ack_publication_observed_candidate_reuse_requires_a_new_root_pro requires_full_scan: false, service_cohort: None, resolved_scope_observer: Some(observer), + post_walk_gate: None, }, ) .await diff --git a/crates/scanner/src/scanner/usage_store.rs b/crates/scanner/src/scanner/usage_store.rs index 21a8a7cad..828999202 100644 --- a/crates/scanner/src/scanner/usage_store.rs +++ b/crates/scanner/src/scanner/usage_store.rs @@ -577,7 +577,10 @@ where outcome = DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::DataMovement); break 'updates; } - let observational = data_usage_info.usage_snapshot_converged == Some(false); + // Routing keys on the explicit producer mark, not on convergence: a + // `Complete` cycle with pending buckets is authoritative but not + // converged, while a vetoed cycle is observational. + let observational = data_usage_info.usage_snapshot_observational; let target_path = if observational { DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str() } else { @@ -1143,7 +1146,6 @@ where } } if !observational - && data_usage_info.usage_snapshot_converged == Some(true) && matches!(outcome, DataUsagePersistOutcome::Saved | DataUsagePersistOutcome::AlreadyDurable) && (outcome == DataUsagePersistOutcome::AlreadyDurable || write_confirmed) && let (Some(expected), Some(epoch)) = (ack_expectation.as_ref(), ack_epoch) diff --git a/crates/scanner/src/scanner_folder.rs b/crates/scanner/src/scanner_folder.rs index 5f664466a..7d1c054ab 100644 --- a/crates/scanner/src/scanner_folder.rs +++ b/crates/scanner/src/scanner_folder.rs @@ -23,6 +23,7 @@ use crate::data_usage_define::{ DATA_USAGE_SCAN_CHECKPOINT_VERSION, DataUsageCache, DataUsageCacheInfo, DataUsageEntry, DataUsageHash, DataUsageHashMap, DataUsageRawEnumerationCursor, DataUsageScanCheckpoint, DataUsageScanCheckpointReason, MAX_DATA_USAGE_CACHE_DEPTH, PendingScannerHeal, PendingScannerHealKind, ScannerSizeSummaryExt, SizeReconciliationEntry, SizeSummary, hash_path, + scan_frontier_covers_path, }; use crate::error::ScannerError; use crate::raw_page_index::{RawEnumerationPageIndex, RawEnumerationPageWriter}; @@ -121,6 +122,14 @@ const MAX_SIZE_RECONCILIATION_ENTRIES_PER_BUCKET: usize = 10_000; const MAX_SIZE_RECONCILIATION_BYTES_PER_BUCKET: usize = 8 * 1024 * 1024; const MAX_SIZE_RECONCILIATION_AGE_SECS: u64 = 7 * 24 * 60 * 60; +fn checkpoint_interval_for_build_time(build_time: Duration) -> Duration { + // Aim to spend at most a tenth of the interval building snapshots, while + // preserving the existing upper bound on periodic checkpoint attempts. + build_time + .saturating_mul(10) + .clamp(SCANNER_CHECKPOINT_MIN_INTERVAL, SCANNER_CHECKPOINT_INTERVAL) +} + // --- scanner excess alerts as S3 notification events (rustfs/backlog#1868) -- // // The excess-versions / excess-version-size / excess-folders alerts were @@ -426,10 +435,6 @@ fn folder_resume_match(folder_name: &str, resume_after: &str) -> Option bool { - folder <= frontier && !matches!(folder_resume_match(folder, frontier), Some(FolderResumeMatch::Descendant)) -} - fn order_items_for_resume(items: &mut [T], resume_after: Option<&str>, name: F) -> FolderResumeOrder where F: Fn(&T) -> &str, @@ -476,6 +481,19 @@ fn checkpoint_reason_from_budget(reason: Option) -> Da } } +// Cache entry failures survive diagnostic retry-cache TTL and capacity eviction. +// Index once per scan; resumed subtree checks must not flatten histograms repeatedly. +fn failed_scan_paths(cache: &DataUsageCache) -> Vec { + let mut paths = cache + .cache + .iter() + .filter(|(_, entry)| entry.failed_objects > 0) + .map(|(path, _)| path.clone()) + .collect::>(); + paths.sort_unstable(); + paths +} + fn data_usage_entry_has_progress(entry: &DataUsageEntry) -> bool { data_usage_root_has_progress(entry) } @@ -744,6 +762,7 @@ pub struct FolderScanner { checkpoint_objects: u64, last_checkpoint_objects: u64, last_checkpoint_at: Instant, + last_checkpoint_build_time: Duration, update_current_path: UpdateCurrentPathFn, @@ -756,6 +775,7 @@ pub struct FolderScanner { pending_heals_changed: bool, coverage_frontier: Option, resume_frontier: Option, + resume_failed_paths: Vec, coverage_gap: bool, raw_enumeration_progress: Vec, pending_heal_sync_deferred: bool, @@ -1111,16 +1131,39 @@ impl FolderScanner { self.update_cache.info.scan_checkpoint = Some(checkpoint); } - fn record_completed_child(&mut self, folder: &str, healthy: bool) { - if self.old_cache.info.scan_progress.is_some() { - self.coverage_gap |= !healthy; - if !self.coverage_gap { - self.coverage_frontier = Some(folder.to_owned()); - } + fn record_completed_child(&mut self, folder: &str) { + if self.old_cache.info.scan_progress.is_some() + && !self.coverage_gap + && !self + .coverage_frontier + .as_deref() + .is_some_and(|frontier| scan_frontier_covers_path(folder, frontier)) + { + self.coverage_frontier = Some(folder.to_owned()); } self.record_scan_resume_hint(folder); } + fn resume_subtree_has_failures(&self, folder: &str) -> bool { + if self.resume_failed_paths.is_empty() { + return false; + } + if self + .resume_failed_paths + .binary_search_by(|path| path.as_str().cmp(folder)) + .is_ok() + { + return true; + } + let prefix = format!("{folder}/"); + let index = self + .resume_failed_paths + .partition_point(|path| path.as_str() < prefix.as_str()); + self.resume_failed_paths + .get(index) + .is_some_and(|path| path.starts_with(&prefix)) + } + fn record_scan_resume_hint_if_not_ancestor(&mut self, folder: &str) { let keep_existing = self .new_cache @@ -1258,7 +1301,7 @@ impl FolderScanner { let elapsed = self.last_checkpoint_at.elapsed(); if into.compacted || self.new_cache.info.scan_progress.is_none() - || elapsed < SCANNER_CHECKPOINT_MIN_INTERVAL + || elapsed < checkpoint_interval_for_build_time(self.last_checkpoint_build_time) || (self.checkpoint_objects.saturating_sub(self.last_checkpoint_objects) < SCANNER_CHECKPOINT_OBJECT_INTERVAL && elapsed < SCANNER_CHECKPOINT_INTERVAL) { @@ -1280,16 +1323,24 @@ impl FolderScanner { // Invalid partial graphs must not trigger a fresh O(N) clone on every object. self.last_checkpoint_objects = self.checkpoint_objects; self.last_checkpoint_at = Instant::now(); + let snapshot = self.checkpoint_snapshot(folder, hash, into); + self.last_checkpoint_build_time = self.last_checkpoint_at.elapsed(); + if let Some(snapshot) = snapshot { + permit.send(snapshot); + } + } + + fn checkpoint_snapshot(&self, folder: &CachedFolder, hash: &DataUsageHash, into: &DataUsageEntry) -> Option { // Heal revisits are not sorted with the forward sweep. An unfinished // earlier subtree, including a suspended ancestor, must not be signed // as completed prefix coverage. if self.coverage_frontier.as_deref().is_some_and(|frontier| { - folder_is_covered_by_frontier(&folder.name, frontier) + scan_frontier_covers_path(&folder.name, frontier) || self.checkpoint_ancestors[..self.checkpoint_depth] .iter() - .any(|ancestor| folder_is_covered_by_frontier(&ancestor.hash.0, frontier)) + .any(|ancestor| scan_frontier_covers_path(&ancestor.hash.0, frontier)) }) { - return; + return None; } let mut snapshot = self.new_cache.clone(); let mut parent = None; @@ -1298,30 +1349,26 @@ impl FolderScanner { parent = Some(ancestor.hash.clone()); } snapshot.replace_hashed(hash, &folder.parent, into); - if snapshot.checked_flatten_complete_scope(&snapshot.info.name).is_none() { - return; - } + snapshot.checked_flatten_complete_scope(&snapshot.info.name)?; snapshot.info.last_update = Some(SystemTime::now()); snapshot.info.snapshot_complete = false; let (cursor, page_index) = self.peek_raw_enumeration_resume_state(); snapshot.info.scan_raw_enumeration_cursor = cursor; snapshot.info.scan_raw_enumeration_page_index = page_index; - if snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).is_err() { - return; - } + snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).ok()?; // A resumed readdir can run before covered subtrees are copied. Keep // its durable receipt until this snapshot can represent that coverage. if self.coverage_frontier.is_some() && snapshot.info.scan_coverage_receipt.is_none() { - return; + return None; } // The receipt was just sealed from this snapshot; do not hash it twice. if snapshot.info.scan_raw_enumeration_cursor.is_none() && snapshot.info.scan_raw_enumeration_page_index.is_none() && snapshot.info.scan_coverage_receipt.is_none() { - return; + return None; } - permit.send(snapshot); + Some(snapshot) } fn carry_forward_old_children(&mut self, parent_hash: &DataUsageHash, entry: &mut DataUsageEntry) { @@ -1362,7 +1409,7 @@ impl FolderScanner { if self .coverage_frontier .as_deref() - .is_some_and(|frontier| folder_is_covered_by_frontier(&child_hash.0, frontier)) + .is_some_and(|frontier| scan_frontier_covers_path(&child_hash.0, frontier)) { self.coverage_frontier = None; self.coverage_gap = true; @@ -1927,10 +1974,23 @@ impl FolderScanner { // A new FolderScanner starts with an empty seen set, so // cached paths are still rechecked once per cycle. Skip // duplicate listings within this same scanner pass. - self.coverage_gap |= self.old_cache.info.scan_progress.is_some(); + into.failed_objects = into.failed_objects.max(1); continue; } + // Charge the disk's background I/O budget before the metadata + // read. The `xl.meta` size is only measured when bytes are + // budgeted, so an uncapped byte axis costs no extra stat. + let budgeted_bytes = if self.sleeper.budgets_disk_io_bytes() { + tokio::fs::metadata(&item.path) + .await + .map(|metadata| metadata.len()) + .unwrap_or(0) + } else { + 0 + }; + self.sleeper.admit_disk_io(1, budgeted_bytes).await; + let sz = match self .local_disk .get_size_with_tier_names(item.clone(), &self.tier_registry.names) @@ -1943,9 +2003,8 @@ impl FolderScanner { if failure_action != GetSizeFailureAction::Skip { self.resolved_failed_object_paths.remove(&item.path); - self.coverage_gap |= self.old_cache.info.scan_progress.is_some(); + into.failed_objects = into.failed_objects.saturating_add(1); if !retry_suppressed { - into.failed_objects += 1; self.record_failed(&item.path); } @@ -2098,14 +2157,13 @@ impl FolderScanner { } if self.is_erasure_mode && found_erasure_data_directory && !found_object_metadata { - self.coverage_gap |= self.old_cache.info.scan_progress.is_some(); found_object_metadata = true; let metadata_path = path_join_buf(&[&dir_path, STORAGE_FORMAT_FILE]); self.mark_failed_path_seen(&metadata_path); let retry_suppressed = self.failed_retry_suppressed(&metadata_path); + into.failed_objects = into.failed_objects.saturating_add(1); if !retry_suppressed { - into.failed_objects = into.failed_objects.saturating_add(1); self.record_failed(&metadata_path); let failed_cache_entries = self.new_cache.info.failed_objects.len(); @@ -2253,8 +2311,9 @@ impl FolderScanner { && !into.compacted && forward_resume_after .as_deref() - .is_some_and(|resume| folder_is_covered_by_frontier(&folder_item.name, resume)) + .is_some_and(|resume| scan_frontier_covers_path(&folder_item.name, resume)) && self.old_cache.find(&folder_item.name).is_some() + && !self.resume_subtree_has_failures(&folder_item.name) { self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent); into.add_child(&h); @@ -2294,6 +2353,12 @@ impl FolderScanner { // Usage sampling is not proof that a Deep check ran. if self.scan_mode != HealScanMode::Deep && !forward_sweep + && !self + .old_cache + .info + .replication + .as_ref() + .is_some_and(|config| config.has_active_rules("", true)) && !into.compacted && self.old_cache.is_compacted(&h) { @@ -2317,7 +2382,7 @@ impl FolderScanner { // In compacted mode child totals are accumulated directly into the parent entry. let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), into)); fut.await.map_err(|e| ScannerError::Other(e.to_string()))?; - self.record_completed_child(&folder_item.name, into.failed_objects == 0); + self.record_completed_child(&folder_item.name); self.send_update_for_entry(&this_hash, &folder.parent, into).await; tokio::task::yield_now().await; } else { @@ -2350,7 +2415,7 @@ impl FolderScanner { tokio::task::yield_now().await; into.add_child(&h); - self.record_completed_child(&folder_item.name, dst.failed_objects == 0); + self.record_completed_child(&folder_item.name); self.maybe_send_checkpoint(&folder, &this_hash, into); // We scanned a folder, optionally send update. self.update_cache.delete_recursive(&h); @@ -2903,6 +2968,7 @@ pub(crate) async fn scan_data_folder_scoped( } let bucket_name = cache.info.name.clone(); let next_cycle = cache.info.next_cycle; + let resume_position_on_entry = cache.has_scan_position(); let had_scan_checkpoint = cache.info.scan_checkpoint.is_some(); let new_cache = DataUsageCache { info: cache.info.clone(), @@ -2912,6 +2978,11 @@ pub(crate) async fn scan_data_folder_scoped( info: cache.info.clone(), ..Default::default() }; + let resume_failed_paths = if resume_frontier.is_some() { + failed_scan_paths(&cache) + } else { + Vec::new() + }; // Create folder scanner let mut scanner = FolderScanner { root: base_path, @@ -2938,6 +3009,7 @@ pub(crate) async fn scan_data_folder_scoped( checkpoint_objects: 0, last_checkpoint_objects: 0, last_checkpoint_at: Instant::now(), + last_checkpoint_build_time: Duration::ZERO, update_current_path, budget: budget.clone(), skip_heal, @@ -2946,6 +3018,7 @@ pub(crate) async fn scan_data_folder_scoped( pending_heals_changed: false, coverage_frontier: resume_frontier.clone(), resume_frontier, + resume_failed_paths, coverage_gap: false, pending_heal_sync_deferred: false, pending_heal_batch_dirty: false, @@ -2989,7 +3062,7 @@ pub(crate) async fn scan_data_folder_scoped( new_cache.info.last_update = Some(SystemTime::now()); new_cache.info.next_cycle = next_cycle; let unresolved_objects = coverage_gap - || root.failed_objects > 0 + || new_cache.cache.values().any(|entry| entry.failed_objects > 0) || !new_cache.info.failed_objects.is_empty() || !new_cache.info.size_reconciliation.is_empty(); let mixed_coverage = new_cache @@ -3020,6 +3093,27 @@ pub(crate) async fn scan_data_folder_scoped( close_disk_guard.close().await; if unresolved_objects || mixed_coverage { + debug!( + target: "rustfs::scanner::folder", + event = EVENT_SCANNER_FOLDER_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_FOLDER, + state = "bucket_sweep_incomplete", + bucket = %bucket_name, + pool = new_cache.info.source.map(|source| source.pool_index), + set = new_cache.info.source.map(|source| source.set_index), + cycle = next_cycle, + leader_epoch = new_cache.info.leader_epoch, + resume_position_on_entry, + resume_position = new_cache.has_scan_position(), + coverage_gap, + unresolved_objects, + root_failed_objects = root.failed_objects, + failed_objects = new_cache.info.failed_objects.len(), + size_reconciliation_entries = new_cache.info.size_reconciliation.len(), + mixed_coverage, + "Scanner bucket sweep requires another verification pass" + ); Err(ScannerError::PartialCache(Box::new(std::mem::take(new_cache)))) } else { Ok(std::mem::take(new_cache)) diff --git a/crates/scanner/src/scanner_folder/tests.rs b/crates/scanner/src/scanner_folder/tests.rs index 106169bd1..731571965 100644 --- a/crates/scanner/src/scanner_folder/tests.rs +++ b/crates/scanner/src/scanner_folder/tests.rs @@ -349,6 +349,7 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) { checkpoint_objects: 0, last_checkpoint_objects: 0, last_checkpoint_at: Instant::now(), + last_checkpoint_build_time: Duration::ZERO, update_current_path, budget: ScannerCycleBudget::new(&CancellationToken::new(), Default::default()), skip_heal: Arc::new(AtomicBool::new(false)), @@ -361,6 +362,7 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) { pending_heals_changed: false, coverage_frontier: None, resume_frontier: None, + resume_failed_paths: Vec::new(), coverage_gap: false, raw_enumeration_progress: Vec::new(), pending_heal_sync_deferred: false, @@ -375,15 +377,8 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) { (scanner, temp_dir) } -#[tokio::test] -async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() { +async fn build_checkpoint_scanner() -> (FolderScanner, std::path::PathBuf) { let (mut scanner, temp_dir) = build_test_scanner().await; - let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); - scanner.checkpoint_tx = Some(checkpoint_tx); - scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; - scanner.last_checkpoint_at = Instant::now() - .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) - .expect("test instant subtraction"); scanner.new_cache.info.name = "bucket".to_string(); scanner.new_cache.info.scan_progress = Some(crate::DataUsageScanProgress { started_plan: crate::DataUsageScanPlanDigest([1; 32]), @@ -411,6 +406,18 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() .seal_scan_frontier(Some("bucket/a")) .expect("fixture frontier"); assert_eq!(scanner.new_cache.validated_scan_frontier(), Some("bucket/a")); + (scanner, temp_dir) +} + +#[tokio::test] +async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() { + let (mut scanner, temp_dir) = build_checkpoint_scanner().await; + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) + .expect("test instant subtraction"); let checkpoint_folder = CachedFolder { name: "bucket".to_string(), parent: None, @@ -421,6 +428,7 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry); + let build_time = scanner.last_checkpoint_build_time; scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL * 2; scanner.last_checkpoint_at = Instant::now() .checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL) @@ -430,6 +438,10 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL, "a full checkpoint queue must reject before cloning or advancing progress" ); + assert_eq!( + scanner.last_checkpoint_build_time, build_time, + "a full queue must not reset the cost estimate" + ); let checkpoint = checkpoint_rx.try_recv().expect("object threshold emits a bounded checkpoint"); assert_eq!(checkpoint.info.name, "bucket"); @@ -460,6 +472,87 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() .expect("remove test scanner directory"); } +#[test] +fn checkpoint_build_cost_interval_is_bounded() { + for (build_time, interval) in [ + (Duration::ZERO, Duration::from_secs(5)), + (Duration::from_millis(500), Duration::from_secs(5)), + (Duration::from_millis(501), Duration::from_millis(5010)), + (Duration::from_secs(3), Duration::from_secs(30)), + (Duration::from_secs(6), Duration::from_secs(60)), + (Duration::MAX, Duration::from_secs(60)), + ] { + assert_eq!(checkpoint_interval_for_build_time(build_time), interval, "build time: {build_time:?}"); + } +} + +#[tokio::test] +async fn periodic_checkpoint_build_cost_backoff_preserves_deadline_and_validation() { + let (mut scanner, temp_dir) = build_checkpoint_scanner().await; + let _guard = TestGuard { + temp_dir: Some(temp_dir), + }; + let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1); + scanner.checkpoint_tx = Some(checkpoint_tx); + let folder = CachedFolder { + name: "bucket".to_string(), + parent: None, + object_heal_prob_div: 1, + }; + let hash = hash_path("bucket"); + let mut entry = scanner.new_cache.root().expect("checkpoint fixture root"); + scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL; + scanner.last_checkpoint_build_time = Duration::from_secs(3); + scanner.last_checkpoint_at = Instant::now() + .checked_sub(Duration::from_secs(10)) + .expect("backdate checkpoint attempt"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + assert!( + checkpoint_rx.try_recv().is_err(), + "expensive checkpoints must not rebuild at the old five-second floor" + ); + assert_eq!(scanner.last_checkpoint_objects, 0, "a deferred checkpoint must not consume progress"); + assert_eq!(scanner.last_checkpoint_build_time, Duration::from_secs(3)); + + scanner.last_checkpoint_at = Instant::now() + .checked_sub(Duration::from_secs(30)) + .expect("reach cost-adjusted interval"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + let snapshot = checkpoint_rx + .try_recv() + .expect("checkpoint is admitted at its adjusted interval"); + assert_eq!(snapshot.validated_scan_frontier(), Some("bucket/a")); + assert!(snapshot.checked_flatten_complete_scope("bucket").is_some()); + assert!(!snapshot.info.snapshot_complete); + assert!( + scanner.last_checkpoint_build_time <= scanner.last_checkpoint_at.elapsed(), + "record actual construction time" + ); + + scanner.last_checkpoint_build_time = Duration::MAX; + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_INTERVAL) + .expect("reach periodic deadline without another object batch"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + assert!( + checkpoint_rx.try_recv().is_ok(), + "even a costly snapshot retains the sixty-second periodic attempt" + ); + + scanner.new_cache.cache.get_mut("bucket/a").expect("fixture child").size = usize::MAX; + entry.size = 1; + scanner.last_checkpoint_build_time = Duration::from_secs(3); + scanner.last_checkpoint_at = Instant::now() + .checked_sub(SCANNER_CHECKPOINT_INTERVAL) + .expect("reach periodic overflow check"); + scanner.maybe_send_checkpoint(&folder, &hash, &entry); + assert!(checkpoint_rx.try_recv().is_err(), "backoff must not bypass overflow validation"); + assert!( + scanner.last_checkpoint_build_time <= scanner.last_checkpoint_at.elapsed(), + "failed construction must also update the cost estimate" + ); +} + #[tokio::test] async fn checkpoint_ancestor_depth_limit_preserves_suspended_accounting() { let (mut scanner, temp_dir) = build_test_scanner().await; @@ -2495,7 +2588,7 @@ async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() { .await .expect("cached metadata failure must still stop erasure data directory descent"); - assert_eq!(retry_into.failed_objects, 0, "cached failure should not be counted twice"); + assert_eq!(retry_into.failed_objects, 1, "a rebuilt entry must retain suppressed failure debt"); assert!(!retry_budget.budget_elapsed()); assert_eq!(retry_budget.reason(), None); @@ -2620,7 +2713,7 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() { .await .expect("cached missing metadata must still stop erasure data directory descent"); - assert_eq!(retry_into.failed_objects, 0, "cached failure should not be counted twice"); + assert_eq!(retry_into.failed_objects, 1, "a rebuilt entry must retain suppressed failure debt"); assert!(!retry_budget.budget_elapsed()); assert_eq!(retry_budget.reason(), None); } @@ -3846,6 +3939,15 @@ async fn test_scan_data_folder_success_clears_resume_hint() { #[tokio::test] #[serial] async fn test_scan_data_folder_keeps_unresolved_objects_partial() { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .json() + .with_max_level(tracing::Level::DEBUG) + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _subscriber_guard = tracing::subscriber::set_default(subscriber); let (scanner, temp_dir) = build_test_scanner().await; let _guard = TestGuard { temp_dir: Some(temp_dir.clone()), @@ -3896,6 +3998,18 @@ async fn test_scan_data_folder_keeps_unresolved_objects_partial() { "corrupt object failure should remain recorded: {:?}", partial.info.failed_objects ); + let events = logs + .contents() + .lines() + .map(|line| serde_json::from_str::(line).expect("scanner JSON event")) + .filter(|event| event["fields"]["state"] == "bucket_sweep_incomplete") + .collect::>(); + assert_eq!(events.len(), 1, "the incomplete sweep must identify its blocker"); + let fields = &events[0]["fields"]; + assert_eq!(fields["bucket"], "bucket"); + assert_eq!(fields["failed_objects"], 1); + assert_eq!(fields["mixed_coverage"], false); + assert!(!events[0].to_string().contains(&failed_path), "do not log individual object paths"); } #[tokio::test] diff --git a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs index bdda4458b..912e57983 100644 --- a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs +++ b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs @@ -31,6 +31,9 @@ const SOURCE: DataUsageCacheSource = DataUsageCacheSource::new(0, 0); const PLAN: DataUsageScanPlanDigest = DataUsageScanPlanDigest([17; 32]); fn scan_options(scan_mode: HealScanMode) -> ScannerDiskScanOptions { + // These fixtures walk thousands of synthetic objects through the real + // local-disk path; the per-disk I/O budget would only slow them down. + crate::SCANNER_SLEEPER.disable_disk_io_budget_for_tests(); ScannerDiskScanOptions { scan_mode, prefix_scan_scope: None, @@ -886,6 +889,25 @@ async fn checkpoint_fixture_hot_digest_retains_partial_progress() { #[tokio::test] #[serial] async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_end() { + hot_bucket_certifies_first_full_sweep_after_mixed_end(false).await; +} + +#[tokio::test] +#[serial] +async fn checkpoint_fixture_hot_bucket_certifies_after_mixed_end_and_leader_handoff() { + hot_bucket_certifies_first_full_sweep_after_mixed_end(true).await; +} + +async fn hot_bucket_certifies_first_full_sweep_after_mixed_end(advance_epoch: bool) { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .json() + .with_max_level(tracing::Level::DEBUG) + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _subscriber_guard = tracing::subscriber::set_default(subscriber); let (scanner, root) = build_test_scanner().await; let _guard = TestGuard { temp_dir: Some(root.clone()), @@ -906,6 +928,8 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en // tail that ends mixed, then fresh unbounded sweeps under new plans. let rounds = [(Some(4), false), (None, false), (None, true), (None, true)]; for (round, (max_objects, expect_complete)) in (0_u64..).zip(rounds) { + let cycle = 11 + round; + let leader_epoch = if advance_epoch { 7 + round } else { 7 }; write_checkpoint_object(&root, "hot/current", &[(None, 1)]).await; let plan = crate::scanner_io::checkpoint_fixture_bucket_digest(PLAN, Some(round)); let mut cache = DataUsageCache::default(); @@ -917,13 +941,14 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en &mut cache, "bucket", SOURCE, - 11, - 7, + cycle, + leader_epoch, plan, crate::scanner_io::DataUsageCacheReuseOptions { require_source: true, tier_registry_generation: Some(identity.tier_registry_generation), checkpoint_identity: Some(identity), + checkpoint_plan_digest: None, }, ); cache.info.skip_healing = true; @@ -961,6 +986,18 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en let saved = store.strict_load().await; assert_eq!(complete, expect_complete, "round {round} outcome"); assert_eq!(saved.info.snapshot_complete, expect_complete, "round {round} saved completion"); + assert_eq!(saved.info.next_cycle, cycle); + assert_eq!(saved.info.leader_epoch, leader_epoch); + if round == 1 { + assert!(saved.info.scan_resume_after.is_none()); + assert!(saved.info.scan_checkpoint.is_none()); + assert!(saved.info.scan_raw_enumeration_cursor.is_none()); + assert!(saved.info.scan_raw_enumeration_page_index.is_none()); + assert!(saved.info.scan_coverage_receipt.is_none()); + let progress = saved.info.scan_progress.expect("mixed tail retains a verification plan"); + assert_eq!(progress.started_plan, plan); + assert_eq!(progress.requested_plan, plan); + } if expect_complete { assert_eq!(saved.info.scan_plan_digest, Some(plan), "round {round} certifies its own plan"); assert!(saved.info.scan_progress.is_none()); @@ -970,6 +1007,53 @@ async fn checkpoint_fixture_hot_bucket_certifies_first_full_sweep_after_mixed_en assert!(saved.info.scan_plan_digest.is_none(), "round {round} must not certify"); } } + let captured = logs.contents(); + let all_events = captured + .lines() + .map(|line| serde_json::from_str::(line).expect("scanner JSON event")) + .collect::>(); + let admissions = all_events + .iter() + .filter(|event| event["fields"]["state"] == "bucket_checkpoint_prepared") + .collect::>(); + assert_eq!(admissions.len(), 4, "every sweep must identify fresh versus resumed admission"); + for (round, event) in admissions.iter().enumerate() { + let fields = &event["fields"]; + let round_cycle = 11 + u64::try_from(round).expect("round fits u64"); + assert_eq!(fields["cycle"], round_cycle); + assert_eq!(fields["leader_epoch"], if advance_epoch { round_cycle - 4 } else { 7 }); + assert_eq!(fields["previous_cycle"], if round == 0 { 0 } else { round_cycle - 1 }); + assert_eq!( + fields["previous_leader_epoch"], + if round == 0 { + 0 + } else if advance_epoch { + round_cycle - 5 + } else { + 7 + } + ); + assert_eq!(fields["resume_position"], round == 1); + assert_eq!(fields["mixed_coverage"], round == 1); + } + let events = all_events + .iter() + .filter(|event| event["fields"]["state"] == "bucket_sweep_incomplete") + .collect::>(); + assert_eq!(events.len(), 1, "only the mixed tail needs a verification-sweep diagnostic"); + let fields = &events[0]["fields"]; + assert_eq!(fields["bucket"], "bucket"); + assert_eq!(fields["pool"], 0); + assert_eq!(fields["set"], 0); + assert_eq!(fields["cycle"], 12); + assert_eq!(fields["leader_epoch"], if advance_epoch { 8 } else { 7 }); + assert_eq!(fields["resume_position_on_entry"], true); + assert_eq!(fields["resume_position"], false); + assert_eq!(fields["mixed_coverage"], true); + assert_eq!(fields["unresolved_objects"], false); + assert_eq!(fields["coverage_gap"], false); + assert_eq!(fields["failed_objects"], 0); + assert_eq!(fields["size_reconciliation_entries"], 0); } async fn write_checkpoint_object(root: &std::path::Path, object: &str, versions: &[(Option, i64)]) { @@ -1124,6 +1208,7 @@ where require_source: true, tier_registry_generation: None, checkpoint_identity: Some(identity), + checkpoint_plan_digest: None, }, ); let prepared = retained(&cache); @@ -1263,6 +1348,7 @@ where require_source: true, tier_registry_generation: Some(identity.tier_registry_generation), checkpoint_identity: Some(identity), + checkpoint_plan_digest: None, }, ); cache.info.skip_healing = true; diff --git a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs index c838d9fc4..93355f7d2 100644 --- a/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs +++ b/crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs @@ -22,6 +22,7 @@ const OBJECTS: u64 = 8; fn prepare_scanner(scanner: &mut FolderScanner, cache: DataUsageCache) { scanner.resume_frontier = cache.validated_scan_frontier().map(str::to_owned); scanner.coverage_frontier = scanner.resume_frontier.clone(); + scanner.resume_failed_paths = failed_scan_paths(&cache); scanner.new_cache = DataUsageCache { info: cache.info.clone(), ..Default::default() @@ -52,7 +53,10 @@ async fn nested_periodic_snapshot( if corrupt_first { write_test_object_metadata_bytes(&root, "bucket", "prefix/a/0000", b"not-valid-filemeta").await; } - let (_, identity) = bound_checkpoint(); + let identity = crate::DataUsageScanIdentity { + tier_registry_generation: crate::runtime_tier_registry_for_cycle(11, 7).await.generation, + ..bound_checkpoint().1 + }; let mut cache = DataUsageCache::default(); cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity); prepare_scanner(&mut scanner, cache); @@ -180,6 +184,8 @@ async fn finish_resumed_walk(cache: DataUsageCache, original: &FolderScanner) -> scanner.root = original.root.clone(); scanner.local_disk = original.local_disk.clone(); prepare_scanner(&mut scanner, cache); + scanner.failed_object_ttl_secs = original.failed_object_ttl_secs; + scanner.failed_objects_max = original.failed_objects_max; let parent = CancellationToken::new(); scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); let mut entry = DataUsageEntry::default(); @@ -199,6 +205,96 @@ async fn finish_resumed_walk(cache: DataUsageCache, original: &FolderScanner) -> (scanner.new_cache, visits, guard) } +#[tokio::test] +#[serial] +async fn periodic_mixed_tail_requires_terminal_persistence_before_fresh_certification() { + let (periodic, identity, scanner, _guard) = nested_periodic_snapshot(false, (0, 0)).await; + assert!(periodic.has_scan_position(), "real periodic snapshot retains traversal position"); + let store = FixtureStore::new(); + let revisions = DataUsageCache::default() + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("initial periodic fixture revisions"); + periodic + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("seed durable periodic snapshot"); + + for round in 0..3 { + let cycle = 12 + round; + let leader_epoch = 9 + round; + let plan = crate::scanner_io::checkpoint_fixture_bucket_digest(PLAN, Some(cycle)); + write_checkpoint_object(std::path::Path::new(&scanner.root), "prefix/b/0003", &[(None, 1)]).await; + let mut cache = DataUsageCache::default(); + let revisions = cache + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("reload last successful save"); + cache.prepare_bucket_checkpoint("bucket", cycle, leader_epoch, SOURCE, plan, identity); + assert_eq!( + cache.has_scan_position(), + round < 2, + "only a saved terminal clears the inherited position" + ); + assert_eq!( + cache + .info + .scan_progress + .is_some_and(|progress| progress.started_plan != progress.requested_plan), + round < 2, + "a surviving periodic prefix must remain mixed under a changed plan" + ); + cache.info.skip_healing = true; + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + let outcome = scanner + .local_disk + .clone() + .nsscanner_disk( + budget.token(), + budget.clone(), + vec![scanner.local_disk.clone()], + cache, + None, + scan_options(HealScanMode::Normal), + ) + .await + .expect("finish the periodic tail or fresh verification sweep"); + let (terminal, complete) = match outcome { + ScannerDiskScanOutcome::Partial(cache) => (cache, false), + ScannerDiskScanOutcome::Complete(cache) => (cache, true), + ScannerDiskScanOutcome::NamespaceNotFound(_) => panic!("fixture namespace exists"), + }; + assert!(!budget.budget_elapsed()); + assert_eq!(complete, round == 2); + assert!(!terminal.has_scan_position(), "completed traversal clears every position before saving"); + let totals = terminal.checked_flatten("bucket").expect("connected terminal root"); + assert_eq!(totals.objects, usize::try_from(OBJECTS).expect("fixture count")); + assert_eq!(totals.size, usize::try_from(OBJECTS).expect("fixture bytes")); + + store.reject_save.store(round == 0, Ordering::SeqCst); + let saved = terminal + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await; + if round == 0 { + assert!(saved.is_err(), "injected terminal persistence failure must be observable"); + let retained = store.strict_load().await; + assert_eq!(retained.info.scan_coverage_receipt, periodic.info.scan_coverage_receipt); + assert_eq!(retained.info.scan_checkpoint, periodic.info.scan_checkpoint); + assert!( + retained.has_scan_position(), + "failed terminal save must leave periodic recovery state intact" + ); + } else { + saved.expect("persist terminal sweep"); + let retained = store.strict_load().await; + assert!(!retained.has_scan_position()); + assert_eq!(retained.info.snapshot_complete, round == 2); + } + store.reject_save.store(false, Ordering::SeqCst); + } +} + #[tokio::test] #[serial] async fn periodic_nested_snapshot_resumes_after_leader_handoff_with_exact_totals() { @@ -216,20 +312,22 @@ async fn periodic_nested_snapshot_resumes_after_leader_handoff_with_exact_totals #[tokio::test] #[serial] -async fn periodic_nested_snapshot_keeps_failed_prefix_outside_resume_frontier() { +async fn periodic_nested_snapshot_resumes_healthy_progress_after_a_failed_object() { let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; - assert!( - loaded.validated_scan_frontier().is_none(), - "an earlier corrupt object must block subtree skipping" + assert_eq!( + loaded.validated_scan_frontier(), + Some("bucket/prefix/a/0001"), + "object health debt must not freeze completed traversal coverage" ); assert!(!loaded.info.failed_objects.is_empty(), "persisted failure debt must survive handoff"); let (finished, visits, _unused_guard) = finish_resumed_walk(loaded, &original).await; - assert_eq!(visits, OBJECTS - 1, "every healthy object must be re-read after the coverage gap"); + assert_eq!(visits, OBJECTS - 2, "the healthy covered sibling must skip its metadata read"); let totals = finished .checked_flatten_complete_scope("bucket") .expect("failed sweep still has connected accounting"); assert_eq!(u64::try_from(totals.objects).expect("healthy count"), OBJECTS - 1); assert_eq!(u64::try_from(totals.size).expect("healthy bytes"), OBJECTS - 1); + assert_eq!(totals.failed_objects, 1, "the retried failed subtree must retain its debt"); assert!(!finished.info.failed_objects.is_empty(), "unrepaired metadata must remain pending"); } @@ -792,3 +890,215 @@ async fn periodic_nested_snapshot_does_not_publish_shared_compacted_accumulator_ "a shared compacted parent accumulator must not be snapshotted as the current child" ); } + +#[tokio::test] +#[serial] +async fn periodic_failed_object_debt_survives_diagnostic_eviction_and_clears_after_repair() { + let (mut loaded, _, mut original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + loaded.info.failed_objects.clear(); + original.failed_object_ttl_secs = 0; + let (failed, visits, _unused_guard) = finish_resumed_walk(loaded.clone(), &original).await; + let totals = failed + .checked_flatten_complete_scope("bucket") + .expect("failed resumed accounting"); + assert_eq!(visits, OBJECTS - 2); + assert_eq!(totals.objects, usize::try_from(OBJECTS - 1).expect("healthy objects")); + assert_eq!(totals.failed_objects, 1, "uncapped entry debt must survive without diagnostic entries"); + assert!(failed.info.failed_objects.is_empty(), "TTL zero must not be needed for correctness"); + + write_checkpoint_object(std::path::Path::new(&original.root), "prefix/a/0000", &[(None, 1)]).await; + let (repaired, visits, _repair_guard) = finish_resumed_walk(loaded, &original).await; + let totals = repaired + .checked_flatten_complete_scope("bucket") + .expect("repaired resumed accounting"); + assert_eq!( + visits, + OBJECTS - 1, + "repair must reread the failed object and retain the covered healthy sibling" + ); + assert_eq!(totals.objects, usize::try_from(OBJECTS).expect("all objects")); + assert_eq!(totals.size, usize::try_from(OBJECTS).expect("all bytes")); + assert_eq!(totals.failed_objects, 0); +} + +#[tokio::test] +#[serial] +async fn periodic_failed_subtree_resumes_healthy_descendants_and_compacted_debt() { + let (loaded, identity, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + let (mut complete_walk, _, _unused_guard) = finish_resumed_walk(loaded, &original).await; + complete_walk.info.failed_objects.clear(); + complete_walk + .seal_scan_frontier(Some("bucket/prefix/a")) + .expect("seal completed unhealthy parent"); + for compacted in [false, true] { + let mut cache = complete_walk.clone(); + if compacted { + cache.reduce_children_of(&hash_path("bucket/prefix/a"), 1, true); + assert!(cache.find("bucket/prefix/a").expect("compacted parent").compacted); + cache + .seal_scan_frontier(Some("bucket/prefix/a")) + .expect("reseal compacted accounting"); + } + let store = FixtureStore::new(); + let revisions = DataUsageCache::default() + .load_with_revisions(store.clone(), CACHE_NAME) + .await + .expect("fresh revisions"); + cache + .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0) + .await + .expect("persist failure-bearing coverage"); + let mut adopted = DataUsageCache::default(); + adopted + .load(store, CACHE_NAME) + .await + .expect("reload failure-bearing coverage"); + assert_eq!( + adopted.prepare_bucket_checkpoint("bucket", 11, 9, SOURCE, PLAN, identity), + crate::DataUsageCachePrepareOutcome::Reused + ); + let (finished, visits, _resumed_guard) = finish_resumed_walk(adopted, &original).await; + assert_eq!( + visits, + if compacted { OBJECTS - 1 } else { 4 }, + "only healthy independently cached covered subtrees may skip reads" + ); + let totals = finished + .checked_flatten_complete_scope("bucket") + .expect("resumed failed parent accounting"); + assert_eq!(totals.objects, usize::try_from(OBJECTS - 1).expect("healthy objects")); + assert_eq!(totals.size, usize::try_from(OBJECTS - 1).expect("healthy bytes")); + assert_eq!(totals.failed_objects, 1); + } +} + +#[tokio::test] +#[serial] +async fn periodic_resume_uses_traversal_order_for_punctuation_siblings() { + let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + write_checkpoint_object(std::path::Path::new(&original.root), "prefix/a-", &[(None, 1)]).await; + let (mut cache, _, _walk_guard) = finish_resumed_walk(loaded, &original).await; + cache.info.failed_objects.clear(); + cache + .seal_scan_frontier(Some("bucket/prefix/a-")) + .expect("seal completed punctuation sibling"); + let (finished, visits, _resumed_guard) = finish_resumed_walk(cache, &original).await; + assert_eq!(visits, 4, "healthy children of the earlier failed sibling must remain covered"); + let totals = finished + .checked_flatten_complete_scope("bucket") + .expect("punctuation sibling accounting"); + assert_eq!( + totals.objects, + usize::try_from(OBJECTS).expect("healthy originals plus punctuation sibling") + ); + assert_eq!(totals.size, totals.objects); + assert_eq!(totals.failed_objects, 1); +} + +#[tokio::test] +#[serial] +async fn periodic_failed_nested_entries_cannot_certify_a_bucket_with_diagnostics_disabled() { + let logs = CapturedLogs::default(); + let subscriber = tracing_subscriber::fmt() + .json() + .with_max_level(tracing::Level::DEBUG) + .with_writer(logs.clone()) + .with_ansi(false) + .without_time() + .finish(); + let _subscriber_guard = tracing::subscriber::set_default(subscriber); + let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await; + let result = temp_env::async_with_vars([(ENV_FAILED_OBJECT_TTL_SECS, Some("0"))], async { + let mut cache = loaded; + cache.info.failed_objects.clear(); + cache.info.skip_healing = true; + let parent = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default()); + scan_data_folder( + parent, + budget, + vec![original.local_disk.clone()], + original.local_disk.clone(), + cache, + None, + HealScanMode::Normal, + DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest), + ) + .await + }) + .await; + let Err(ScannerError::PartialCache(cache)) = result else { + panic!("nested unresolved object must keep the bucket partial"); + }; + assert!(!cache.info.snapshot_complete); + assert!(cache.info.failed_objects.is_empty()); + assert_eq!( + cache + .checked_flatten_complete_scope("bucket") + .expect("partial accounting") + .failed_objects, + 1 + ); + let events = logs + .contents() + .lines() + .map(|line| serde_json::from_str::(line).expect("scanner JSON event")) + .filter(|event| event["fields"]["state"] == "bucket_sweep_incomplete") + .collect::>(); + assert_eq!(events.len(), 1, "the completed sweep must identify its uncapped failure debt"); + let fields = &events[0]["fields"]; + assert_eq!(fields["coverage_gap"], false); + assert_eq!(fields["root_failed_objects"], 0); + assert_eq!(fields["failed_objects"], 0); + assert_eq!(fields["size_reconciliation_entries"], 0); + assert_eq!(fields["mixed_coverage"], false); + assert_eq!( + fields["unresolved_objects"], true, + "child-entry failure debt remains a certification blocker" + ); +} + +#[tokio::test] +#[serial] +async fn periodic_resume_debt_index_and_frontier_keep_subtree_boundaries() { + let (mut scanner, root) = build_test_scanner().await; + let _guard = TestGuard { temp_dir: Some(root) }; + scanner.resume_failed_paths = vec!["bucket/a-".into(), "bucket/a/z".into(), "bucket/b".into()]; + assert!( + scanner.resume_subtree_has_failures("bucket/a"), + "punctuation sibling must not hide a failing descendant" + ); + assert!(scanner.resume_subtree_has_failures("bucket/b")); + assert!(!scanner.resume_subtree_has_failures("bucket/ab")); + assert!(!scanner.resume_subtree_has_failures("bucket/a/healthy")); + let (cache, _) = bound_checkpoint(); + prepare_scanner(&mut scanner, cache); + scanner.coverage_frontier = Some("bucket/prefix/a".into()); + scanner.record_completed_child("bucket/prefix/a/0000"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/prefix/a"), + "debt retry must not narrow inherited coverage" + ); + scanner.record_completed_child("bucket/prefix"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/prefix"), + "completed ancestor widens coverage" + ); + scanner.coverage_frontier = Some("bucket/a-".into()); + assert!(scan_frontier_covers_path("bucket/a/healthy", "bucket/a-")); + scanner.record_completed_child("bucket/a/healthy"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/a-"), + "component order must preserve a later sibling's frontier" + ); + scanner.coverage_gap = true; + scanner.record_completed_child("bucket/z"); + assert_eq!( + scanner.coverage_frontier.as_deref(), + Some("bucket/a-"), + "structural gaps must still freeze coverage" + ); +} diff --git a/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs b/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs index 9390306db..56fc7be53 100644 --- a/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs +++ b/crates/scanner/src/scanner_folder/tests/incremental_enumeration.rs @@ -91,3 +91,42 @@ fn raw_enumeration_progress_invalid_entry_discards_index_but_retains_diagnostic_ assert!(progress.page_index().is_none()); assert_eq!(progress.cursor().expect("diagnostic cursor").entries_seen, 3); } + +// A flat prefix of tens of thousands of keys must cost the walker a bounded +// amount of bookkeeping per entry. The pre-incremental index re-normalized +// the cumulative entry list on every entry (rustfs#8113: 8,000 entries took +// about 40 s); the page writer hashes each entry once when its page commits. +#[test] +fn raw_enumeration_progress_record_entry_stays_linear_for_large_prefixes() { + const ENTRIES: usize = 20_000; + let committed_page_entries = ENTRIES - ENTRIES % SCANNER_RAW_ENUMERATION_PAGE_ENTRY_LIMIT; + let mut progress = RawEnumerationProgress::new("bucket", None); + RAW_PAGE_DIGEST_ENTRIES.set(0); + let start = Instant::now(); + // Reverse order is the worst case for the sorted insert into the building page. + for i in (0..ENTRIES).rev() { + progress.record_entry(&format!("entry-{i:06}")); + } + let elapsed = start.elapsed(); + assert_eq!(progress.checkpointable_entry_count(), ENTRIES); + assert_eq!( + RAW_PAGE_DIGEST_ENTRIES.get(), + committed_page_entries, + "each entry is hashed exactly once, when its page commits" + ); + assert!( + elapsed < Duration::from_secs(1), + "recording {ENTRIES} entries took {elapsed:?}; per-entry bookkeeping must stay amortised constant" + ); + let checkpoint = progress.page_index().expect("checkpoint with committed pages"); + assert_eq!( + RAW_PAGE_DIGEST_ENTRIES.get(), + ENTRIES, + "a checkpoint hashes only the building page it commits" + ); + assert_eq!(checkpoint.indexed_entries().expect("validated checkpoint").len(), ENTRIES); + assert_eq!( + progress.cursor().expect("cursor").entries_seen, + u64::try_from(ENTRIES).expect("entry count fits") + ); +} diff --git a/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs b/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs index 3e7f9f570..a166a20ef 100644 --- a/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs +++ b/crates/scanner/src/scanner_folder/tests/mrf_ownership.rs @@ -243,6 +243,13 @@ impl HealStorageAPI for NoticeStorage { async fn bucket_incarnation_id(&self, _: &str) -> rustfs_heal::Result> { Ok(Some(self.bucket_incarnation_id)) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(Vec::new()) } diff --git a/crates/scanner/src/scanner_io.rs b/crates/scanner/src/scanner_io.rs index 1af655211..094ab0785 100644 --- a/crates/scanner/src/scanner_io.rs +++ b/crates/scanner/src/scanner_io.rs @@ -26,7 +26,7 @@ use futures::future::join_all; use metrics::counter; use rand::seq::SliceRandom as _; #[cfg(test)] -use rustfs_config::{ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS}; +use rustfs_config::{ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, ENV_SCANNER_WALKERS_PER_DISK}; use rustfs_data_usage::{BucketTargetUsageInfo, BucketUsageInfo, observed_data_usage_is_newer}; use rustfs_filemeta::FileMeta; use rustfs_heal_contracts::heal_channel::HealScanMode; @@ -39,7 +39,7 @@ use s3s::dto::{ BucketLifecycleConfiguration, ObjectLockConfiguration, ObjectLockEnabled, ReplicationConfiguration, VersioningConfiguration, }; use sha2::{Digest as _, Sha256}; -use std::collections::{BTreeMap, HashMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::future::Future; use std::path::Path; use std::pin::Pin; @@ -409,14 +409,26 @@ fn resolve_remote_dirty_usage_scope( } } +/// `expected_converged` is `None` for the authoritative object: a `Complete` +/// cycle with pending buckets publishes there unconverged and is still a +/// structural baseline, because the buckets it could not certify stay in the +/// dirty map and are rescanned rather than reused. The convergence marker +/// must still be present; a snapshot without one was not written by a +/// certifying scanner. fn complete_scanner_cache_snapshot_plan_digest( snapshot: &DataUsageInfo, proof: ScannerCacheBaselineProof<'_>, - expected_converged: bool, + expected_converged: Option, ) -> Option { + // The observational mark disqualifies only the authoritative slot: an + // observed candidate carries it by construction. + let marks_accepted = match expected_converged { + Some(expected) => snapshot.usage_snapshot_converged == Some(expected), + None => snapshot.usage_snapshot_converged.is_some() && !snapshot.usage_snapshot_observational, + }; if !snapshot.is_complete_bucket_usage_snapshot() || snapshot.usage_snapshot_partial - || snapshot.usage_snapshot_converged != Some(expected_converged) + || !marks_accepted || snapshot.scanner_epoch != Some(proof.leader_epoch) || snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len() { @@ -426,13 +438,7 @@ fn complete_scanner_cache_snapshot_plan_digest( // Completed maintenance also covers ordinary usage. Keep its exact stored // proof for cache reuse, and reject mixtures of different set work proofs. let baseline_plan_digest = DataUsageScanPlanDigest(snapshot.usage_snapshot_set_states.first()?.scan_plan_digest?); - if ![ - proof.scan_plan_digest, - scanner_bucket_work_digest(proof.scan_plan_digest, HealScanMode::Normal, true), - scanner_bucket_work_digest(proof.scan_plan_digest, HealScanMode::Deep, true), - ] - .contains(&baseline_plan_digest) - { + if !scanner_plan_digest_is_work_variant(proof.scan_plan_digest, baseline_plan_digest) { return None; } let mut states = HashSet::with_capacity(snapshot.usage_snapshot_set_states.len()); @@ -453,9 +459,35 @@ fn complete_scanner_cache_snapshot_plan_digest( (states == *proof.expected_sources).then_some(baseline_plan_digest) } +/// Whether `digest` is `structural` under any work flag: the same bucket +/// list, topology and movement state walked as a scoped, full or Deep sweep. +/// The flag changes what a walk must visit, not the namespace it describes. +fn scanner_plan_digest_is_work_variant(structural: DataUsageScanPlanDigest, digest: DataUsageScanPlanDigest) -> bool { + [ + structural, + scanner_bucket_work_digest(structural, HealScanMode::Normal, true), + scanner_bucket_work_digest(structural, HealScanMode::Deep, true), + ] + .contains(&digest) +} + +/// Buckets the last authoritative publication left pending. A `Complete` +/// cycle names every bucket whose value it could not certify, including a +/// partial walk that published the bucket's last certified value, so the +/// next scoped cycle must walk them even when the dirty map no longer names +/// them. Over-inclusion only costs a walk; a missing baseline contributes +/// nothing and the dirty map decides alone. +fn scanner_baseline_pending_buckets(proof: &ScannerCacheBaselineProof<'_>) -> Vec { + proof + .authoritative_data + .and_then(|data| serde_json::from_slice::(data).ok()) + .map(|snapshot| snapshot.usage_snapshot_pending_buckets) + .unwrap_or_default() +} + fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option { let authoritative = serde_json::from_slice::(proof.authoritative_data?).ok()?; - if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, true) { + if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) { return Some(validated_digest); } @@ -464,7 +496,7 @@ fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof< // remains observational: this proof grants bucket-scope reuse only and // never changes authoritative usage publication or dirty acknowledgement. let authoritative_has_identity = (crate::scanner::data_usage_info_has_persisted_baseline_identity(&authoritative) - && authoritative.usage_snapshot_converged != Some(false)) + && !authoritative.usage_snapshot_observational) || crate::scanner::data_usage_info_is_bootstrap_pending(&authoritative); if !authoritative_has_identity { return None; @@ -474,7 +506,7 @@ fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof< return None; } - complete_scanner_cache_snapshot_plan_digest(&observed, proof, false) + complete_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false)) } fn scanner_segment_reuse_baseline_producer_evidence( @@ -492,7 +524,9 @@ fn scanner_segment_reuse_baseline_producer_evidence( let Ok(authoritative) = serde_json::from_slice::(authoritative_data) else { return (evidence, false); }; - if complete_scanner_cache_snapshot_plan_digest(&authoritative, baseline_proof, true).is_none() { + // Segment (prefix) reuse keeps requiring a converged baseline: its + // producer evidence is proven against a dirty snapshot that was current. + if complete_scanner_cache_snapshot_plan_digest(&authoritative, baseline_proof, Some(true)).is_none() { return (evidence, false); } let Some(cold_zero_walk_oracle) = @@ -550,6 +584,7 @@ fn scoped_scan_scope_from_dirty_buckets( let current_buckets = all_buckets.iter().map(|bucket| bucket.name.as_str()).collect::>(); let selected_buckets = dirty_buckets .into_iter() + .chain(scanner_baseline_pending_buckets(&baseline_proof)) .filter(|bucket| current_buckets.contains(bucket.as_str())) .collect::>(); if selected_buckets.is_empty() { @@ -807,6 +842,12 @@ pub struct ScannerBucketScanPlan { digest: DataUsageScanPlanDigest, /// Includes mutation generations even when the set planner uses a structural digest. bucket_coverage_digest: DataUsageScanPlanDigest, + /// The plan a bucket sweep is started and resumed under: the coverage + /// identity (`scanner_activity_coverage_digest`) with the bucket list. + /// It leaves out what `bucket_coverage_digest` binds for reuse, so a + /// coordinator restart or a maintenance publication does not turn a + /// resumed sweep into mixed coverage. + checkpoint_coverage_digest: DataUsageScanPlanDigest, requires_full_scan: bool, service_cohort: Option>>, // Cache work must invalidate on namespace completion even when its scoped baseline remains reusable. @@ -829,10 +870,13 @@ pub struct ScannerBucketScanPlan { struct ScannerBucketFailureState { hard: Arc>>, partial: Arc>>, + /// Partial buckets whose set scan published the bucket's last certified + /// value in place of the unfinished walk; a subset of `partial`. + partial_carried: Arc>>, namespace_not_found: Arc>>, } -fn scanner_bucket_plan_digest(buckets: &[BucketInfo], activity_digest: [u8; 32]) -> DataUsageScanPlanDigest { +pub(crate) fn scanner_bucket_plan_digest(buckets: &[BucketInfo], activity_digest: [u8; 32]) -> DataUsageScanPlanDigest { let mut buckets = buckets.iter().collect::>(); buckets.sort_unstable_by(|left, right| left.name.cmp(&right.name)); @@ -939,22 +983,50 @@ fn finalize_nsscanner_result(results: &[DataUsageCache], first_err: Option ScannerBucketScanStatus { +fn scanner_bucket_scan_status( + has_failed: bool, + has_uncarried_partial: bool, + has_namespace_not_found: bool, + has_carried_partial: bool, +) -> ScannerBucketScanStatus { if has_failed { ScannerBucketScanStatus::Failed - } else if has_partial { + } else if has_uncarried_partial { ScannerBucketScanStatus::Partial } else if has_namespace_not_found { ScannerBucketScanStatus::NamespaceNotFound + } else if has_carried_partial { + ScannerBucketScanStatus::PartialCarried } else { ScannerBucketScanStatus::Complete } } +fn scanner_partial_progress_is_usable( + bucket_plan_complete: bool, + scan_scope_matches: bool, + bucket_status: ScannerBucketScanStatus, + has_set_failure: bool, + has_missing_namespace: bool, +) -> bool { + bucket_plan_complete + && scan_scope_matches + && matches!(bucket_status, ScannerBucketScanStatus::Partial | ScannerBucketScanStatus::PartialCarried) + && !has_set_failure + && !has_missing_namespace +} + fn scanner_cycle_cold_zero_walk_oracle( scan_scope: &ScannerBucketScanScope, all_buckets: &[BucketInfo], @@ -989,9 +1061,16 @@ fn classify_nsscanner_cycle( if activity_status == ScannerCycleActivityStatus::Unverified { return ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable); } + // A bucket walk that stopped early keeps the cycle incomplete only when + // no certified value stands in for it (`Partial`). `PartialCarried` + // means every such bucket published its last certified value and is + // reported pending; `completed_all_sets` proves the sets agreed. if budget_elapsed || cancelled - || !matches!(bucket_scan_status, ScannerBucketScanStatus::Complete) + || !matches!( + bucket_scan_status, + ScannerBucketScanStatus::Complete | ScannerBucketScanStatus::PartialCarried + ) || dirty_usage_status == DirtyUsageSnapshotStatus::Unverified { return ScannerCycleStatus::Incomplete; @@ -1000,9 +1079,14 @@ fn classify_nsscanner_cycle( return ScannerCycleStatus::Incomplete; } - match (activity_status, dirty_usage_status) { - (ScannerCycleActivityStatus::Unchanged, DirtyUsageSnapshotStatus::Current) => ScannerCycleStatus::Complete, - _ => ScannerCycleStatus::Superseded, + // Termination is snapshot-at-the-beginning: a structurally complete walk + // that the guardrail did not veto is `Complete`. Buckets whose dirty + // generation moved during the walk, and buckets whose walk stopped early + // behind a certified value, stay pending for the next cycle and are + // reported in the publication; they no longer supersede the cycle. + match activity_status { + ScannerCycleActivityStatus::Unchanged => ScannerCycleStatus::Complete, + ScannerCycleActivityStatus::Changed | ScannerCycleActivityStatus::Unverified => ScannerCycleStatus::Superseded, } } @@ -1014,24 +1098,52 @@ fn should_publish_observational_snapshot(status: ScannerCycleStatus) -> bool { matches!(status, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable)) } +/// Pending buckets of a cycle, sorted so the publication is deterministic. +pub(crate) type ScannerCyclePendingBuckets = BTreeSet; + +/// Stamp the certification outcome of a cycle on its usage snapshot. A +/// `Complete` cycle publishes an authoritative snapshot that is converged +/// only when no bucket is pending; any other publishable status is an +/// observational snapshot routed to the observed object. The marks are part +/// of the bytes the publication expectation digests, so every writer applies +/// exactly this function. +fn apply_scanner_cycle_publication_marks( + data_usage_info: &mut DataUsageInfo, + status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, +) { + let complete = status == ScannerCycleStatus::Complete; + data_usage_info.usage_snapshot_observational = !complete; + data_usage_info.usage_snapshot_converged = Some(complete && pending_buckets.is_empty()); + if complete { + data_usage_info.usage_snapshot_pending_buckets = pending_buckets.iter().cloned().collect(); + data_usage_info.usage_snapshot_pending_bucket_count = u64::try_from(pending_buckets.len()).unwrap_or(u64::MAX); + } else { + data_usage_info.usage_snapshot_pending_buckets.clear(); + data_usage_info.usage_snapshot_pending_bucket_count = 0; + } +} + fn prepare_usage_snapshot_for_publication( status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, mut data_usage_info: DataUsageInfo, ) -> Option { if !should_publish_usage_snapshot(status) { return None; } - data_usage_info.usage_snapshot_converged = Some(status == ScannerCycleStatus::Complete); + apply_scanner_cycle_publication_marks(&mut data_usage_info, status, pending_buckets); Some(data_usage_info) } async fn publish_usage_snapshot( updates: &mpsc::Sender, status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, data_usage_info: DataUsageInfo, ) -> Result { - let Some(data_usage_info) = prepare_usage_snapshot_for_publication(status, data_usage_info) else { + let Some(data_usage_info) = prepare_usage_snapshot_for_publication(status, pending_buckets, data_usage_info) else { return Ok(false); }; send_data_usage_update(updates, data_usage_info).await?; @@ -1045,10 +1157,17 @@ async fn publish_observational_snapshot( data_usage_info.usage_snapshot_complete = false; data_usage_info.usage_snapshot_partial = true; data_usage_info.usage_snapshot_converged = Some(false); + data_usage_info.usage_snapshot_observational = true; + data_usage_info.usage_snapshot_pending_buckets.clear(); + data_usage_info.usage_snapshot_pending_bucket_count = 0; send_data_usage_update(updates, data_usage_info).await?; Ok(true) } +/// Post-walk verdict of the cluster activity guardrail. `Changed` is an +/// attributed veto (or, for peers, the phase-2 dirty-usage fallback); local +/// namespace and dirty-usage generation movement is not a veto and is +/// certified per bucket instead. #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum ScannerCycleActivityStatus { Unchanged, @@ -1064,15 +1183,54 @@ async fn scanner_cycle_activity_status( where S: ScannerStorage, { - // Read the pending-commit barrier before sampling its completion generation. - // A tail that drains during this await must invalidate the earlier baseline. + // Read the storage movement barrier before sampling the activity + // generations. A movement that starts during this await must invalidate + // the earlier baseline. Ordinary PUT tails are not part of the barrier: + // they shade their bucket and are certified per bucket below. let publication_blocked = store.scanner_data_usage_publication_blocked().await; match crate::scanner::probe_scanner_activity(store, distributed).await { Ok(after) => { - let status = if !publication_blocked && after == *before { - ScannerCycleActivityStatus::Unchanged + let veto = if publication_blocked { + Some(crate::scanner::ScannerActivityGuardrailVeto::new( + crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE, + "publication_blocked", + )) } else { + crate::scanner::scanner_activity_guardrail_veto(before, &after) + }; + let status = if let Some(veto) = veto { + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_SET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "cycle_guardrail_veto", + stage = "post_scan_probe", + field = veto.field, + host = %veto.host, + "Scanner cycle publication vetoed by the cluster activity guardrail" + ); + global_metrics().record_scanner_guardrail_veto(veto.reason()); ScannerCycleActivityStatus::Changed + } else if let Some(moved) = crate::scanner::scanner_peer_dirty_usage_moved(before, &after) { + // Per-bucket certification covers the local dirty map only; + // a peer whose dirty state moved keeps today's whole-cycle + // outcome until scoped acknowledgements can name its buckets. + tracing::info!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_SET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "cycle_peer_dirty_usage_moved", + stage = "post_scan_probe", + field = moved.field, + host = %moved.host, + "Scanner cycle superseded because a peer's dirty usage moved during the walk" + ); + global_metrics().record_scanner_guardrail_veto(moved.reason()); + ScannerCycleActivityStatus::Changed + } else { + ScannerCycleActivityStatus::Unchanged }; (status, crate::scanner::scanner_activity_publication_lease_targets(&after)) } @@ -1134,12 +1292,30 @@ pub(crate) fn cache_root_entry_info(cache: &DataUsageCache) -> std::result::Resu }) } -fn apply_bucket_result_to_cache(cache: &mut DataUsageCache, result: DataUsageEntryInfo, update_time: SystemTime) -> bool { +fn apply_bucket_result_to_cache( + cache: &mut DataUsageCache, + result: DataUsageEntryInfo, + update_time: SystemTime, +) -> Option { if cache.info.tier_registry_generation != result.tier_registry_generation { // A result from another registry generation must never be folded into // this cycle. Leaving it unapplied makes the cycle incomplete and // forces the caller to re-account it under one frozen registry. - return false; + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_CACHE_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "bucket_result_rejected", + bucket = %result.name, + pool = cache.info.source.map(|source| source.pool_index), + set = cache.info.source.map(|source| source.set_index), + cycle = cache.info.next_cycle, + expected_tier_registry_generation = cache.info.tier_registry_generation, + result_tier_registry_generation = result.tier_registry_generation, + "Scanner bucket result rejected because the tier registry generation differs" + ); + return None; } match result.bucket_incarnation { Some(incarnation) if !incarnation.is_nil() => { @@ -1151,7 +1327,7 @@ fn apply_bucket_result_to_cache(cache: &mut DataUsageCache, result: DataUsageEnt } cache.replace(&result.name, &result.parent, result.entry); cache.info.last_update = Some(update_time); - true + Some(result.name) } fn should_publish_completed_snapshot(completed_count: usize, total_count: usize, budget_elapsed: bool, cancelled: bool) -> bool { @@ -1182,6 +1358,42 @@ fn namespace_scanner_workers( workers } +struct NamespaceScannerWalker { + disk: T, + mode: NamespaceScannerWorkerMode, + /// Shared by every walker of one local disk; `None` for disks the + /// coordinator reaches over the network, which keep today's one walker. + local_admission: Option>, +} + +/// Expands each worker into its bucket walkers. A local coordinator disk gets +/// `walkers_per_disk` walkers sharing one disk admission; every other worker +/// keeps a single walker, so the walker count never drops below the disk count. +fn namespace_scanner_walkers( + workers: Vec<(T, NamespaceScannerWorkerMode)>, + walkers_per_disk: usize, + is_local: impl Fn(&T) -> bool, +) -> Vec> { + let mut walkers = Vec::with_capacity(workers.len()); + for (disk, mode) in workers { + if mode == NamespaceScannerWorkerMode::Coordinator && is_local(&disk) { + let local_admission = crate::remote_scanner::LocalDiskAdmissionShare::new(); + walkers.extend((0..walkers_per_disk.max(1)).map(|_| NamespaceScannerWalker { + disk: disk.clone(), + mode, + local_admission: Some(local_admission.clone()), + })); + } else { + walkers.push(NamespaceScannerWalker { + disk, + mode, + local_admission: None, + }); + } + } + walkers +} + fn group_remote_disks_by_peer(disks: Vec, peer_key: impl Fn(&T) -> String) -> Vec> { let mut groups = HashMap::>::new(); for disk in disks { @@ -1344,8 +1556,10 @@ pub(crate) async fn scanner_bucket_checkpoint_identity( publication_epoch: u64, tier_registry_generation: u64, scan_mode: HealScanMode, -) -> Result { - let bucket_incarnation = set.bucket_incarnation_id_from_disk(bucket).await?; +) -> Result> { + let Some(bucket_incarnation) = set.find_bucket_incarnation_id_from_disk(bucket).await? else { + return Ok(None); + }; let disks = set .format .erasure @@ -1361,14 +1575,14 @@ pub(crate) async fn scanner_bucket_checkpoint_identity( for disk in disks { digest.update(disk.as_bytes()); } - Ok(crate::DataUsageScanIdentity { + Ok(Some(crate::DataUsageScanIdentity { version: 1, bucket_incarnation, set_layout: crate::DataUsageScanPlanDigest(digest.finalize().into()), publication_epoch, tier_registry_generation, scan_mode, - }) + })) } #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -1426,7 +1640,10 @@ pub(crate) struct ScannerCycleResult { pub(crate) status: ScannerCycleStatus, publication_epoch: Option, activity_digest: Option<[u8; 32]>, - observational_snapshot_published: bool, + activity_baseline: Option>, + pending_dirty_usage_buckets: usize, + usage_update_emitted: bool, + partial_bucket_progress: bool, dirty_usage_clear: Option, remote_dirty_usage_acknowledgements: Vec, distributed_segment_invalidation_evidence: Option, @@ -1444,7 +1661,10 @@ impl ScannerCycleResult { status, publication_epoch: None, activity_digest: None, - observational_snapshot_published: false, + activity_baseline: None, + pending_dirty_usage_buckets: 0, + usage_update_emitted: false, + partial_bucket_progress: false, dirty_usage_clear, remote_dirty_usage_acknowledgements: Vec::new(), distributed_segment_invalidation_evidence: None, @@ -1473,18 +1693,52 @@ impl ScannerCycleResult { self } + #[cfg(test)] pub(crate) fn activity_digest(&self) -> Option<[u8; 32]> { self.activity_digest } - pub(crate) fn with_observational_snapshot_published(mut self, published: bool) -> Self { + fn with_activity_baseline(mut self, baseline: Arc) -> Self { self.publication_expectation = None; - self.observational_snapshot_published = published; + self.activity_baseline = Some(baseline); self } - pub(crate) fn has_observational_snapshot(&self) -> bool { - self.observational_snapshot_published + /// The pre-walk activity probe the post-lease guardrail recheck compares + /// against. + pub(crate) fn activity_baseline(&self) -> Option<&crate::scanner::ScannerActivitySnapshot> { + self.activity_baseline.as_deref() + } + + fn with_pending_dirty_usage_buckets(mut self, pending: usize) -> Self { + self.publication_expectation = None; + self.pending_dirty_usage_buckets = pending; + self + } + + /// Buckets a `Complete` cycle left pending for the next cycle. + pub(crate) fn pending_dirty_usage_buckets(&self) -> usize { + self.pending_dirty_usage_buckets + } + + pub(crate) fn with_usage_update_emitted(mut self, emitted: bool) -> Self { + self.publication_expectation = None; + self.usage_update_emitted = emitted; + self + } + + pub(crate) fn has_usage_update(&self) -> bool { + self.usage_update_emitted + } + + pub(crate) fn with_partial_bucket_progress(mut self, partial_bucket_progress: bool) -> Self { + self.publication_expectation = None; + self.partial_bucket_progress = partial_bucket_progress; + self + } + + pub(crate) fn has_partial_bucket_progress(&self) -> bool { + self.partial_bucket_progress } fn with_failed_dirty_usage(mut self, failed_dirty_usage: bool) -> Self { @@ -1611,7 +1865,8 @@ use guards::*; pub(crate) use cache::{ DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult, - acquire_scanner_cache_locks, current_cache_root_or_prepare_with_generation, persist_scanner_checkpoint, + acquire_scanner_cache_locks, current_cache_root_or_prepare_with_generation, log_saved_partial_bucket_cache, + persist_scanner_checkpoint, }; pub use dirty_usage::{ ScannerDirtyUsageAckError, ScannerDirtyUsageBucket, ScannerDirtyUsageClearObserver, ScannerDirtyUsageMutationObserver, @@ -1620,9 +1875,9 @@ pub use dirty_usage::{ acknowledge_scoped_dirty_usage, clear_dirty_usage_bucket, encode_durable_dirty_usage_producer_replay_record, record_dirty_usage_bucket, record_dirty_usage_bucket_from_producer, record_dirty_usage_bucket_from_producers, record_dirty_usage_object, record_dirty_usage_object_from_producer, record_scanner_maintenance_change, - replay_durable_dirty_usage_producer_record, scanner_activity_epoch, scanner_dirty_usage_bucket_generation, - scanner_dirty_usage_snapshot, scanner_dirty_usage_state, scanner_maintenance_generation, - set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, + record_status_class_mutation, replay_durable_dirty_usage_producer_record, scanner_activity_epoch, + scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state, + scanner_maintenance_generation, set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer, }; #[cfg(test)] pub(crate) use dirty_usage::{clear_dirty_usage_buckets_for_tests, dirty_usage_buckets_for_tests}; diff --git a/crates/scanner/src/scanner_io/cache.rs b/crates/scanner/src/scanner_io/cache.rs index ff2da937c..df5caf093 100644 --- a/crates/scanner/src/scanner_io/cache.rs +++ b/crates/scanner/src/scanner_io/cache.rs @@ -15,6 +15,24 @@ use super::*; use crate::{ScannerConfigObjectDelete, ScannerObjectIO}; +pub(crate) fn log_saved_partial_bucket_cache(cache: &DataUsageCache, cache_name: &str) { + debug!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_CACHE_PERSIST_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "partial_saved_not_published", + bucket = %cache.info.name, + pool = cache.info.source.map(|source| source.pool_index), + set = cache.info.source.map(|source| source.set_index), + cycle = cache.info.next_cycle, + leader_epoch = cache.info.leader_epoch, + cache_name, + resume_position = cache.has_scan_position(), + "Scanner partial bucket cache saved without publishing usage aggregate" + ); +} + pub(crate) fn scanner_cache_lock_resource(cache_name: &str, source: DataUsageCacheSource) -> String { let lock_name = format!("{SCANNER_CACHE_LOCK_SUFFIX}.pool-{}.set-{}", source.pool_index, source.set_index); path_join_buf(&[crate::BUCKET_META_PREFIX, cache_name, &lock_name]) @@ -268,6 +286,10 @@ pub(crate) struct DataUsageCacheReuseOptions { pub(crate) require_source: bool, pub(crate) tier_registry_generation: Option, pub(crate) checkpoint_identity: Option, + /// The coverage plan a sweep is started and resumed under + /// (`scanner_activity_coverage_digest`); `None` resumes under the reuse + /// plan itself, as a remote worker does with the single digest on the wire. + pub(crate) checkpoint_plan_digest: Option, } #[cfg(test)] @@ -291,6 +313,7 @@ pub(crate) fn current_cache_root_or_prepare( require_source, tier_registry_generation: None, checkpoint_identity: None, + checkpoint_plan_digest: None, }, ) } @@ -304,6 +327,7 @@ pub(crate) fn current_cache_root_or_prepare_with_generation( scan_plan_digest: DataUsageScanPlanDigest, options: DataUsageCacheReuseOptions, ) -> DataUsageCacheScanState { + let checkpoint_plan_digest = options.checkpoint_plan_digest.unwrap_or(scan_plan_digest); if cache.info.next_cycle <= next_cycle && cache.info.leader_epoch <= leader_epoch && cache.info.scan_identity != options.checkpoint_identity @@ -319,6 +343,24 @@ pub(crate) fn current_cache_root_or_prepare_with_generation( // an older registry cannot be reused under the new cycle generation. cache.info.scan_plan_digest = None; } + if cache.info.name == name + && cache.info.next_cycle <= next_cycle + && cache.info.leader_epoch <= leader_epoch + && cache.info.snapshot_complete + && cache + .info + .replication + .as_ref() + .is_some_and(|config| config.has_active_rules("", true)) + { + // Replication writebacks leave namespace generations unchanged. A + // completed cache must start a fresh coverage sweep, even on retry. + cache.info.scan_progress = Some(crate::DataUsageScanProgress { + started_plan: checkpoint_plan_digest, + requested_plan: checkpoint_plan_digest, + }); + cache.info.scan_plan_digest = None; + } match current_cache_root_entry_with_generation( cache, name, @@ -332,9 +374,15 @@ pub(crate) fn current_cache_root_or_prepare_with_generation( current => DataUsageCacheScanState::Prepared { invalid_current: current.err(), outcome: match options.checkpoint_identity.filter(crate::DataUsageScanIdentity::is_valid) { - Some(identity) => { - cache.prepare_bucket_checkpoint(name, next_cycle, leader_epoch, source, scan_plan_digest, identity) - } + Some(identity) => cache.prepare_bucket_checkpoint_with_coverage( + name, + next_cycle, + leader_epoch, + source, + scan_plan_digest, + checkpoint_plan_digest, + identity, + ), None => cache.prepare_for_scan(name, next_cycle, leader_epoch, source, scan_plan_digest, options.require_source), }, }, @@ -356,6 +404,96 @@ pub(super) fn cache_snapshot_is_current( ) } +/// A set cache that stands in for buckets whose walk stopped early is not +/// the execution a full walk of the same cycle produces. Bind the carried +/// bucket names so neither snapshot is adopted in place of the other by the +/// same-execution reuse in `persist_and_publish_cache_snapshot`. +pub(super) fn scanner_carried_execution_digest( + execution_digest: DataUsageScanPlanDigest, + carried_buckets: &[String], +) -> DataUsageScanPlanDigest { + if carried_buckets.is_empty() { + return execution_digest; + } + let mut carried = carried_buckets.to_vec(); + carried.sort_unstable(); + let mut hasher = Sha256::new(); + hasher.update(b"scanner-carried-execution-v1"); + hasher.update(execution_digest.0); + hasher.update(u64::try_from(carried.len()).unwrap_or(u64::MAX).to_be_bytes()); + for bucket in carried { + hasher.update(u64::try_from(bucket.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(bucket.as_bytes()); + } + DataUsageScanPlanDigest(hasher.finalize().into()) +} + +/// Replication statistics are observational and may change within one +/// namespace execution. Bind them without relaxing same-execution reuse. +pub(super) fn scanner_replication_execution_digest( + execution_digest: DataUsageScanPlanDigest, + cache: &DataUsageCache, +) -> DataUsageScanPlanDigest { + let mut entries = cache + .cache + .iter() + .filter_map(|(key, entry)| { + entry + .replication_stats + .as_ref() + .filter(|stats| stats.replica_size != 0 || stats.replica_count != 0 || !stats.targets.is_empty()) + .map(|stats| (key, stats)) + }) + .collect::>(); + if entries.is_empty() { + return execution_digest; + } + entries.sort_unstable_by(|left, right| left.0.cmp(right.0)); + let mut hasher = Sha256::new(); + hasher.update(b"scanner-replication-execution-v1"); + hasher.update(execution_digest.0); + for (key, stats) in entries { + hasher.update(u64::try_from(key.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(key.as_bytes()); + hasher.update(stats.replica_size.to_be_bytes()); + hasher.update(stats.replica_count.to_be_bytes()); + let mut targets = stats.targets.iter().collect::>(); + targets.sort_unstable_by(|left, right| left.0.cmp(right.0)); + hasher.update(u64::try_from(targets.len()).unwrap_or(u64::MAX).to_be_bytes()); + for (arn, target) in targets { + hasher.update(u64::try_from(arn.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(arn.as_bytes()); + let rustfs_data_usage::ReplicationTargetUsage { + pending_size, + replicated_size, + failed_size, + failed_count, + pending_count, + missed_threshold_size, + after_threshold_size, + missed_threshold_count, + after_threshold_count, + replicated_count, + } = target; + for value in [ + pending_size, + replicated_size, + failed_size, + failed_count, + pending_count, + missed_threshold_size, + after_threshold_size, + missed_threshold_count, + after_threshold_count, + replicated_count, + ] { + hasher.update(value.to_be_bytes()); + } + } + } + DataUsageScanPlanDigest(hasher.finalize().into()) +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) struct ScannerSnapshotIdentity { pub(super) cycle: u64, @@ -494,8 +632,12 @@ pub(super) fn empty_namespace_usage_candidate( } impl ValidatedUsageCandidate { - pub(super) fn prepare(mut self, status: ScannerCycleStatus) -> (DataUsageInfo, Option) { - self.data.usage_snapshot_converged = Some(status == ScannerCycleStatus::Complete); + pub(super) fn prepare( + mut self, + status: ScannerCycleStatus, + pending_buckets: &ScannerCyclePendingBuckets, + ) -> (DataUsageInfo, Option) { + apply_scanner_cycle_publication_marks(&mut self.data, status, pending_buckets); let expectation = if status == ScannerCycleStatus::Complete { struct DigestWriter(Sha256); impl std::io::Write for DigestWriter { diff --git a/crates/scanner/src/scanner_io/dirty_usage.rs b/crates/scanner/src/scanner_io/dirty_usage.rs index 43af3e1df..ed2920142 100644 --- a/crates/scanner/src/scanner_io/dirty_usage.rs +++ b/crates/scanner/src/scanner_io/dirty_usage.rs @@ -457,6 +457,88 @@ fn apply_scoped_dirty_usage_ack( Ok(cleared) } +#[cfg(test)] +mod status_class_mutation_tests { + use super::*; + use crate::segment_invalidation::SegmentInvalidationProducerIdentity; + use serial_test::serial; + + fn bucket(name: &str) -> BucketInfo { + BucketInfo { + name: name.to_string(), + created: None, + deleted: None, + versioning: false, + object_locking: false, + } + } + + #[test] + #[serial] + fn status_class_mutation_is_not_namespace_activity() { + clear_dirty_usage_buckets_for_tests(); + let generation_before = dirty_usage_generation(); + let snapshot = snapshot_dirty_usage_buckets(&[bucket("photos")], generation_before); + assert_eq!(dirty_usage_snapshot_status(&snapshot), DirtyUsageSnapshotStatus::Current); + + record_status_class_mutation("photos", "2026/image.jpg", SegmentInvalidationProducerIdentity::Replication); + + assert_eq!( + dirty_usage_generation(), + generation_before, + "a status-class writeback must not advance the dirty generation" + ); + assert!( + !dirty_usage_buckets_pending(), + "a status-class writeback must not count as pending dirty usage" + ); + assert!( + dirty_usage_buckets_for_tests().is_empty(), + "a status-class writeback must not enter the dirty map" + ); + assert_eq!( + dirty_usage_snapshot_status(&snapshot), + DirtyUsageSnapshotStatus::Current, + "a status-class writeback during a walk must not supersede the cycle" + ); + assert_ne!( + DIRTY_USAGE_PRODUCER_COVERAGE.load(Ordering::Acquire) + & SegmentInvalidationProducerIdentity::Replication + .production_coverage_bit() + .expect("replication has a coverage bit"), + 0, + "producer coverage keeps recording the reclassified producer" + ); + clear_dirty_usage_buckets_for_tests(); + } + + #[test] + #[serial] + fn status_class_mutation_does_not_reach_the_mutation_observer() { + use std::sync::{Arc, Mutex}; + + clear_dirty_usage_buckets_for_tests(); + let observed = Arc::new(Mutex::new(Vec::new())); + let sink = observed.clone(); + let previous = set_scanner_dirty_usage_mutation_observer(Some(Arc::new(move |bucket, object, producer| { + sink.lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .push((bucket.to_string(), object.to_string(), producer)); + }))); + + record_status_class_mutation("photos", "2026/image.jpg", SegmentInvalidationProducerIdentity::Replication); + record_status_class_mutation("", "2026/image.jpg", SegmentInvalidationProducerIdentity::Replication); + record_status_class_mutation("photos", "", SegmentInvalidationProducerIdentity::Replication); + + set_scanner_dirty_usage_mutation_observer(previous); + assert!( + observed.lock().unwrap_or_else(|poisoned| poisoned.into_inner()).is_empty(), + "the durable dirty journal must never see a status-class writeback" + ); + clear_dirty_usage_buckets_for_tests(); + } +} + #[cfg(test)] mod scoped_dirty_usage_tests { use super::*; @@ -934,6 +1016,31 @@ where DIRTY_USAGE_BUCKET_NOTIFY.notify_one(); } +/// Record a status-class barrier hit: a committed writeback that changed only +/// replication or restore state on an existing version. +/// +/// Such a writeback changes no size, version count or delete marker, so it is +/// not namespace activity: it must not advance the dirty-usage generation, +/// enter the activity snapshot, wake the scanner or reach the durable dirty +/// journal, because any of those makes a cycle `Superseded` for as long as +/// replication runs. It needs no rescan queue either: a deployment with an +/// active replication rule walks every bucket on every cycle +/// (`ScannerMaintenanceFeatures::needs_regular_cycle`), and that walk is what +/// refreshes `replication_stats`. Only the process-local producer coverage is +/// recorded so segment-reuse evidence keeps its meaning. +pub fn record_status_class_mutation( + bucket: &str, + object: &str, + producer: crate::segment_invalidation::SegmentInvalidationProducerIdentity, +) { + if bucket.is_empty() || object.is_empty() { + return; + } + if let Some(coverage_bit) = producer.production_coverage_bit() { + DIRTY_USAGE_PRODUCER_COVERAGE.fetch_or(coverage_bit, Ordering::AcqRel); + } +} + fn record_segment_invalidation_producer_identities_for_generation( identities: &mut DirtyUsageProducerIdentities, bucket: &str, @@ -1218,21 +1325,43 @@ pub(super) fn dirty_usage_buckets_excluding_failed( .collect() } +/// Plan the per-bucket certification of a walk. The returned snapshot is +/// applied by `clear_dirty_usage_buckets`, which removes only the buckets +/// whose generation still equals the start snapshot: a bucket written during +/// the walk keeps its newer generation and stays pending. The dirty-usage +/// generation therefore does not need to be unchanged; only the guardrail +/// must have passed. pub(super) fn should_clear_dirty_usage_snapshot( result_ok: bool, completed_all_sets: bool, budget_elapsed: bool, - activity_and_generation_current: bool, + guardrail_passed: bool, dirty_buckets: &DirtyUsageBuckets, failed_buckets: &HashSet, ) -> Option { - if result_ok && completed_all_sets && !budget_elapsed && activity_and_generation_current { + if result_ok && completed_all_sets && !budget_elapsed && guardrail_passed { return Some(dirty_usage_buckets_excluding_failed(dirty_buckets, failed_buckets)); } None } +/// Buckets the walk cannot certify against its start snapshot: every dirty +/// bucket whose generation differs from the snapshot's, including buckets +/// dirtied after the snapshot was taken. Only a `Complete` cycle reads this, +/// and a failed bucket never reaches `Complete`, so the failed set that +/// `dirty_usage_buckets_excluding_failed` removes is empty here. The caller +/// adds the buckets whose walk stopped early behind a certified value; they +/// are pending whether or not the dirty map names them. +pub(super) fn pending_dirty_usage_buckets(snapshot: &DirtyUsageBuckets) -> BTreeSet { + let dirty_buckets = dirty_usage_buckets(); + dirty_buckets + .iter() + .filter(|(bucket, generation)| snapshot.get(bucket.as_str()) != Some(*generation)) + .map(|(bucket, _)| bucket.clone()) + .collect() +} + pub(super) async fn record_failed_dirty_bucket(failed_buckets: &Arc>>, bucket: &str) { failed_buckets.lock().await.insert(bucket.to_string()); } diff --git a/crates/scanner/src/scanner_io/io_cache.rs b/crates/scanner/src/scanner_io/io_cache.rs index f6a9d4146..f8a7f863b 100644 --- a/crates/scanner/src/scanner_io/io_cache.rs +++ b/crates/scanner/src/scanner_io/io_cache.rs @@ -14,6 +14,90 @@ /// ScannerIOCache implementation for SetDisks: bucket ordering, worker fan-out, merge, and publish. use super::*; +pub(super) async fn collect_completed_bucket_results( + ctx: CancellationToken, + mut cache: DataUsageCache, + mut results: mpsc::Receiver, +) -> (DataUsageCache, HashSet) { + let mut cancelled = false; + let mut completed = HashSet::new(); + loop { + tokio::select! { + _ = ctx.cancelled(), if !cancelled => cancelled = true, + result = results.recv() => { + let Some(result) = result else { + return (cache, completed); + }; + if let Some(bucket) = apply_bucket_result_to_cache(&mut cache, result, SystemTime::now()) { + completed.insert(bucket); + } + } + } + } +} + +/// Stand in for every bucket of this set whose walk stopped early with the +/// bucket's last certified value from the previous complete set cache. +/// +/// Usage is a sum: a certified bucket's value plus an uncertified bucket's +/// last certified value is a valid snapshot (see +/// `docs/architecture/scanner-per-bucket-convergence.md`), so a walk that +/// stops early no longer makes the whole set, and with it the whole cycle, +/// incomplete. The cycle reports the bucket pending and walks it again. A +/// bucket that never certified on this set, failed, was not found, or +/// changed incarnation is not covered and keeps the set incomplete. The +/// stand-in is the previous certified value, never this walk's running +/// total. +async fn carry_partial_bucket_values( + cache: &mut DataUsageCache, + old_cache: &DataUsageCache, + old_cache_certified: bool, + buckets: &[BucketInfo], + completed_buckets: &HashSet, + bucket_failures: &ScannerBucketFailureState, + current_bucket_incarnations: Option<&HashMap>, +) -> Vec { + if !old_cache_certified { + return Vec::new(); + } + let Some(current_bucket_incarnations) = current_bucket_incarnations else { + return Vec::new(); + }; + // Snapshot each set in turn; the workers of other sets take these locks + // one at a time and must never wait behind a finalising set. + let partial = bucket_failures.partial.lock().await.clone(); + let failed = bucket_failures.hard.lock().await.clone(); + let namespace_not_found = bucket_failures.namespace_not_found.lock().await.clone(); + let mut carried = Vec::new(); + for bucket in buckets { + let name = bucket.name.as_str(); + if completed_buckets.contains(name) + || !partial.contains(name) + || failed.contains(name) + || namespace_not_found.contains(name) + { + continue; + } + let current_incarnation = current_bucket_incarnations + .get(name) + .filter(|incarnation| !incarnation.is_nil()); + let certified_incarnation = old_cache + .info + .scan_bucket_incarnations + .get(name) + .filter(|incarnation| !incarnation.is_nil()); + if current_incarnation.is_none() || certified_incarnation != current_incarnation { + continue; + } + let Some(certified) = old_cache.checked_flatten(name) else { + continue; + }; + cache.replace(name, DATA_USAGE_ROOT, certified); + carried.push(bucket.name.clone()); + } + carried +} + #[derive(Clone, Copy)] pub(super) struct ScannerSetCacheGeneration { pub(super) want_cycle: u64, @@ -189,6 +273,7 @@ impl ScannerIOCache for SetDisks { scope, digest: scan_plan_digest, bucket_coverage_digest, + checkpoint_coverage_digest, requires_full_scan, service_cohort, execution_digest, @@ -202,8 +287,10 @@ impl ScannerIOCache for SetDisks { cold_zero_walk_reuse_observed, segment_invalidation_proof, } = scan_plan; + let structural_scan_plan_digest = scan_plan_digest; let scan_plan_digest = scanner_bucket_work_digest(scan_plan_digest, scan_mode, requires_full_scan); let bucket_work_digest = scanner_bucket_work_digest(bucket_coverage_digest, scan_mode, requires_full_scan); + let bucket_checkpoint_work_digest = scanner_bucket_work_digest(checkpoint_coverage_digest, scan_mode, requires_full_scan); let pool_label = self.pool_index.to_string(); let set_label = self.set_index.to_string(); @@ -233,6 +320,9 @@ impl ScannerIOCache for SetDisks { None } }; + // Captured before `prepare_for_scan` clears the mark: only a complete + // snapshot may stand in for a bucket whose walk stops early. + let old_cache_certified = old_cache.info.snapshot_complete && old_cache.info.last_update.is_some(); let current_bucket_incarnations = scanner_current_bucket_incarnations(self.as_ref(), &all_buckets).await; let scoped_scan = prepare_scoped_set_scan( &old_cache, @@ -294,7 +384,8 @@ impl ScannerIOCache for SetDisks { }; cache.info.last_update = Some(now); cache.info.snapshot_complete = true; - cache.info.scan_execution_digest = Some(execution_digest); + let replication_execution = super::cache::scanner_replication_execution_digest(execution_digest, &cache); + cache.info.scan_execution_digest = Some(replication_execution); cache.info.segment_invalidation_proof = completed_segment_invalidation_proof; cache.info.lkg_snapshot_complete = false; cache.info.lkg_next_cycle = None; @@ -469,7 +560,11 @@ impl ScannerIOCache for SetDisks { "Scanner set assigned remote disks without namespace scanner support to coordinator-driven workers" ); } - let dispatch_limit = scanner_max_concurrent_disk_scans(workers.len()); + let online_disks = workers.len(); + let walkers_per_disk = crate::runtime_config::scanner_walkers_per_disk_configured(); + let walkers = namespace_scanner_walkers(workers, walkers_per_disk, |disk| disk.is_local()); + let dispatch_limit = scanner_max_concurrent_disk_scans(walkers.len()); + let configured_disk_scans = crate::runtime_config::scanner_max_concurrent_disk_scans_configured(); let serial_progress_accounting = budget.requires_serial_progress_accounting(); let disk_scan_limit = scanner_budgeted_concurrency_limit(dispatch_limit, serial_progress_accounting); record_disk_scan_concurrency_limit(&pool_label, &set_label, disk_scan_limit); @@ -479,10 +574,18 @@ impl ScannerIOCache for SetDisks { component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, state = "disk_concurrency_budget", - reason = if serial_progress_accounting { "count_budget" } else { "configured_or_available_work" }, + reason = if serial_progress_accounting { + "count_budget" + } else if configured_disk_scans != 0 && configured_disk_scans < walkers.len() { + "configured_cap" + } else { + "available_walkers" + }, pool = self.pool_index, set = self.set_index, - online_disks = workers.len(), + online_disks, + walkers_per_disk, + walkers = walkers.len(), dispatch_limit, concurrency_limit = disk_scan_limit, "Scanner disk concurrency budget resolved" @@ -512,6 +615,23 @@ impl ScannerIOCache for SetDisks { old_cache.info.scan_plan_digest, ) }); + // A complete set cache written under the same structural plan with + // another work flag (a full or Deep sweep against a scoped one) + // describes the same namespace. Let `prepare_for_scan` keep its + // entries as seeds, so a bucket whose walk stops early in this + // cycle can publish its last certified value + // (`carry_partial_bucket_values`); the cycle-level baseline proof + // accepts the same variants. The LKG identity above keeps the + // digest the cache was written under. + if old_cache_certified + && old_cache.info.scan_plan_digest != Some(scan_plan_digest) + && old_cache + .info + .scan_plan_digest + .is_some_and(|digest| scanner_plan_digest_is_work_variant(structural_scan_plan_digest, digest)) + { + old_cache.info.scan_plan_digest = Some(scan_plan_digest); + } let prepare_outcome = match old_cache.prepare_for_scan( DATA_USAGE_ROOT, want_cycle, @@ -626,42 +746,21 @@ impl ScannerIOCache for SetDisks { } } - let cache_mutex: Arc> = Arc::new(Mutex::new(cache)); - - let (bucket_result_tx, mut bucket_result_rx) = mpsc::channel::(workers.len()); - - let cache_mutex_clone = cache_mutex.clone(); - let ctx_clone = ctx.clone(); - let completed_bucket_count = Arc::new(AtomicUsize::new(0)); - let completed_bucket_count_clone = completed_bucket_count.clone(); - let collect_bucket_results_fut = AbortOnDropHandle::new(tokio::spawn(async move { - let mut cancelled = false; - - loop { - tokio::select! { - _ = ctx_clone.cancelled(), if !cancelled => { - cancelled = true; - } - result = bucket_result_rx.recv() => { - let Some(result) = result else { - return; - }; - - let mut cache = cache_mutex_clone.lock().await; - if apply_bucket_result_to_cache(&mut cache, result, SystemTime::now()) { - completed_bucket_count_clone.fetch_add(1, Ordering::Relaxed); - } - } - } - } - })); + let (bucket_result_tx, bucket_result_rx) = mpsc::channel::(walkers.len()); + let collect_bucket_results_fut = + AbortOnDropHandle::new(tokio::spawn(collect_completed_bucket_results(ctx.clone(), cache, bucket_result_rx))); let mut futs = Vec::new(); let bucket_rx_mutex: Arc>> = Arc::new(Mutex::new(bucket_rx)); let remaining_bucket_work = Arc::new(AtomicUsize::new(buckets.len())); let bucket_work_complete = CancellationToken::new(); - for (disk, worker_mode) in workers { + for NamespaceScannerWalker { + disk, + mode: worker_mode, + local_admission, + } in walkers + { let service_cohort_clone = service_cohort.clone(); let bucket_rx_mutex_clone = bucket_rx_mutex.clone(); let bucket_tx_clone = bucket_tx.clone(); @@ -753,8 +852,12 @@ impl ScannerIOCache for SetDisks { ); let cache_name = path_join_buf(&[&bucket.name, DATA_USAGE_CACHE_NAME]); - let bucket_scan_plan_digest = - scanner_bucket_cache_digest(bucket_work_digest, dirty_usage_buckets_clone.get(&bucket.name).copied()); + let bucket_dirty_generation = dirty_usage_buckets_clone.get(&bucket.name).copied(); + // The reuse plan a complete cache is current under, and the + // coverage plan a sweep is started and resumed under. + let bucket_scan_plan_digest = scanner_bucket_cache_digest(bucket_work_digest, bucket_dirty_generation); + let bucket_checkpoint_plan_digest = + scanner_bucket_cache_digest(bucket_checkpoint_work_digest, bucket_dirty_generation); if let Some(server_epoch) = remote_server_epoch { let request_sequence = remote_session_sequence; @@ -814,6 +917,19 @@ impl ScannerIOCache for SetDisks { } Ok(crate::remote_scanner::RemoteScannerOutcome::Partial) => { record_partial_dirty_bucket(&partial_dirty_buckets_clone, &bucket.name).await; + debug!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_DISK_BUCKET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "remote_scan_partial", + bucket = %bucket.name, + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + disk = %disk.path().display(), + "Remote scanner bucket returned partial coverage" + ); } Ok(crate::remote_scanner::RemoteScannerOutcome::NamespaceNotFound) => { if requeue_bucket_work(&bucket_tx_clone, &bucket, &mut work_guard).await { @@ -867,10 +983,10 @@ impl ScannerIOCache for SetDisks { event = EVENT_SCANNER_DISK_BUCKET_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, - bucket = %bucket.name, state = "remote_zero_progress_requeued", + bucket = %bucket.name, error = %e, - "Remote scanner requeued bucket rejected before execution" + "Remote scanner requeued bucket work" ); break; } @@ -906,8 +1022,8 @@ impl ScannerIOCache for SetDisks { continue; } - let _local_admission = if disk.is_local() { - match crate::remote_scanner::try_admit_remote_scanner(&disk) { + let _local_admission = if let Some(share) = &local_admission { + match share.acquire(&disk) { Ok(admission) => Some(admission), Err(e) => { if requeue_bucket_work(&bucket_tx_clone, &bucket, &mut work_guard).await { @@ -935,6 +1051,8 @@ impl ScannerIOCache for SetDisks { } else { None }; + #[cfg(test)] + super::tests::observe_coordinator_bucket_scan_admitted(&bucket.name).await; // Lock order: scanner leader fence -> set-scoped per-bucket cache lock -> // cache object read/write. @@ -1002,7 +1120,7 @@ impl ScannerIOCache for SetDisks { }; // Lack of an authoritative legacy identity disables the new // checkpoint protocol; the existing full rebuild remains available. - let checkpoint_identity = scanner_bucket_checkpoint_identity( + let checkpoint_identity = match scanner_bucket_checkpoint_identity( &store_clone_clone, &bucket.name, expected_publication_epoch_clone, @@ -1010,7 +1128,28 @@ impl ScannerIOCache for SetDisks { scan_mode, ) .await - .ok(); + { + Ok(identity) => identity, + Err(error) => { + record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await; + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_DISK_BUCKET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "checkpoint_identity_unavailable", + stage = "admission", + bucket = %bucket.name, + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + leader_epoch, + error = %error, + "Scanner retained the bucket checkpoint after an identity lookup failure" + ); + continue; + } + }; let scan_state = current_cache_root_or_prepare_with_generation( &mut cache, &bucket.name, @@ -1022,6 +1161,7 @@ impl ScannerIOCache for SetDisks { require_source: require_cache_source, tier_registry_generation: Some(tier_registry_generation), checkpoint_identity, + checkpoint_plan_digest: Some(bucket_checkpoint_plan_digest), }, ); let outcome = match scan_state { @@ -1231,8 +1371,8 @@ impl ScannerIOCache for SetDisks { } } }; - if let Some(expected) = checkpoint_identity - && scanner_bucket_checkpoint_identity( + if let Some(expected) = checkpoint_identity { + let current_identity = match scanner_bucket_checkpoint_identity( &store_clone_clone, &bucket.name, expected_publication_epoch_clone, @@ -1240,11 +1380,46 @@ impl ScannerIOCache for SetDisks { scan_mode, ) .await - .ok() - != Some(expected) - { - record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await; - continue; + { + Ok(identity) => identity, + Err(error) => { + record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await; + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_DISK_BUCKET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "checkpoint_identity_unavailable", + stage = "post_scan", + bucket = %bucket.name, + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + leader_epoch, + error = %error, + "Scanner could not verify the bucket identity before persistence" + ); + continue; + } + }; + if current_identity != Some(expected) { + record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await; + warn!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_DISK_BUCKET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "checkpoint_identity_changed", + stage = "post_scan", + bucket = %bucket.name, + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + leader_epoch, + "Scanner rejected the bucket result after an identity change" + ); + continue; + } } let scan_outcome = match scan_result { Ok(scan_outcome) => scan_outcome, @@ -1308,7 +1483,14 @@ impl ScannerIOCache for SetDisks { }; let partial = match scan_outcome { - ScannerDiskScanOutcome::Complete(completed_cache) => { + ScannerDiskScanOutcome::Complete(mut completed_cache) => { + // The sweep finished under its coverage plan; the + // complete cache is current under the reuse plan, + // which a restart or a maintenance publication + // changes so the bucket is walked again next time. + if completed_cache.info.snapshot_complete { + completed_cache.info.scan_plan_digest = Some(bucket_scan_plan_digest); + } cache = completed_cache; None } @@ -1372,16 +1554,7 @@ impl ScannerIOCache for SetDisks { record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await; } if partial_saved { - debug!( - target: "rustfs::scanner::io", - event = EVENT_SCANNER_CACHE_PERSIST_STATE, - component = LOG_COMPONENT_SCANNER, - subsystem = LOG_SUBSYSTEM_IO, - bucket = %bucket.name, - cache_name = %cache_name, - state = "partial_saved_not_published", - "Scanner partial bucket cache saved without publishing usage aggregate" - ); + log_saved_partial_bucket_cache(&partial_cache, &cache_name); } continue; @@ -1549,46 +1722,82 @@ impl ScannerIOCache for SetDisks { record_disk_bucket_scans_queued(0, &pool_label, &set_label); record_disk_bucket_scans_active(0, &pool_label, &set_label); - if let Err(err) = collect_bucket_results_fut.await { - return Err(scanner_task_join_error("scanner bucket result collector", err)); - } + let (mut cache, completed_buckets) = collect_bucket_results_fut + .await + .map_err(|err| scanner_task_join_error("scanner bucket result collector", err))?; if let Some(err) = first_join_err { return Err(err); } - let completed_count = completed_bucket_count.load(Ordering::Relaxed); - if should_publish_completed_snapshot(completed_count, buckets.len(), budget.budget_elapsed(), ctx.is_cancelled()) { - let completed_segment_invalidation_proof = - scanner_completed_set_segment_invalidation_proof(&segment_invalidation_proof, cold_zero_walk_reuse_candidate); - let cache_snapshot = { - let mut cache = cache_mutex.lock().await; - cache.info.next_cycle = want_cycle; - cache.info.last_update.get_or_insert_with(SystemTime::now); - cache.info.snapshot_complete = true; - cache.info.scan_execution_digest = Some(execution_digest); - cache.info.segment_invalidation_proof = completed_segment_invalidation_proof; - cache.info.lkg_snapshot_complete = false; - cache.info.lkg_next_cycle = None; - cache.info.lkg_last_update = None; - cache.info.lkg_leader_epoch = None; - cache.info.lkg_scan_plan_digest = None; - if cold_zero_walk_reuse_candidate { - cold_zero_walk_reuse_observed.store(true, Ordering::Release); - } - cache.clone() + let carried_buckets = carry_partial_bucket_values( + &mut cache, + &old_cache, + old_cache_certified, + &buckets, + &completed_buckets, + &bucket_failures, + current_bucket_incarnations.as_ref(), + ) + .await; + if !carried_buckets.is_empty() { + debug!( + target: "rustfs::scanner::io", + event = EVENT_SCANNER_SET_STATE, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_IO, + state = "partial_buckets_carried", + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + carried_buckets = carried_buckets.len(), + first_carried_bucket = carried_buckets.iter().min(), + "Scanner set published the last certified value for buckets whose walk stopped early" + ); + bucket_failures + .partial_carried + .lock() + .await + .extend(carried_buckets.iter().cloned()); + } + let completed_count = completed_buckets.len(); + let covered_count = completed_count.saturating_add(carried_buckets.len()); + let coverage_complete = + should_publish_completed_snapshot(covered_count, buckets.len(), budget.budget_elapsed(), ctx.is_cancelled()); + if coverage_complete { + // A carried bucket was not walked to its end, so this set cannot + // prove the producer window that prefix-level reuse would need. + let completed_segment_invalidation_proof = if carried_buckets.is_empty() { + scanner_completed_set_segment_invalidation_proof(&segment_invalidation_proof, cold_zero_walk_reuse_candidate) + } else { + None }; + cache.info.next_cycle = want_cycle; + cache.info.last_update.get_or_insert_with(SystemTime::now); + cache.info.snapshot_complete = true; + let carried_execution = super::cache::scanner_carried_execution_digest(execution_digest, &carried_buckets); + let replication_execution = super::cache::scanner_replication_execution_digest(carried_execution, &cache); + cache.info.scan_execution_digest = Some(replication_execution); + cache.info.segment_invalidation_proof = completed_segment_invalidation_proof; + cache.info.lkg_snapshot_complete = false; + cache.info.lkg_next_cycle = None; + cache.info.lkg_last_update = None; + cache.info.lkg_leader_epoch = None; + cache.info.lkg_scan_plan_digest = None; + if cold_zero_walk_reuse_candidate { + cold_zero_walk_reuse_observed.store(true, Ordering::Release); + } let _ = persist_and_publish_cache_snapshot( self.clone(), &updates, - cache_snapshot, + cache, initial_revisions.as_ref(), cache_cycle_floor.as_ref(), expected_publication_epoch, ) .await; } else { - let mut incomplete_scope = cache_mutex.lock().await.clone(); + let mut incomplete_scope = cache; incomplete_scope.info.name = DATA_USAGE_ROOT.to_string(); incomplete_scope.info.next_cycle = want_cycle; incomplete_scope.info.last_update = None; @@ -1617,11 +1826,21 @@ impl ScannerIOCache for SetDisks { event = EVENT_SCANNER_CACHE_PERSIST_STATE, component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, + state = "set_cache_publish_skipped", + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, completed_buckets = completed_count, + carried_buckets = carried_buckets.len(), total_buckets = buckets.len(), + first_incomplete_bucket = buckets + .iter() + .map(|bucket| bucket.name.as_str()) + .filter(|bucket| !completed_buckets.contains(*bucket) && !carried_buckets.iter().any(|carried| carried == bucket)) + .min(), + unprocessed_buckets, budget_elapsed = budget.budget_elapsed(), cancelled = ctx.is_cancelled(), - state = "set_cache_publish_skipped", "Scanner set cache publish skipped because cycle did not complete cleanly" ); } @@ -1632,7 +1851,13 @@ impl ScannerIOCache for SetDisks { component = LOG_COMPONENT_SCANNER, subsystem = LOG_SUBSYSTEM_IO, state = "set_scan_completed", - "Scanner set scan completed" + pool = source.pool_index, + set = source.set_index, + cycle = want_cycle, + completed_buckets = completed_count, + total_buckets = buckets.len(), + coverage_complete, + "Scanner set workers finished" ); Ok(()) diff --git a/crates/scanner/src/scanner_io/io_cycle.rs b/crates/scanner/src/scanner_io/io_cycle.rs index 577cb1094..db55aa91c 100644 --- a/crates/scanner/src/scanner_io/io_cycle.rs +++ b/crates/scanner/src/scanner_io/io_cycle.rs @@ -77,6 +77,8 @@ where service_cohort: None, #[cfg(test)] resolved_scope_observer: None, + #[cfg(test)] + post_walk_gate: None, }; nsscanner_with_storage_status_scoped(store, request).await } @@ -96,6 +98,11 @@ pub(crate) struct ScannerCycleRequest { pub(crate) service_cohort: Option>>, #[cfg(test)] pub(crate) resolved_scope_observer: Option>, + /// Holds the post-walk guardrail probe until the test releases it, so a + /// write issued after `resolved_scope_observer` fired is ordered before + /// certification regardless of how fast the walk finishes. + #[cfg(test)] + pub(crate) post_walk_gate: Option>, } pub(super) struct ScannerBucketScopeResolution<'a> { @@ -287,6 +294,8 @@ where service_cohort, #[cfg(test)] resolved_scope_observer, + #[cfg(test)] + post_walk_gate, } = request; let child_token = ctx.child_token(); let _tier_cycle_guard = begin_tier_registry_cycle(want_cycle, leader_epoch); @@ -325,7 +334,7 @@ where let distributed = store.setup_is_dist_erasure().await; let activity_before = match scanner_activity_preflight(crate::scanner::probe_scanner_activity(store, distributed).await) { - ScannerActivityPreflight::Ready(snapshot) => snapshot, + ScannerActivityPreflight::Ready(snapshot) => Arc::new(snapshot), ScannerActivityPreflight::ActivityBaselineUnavailable(err) => { warn!( target: "rustfs::scanner::io", @@ -388,6 +397,8 @@ where let scan_plan_digest = scanner_bucket_work_digest(structural_scan_plan_digest, scan_mode, requires_full_scan); let activity_digest = crate::scanner::scanner_activity_snapshot_digest(&activity_before); let bucket_coverage_digest = scanner_bucket_plan_digest(&all_buckets, activity_digest); + let checkpoint_coverage_digest = + scanner_bucket_plan_digest(&all_buckets, crate::scanner::scanner_activity_coverage_digest(&activity_before)); let execution_digest = scanner_bucket_work_digest(bucket_coverage_digest, scan_mode, requires_full_scan); let dirty_usage_snapshot = Arc::new(snapshot_dirty_usage_buckets(&all_buckets, dirty_generation_before_bucket_list)); let scope_resolution = resolve_scanner_bucket_scan_scope( @@ -442,6 +453,11 @@ where dirty_usage_status, activity_status, ); + let pending_buckets = if status == ScannerCycleStatus::Complete { + pending_dirty_usage_buckets(&dirty_usage_snapshot.buckets) + } else { + ScannerCyclePendingBuckets::new() + }; let Some(candidate) = empty_namespace_usage_candidate( &all_buckets, &expected_sources, @@ -456,13 +472,13 @@ where ) else { return Ok(ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_publication_epoch(publication_epoch)); }; - let (empty_usage, publication_expectation) = candidate.prepare(status); - let observational_snapshot_published = if should_publish_observational_snapshot(status) { + let (empty_usage, publication_expectation) = candidate.prepare(status, &pending_buckets); + let usage_update_emitted = if should_publish_observational_snapshot(status) { publish_observational_snapshot(&updates, empty_usage).await? } else { - publish_usage_snapshot(&updates, status, empty_usage).await? + publish_usage_snapshot(&updates, status, &pending_buckets, empty_usage).await? }; - if !observational_snapshot_published { + if !usage_update_emitted { return Ok(ScannerCycleResult::new(status, None).with_publication_epoch(publication_epoch)); } if status == ScannerCycleStatus::Complete { @@ -484,7 +500,9 @@ where return Ok(ScannerCycleResult::new(status, dirty_usage_clear) .with_publication_epoch(publication_epoch) .with_activity_digest(activity_digest) - .with_observational_snapshot_published(observational_snapshot_published) + .with_activity_baseline(activity_before) + .with_pending_dirty_usage_buckets(pending_buckets.len()) + .with_usage_update_emitted(usage_update_emitted) .with_remote_publication_lease_targets(remote_publication_lease_targets) .with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements) .with_segment_reuse_activation_preflight(segment_reuse_activation_preflight) @@ -597,6 +615,7 @@ where scope: scan_scope.clone(), digest: structural_scan_plan_digest, bucket_coverage_digest, + checkpoint_coverage_digest, requires_full_scan, service_cohort: service_cohort.clone(), execution_digest, @@ -698,19 +717,27 @@ where record_set_scan_concurrency_limit(0); record_set_scans_queued(0); record_set_scans_active(0); + #[cfg(test)] + if let Some(gate) = post_walk_gate { + let _ = gate.await; + } let first_err = first_err_mutex.lock().await.take(); + let has_set_failure = first_err.is_some(); let results = results_mutex.lock().await.clone(); let completed_all_sets = bucket_plan_complete && scanner_results_form_complete_snapshot(&results, &expected_sources); let result = finalize_nsscanner_result(&results, first_err); let failed_buckets = bucket_failures.hard.lock().await.clone(); let partial_buckets = bucket_failures.partial.lock().await.clone(); + let carried_partial_buckets = bucket_failures.partial_carried.lock().await.clone(); let namespace_not_found_buckets = bucket_failures.namespace_not_found.lock().await.clone(); let scan_scope_matches = scanner_results_match_scan_scope(&results, &expected_sources); + let has_uncarried_partial = partial_buckets.iter().any(|bucket| !carried_partial_buckets.contains(bucket)); let bucket_scan_status = scanner_bucket_scan_status( !failed_buckets.is_empty(), - scan_scope_matches && !partial_buckets.is_empty(), + scan_scope_matches && has_uncarried_partial, scan_scope_matches && !namespace_not_found_buckets.is_empty(), + scan_scope_matches && !partial_buckets.is_empty(), ); let cold_zero_walk_oracle = scanner_cycle_cold_zero_walk_oracle( &scan_scope, @@ -731,8 +758,19 @@ where let observed_cycle_floor = cache_cycle_floor.load(Ordering::Acquire); let required_cycle_floor = (observed_cycle_floor > want_cycle).then_some(observed_cycle_floor); let budget_elapsed = budget.budget_elapsed(); + // A mixed-plan tail can save its checkpoint without producing a root + // candidate. Do not infer progress from an arbitrary missing update, or + // let a successful set hide another set's worker/persistence failure. + let partial_bucket_progress = !budget_elapsed + && !ctx.is_cancelled() + && scanner_partial_progress_is_usable( + bucket_plan_complete, + scan_scope_matches, + bucket_scan_status, + has_set_failure, + !namespace_not_found_buckets.is_empty(), + ); let dirty_usage_status = dirty_usage_snapshot_status(&dirty_usage_snapshot); - let dirty_usage_current = dirty_usage_status == DirtyUsageSnapshotStatus::Current; let (activity_status, remote_publication_lease_targets) = scanner_cycle_activity_status(store, distributed, &activity_before).await; let all_bucket_names = all_buckets.iter().map(|bucket| bucket.name.clone()).collect::>(); @@ -777,14 +815,27 @@ where dirty_usage_status, activity_status, ); + // Certification is per bucket: the pending set is read after the + // guardrail verdict so a bucket written during the walk is reported in + // the same publication that certifies the others. A bucket whose walk + // stopped early is pending for the same reason: the publication carries + // its last certified value, not this walk's (a `Complete` cycle has only + // carried partial buckets, see `classify_nsscanner_cycle`). + let pending_buckets = if cycle_status == ScannerCycleStatus::Complete { + let mut pending = pending_dirty_usage_buckets(&dirty_usage_snapshot.buckets); + pending.extend(partial_buckets.iter().cloned()); + pending + } else { + ScannerCyclePendingBuckets::new() + }; let mut publication_expectation = None; - let observational_snapshot_published = if let Some(candidate) = completed_usage { - let (data_usage_info, expectation) = candidate.prepare(cycle_status); + let usage_update_emitted = if let Some(candidate) = completed_usage { + let (data_usage_info, expectation) = candidate.prepare(cycle_status, &pending_buckets); publication_expectation = expectation; if should_publish_observational_snapshot(cycle_status) { publish_observational_snapshot(&updates, data_usage_info).await? } else { - publish_usage_snapshot(&updates, cycle_status, data_usage_info).await? + publish_usage_snapshot(&updates, cycle_status, &pending_buckets, data_usage_info).await? } } else if !ctx.is_cancelled() && let Some((data_usage_info, _)) = observational_usage @@ -793,13 +844,16 @@ where } else { false }; + // A walk that stopped early certifies nothing: the bucket keeps its + // dirty generation like a failed one and is walked again next cycle. + let uncertified_buckets = failed_buckets.union(&partial_buckets).cloned().collect::>(); let dirty_usage_clear = should_clear_dirty_usage_snapshot( result.is_ok(), structurally_complete_snapshot, budget_elapsed, - activity_status == ScannerCycleActivityStatus::Unchanged && dirty_usage_current, + activity_status == ScannerCycleActivityStatus::Unchanged, &dirty_usage_snapshot.buckets, - &failed_buckets, + &uncertified_buckets, ); result?; if cycle_status == ScannerCycleStatus::Complete { @@ -816,7 +870,10 @@ where Ok(ScannerCycleResult::new(cycle_status, dirty_usage_clear) .with_publication_epoch(publication_epoch) .with_activity_digest(activity_digest) - .with_observational_snapshot_published(observational_snapshot_published) + .with_activity_baseline(activity_before) + .with_pending_dirty_usage_buckets(pending_buckets.len()) + .with_usage_update_emitted(usage_update_emitted) + .with_partial_bucket_progress(partial_bucket_progress) .with_remote_publication_lease_targets(remote_publication_lease_targets) .with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements) .with_distributed_segment_invalidation_evidence(distributed_segment_invalidation_evidence) diff --git a/crates/scanner/src/scanner_io/io_disk.rs b/crates/scanner/src/scanner_io/io_disk.rs index a77639c49..403c05c39 100644 --- a/crates/scanner/src/scanner_io/io_disk.rs +++ b/crates/scanner/src/scanner_io/io_disk.rs @@ -180,9 +180,9 @@ impl ScannerIODisk for Disk { .await .unwrap_or_else(|_| (BucketLifecycleConfiguration::default(), OffsetDateTime::now_utc())); - if lifecycle_config.has_active_rules("") { - cache.info.lifecycle = Some(Arc::new(lifecycle_config)); - } + // Persisted usage caches are not an authority for lifecycle policy. + // Clear their old snapshot when configuration is absent, disabled, or unreadable. + cache.info.lifecycle = lifecycle_config.has_active_rules("").then(|| Arc::new(lifecycle_config)); let (replication_config, _) = crate::storage_api::ecstore_with_background_disk_io(ctx.clone(), get_replication_config(&cache.info.name)) @@ -230,7 +230,8 @@ impl ScannerIODisk for Disk { cache, updates, scan_mode, - SCANNER_SLEEPER.clone(), + // Every walker of this local disk shares its background I/O budget. + SCANNER_SLEEPER.for_disk(&self.path()), prefix_scan_scope, checkpoint_tx, ), diff --git a/crates/scanner/src/scanner_io/publish_gate_tests.rs b/crates/scanner/src/scanner_io/publish_gate_tests.rs index 564ea2baf..388679f31 100644 --- a/crates/scanner/src/scanner_io/publish_gate_tests.rs +++ b/crates/scanner/src/scanner_io/publish_gate_tests.rs @@ -19,6 +19,41 @@ use rustfs_data_usage::{ReplicationAllStats, ReplicationTargetUsage, TierAccount const TEST_PLAN_DIGEST: DataUsageScanPlanDigest = DataUsageScanPlanDigest([7; 32]); const TEST_COVERAGE_DIGEST: DataUsageScanPlanDigest = DataUsageScanPlanDigest([6; 32]); +#[test] +fn partial_progress_requires_complete_scope_and_no_hard_failure() { + for plan_complete in [false, true] { + for scope_matches in [false, true] { + for set_failed in [false, true] { + for missing_namespace in [false, true] { + for status in [ + ScannerBucketScanStatus::Complete, + ScannerBucketScanStatus::Partial, + ScannerBucketScanStatus::Failed, + ScannerBucketScanStatus::NamespaceNotFound, + ] { + let expected = plan_complete + && scope_matches + && !set_failed + && !missing_namespace + && status == ScannerBucketScanStatus::Partial; + assert_eq!( + scanner_partial_progress_is_usable( + plan_complete, + scope_matches, + status, + set_failed, + missing_namespace + ), + expected, + "plan={plan_complete}, scope={scope_matches}, status={status:?}, set_failed={set_failed}, missing_namespace={missing_namespace}" + ); + } + } + } + } + } +} + #[test] fn scanner_bucket_inventory_requires_exact_unique_set_union() { let first = BucketInfo { @@ -803,6 +838,7 @@ fn current_cache_root_with_new_tier_generation_resets_old_cache() { require_source: false, tier_registry_generation: Some(2), checkpoint_identity: None, + checkpoint_plan_digest: None, }, ); @@ -899,12 +935,14 @@ fn completed_data_usage_info_publishes_confirmed_empty_namespace() { } #[test] -fn complete_usage_candidate_with_changed_generation_is_superseded() { +fn complete_usage_candidate_with_changed_generation_is_certified_per_bucket() { let all_buckets = vec!["bucket".to_string()]; let completed_set = completed_root_cache("bucket", 2, 10, DataUsageCacheSource::new(0, 0)); let completed_usage = completed_data_usage_info_for_test(&[completed_set], &all_buckets, false, false); assert!(completed_usage.is_some()); + // A moved dirty generation shades buckets for the next cycle; only a + // guardrail veto supersedes a structurally complete candidate. assert_eq!( classify_nsscanner_cycle( completed_usage.is_some(), @@ -914,6 +952,17 @@ fn complete_usage_candidate_with_changed_generation_is_superseded() { DirtyUsageSnapshotStatus::Changed, ScannerCycleActivityStatus::Unchanged, ), + ScannerCycleStatus::Complete + ); + assert_eq!( + classify_nsscanner_cycle( + completed_usage.is_some(), + false, + false, + ScannerBucketScanStatus::Complete, + DirtyUsageSnapshotStatus::Changed, + ScannerCycleActivityStatus::Changed, + ), ScannerCycleStatus::Superseded ); } @@ -1528,3 +1577,55 @@ async fn requeued_bucket_remains_pending_for_another_worker() { assert!(!complete.is_cancelled()); assert_eq!(rx.recv().await.expect("requeued bucket").name, "photos"); } + +#[test] +fn replication_execution_digest_is_canonical_and_tracks_statistics() { + use rustfs_data_usage::{ReplicationAllStats, ReplicationTargetUsage}; + let execution = DataUsageScanPlanDigest([9; 32]); + let mut cache = DataUsageCache::default(); + assert_eq!(super::cache::scanner_replication_execution_digest(execution, &cache), execution); + let first = ReplicationTargetUsage { + replicated_size: 7, + replicated_count: 1, + ..Default::default() + }; + let second = ReplicationTargetUsage { + pending_size: 3, + pending_count: 1, + ..Default::default() + }; + cache.cache.insert( + "photos".to_string(), + DataUsageEntry { + replication_stats: Some(ReplicationAllStats { + replica_size: 0, + replica_count: 0, + targets: HashMap::from([("arn:a".to_string(), first.clone()), ("arn:b".to_string(), second.clone())]), + }), + ..Default::default() + }, + ); + let digest = super::cache::scanner_replication_execution_digest(execution, &cache); + assert_ne!(digest, execution); + let stats = cache + .cache + .get_mut("photos") + .expect("cached bucket") + .replication_stats + .as_mut() + .expect("replication stats"); + stats.targets = HashMap::from([("arn:b".to_string(), second), ("arn:a".to_string(), first)]); + assert_eq!(super::cache::scanner_replication_execution_digest(execution, &cache), digest); + cache + .cache + .get_mut("photos") + .expect("cached bucket") + .replication_stats + .as_mut() + .expect("replication stats") + .targets + .get_mut("arn:a") + .expect("target stats") + .failed_count = 1; + assert_ne!(super::cache::scanner_replication_execution_digest(execution, &cache), digest); +} diff --git a/crates/scanner/src/scanner_io/tests.rs b/crates/scanner/src/scanner_io/tests.rs index 1a7eef5bb..29a5652d1 100644 --- a/crates/scanner/src/scanner_io/tests.rs +++ b/crates/scanner/src/scanner_io/tests.rs @@ -23,7 +23,7 @@ use crate::scanner_folder::ScannerItem; use crate::storage_api::EcstoreScannerPeerDirtyUsageSnapshot; use crate::storage_api::owner::{ EcstorePoolDecommissionInfo, EcstoreRebalStatus, EcstoreRebalanceInfo, EcstoreRebalanceMeta, EcstoreRebalanceStats, - WriteCommitGuard, WriteCompletion, ecstore_hold_namespace_commit, + WriteCommitGuard, WriteCompletion, ecstore_hold_namespace_commit, ecstore_namespace_commits_pending, }; use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _}; use crate::{ @@ -44,6 +44,9 @@ use temp_env::with_var; use time::OffsetDateTime; use uuid::Uuid; +mod checkpoint_coverage_identity; +mod partial_bucket_carry; +mod per_bucket_certification; mod scoped_entry_fallback; mod service_cohort; mod usage_expiration; @@ -1040,7 +1043,7 @@ async fn checkpoint_rejects_unreadable_or_changed_global_fence() { async fn wait_for_namespace_commit_tails(store: &ECStore) { tokio::time::timeout(Duration::from_secs(30), async { - while store.scanner_data_usage_publication_blocked().await { + while ecstore_namespace_commits_pending(store) { tokio::time::sleep(Duration::from_millis(1)).await; } }) @@ -1059,6 +1062,7 @@ async fn checkpoint_fixture_bucket_identity_uses_its_set_instance_owner() { let first_identity = scanner_bucket_checkpoint_identity(&first.pools[0].disk_set[0], "checkpoint-identity", 0, 7, HealScanMode::Normal) .await + .expect("first durable identity lookup") .expect("first durable identity"); let (_second_dir, second) = setup_two_pool_scanner_store().await; second @@ -1068,18 +1072,20 @@ async fn checkpoint_fixture_bucket_identity_uses_its_set_instance_owner() { let second_identity = scanner_bucket_checkpoint_identity(&second.pools[0].disk_set[0], "checkpoint-identity", 0, 7, HealScanMode::Normal) .await + .expect("second durable identity lookup") .expect("second durable identity"); assert_ne!(first_identity.bucket_incarnation, second_identity.bucket_incarnation); assert_eq!( scanner_bucket_checkpoint_identity(&first.pools[0].disk_set[0], "checkpoint-identity", 0, 7, HealScanMode::Normal) .await .expect("first owner remains bound"), - first_identity + Some(first_identity) ); assert!( scanner_bucket_checkpoint_identity(&first.pools[0].disk_set[0], "missing-checkpoint-bucket", 0, 7, HealScanMode::Normal) .await - .is_err() + .expect("missing identity must be distinguished from a read failure") + .is_none() ); } @@ -1318,6 +1324,7 @@ async fn scoped_scan_production_entry_preserves_deep_and_full_maintenance_work() observed_usage_candidate: None, requires_full_scan, resolved_scope_observer: Some(observer), + post_walk_gate: None, service_cohort: None, }, ), @@ -1408,6 +1415,7 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure( observed_usage_candidate: None, requires_full_scan: false, resolved_scope_observer: None, + post_walk_gate: None, service_cohort: None, }, ), @@ -1461,6 +1469,7 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure( observed_usage_candidate: None, requires_full_scan, resolved_scope_observer: None, + post_walk_gate: None, service_cohort: None, }, ), @@ -1504,17 +1513,20 @@ async fn data_usage_publish_rejects_a_second_terminal_update_without_blocking() ), ] { let (updates, mut receiver) = mpsc::channel(1); + let pending = ScannerCyclePendingBuckets::new(); assert!( - publish_usage_snapshot(&updates, status, data_usage_info) + publish_usage_snapshot(&updates, status, &pending, data_usage_info) .await .expect("first terminal update should be accepted") ); - let err = - tokio::time::timeout(Duration::from_secs(1), publish_usage_snapshot(&updates, status, DataUsageInfo::default())) - .await - .expect("a full terminal update must fail without waiting") - .expect_err("a second terminal update must be rejected"); + let err = tokio::time::timeout( + Duration::from_secs(1), + publish_usage_snapshot(&updates, status, &pending, DataUsageInfo::default()), + ) + .await + .expect("a full terminal update must fail without waiting") + .expect_err("a second terminal update must be rejected"); assert!(err.to_string().contains("already queued")); assert!(receiver.try_recv().is_ok(), "the first terminal update must remain owned by the receiver"); assert!(receiver.try_recv().is_err(), "the rejected second update must not enter the channel"); @@ -1581,7 +1593,7 @@ async fn multi_pool_scanner_cycle_publishes_combined_usage() { #[tokio::test] #[serial] -async fn pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage() { +async fn pending_put_commit_keeps_scanner_walk_live_and_publishes_authority() { let (_temp_dir, store) = setup_two_pool_scanner_store().await; let bucket = format!("scanner-pending-put-{}", Uuid::new_v4().simple()); store @@ -1611,11 +1623,15 @@ async fn pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage( let mut pending = Some(ecstore_hold_namespace_commit(store.as_ref())); let mut previous_activity_digest = None; let mut structural_plan_digest = None; - for (cycle, converged) in [(1, false), (2, true)] { - if converged { + for (cycle, commit_pending) in [(1, true), (2, false)] { + if !commit_pending { drop(pending.take()); } - assert_eq!(store.scanner_data_usage_publication_blocked().await, !converged); + assert_eq!(ecstore_namespace_commits_pending(store.as_ref()), commit_pending); + assert!( + !store.scanner_data_usage_publication_blocked().await, + "an ordinary PUT commit is namespace activity, not a publication barrier" + ); assert!(!store.scanner_data_movement_pause_status().await.paused); let activity = crate::scanner::probe_scanner_activity(store.as_ref(), false) .await @@ -1643,22 +1659,20 @@ async fn pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage( .expect("namespace scanning must finish while a PUT commit is pending") .expect("namespace scanning must remain available during a pending PUT commit"); assert_eq!(result.activity_digest(), Some(activity_digest)); - if !converged { + if commit_pending { assert_eq!(budget.progress().0, 2, "the pending commit must not suppress actual object traversal"); } assert_eq!( result.status, - if converged { - ScannerCycleStatus::Complete - } else { - ScannerCycleStatus::Superseded - } + ScannerCycleStatus::Complete, + "a pending PUT commit shades its bucket; it no longer supersedes the walk" ); let usage = receiver .recv() .await .expect("the completed walk should produce a usage candidate"); - assert_eq!(usage.usage_snapshot_converged, Some(converged)); + assert_eq!(usage.usage_snapshot_converged, Some(true)); + assert!(!usage.usage_snapshot_observational); assert_eq!(usage.scanner_cycle, Some(cycle)); assert_eq!(usage.objects_total_count, 2); assert_eq!(usage.objects_total_size, 11); @@ -2297,6 +2311,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers scope: ScannerBucketScanScope::default(), digest: DataUsageScanPlanDigest([6; 32]), bucket_coverage_digest: DataUsageScanPlanDigest([6; 32]), + checkpoint_coverage_digest: DataUsageScanPlanDigest([6; 32]), requires_full_scan: false, execution_digest: empty_execution, leader_epoch: 11, @@ -2391,7 +2406,64 @@ fn complete_segment_reuse_baseline( } #[test] -fn scoped_scan_requires_a_converged_complete_baseline_with_exact_set_provenance() { +fn scoped_scan_includes_the_baseline_pending_buckets() { + let source = DataUsageCacheSource::new(1, 2); + let expected_sources = HashSet::from([source]); + let scan_plan_digest = DataUsageScanPlanDigest([9; 32]); + let mut baseline: DataUsageInfo = + serde_json::from_slice(&complete_usage_baseline(source, scan_plan_digest, 7, 11)).expect("test baseline should decode"); + baseline.buckets_usage.insert("stale".to_string(), Default::default()); + baseline.buckets_count = 2; + baseline.usage_snapshot_converged = Some(false); + baseline.usage_snapshot_pending_buckets = vec!["stale".to_string(), "deleted".to_string()]; + baseline.usage_snapshot_pending_bucket_count = 2; + let baseline = bytes::Bytes::from(serde_json::to_vec(&baseline).expect("test baseline should encode")); + let proof = ScannerCacheBaselineProof { + authoritative_data: Some(&baseline), + observed_candidate_data: None, + expected_sources: &expected_sources, + leader_epoch: 11, + want_cycle: 8, + scan_plan_digest, + }; + let all_buckets = vec![bucket_info("photos"), bucket_info("stale"), bucket_info("cold")]; + + // A bucket the last publication left pending (a partial walk that + // published its last certified value) is walked with the dirty buckets. + let scope = scoped_scan_scope_from_dirty_buckets( + ScannerBucketScanScope::default(), + HashSet::from(["photos".to_string()]), + None, + true, + false, + &all_buckets, + proof, + ); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["photos".to_string(), "stale".to_string()])), + "pending buckets join the dirty scope; a deleted bucket is dropped" + ); + + // With nothing dirty, the pending list alone scopes the walk. + let scope = scoped_scan_scope_from_dirty_buckets( + ScannerBucketScanScope::default(), + HashSet::new(), + None, + true, + false, + &all_buckets, + proof, + ); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["stale".to_string()])), + "a pending bucket is rescanned without a dirty wake" + ); +} + +#[test] +fn scoped_scan_requires_a_complete_authoritative_baseline_with_exact_set_provenance() { let source = DataUsageCacheSource::new(1, 2); let expected_sources = HashSet::from([source]); let scan_plan_digest = DataUsageScanPlanDigest([9; 32]); @@ -2409,19 +2481,41 @@ fn scoped_scan_requires_a_converged_complete_baseline_with_exact_set_provenance( Some(scan_plan_digest) ); - let mut incomplete = serde_json::from_slice::(&baseline).expect("test baseline should decode"); - incomplete.usage_snapshot_converged = Some(false); - let incomplete = bytes::Bytes::from(serde_json::to_vec(&incomplete).expect("test baseline should encode")); + // A complete cycle that left buckets pending publishes unconverged on + // the authoritative object and is still the structural baseline: its + // pending buckets stay dirty and are rescanned, the rest reuse. + let mut pending = serde_json::from_slice::(&baseline).expect("test baseline should decode"); + pending.usage_snapshot_converged = Some(false); + pending.usage_snapshot_pending_buckets = vec!["bucket-a".to_string()]; + pending.usage_snapshot_pending_bucket_count = 1; + let pending = bytes::Bytes::from(serde_json::to_vec(&pending).expect("test baseline should encode")); assert_eq!( complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { - authoritative_data: Some(&incomplete), + authoritative_data: Some(&pending), observed_candidate_data: None, expected_sources: &expected_sources, leader_epoch: 11, want_cycle: 8, scan_plan_digest, }), - None + Some(scan_plan_digest) + ); + + let mut observational = serde_json::from_slice::(&baseline).expect("test baseline should decode"); + observational.usage_snapshot_converged = Some(false); + observational.usage_snapshot_observational = true; + let observational = bytes::Bytes::from(serde_json::to_vec(&observational).expect("test baseline should encode")); + assert_eq!( + complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { + authoritative_data: Some(&observational), + observed_candidate_data: None, + expected_sources: &expected_sources, + leader_epoch: 11, + want_cycle: 8, + scan_plan_digest, + }), + None, + "an observational snapshot in the authoritative slot is not a baseline" ); let mut wrong_provenance = serde_json::from_slice::(&baseline).expect("test baseline should decode"); @@ -2528,10 +2622,18 @@ fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_bas observation_of_nonconverged_authoritative.usage_snapshot_set_states[0].scanner_cycle = Some(8); observation_of_nonconverged_authoritative.usage_snapshot_authoritative_baseline = Some(nonconverged_authoritative.snapshot_identity()); + observation_of_nonconverged_authoritative.usage_snapshot_observational = true; + let mut observational_in_authoritative_slot = nonconverged_authoritative.clone(); + observational_in_authoritative_slot.usage_snapshot_observational = true; let nonconverged_authoritative = bytes::Bytes::from(serde_json::to_vec(&nonconverged_authoritative).expect("nonconverged baseline should encode")); + let observational_in_authoritative_slot = bytes::Bytes::from( + serde_json::to_vec(&observational_in_authoritative_slot).expect("observational baseline should encode"), + ); let observation_of_nonconverged_authoritative = bytes::Bytes::from(serde_json::to_vec(&observation_of_nonconverged_authoritative).expect("observation should encode")); + // The unconverged authoritative snapshot is itself the baseline, so the + // observation chained on it is not consulted. assert_eq!( complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { authoritative_data: Some(&nonconverged_authoritative), @@ -2541,6 +2643,19 @@ fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_bas want_cycle: 9, scan_plan_digest, }), + Some(scan_plan_digest) + ); + // An observational snapshot that somehow sits in the authoritative slot + // is neither a baseline nor an anchor for a chained observation. + assert_eq!( + complete_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof { + authoritative_data: Some(&observational_in_authoritative_slot), + observed_candidate_data: Some(&observation_of_nonconverged_authoritative), + expected_sources: &expected_sources, + leader_epoch: 11, + want_cycle: 9, + scan_plan_digest, + }), None ); } @@ -3583,6 +3698,8 @@ fn scanner_cycle_status_requires_a_clean_complete_snapshot() { ), ScannerCycleStatus::Complete ); + // Dirty-usage movement during the walk shades buckets; it is certified + // per bucket and no longer supersedes a walk the guardrail accepted. assert_eq!( classify_nsscanner_cycle( true, @@ -3592,19 +3709,22 @@ fn scanner_cycle_status_requires_a_clean_complete_snapshot() { DirtyUsageSnapshotStatus::Changed, ScannerCycleActivityStatus::Unchanged, ), - ScannerCycleStatus::Superseded - ); - assert_eq!( - classify_nsscanner_cycle( - true, - false, - false, - ScannerBucketScanStatus::Complete, - DirtyUsageSnapshotStatus::Current, - ScannerCycleActivityStatus::Changed, - ), - ScannerCycleStatus::Superseded + ScannerCycleStatus::Complete ); + for dirty_usage_status in [DirtyUsageSnapshotStatus::Current, DirtyUsageSnapshotStatus::Changed] { + assert_eq!( + classify_nsscanner_cycle( + true, + false, + false, + ScannerBucketScanStatus::Complete, + dirty_usage_status, + ScannerCycleActivityStatus::Changed, + ), + ScannerCycleStatus::Superseded, + "a guardrail veto supersedes regardless of dirty state ({dirty_usage_status:?})" + ); + } assert_eq!( classify_nsscanner_cycle( true, @@ -3663,6 +3783,73 @@ fn scanner_cycle_status_requires_a_clean_complete_snapshot() { } } +#[test] +fn carried_partial_buckets_complete_the_cycle_as_pending() { + for dirty_usage_status in [DirtyUsageSnapshotStatus::Current, DirtyUsageSnapshotStatus::Changed] { + assert_eq!( + classify_nsscanner_cycle( + true, + false, + false, + ScannerBucketScanStatus::PartialCarried, + dirty_usage_status, + ScannerCycleActivityStatus::Unchanged, + ), + ScannerCycleStatus::Complete, + "a partial walk behind a certified value is pending, not incomplete ({dirty_usage_status:?})" + ); + } + assert_eq!( + classify_nsscanner_cycle( + true, + false, + false, + ScannerBucketScanStatus::PartialCarried, + DirtyUsageSnapshotStatus::Current, + ScannerCycleActivityStatus::Changed, + ), + ScannerCycleStatus::Superseded, + "the guardrail still vetoes a carried cycle" + ); + assert_eq!( + classify_nsscanner_cycle( + true, + false, + false, + ScannerBucketScanStatus::PartialCarried, + DirtyUsageSnapshotStatus::Current, + ScannerCycleActivityStatus::Unverified, + ), + ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable) + ); + for (completed_all_sets, budget, cancelled, bucket_status) in [ + (false, false, false, ScannerBucketScanStatus::PartialCarried), + (true, true, false, ScannerBucketScanStatus::PartialCarried), + (true, false, true, ScannerBucketScanStatus::PartialCarried), + (true, false, false, ScannerBucketScanStatus::Partial), + ] { + assert_eq!( + classify_nsscanner_cycle( + completed_all_sets, + budget, + cancelled, + bucket_status, + DirtyUsageSnapshotStatus::Current, + ScannerCycleActivityStatus::Unchanged, + ), + ScannerCycleStatus::Incomplete, + "a set that could not stand in, a budget, a cancellation, or an uncovered partial walk stays incomplete \ + ({completed_all_sets}, {budget}, {cancelled}, {bucket_status:?})" + ); + } + for bucket_status in [ScannerBucketScanStatus::Partial, ScannerBucketScanStatus::PartialCarried] { + assert!( + scanner_partial_progress_is_usable(true, true, bucket_status, false, false), + "{bucket_status:?} saved checkpoints the retry can resume" + ); + } +} + #[test] fn checkpoint_fixture_superseded_is_distinct_from_partial_and_cancel() { for (budget, cancelled, bucket, expected) in [ @@ -3677,11 +3864,26 @@ fn checkpoint_fixture_superseded_is_distinct_from_partial_and_cancel() { cancelled, bucket, DirtyUsageSnapshotStatus::Changed, - ScannerCycleActivityStatus::Unchanged + ScannerCycleActivityStatus::Changed ), expected, ); } + // Budget and cancellation stay `Incomplete` even when the guardrail + // passed and only dirty usage moved. + for (budget, cancelled) in [(true, false), (false, true)] { + assert_eq!( + classify_nsscanner_cycle( + true, + budget, + cancelled, + ScannerBucketScanStatus::Complete, + DirtyUsageSnapshotStatus::Changed, + ScannerCycleActivityStatus::Unchanged + ), + ScannerCycleStatus::Incomplete, + ); + } } #[test] @@ -3714,20 +3916,28 @@ fn unverified_activity_defers_partial_and_floor_cycles() { #[tokio::test] async fn structurally_complete_superseded_cycles_publish_without_claiming_convergence() { - let (updates, mut receiver) = mpsc::channel(2); + let (updates, mut receiver) = mpsc::channel(3); + let no_pending = ScannerCyclePendingBuckets::new(); + let pending = ScannerCyclePendingBuckets::from(["hot-b".to_string(), "hot-a".to_string()]); assert!( - publish_usage_snapshot(&updates, ScannerCycleStatus::Complete, DataUsageInfo::default()) + publish_usage_snapshot(&updates, ScannerCycleStatus::Complete, &no_pending, DataUsageInfo::default()) .await .expect("complete snapshot publication should succeed") ); assert!( - publish_usage_snapshot(&updates, ScannerCycleStatus::Superseded, DataUsageInfo::default()) + publish_usage_snapshot(&updates, ScannerCycleStatus::Complete, &pending, DataUsageInfo::default()) + .await + .expect("complete snapshot with pending buckets should publish") + ); + // A vetoed cycle never carries a pending list even when one is supplied. + assert!( + publish_usage_snapshot(&updates, ScannerCycleStatus::Superseded, &pending, DataUsageInfo::default()) .await .expect("superseded snapshot publication should succeed") ); assert!( - !publish_usage_snapshot(&updates, ScannerCycleStatus::Incomplete, DataUsageInfo::default()) + !publish_usage_snapshot(&updates, ScannerCycleStatus::Incomplete, &no_pending, DataUsageInfo::default()) .await .expect("incomplete snapshot suppression should succeed") ); @@ -3735,28 +3945,33 @@ async fn structurally_complete_superseded_cycles_publish_without_claiming_conver !publish_usage_snapshot( &updates, ScannerCycleStatus::Deferred(ScannerCycleDeferReason::ActivityBaselineUnavailable), + &no_pending, DataUsageInfo::default(), ) .await .expect("unverified activity suppression should succeed") ); - assert_eq!( - receiver - .recv() - .await - .expect("complete update should be sent") - .usage_snapshot_converged, - Some(true) - ); - assert_eq!( - receiver - .recv() - .await - .expect("superseded update should be sent") - .usage_snapshot_converged, - Some(false) - ); + let converged = receiver.recv().await.expect("complete update should be sent"); + assert_eq!(converged.usage_snapshot_converged, Some(true)); + assert!(!converged.usage_snapshot_observational); + assert!(converged.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(converged.usage_snapshot_pending_bucket_count, 0); + + let certified = receiver + .recv() + .await + .expect("complete update with pending buckets should be sent"); + assert_eq!(certified.usage_snapshot_converged, Some(false)); + assert!(!certified.usage_snapshot_observational, "pending buckets keep the authoritative route"); + assert_eq!(certified.usage_snapshot_pending_buckets, vec!["hot-a".to_string(), "hot-b".to_string()]); + assert_eq!(certified.usage_snapshot_pending_bucket_count, 2); + + let superseded = receiver.recv().await.expect("superseded update should be sent"); + assert_eq!(superseded.usage_snapshot_converged, Some(false)); + assert!(superseded.usage_snapshot_observational); + assert!(superseded.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(superseded.usage_snapshot_pending_bucket_count, 0); } #[tokio::test] @@ -3854,12 +4069,41 @@ fn scanner_cycle_fails_closed_when_dirty_generation_is_unverified() { ); } +#[test] +fn carried_execution_identity_differs_from_a_full_walk_and_is_order_independent() { + let execution = DataUsageScanPlanDigest([7; 32]); + assert_eq!(super::cache::scanner_carried_execution_digest(execution, &[]), execution); + let carried = super::cache::scanner_carried_execution_digest(execution, &["b".to_string(), "a".to_string()]); + assert_ne!(carried, execution, "a set that stood in for a bucket is not the full walk's execution"); + assert_eq!( + carried, + super::cache::scanner_carried_execution_digest(execution, &["a".to_string(), "b".to_string()]) + ); + assert_ne!(carried, super::cache::scanner_carried_execution_digest(execution, &["a".to_string()])); +} + #[test] fn scanner_bucket_failure_status_preserves_the_strongest_failure() { - assert_eq!(scanner_bucket_scan_status(false, false, false), ScannerBucketScanStatus::Complete); - assert_eq!(scanner_bucket_scan_status(false, false, true), ScannerBucketScanStatus::NamespaceNotFound); - assert_eq!(scanner_bucket_scan_status(false, true, true), ScannerBucketScanStatus::Partial); - assert_eq!(scanner_bucket_scan_status(true, true, true), ScannerBucketScanStatus::Failed); + assert_eq!(scanner_bucket_scan_status(false, false, false, false), ScannerBucketScanStatus::Complete); + assert_eq!( + scanner_bucket_scan_status(false, false, true, false), + ScannerBucketScanStatus::NamespaceNotFound + ); + assert_eq!(scanner_bucket_scan_status(false, true, true, false), ScannerBucketScanStatus::Partial); + assert_eq!(scanner_bucket_scan_status(true, true, true, false), ScannerBucketScanStatus::Failed); + // A partial walk covered by its last certified value may complete, but + // never ahead of an uncovered partial walk, a failure, or a missing + // namespace. + assert_eq!( + scanner_bucket_scan_status(false, false, false, true), + ScannerBucketScanStatus::PartialCarried + ); + assert_eq!(scanner_bucket_scan_status(false, true, false, true), ScannerBucketScanStatus::Partial); + assert_eq!( + scanner_bucket_scan_status(false, false, true, true), + ScannerBucketScanStatus::NamespaceNotFound + ); + assert_eq!(scanner_bucket_scan_status(true, false, false, true), ScannerBucketScanStatus::Failed); } #[test] @@ -4393,6 +4637,53 @@ fn cache_root_entry_info_rejects_missing_or_dangling_roots() { assert!(cache_root_entry_info(&compacted_with_child).is_err()); } +#[tokio::test] +async fn completed_bucket_results_count_distinct_current_roots_even_after_cancellation() { + for cancelled in [false, true] { + let ctx = CancellationToken::new(); + if cancelled { + ctx.cancel(); + } + let mut cache = DataUsageCache { + info: DataUsageCacheInfo { + name: DATA_USAGE_ROOT.to_string(), + tier_registry_generation: Some(7), + ..Default::default() + }, + ..Default::default() + }; + cache.replace( + "missing", + DATA_USAGE_ROOT, + DataUsageEntry { + size: 3, + ..Default::default() + }, + ); + let (tx, rx) = mpsc::channel(3); + for (bucket, generation, size) in [("complete", 7, 1), ("complete", 7, 2), ("missing", 8, 11)] { + tx.send(DataUsageEntryInfo { + name: bucket.to_string(), + parent: DATA_USAGE_ROOT.to_string(), + entry: DataUsageEntry { + size, + ..Default::default() + }, + bucket_incarnation: None, + tier_registry_generation: Some(generation), + }) + .await + .expect("fixture result channel"); + } + drop(tx); + let (cache, complete) = super::io_cache::collect_completed_bucket_results(ctx, cache, rx).await; + assert_eq!(complete, HashSet::from(["complete".to_string()])); + assert!(!should_publish_completed_snapshot(complete.len(), 2, false, false)); + assert_eq!(cache.find("complete").expect("applied root").size, 2); + assert_eq!(cache.find("missing").expect("retained old root").size, 3); + } +} + #[test] fn apply_bucket_result_to_cache_updates_bucket_entry() { let mut cache = DataUsageCache { @@ -4413,10 +4704,12 @@ fn apply_bucket_result_to_cache_updates_bucket_entry() { ); let update_time = SystemTime::now(); - assert!(apply_bucket_result_to_cache( + let bucket = "bucket".to_string(); + let bucket_name_ptr = bucket.as_ptr(); + let applied_bucket = apply_bucket_result_to_cache( &mut cache, DataUsageEntryInfo { - name: "bucket".to_string(), + name: bucket, parent: DATA_USAGE_ROOT.to_string(), entry: DataUsageEntry { size: 10, @@ -4427,7 +4720,10 @@ fn apply_bucket_result_to_cache_updates_bucket_entry() { tier_registry_generation: None, }, update_time, - )); + ) + .expect("accepted result returns its bucket name"); + assert_eq!(applied_bucket, "bucket"); + assert_eq!(applied_bucket.as_ptr(), bucket_name_ptr, "reuse the result's name allocation"); assert_eq!(cache.info.last_update, Some(update_time)); let entry = cache.find("bucket").expect("bucket entry should remain present"); @@ -4470,7 +4766,357 @@ fn apply_bucket_result_to_cache_rejects_a_different_tier_generation() { SystemTime::now(), ); - assert!(!applied); + assert!(applied.is_none()); assert_eq!(cache.find("bucket").map(|entry| entry.size), Some(3)); assert!(cache.info.last_update.is_none()); } + +#[tokio::test] +#[serial] +async fn replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries() { + use crate::storage_api::owner::ObjectOperations as _; + for same_cycle in [false, true] { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + let bucket = "status-bucket"; + let object = "cold/object"; + let set = store.pools[0].disk_set[0].clone(); + store + .make_bucket( + bucket, + &MakeBucketOptions { + versioning_enabled: true, + ..Default::default() + }, + ) + .await + .expect("create status bucket"); + let mut reader = ScannerPutObjReader::from_vec(b"initial".to_vec()); + set.put_object( + bucket, + object, + &mut reader, + &ScannerObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("persist initial object"); + wait_for_namespace_commit_tails(store.as_ref()).await; + + let ctx = CancellationToken::new(); + let (updates, mut receiver) = mpsc::channel(1); + let initial = nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + budget: ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()), + ctx, + updates, + want_cycle: 1, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + service_cohort: None, + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await + .expect("initial full usage scan"); + assert_eq!(initial.status, ScannerCycleStatus::Complete); + let baseline = receiver.recv().await.expect("initial usage publication"); + assert!(baseline.buckets_usage[bucket].replication_info.is_empty()); + + let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]); + let mut cache = DataUsageCache::default(); + cache + .load(set.clone(), &cache_name) + .await + .expect("load complete bucket cache"); + assert!(cache.info.snapshot_complete); + // Keep an accounting context with no remote work in this local fixture. + let replication: ReplicationConfiguration = serde_json::from_value(serde_json::json!({ + "role": "", + "rules": [{ + "destination": { "bucket": "arn:test" }, + "id": "status", + "prefix": "", + "priority": 1, + "status": "Enabled" + }] + })) + .expect("decode the local replication accounting fixture"); + cache.info.replication = Some(Arc::new(ReplicationConfig::new( + Some(replication), + Some(crate::storage_api::EcstoreBucketTargets::default()), + ))); + // A complete compacted prefix is eligible for normal usage sampling. + let mut compacted = cache.checked_flatten(bucket).expect("complete cached usage"); + compacted.children.clear(); + compacted.compacted = true; + cache.cache.clear(); + cache.replace(bucket, "", DataUsageEntry::default()); + cache.replace("status-bucket/cold", bucket, compacted); + cache + .save(set.clone(), &cache_name) + .await + .expect("persist compacted prefix fixture"); + let cycle = (2..18) + .find(|cycle| { + !rustfs_data_usage::hash_path("status-bucket/cold") + .mod_(*cycle, crate::scanner_folder::data_usage_update_dir_cycles()) + }) + .expect("choose a cycle outside the compacted prefix sampling window"); + + let mut metadata = HashMap::new(); + rustfs_utils::http::metadata_compat::insert_str( + &mut metadata, + rustfs_utils::http::metadata_compat::SUFFIX_REPLICATION_STATUS, + "arn:test=COMPLETED;".to_string(), + ); + let updated = set + .put_object_metadata( + bucket, + object, + &ScannerObjectOptions { + eval_metadata: Some(metadata), + versioned: true, + ..Default::default() + }, + ) + .await + .expect("persist replication-only metadata update"); + assert_eq!(updated.replication_status.as_str(), "COMPLETED"); + assert!(updated.version_id.is_some_and(|version| !version.is_nil())); + let persisted = set + .get_object_info( + bucket, + object, + &ScannerObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("read back status writeback"); + assert_eq!(persisted.replication_status.as_str(), "COMPLETED"); + let namespace_generation = dirty_usage_generation(); + record_status_class_mutation( + bucket, + object, + crate::segment_invalidation::SegmentInvalidationProducerIdentity::Replication, + ); + assert_eq!(dirty_usage_generation(), namespace_generation); + + let ctx = CancellationToken::new(); + let (updates, mut receiver) = mpsc::channel(1); + let result = nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + budget: ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()), + ctx, + updates, + want_cycle: if same_cycle { 1 } else { u64::from(cycle) }, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + service_cohort: None, + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await + .expect("normal scan must refresh the status bucket"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + let refreshed = receiver.recv().await.expect("refreshed usage publication"); + assert_eq!(refreshed.buckets_usage[bucket].objects_count, 1); + assert_eq!(refreshed.buckets_usage[bucket].size, 7); + assert_eq!( + refreshed.buckets_usage[bucket] + .replication_info + .get("arn:test") + .map_or(0, |target| target.replicated_size), + 7, + "a replication maintenance walk must refresh compacted statistics before publishing" + ); + clear_dirty_usage_buckets_for_tests(); + } +} + +pub(super) struct CoordinatorBucketScanProbe { + active: AtomicUsize, + peak: AtomicUsize, + hold: Duration, +} + +static COORDINATOR_BUCKET_SCAN_PROBE: std::sync::Mutex>> = std::sync::Mutex::new(None); + +/// Called by the coordinator walker loop once a bucket scan is admitted on +/// its disk. The hook is absent from non-test builds and a no-op until a test +/// installs a probe; the probe then holds the walker so overlapping +/// admissions become observable as a peak. +pub(super) async fn observe_coordinator_bucket_scan_admitted(_bucket: &str) { + let probe = COORDINATOR_BUCKET_SCAN_PROBE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone(); + let Some(probe) = probe else { return }; + let active = probe.active.fetch_add(1, Ordering::SeqCst) + 1; + probe.peak.fetch_max(active, Ordering::SeqCst); + tokio::time::sleep(probe.hold).await; + probe.active.fetch_sub(1, Ordering::SeqCst); +} + +fn install_coordinator_bucket_scan_probe(hold: Duration) -> Arc { + let probe = Arc::new(CoordinatorBucketScanProbe { + active: AtomicUsize::new(0), + peak: AtomicUsize::new(0), + hold, + }); + *COORDINATOR_BUCKET_SCAN_PROBE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(probe.clone()); + probe +} + +fn clear_coordinator_bucket_scan_probe() { + *COORDINATOR_BUCKET_SCAN_PROBE + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = None; +} + +#[test] +fn namespace_scanner_walkers_multiply_local_disks_and_share_their_admission() { + let server_epoch = Uuid::from_u128(7); + let workers = vec![ + ("local-a", NamespaceScannerWorkerMode::Coordinator), + ("local-b", NamespaceScannerWorkerMode::Coordinator), + ("legacy-peer", NamespaceScannerWorkerMode::Coordinator), + ("v4-peer", NamespaceScannerWorkerMode::RemoteV4(server_epoch)), + ]; + let is_local = |disk: &&str| disk.starts_with("local"); + + let walkers = namespace_scanner_walkers(workers.clone(), 3, is_local); + assert_eq!(walkers.len(), 2 * 3 + 2, "two local disks fan out, both remote disks keep one walker"); + let local_a: Vec<_> = walkers.iter().filter(|walker| walker.disk == "local-a").collect(); + assert_eq!(local_a.len(), 3); + let share = local_a[0] + .local_admission + .as_ref() + .expect("local walkers carry an admission share"); + assert!( + local_a + .iter() + .all(|walker| walker.local_admission.as_ref().is_some_and(|other| Arc::ptr_eq(other, share))), + "walkers of one disk share one admission" + ); + let local_b = walkers + .iter() + .find(|walker| walker.disk == "local-b") + .expect("local-b walker"); + assert!( + !Arc::ptr_eq(local_b.local_admission.as_ref().expect("local-b share"), share), + "each disk owns its own admission share" + ); + assert!( + walkers + .iter() + .filter(|walker| !is_local(&walker.disk)) + .all(|walker| walker.local_admission.is_none()), + "remote disks admit through the peer, not a local share" + ); + assert_eq!( + walkers + .iter() + .filter(|walker| walker.mode == NamespaceScannerWorkerMode::RemoteV4(server_epoch)) + .count(), + 1 + ); + // The walker count, not the disk count, is the available work the + // configured cap applies to; the disk count stays the floor. + assert_eq!(scanner_concurrency_limit(0, walkers.len()), 8); + assert_eq!(scanner_concurrency_limit(4, walkers.len()), 4); + for factor in [0, 1] { + let walkers = namespace_scanner_walkers(workers.clone(), factor, is_local); + assert_eq!(walkers.len(), workers.len(), "factor {factor} keeps one walker per disk"); + assert_eq!(scanner_concurrency_limit(0, walkers.len()), workers.len()); + } +} + +async fn run_full_cycle(store: &Arc, cycle: u64) -> ScannerCycleStatus { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let (updates, _receiver) = mpsc::channel(8); + tokio::time::timeout( + Duration::from_secs(60), + nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + ctx, + budget, + updates, + want_cycle: cycle, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + resolved_scope_observer: None, + service_cohort: None, + post_walk_gate: None, + }, + ), + ) + .await + .expect("cycle should finish within the test deadline") + .expect("cycle should succeed") + .status +} + +#[tokio::test] +#[serial] +async fn walkers_per_disk_lets_one_disk_walk_two_buckets_concurrently() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in ["walk-a", "walk-b"] { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + let mut reader = ScannerPutObjReader::from_vec(b"object".to_vec()); + store + .put_object(bucket, "object", &mut reader, &ScannerObjectOptions::default()) + .await + .expect("object should persist"); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + + // The probe holds every admitted bucket scan long enough that two walkers + // on the one disk must overlap; a single walker cannot overlap with itself. + let hold = Duration::from_millis(750); + for (cycle, factor, expected_peak) in [(1, "1", 1), (2, "2", 2)] { + let probe = install_coordinator_bucket_scan_probe(hold); + let status = temp_env::async_with_vars([(ENV_SCANNER_WALKERS_PER_DISK, Some(factor))], async { + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + run_full_cycle(&store, cycle).await + }) + .await; + crate::runtime_config::refresh_scanner_runtime_config_for_tests(); + clear_coordinator_bucket_scan_probe(); + assert_eq!(status, ScannerCycleStatus::Complete, "walkers_per_disk={factor}"); + assert_eq!( + probe.peak.load(Ordering::SeqCst), + expected_peak, + "walkers_per_disk={factor}: concurrent bucket scans admitted on the single disk" + ); + assert_eq!(probe.active.load(Ordering::SeqCst), 0, "every admitted scan left the probe"); + } +} diff --git a/crates/scanner/src/scanner_io/tests/checkpoint_coverage_identity.rs b/crates/scanner/src/scanner_io/tests/checkpoint_coverage_identity.rs new file mode 100644 index 000000000..d6490f1cf --- /dev/null +++ b/crates/scanner/src/scanner_io/tests/checkpoint_coverage_identity.rs @@ -0,0 +1,171 @@ +// Copyright 2026 RustFS Team +// SPDX-License-Identifier: Apache-2.0 + +//! A bucket sweep interrupted by the cycle budget resumes from its checkpoint +//! in a later cycle even after an event that changes the reuse identity but +//! not the namespace. The event here is a maintenance publication; a +//! coordinator restart is the same case for the coverage identity, but a +//! process cannot change its own instance id in a test. The other buckets +//! are walked again; the sweep in progress is not restarted as mixed +//! coverage (rustfs/rustfs#8494). + +use super::*; + +const HOT_OBJECTS: [&str; 6] = ["p0/a", "p1/b", "p2/c", "p3/d", "p4/e", "p5/f"]; +const LEADER_EPOCH: u64 = 11; + +async fn put_fixture_object(store: &Arc, bucket: &str, object: &str) { + let mut reader = ScannerPutObjReader::from_vec(vec![7; 16]); + store + .put_object(bucket, object, &mut reader, &ScannerObjectOptions::default()) + .await + .expect("fixture object should persist"); +} + +async fn run_full_cycle( + store: &Arc, + cycle: u64, + max_objects: Option, +) -> (ScannerCycleResult, Option, Arc) { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking( + &ctx, + ScannerCycleBudgetConfig { + max_objects, + ..Default::default() + }, + ); + let (updates, mut receiver) = mpsc::channel(1); + let result = tokio::time::timeout( + Duration::from_secs(60), + nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + ctx: budget.token(), + budget: Arc::clone(&budget), + updates, + want_cycle: cycle, + leader_epoch: LEADER_EPOCH, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: None, + observed_usage_candidate: None, + requires_full_scan: true, + service_cohort: None, + resolved_scope_observer: None, + post_walk_gate: None, + }, + ), + ) + .await + .expect("cycle should finish within the test deadline") + .expect("cycle should succeed"); + let snapshot = receiver.recv().await; + assert!(receiver.recv().await.is_none(), "a cycle publishes at most one terminal snapshot"); + (result, snapshot, budget) +} + +#[tokio::test] +#[serial] +async fn checkpoint_resumes_across_a_maintenance_publication() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in ["hot-bucket", "idle-bucket"] { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + } + for object in HOT_OBJECTS { + put_fixture_object(&store, "hot-bucket", object).await; + } + put_fixture_object(&store, "idle-bucket", "p0/a").await; + record_dirty_usage_bucket("hot-bucket"); + record_dirty_usage_bucket("idle-bucket"); + wait_for_namespace_commit_tails(store.as_ref()).await; + + // Cycle 1: a quiet full walk certifies both buckets. + let (result, first, _) = run_full_cycle(&store, 1, None).await; + let first = first.expect("a complete cycle publishes"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(first.usage_snapshot_converged, Some(true)); + assert_eq!(first.buckets_usage["hot-bucket"].objects_count, 6); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + + // Cycle 2: the written bucket is the only dirty one, so it is walked + // first and the object budget stops its sweep with a checkpoint; the + // cycle itself is incomplete. + for object in ["p6/g", "p7/h"] { + put_fixture_object(&store, "hot-bucket", object).await; + } + record_dirty_usage_bucket("hot-bucket"); + wait_for_namespace_commit_tails(store.as_ref()).await; + let (result, _, budget) = run_full_cycle(&store, 2, Some(2)).await; + assert_eq!(result.status, ScannerCycleStatus::Incomplete); + assert!(budget.budget_elapsed()); + let set = Arc::clone(&store.pools[0].disk_set[0]); + let mut checkpoint = DataUsageCache::default(); + checkpoint + .load(Arc::clone(&set), &path_join_buf(&["hot-bucket", DATA_USAGE_CACHE_NAME])) + .await + .expect("the interrupted sweep should leave a bucket cache"); + assert!(!checkpoint.info.snapshot_complete); + assert!(checkpoint.has_scan_position(), "the interrupted sweep must keep a durable position"); + let started_plan = checkpoint + .info + .scan_progress + .expect("the interrupted sweep records its coverage plan") + .started_plan; + + // A maintenance publication between the cycles changes the reuse + // identity (finished results are walked again) but not the coverage + // identity (a sweep in progress keeps its plan). + let before = crate::scanner::probe_scanner_activity(store.as_ref(), false) + .await + .expect("the fixture activity should be observable"); + record_scanner_maintenance_change("idle-bucket"); + let after = crate::scanner::probe_scanner_activity(store.as_ref(), false) + .await + .expect("the fixture activity should be observable"); + assert_ne!( + crate::scanner::scanner_activity_structural_digest(&before), + crate::scanner::scanner_activity_structural_digest(&after), + "a maintenance publication invalidates reusable results" + ); + assert_eq!( + crate::scanner::scanner_activity_coverage_digest(&before), + crate::scanner::scanner_activity_coverage_digest(&after), + "a maintenance publication keeps the coverage plan" + ); + + // Cycle 3: the hot bucket resumes from its checkpoint and certifies; + // a fresh walk would have read all nine objects. + let (result, third, budget) = run_full_cycle(&store, 3, None).await; + let third = third.expect("a complete cycle publishes"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert_eq!(third.usage_snapshot_converged, Some(true)); + assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 8); + assert_eq!(third.buckets_usage["idle-bucket"].objects_count, 1); + assert!( + budget.progress().0 < 9, + "the resumed sweep must not restart from the first object: {} objects read", + budget.progress().0 + ); + let mut completed = DataUsageCache::default(); + completed + .load(set, &path_join_buf(&["hot-bucket", DATA_USAGE_CACHE_NAME])) + .await + .expect("the certified bucket cache should load"); + assert!(completed.info.snapshot_complete); + assert!(completed.info.scan_progress.is_none()); + assert_ne!( + completed.info.scan_plan_digest, + Some(started_plan), + "a complete cache is current under the reuse plan, not the coverage plan" + ); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + clear_dirty_usage_buckets_for_tests(); +} diff --git a/crates/scanner/src/scanner_io/tests/partial_bucket_carry.rs b/crates/scanner/src/scanner_io/tests/partial_bucket_carry.rs new file mode 100644 index 000000000..bd1fec3ad --- /dev/null +++ b/crates/scanner/src/scanner_io/tests/partial_bucket_carry.rs @@ -0,0 +1,270 @@ +// Copyright 2026 RustFS Team +// SPDX-License-Identifier: Apache-2.0 + +//! A bucket whose walk stops early without exhausting the cycle budget (here +//! a sweep resumed under a newer plan, the `mixed_coverage` case of +//! rustfs/rustfs#8494) no longer makes the whole cycle `Incomplete`. The set +//! publishes the bucket's last certified value, the cycle completes with the +//! bucket pending, and the next cycle walks it again. A bucket that never +//! certified keeps the cycle incomplete. + +use super::*; + +const BUCKETS: [&str; 3] = ["cold-bucket", "hot-bucket", "idle-bucket"]; +const LEADER_EPOCH: u64 = 11; + +async fn put_fixture_object(store: &Arc, bucket: &str, object: &str) { + let mut reader = ScannerPutObjReader::from_vec(vec![7; 16]); + store + .put_object(bucket, object, &mut reader, &ScannerObjectOptions::default()) + .await + .expect("fixture object should persist"); +} + +struct CycleOptions { + baseline: Option, + requires_full_scan: bool, + max_objects: Option, + resolved_scope_observer: Option>, +} + +/// Run one cycle; an incomplete cycle may publish nothing or an observation. +async fn run_cycle(store: &Arc, cycle: u64, options: CycleOptions) -> (ScannerCycleResult, Option) { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new_with_progress_tracking( + &ctx, + ScannerCycleBudgetConfig { + max_objects: options.max_objects, + ..Default::default() + }, + ); + let (updates, mut receiver) = mpsc::channel(1); + let result = tokio::time::timeout( + Duration::from_secs(60), + nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + ctx: budget.token(), + budget, + updates, + want_cycle: cycle, + leader_epoch: LEADER_EPOCH, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: options.baseline, + observed_usage_candidate: None, + requires_full_scan: options.requires_full_scan, + service_cohort: None, + resolved_scope_observer: options.resolved_scope_observer, + post_walk_gate: None, + }, + ), + ) + .await + .expect("cycle should finish within the test deadline") + .expect("cycle should succeed"); + let snapshot = receiver.recv().await; + assert!(receiver.recv().await.is_none(), "a cycle publishes at most one terminal snapshot"); + (result, snapshot) +} + +fn baseline_bytes(snapshot: &DataUsageInfo) -> Bytes { + Bytes::from(serde_json::to_vec(snapshot).expect("snapshot should encode")) +} + +fn complete_cycle(requires_full_scan: bool, baseline: Option<&DataUsageInfo>) -> CycleOptions { + CycleOptions { + baseline: baseline.map(baseline_bytes), + requires_full_scan, + max_objects: None, + resolved_scope_observer: None, + } +} + +#[tokio::test] +#[serial] +async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in BUCKETS { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + put_fixture_object(&store, bucket, "p0/initial").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + + // Cycle 1: a quiet full walk certifies every bucket. + let (result, first) = run_cycle(&store, 1, complete_cycle(true, None)).await; + let first = first.expect("a complete cycle publishes"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(first.usage_snapshot_converged, Some(true)); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + + // Cycle 2: a scoped walk of the written bucket leaves the set cache + // under the scoped plan, which the full walk below must still seed from. + for object in ["p1/second", "p2/second", "p3/second"] { + put_fixture_object(&store, "hot-bucket", object).await; + } + record_dirty_usage_bucket("hot-bucket"); + wait_for_namespace_commit_tails(store.as_ref()).await; + let (result, second) = run_cycle(&store, 2, complete_cycle(false, Some(&first))).await; + let second = second.expect("a complete cycle publishes"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(second.usage_snapshot_converged, Some(true)); + assert_eq!(second.buckets_usage["hot-bucket"].objects_count, 4); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + + // Cycle 3: the object budget stops the hot bucket's walk and leaves a + // checkpoint under this cycle's plan; the cycle itself is incomplete. + for object in ["p4/third", "p5/third"] { + put_fixture_object(&store, "hot-bucket", object).await; + } + record_dirty_usage_bucket("hot-bucket"); + wait_for_namespace_commit_tails(store.as_ref()).await; + let (result, _) = run_cycle( + &store, + 3, + CycleOptions { + baseline: Some(baseline_bytes(&second)), + requires_full_scan: false, + max_objects: Some(2), + resolved_scope_observer: None, + }, + ) + .await; + assert_eq!(result.status, ScannerCycleStatus::Incomplete, "an exhausted budget stays incomplete"); + let set = Arc::clone(&store.pools[0].disk_set[0]); + let mut interrupted = DataUsageCache::default(); + interrupted + .load(set, &path_join_buf(&["hot-bucket", DATA_USAGE_CACHE_NAME])) + .await + .expect("the interrupted sweep should leave a bucket cache"); + assert!(!interrupted.info.snapshot_complete); + assert!(interrupted.has_scan_position(), "the interrupted sweep keeps a durable position"); + + // Cycle 4: the checkpoint resumes under a different plan, so the sweep + // is mixed coverage and returns Partial without exhausting any budget. + // The set stands in with the last certified value and the cycle + // completes with the bucket pending instead of being incomplete. + let (result, fourth) = run_cycle(&store, 4, complete_cycle(true, Some(&second))).await; + let fourth = fourth.expect("a complete cycle publishes"); + assert_eq!( + result.status, + ScannerCycleStatus::Complete, + "a partial walk behind a certified value no longer makes the cycle incomplete" + ); + assert_eq!(result.pending_dirty_usage_buckets(), 1); + assert!(!fourth.usage_snapshot_observational, "the publication stays authoritative"); + assert_eq!(fourth.usage_snapshot_converged, Some(false)); + assert_eq!(fourth.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]); + assert_eq!(fourth.usage_snapshot_pending_bucket_count, 1); + assert!(fourth.is_complete_bucket_usage_snapshot()); + assert_eq!( + fourth.buckets_usage["hot-bucket"].objects_count, 4, + "the pending bucket carries its last certified value, not the partial walk" + ); + assert_eq!(fourth.buckets_usage["cold-bucket"].objects_count, 1); + assert_eq!(fourth.buckets_usage["idle-bucket"].objects_count, 1); + assert_eq!(fourth.objects_total_count, 6); + assert!(result.clear_verified_usage().is_empty()); + assert_eq!( + dirty_usage_buckets_for_tests().keys().cloned().collect::>(), + vec!["hot-bucket".to_string()], + "a partial walk certifies nothing: the bucket stays dirty" + ); + + // Cycle 5: even with the dirty map gone, the baseline's pending list + // scopes the walk to the bucket, whose fresh sweep certifies it. + clear_dirty_usage_buckets_for_tests(); + let (observer, resolved_scope) = tokio::sync::oneshot::channel(); + let (result, fifth) = run_cycle( + &store, + 5, + CycleOptions { + baseline: Some(baseline_bytes(&fourth)), + requires_full_scan: false, + max_objects: None, + resolved_scope_observer: Some(observer), + }, + ) + .await; + let fifth = fifth.expect("a complete cycle publishes"); + let scope = resolved_scope.await.expect("the cycle should report its resolved scope"); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["hot-bucket".to_string()])), + "the next cycle walks the pending bucket" + ); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert_eq!(fifth.usage_snapshot_converged, Some(true)); + assert!(fifth.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(fifth.buckets_usage["hot-bucket"].objects_count, 6); + assert_eq!(fifth.buckets_usage["cold-bucket"].objects_count, 1); + assert_eq!(fifth.objects_total_count, 8); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + clear_dirty_usage_buckets_for_tests(); +} + +#[tokio::test] +#[serial] +async fn partial_bucket_without_a_certified_value_keeps_the_cycle_incomplete() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + store + .make_bucket("fresh-bucket", &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + for object in ["p0/a", "p1/b", "p2/c", "p3/d"] { + put_fixture_object(&store, "fresh-bucket", object).await; + } + record_dirty_usage_bucket("fresh-bucket"); + wait_for_namespace_commit_tails(store.as_ref()).await; + + // Cycle 1: the budget stops the only bucket's first walk; no set cache + // was ever published, so nothing can stand in for the bucket. + let (result, _) = run_cycle( + &store, + 1, + CycleOptions { + baseline: None, + requires_full_scan: true, + max_objects: Some(2), + resolved_scope_observer: None, + }, + ) + .await; + assert_eq!(result.status, ScannerCycleStatus::Incomplete); + + // Cycle 2: the checkpoint resumes under another plan and returns + // Partial. The bucket has no certified value, so the cycle stays + // incomplete and publishes no authoritative snapshot. + let (result, second) = run_cycle(&store, 2, complete_cycle(false, None)).await; + assert_eq!( + result.status, + ScannerCycleStatus::Incomplete, + "a bucket that never certified cannot be carried" + ); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert!( + second.is_none_or(|snapshot| snapshot.usage_snapshot_observational || !snapshot.usage_snapshot_complete), + "an incomplete cycle publishes at most an observation" + ); + assert_eq!(dirty_usage_buckets_for_tests().len(), 1, "the bucket stays dirty"); + + // Cycle 3: the fresh sweep under one plan certifies the bucket. + let (result, third) = run_cycle(&store, 3, complete_cycle(false, None)).await; + let third = third.expect("a complete cycle publishes"); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(third.usage_snapshot_converged, Some(true)); + assert_eq!(third.buckets_usage["fresh-bucket"].objects_count, 4); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + clear_dirty_usage_buckets_for_tests(); +} diff --git a/crates/scanner/src/scanner_io/tests/per_bucket_certification.rs b/crates/scanner/src/scanner_io/tests/per_bucket_certification.rs new file mode 100644 index 000000000..c3eca6776 --- /dev/null +++ b/crates/scanner/src/scanner_io/tests/per_bucket_certification.rs @@ -0,0 +1,308 @@ +// Copyright 2026 RustFS Team +// SPDX-License-Identifier: Apache-2.0 + +//! Per-bucket certification on the single-disk fixture: a bucket written +//! during the walk is reported pending while the other buckets are certified, +//! the publication stays authoritative, and the next cycle is scoped to the +//! pending bucket. A PUT tail pending across the walk is ordinary namespace +//! activity; a guardrail veto still supersedes the whole cycle. + +use super::*; + +const BUCKETS: [&str; 3] = ["cold-bucket", "hot-bucket", "idle-bucket"]; + +async fn put_fixture_object(store: &Arc, bucket: &str, object: &str) { + let mut reader = ScannerPutObjReader::from_vec(vec![7; 16]); + store + .put_object(bucket, object, &mut reader, &ScannerObjectOptions::default()) + .await + .expect("fixture object should persist"); +} + +struct CycleHooks { + resolved_scope_observer: Option>, + post_walk_gate: Option>, +} + +async fn run_cycle( + store: &Arc, + cycle: u64, + baseline: Option, + requires_full_scan: bool, + hooks: CycleHooks, +) -> (ScannerCycleResult, DataUsageInfo) { + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let (updates, mut receiver) = mpsc::channel(1); + let result = tokio::time::timeout( + Duration::from_secs(60), + nsscanner_with_storage_status_scoped( + store.as_ref(), + ScannerCycleRequest { + ctx, + budget, + updates, + want_cycle: cycle, + leader_epoch: 11, + scan_mode: HealScanMode::Normal, + scan_scope: ScannerBucketScanScope::default(), + persisted_usage_baseline: baseline, + observed_usage_candidate: None, + requires_full_scan, + service_cohort: None, + resolved_scope_observer: hooks.resolved_scope_observer, + post_walk_gate: hooks.post_walk_gate, + }, + ), + ) + .await + .expect("cycle should finish within the test deadline") + .expect("cycle should succeed"); + let snapshot = receiver.recv().await.expect("cycle should publish a snapshot"); + assert!(receiver.recv().await.is_none(), "a cycle publishes one terminal snapshot"); + (result, snapshot) +} + +fn baseline_bytes(snapshot: &DataUsageInfo) -> Bytes { + Bytes::from(serde_json::to_vec(snapshot).expect("snapshot should encode")) +} + +#[tokio::test] +#[serial] +async fn write_during_walk_leaves_only_that_bucket_pending_and_scopes_the_next_cycle() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in BUCKETS { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + put_fixture_object(&store, bucket, "initial").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + + // Cycle 1: a quiet full walk certifies every bucket. + let (result, first) = run_cycle( + &store, + 1, + None, + true, + CycleHooks { + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await; + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert_eq!(first.usage_snapshot_converged, Some(true)); + assert!(!first.usage_snapshot_observational); + assert!(first.usage_snapshot_pending_buckets.is_empty()); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty(), "a converged cycle clears the dirty map"); + + // Cycle 2: cold and idle are dirty at the start snapshot; hot is written + // after the scope resolved and before the post-walk proof runs. + for bucket in ["cold-bucket", "idle-bucket"] { + put_fixture_object(&store, bucket, "second").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + let (observer, resolved_scope) = tokio::sync::oneshot::channel(); + let (release, gate) = tokio::sync::oneshot::channel(); + let writer_store = store.clone(); + let writer = tokio::spawn(async move { + let scope = resolved_scope.await.expect("the cycle should report its resolved scope"); + put_fixture_object(&writer_store, "hot-bucket", "during-walk").await; + record_dirty_usage_bucket("hot-bucket"); + wait_for_namespace_commit_tails(writer_store.as_ref()).await; + release.send(()).expect("the cycle should still be waiting on the gate"); + scope + }); + let (result, second) = run_cycle( + &store, + 2, + Some(baseline_bytes(&first)), + false, + CycleHooks { + resolved_scope_observer: Some(observer), + post_walk_gate: Some(gate), + }, + ) + .await; + let scope = writer.await.expect("writer task should finish"); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["cold-bucket".to_string(), "idle-bucket".to_string()])), + "the converged baseline scopes cycle 2 to the buckets dirty at its start" + ); + + assert_eq!( + result.status, + ScannerCycleStatus::Complete, + "a write during the walk no longer supersedes" + ); + assert_eq!(result.pending_dirty_usage_buckets(), 1); + assert!(!second.usage_snapshot_observational, "pending buckets publish on the authoritative route"); + assert_eq!(second.usage_snapshot_converged, Some(false)); + assert_eq!(second.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]); + assert_eq!(second.usage_snapshot_pending_bucket_count, 1); + assert!(second.is_complete_bucket_usage_snapshot()); + assert_eq!(second.buckets_usage["cold-bucket"].objects_count, 2); + assert_eq!(second.buckets_usage["idle-bucket"].objects_count, 2); + assert_eq!( + second.buckets_usage["hot-bucket"].objects_count, 1, + "the pending bucket carries its last walked value" + ); + assert!(result.has_dirty_usage_to_acknowledge()); + assert!(result.clear_verified_usage().is_empty()); + let dirty = dirty_usage_buckets_for_tests(); + assert_eq!( + dirty.keys().cloned().collect::>(), + vec!["hot-bucket".to_string()], + "certification clears the unchanged buckets and keeps the written one" + ); + + // Cycle 3: the unconverged authoritative snapshot is still a structural + // baseline, so the walk is scoped to the pending bucket. + let (observer, resolved_scope) = tokio::sync::oneshot::channel(); + let (result, third) = run_cycle( + &store, + 3, + Some(baseline_bytes(&second)), + false, + CycleHooks { + resolved_scope_observer: Some(observer), + post_walk_gate: None, + }, + ) + .await; + let scope = resolved_scope.await.expect("the cycle should report its resolved scope"); + assert_eq!( + scope.selected_buckets_for_tests(), + Some(&HashSet::from(["hot-bucket".to_string()])), + "the next cycle walks only the pending bucket" + ); + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert_eq!(third.usage_snapshot_converged, Some(true)); + assert!(third.usage_snapshot_pending_buckets.is_empty()); + assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 2); + assert_eq!(third.buckets_usage["cold-bucket"].objects_count, 2); + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty()); + clear_dirty_usage_buckets_for_tests(); +} + +#[tokio::test] +#[serial] +async fn pending_put_tail_across_the_walk_no_longer_vetoes_the_cycle() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in BUCKETS { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + put_fixture_object(&store, bucket, "initial").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + let vetoes_before = global_metrics().report().await.usage_freshness.guardrail_veto_total; + + // A PUT whose rename fanout is still pending when the post-walk probe + // runs used to flip the local publication barrier and supersede the + // whole cycle (rustfs/rustfs#8494). It is ordinary namespace activity: + // the walk certifies every bucket and publishes authority. + let pending_commit = ecstore_hold_namespace_commit(store.as_ref()); + assert!(ecstore_namespace_commits_pending(store.as_ref())); + assert!(!store.scanner_data_usage_publication_blocked().await); + let (result, snapshot) = run_cycle( + &store, + 1, + None, + true, + CycleHooks { + resolved_scope_observer: None, + post_walk_gate: None, + }, + ) + .await; + drop(pending_commit); + + assert_eq!(result.status, ScannerCycleStatus::Complete); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert!(!snapshot.usage_snapshot_observational, "the publication takes the authoritative route"); + assert_eq!(snapshot.usage_snapshot_converged, Some(true)); + assert!(snapshot.usage_snapshot_pending_buckets.is_empty()); + assert!(snapshot.is_complete_bucket_usage_snapshot()); + for bucket in BUCKETS { + assert_eq!(snapshot.buckets_usage[bucket].objects_count, 1); + } + assert!(result.clear_verified_usage().is_empty()); + assert!(dirty_usage_buckets_for_tests().is_empty(), "a converged cycle clears the dirty map"); + let freshness = global_metrics().report().await.usage_freshness; + assert_eq!( + freshness.guardrail_veto_total, vetoes_before, + "a pending PUT tail is not a guardrail veto" + ); + clear_dirty_usage_buckets_for_tests(); +} + +#[tokio::test] +#[serial] +async fn guardrail_veto_still_supersedes_the_cycle_onto_the_observed_route() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + clear_dirty_usage_buckets_for_tests(); + for bucket in BUCKETS { + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + put_fixture_object(&store, bucket, "initial").await; + record_dirty_usage_bucket(bucket); + } + wait_for_namespace_commit_tails(store.as_ref()).await; + let vetoes_before = global_metrics().report().await.usage_freshness.guardrail_veto_total; + + // A maintenance publication during the walk moves a guardrail field. It + // is not part of the coverage plan, so bucket checkpoints keep their + // plan, but the cycle it moved in is still superseded regardless of the + // dirty map: no cycle records whether it published that work. + let (observer, resolved_scope) = tokio::sync::oneshot::channel(); + let (release, gate) = tokio::sync::oneshot::channel(); + let maintainer = tokio::spawn(async move { + resolved_scope.await.expect("the cycle should report its resolved scope"); + record_scanner_maintenance_change("hot-bucket"); + release.send(()).expect("the cycle should still be waiting on the gate"); + }); + let (result, snapshot) = run_cycle( + &store, + 1, + None, + true, + CycleHooks { + resolved_scope_observer: Some(observer), + post_walk_gate: Some(gate), + }, + ) + .await; + maintainer.await.expect("the maintenance task should finish"); + + assert_eq!(result.status, ScannerCycleStatus::Superseded); + assert_eq!(result.pending_dirty_usage_buckets(), 0); + assert!(snapshot.usage_snapshot_observational, "a vetoed cycle routes to the observed object"); + assert_eq!(snapshot.usage_snapshot_converged, Some(false)); + assert!(snapshot.usage_snapshot_pending_buckets.is_empty()); + assert!(snapshot.is_complete_bucket_usage_snapshot()); + assert!(!result.has_dirty_usage_to_acknowledge(), "nothing is certified under a veto"); + assert_eq!(dirty_usage_buckets_for_tests().len(), BUCKETS.len(), "the dirty map is untouched"); + let freshness = global_metrics().report().await.usage_freshness; + assert_eq!(freshness.guardrail_veto_total, vetoes_before + 1); + assert_eq!( + freshness.last_guardrail_veto_reason, + format!("maintenance_generation@{}", crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE) + ); + clear_dirty_usage_buckets_for_tests(); +} diff --git a/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs b/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs index d974bd923..f31ae0f0d 100644 --- a/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs +++ b/crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs @@ -138,6 +138,7 @@ async fn run_entry( requires_full_scan: false, service_cohort: None, resolved_scope_observer: Some(observer), + post_walk_gate: None, }, ), ) @@ -366,7 +367,7 @@ async fn scoped_entry_fallback_rejects_invalid_persisted_baseline_at_the_walker_ let baseline = run_entry(&store, 1, None, true, false, false).await; for (index, kind) in [ "malformed", - "unconverged", + "observational", "missing-set", "wrong-source", "mixed-plan", @@ -378,7 +379,12 @@ async fn scoped_entry_fallback_rejects_invalid_persisted_baseline_at_the_walker_ let mut candidate = baseline.clone(); candidate.usage_snapshot_converged = Some(true); match kind { - "unconverged" => candidate.usage_snapshot_converged = Some(false), + // An unconverged authoritative snapshot is a valid baseline since + // per-bucket certification; only the observational mark rejects. + "observational" => { + candidate.usage_snapshot_converged = Some(false); + candidate.usage_snapshot_observational = true; + } "missing-set" => { candidate.usage_snapshot_set_states.pop(); } diff --git a/crates/scanner/src/scanner_io/tests/service_cohort.rs b/crates/scanner/src/scanner_io/tests/service_cohort.rs index deb1da547..1deb5d6d1 100644 --- a/crates/scanner/src/scanner_io/tests/service_cohort.rs +++ b/crates/scanner/src/scanner_io/tests/service_cohort.rs @@ -73,6 +73,7 @@ async fn run_cohort_cycle( requires_full_scan: false, service_cohort: Some(cohort), resolved_scope_observer: None, + post_walk_gate: None, }, ), ) diff --git a/crates/scanner/src/scanner_io/tests/usage_expiration.rs b/crates/scanner/src/scanner_io/tests/usage_expiration.rs index c914378b1..d3dcd8b45 100644 --- a/crates/scanner/src/scanner_io/tests/usage_expiration.rs +++ b/crates/scanner/src/scanner_io/tests/usage_expiration.rs @@ -9,6 +9,77 @@ use crate::storage_api::scanner_io::{ record_bucket_object_write_memory, replace_bucket_usage_memory_from_info, seed_persisted_usage, update_bucket_metadata, }; +#[tokio::test] +#[serial] +async fn disk_scan_withdraws_persisted_lifecycle_policy() { + let (_temp_dir, store) = setup_local_scanner_store(1, 1).await; + let bucket = "withdrawn-lifecycle"; + store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap(); + let mut reader = ScannerPutObjReader::from_vec(b"retained".to_vec()); + store + .put_object(bucket, "object", &mut reader, &ScannerObjectOptions::default()) + .await + .unwrap(); + let xml = br#"expireEnabled0"#; + update_bucket_metadata(bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec()) + .await + .unwrap(); + let (policy, _) = get_lifecycle_config(bucket).await.unwrap(); + let set = Arc::clone(&store.pools[0].disk_set[0]); + let disks = scanner_set_disk_inventory(&set).await; + let cache_name = format!("{bucket}/{DATA_USAGE_CACHE_NAME}"); + for config in [ + Vec::new(), + String::from_utf8(xml.to_vec()) + .unwrap() + .replace("Enabled", "Disabled") + .into_bytes(), + ] { + let mut old_cache = DataUsageCache::default(); + old_cache.info.name = bucket.to_string(); + old_cache.info.lifecycle = Some(Arc::new(policy.clone())); + old_cache.replace(bucket, "", DataUsageEntry::default()); + old_cache.save(Arc::clone(&set), &cache_name).await.unwrap(); + update_bucket_metadata(bucket, BUCKET_LIFECYCLE_CONFIG, config).await.unwrap(); + + // A fresh cache instance models restart: the obsolete policy survives + // serialization and must be withdrawn by the actual disk scan. + let mut loaded = DataUsageCache::default(); + loaded.load(Arc::clone(&set), &cache_name).await.unwrap(); + assert!(loaded.info.lifecycle.is_some()); + let ctx = CancellationToken::new(); + let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default()); + let outcome = disks[0] + .clone() + .nsscanner_disk( + ctx, + budget, + disks.clone(), + loaded, + None, + ScannerDiskScanOptions { + scan_mode: HealScanMode::Normal, + prefix_scan_scope: None, + checkpoint_tx: None, + }, + ) + .await + .unwrap(); + let ScannerDiskScanOutcome::Complete(refreshed) = outcome else { + panic!("the single-drive scan must complete"); + }; + assert!(refreshed.info.lifecycle.is_none()); + refreshed.save(Arc::clone(&set), &cache_name).await.unwrap(); + let mut restarted = DataUsageCache::default(); + restarted.load(Arc::clone(&set), &cache_name).await.unwrap(); + assert!(restarted.info.lifecycle.is_none(), "another restart must not resurrect policy"); + store + .get_object_info(bucket, "object", &ScannerObjectOptions::default()) + .await + .unwrap(); + } +} + // Lifecycle expiry goes through the process-global store, which the first // store created in the process claims. Nextest runs this test in its own // process; the lifecycle scenarios share the store that owns expiry processing. diff --git a/crates/scanner/src/sleeper.rs b/crates/scanner/src/sleeper.rs index 2af363f7f..115b348eb 100644 --- a/crates/scanner/src/sleeper.rs +++ b/crates/scanner/src/sleeper.rs @@ -12,17 +12,25 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::path::Path; use std::sync::atomic::{AtomicBool, AtomicU8, Ordering}; use std::sync::{Arc, LazyLock, RwLock}; use std::time::Instant; use rustfs_config::{ - DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, ENV_SCANNER_IDLE_MODE, ENV_SCANNER_SPEED, - ENV_SCANNER_YIELD_EVERY_N_OBJECTS, ScannerSpeed, + DEFAULT_SCANNER_IDLE_MODE, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, + ENV_SCANNER_DISK_IO_BUDGET_IOPS, ENV_SCANNER_IDLE_MODE, ENV_SCANNER_SPEED, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, ScannerSpeed, }; use rustfs_scanner_metrics::metrics::global_metrics; use tokio::time::Duration; +mod io_budget; + +pub use io_budget::{ + SCANNER_DISK_IO_BUDGET_MAX_WAIT, ScannerDiskIoBudget, ScannerDiskIoBudgetConfig, ScannerDiskIoBudgetLimits, + ScannerDiskIoBudgetStatus, ScannerDiskMedia, ScannerIoBudgetLimit, resolve_disk_io_budget, scanner_disk_io_budget_status, +}; + const MIN_SLEEP: Duration = Duration::from_millis(1); const SCANNER_SPEED_FASTEST: u8 = 0; const SCANNER_SPEED_FAST: u8 = 1; @@ -74,6 +82,19 @@ fn scanner_env_config() -> (ScannerSpeed, bool) { (speed, idle_mode) } +fn disk_io_budget_env_config(speed: ScannerSpeed) -> ScannerDiskIoBudgetConfig { + let limit = |key: &str| { + rustfs_utils::get_env_opt_str(key) + .and_then(|value| ScannerIoBudgetLimit::parse(&value)) + .unwrap_or(ScannerIoBudgetLimit::Auto) + }; + ScannerDiskIoBudgetConfig { + iops: limit(ENV_SCANNER_DISK_IO_BUDGET_IOPS), + bytes_per_sec: limit(ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC), + speed, + } +} + pub(crate) fn scanner_yield_every_n_objects() -> u64 { rustfs_utils::get_env_u64(ENV_SCANNER_YIELD_EVERY_N_OBJECTS, DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS) } @@ -102,6 +123,7 @@ pub static SCANNER_SLEEPER: LazyLock = LazyLock::new(|| { let sleeper = DynamicSleeper::new(speed); sleeper.record_throttle_config(); + sleeper.update_disk_io_budget(disk_io_budget_env_config(speed)); sleeper }); @@ -111,9 +133,12 @@ pub static SCANNER_SLEEPER: LazyLock = LazyLock::new(|| { /// `elapsed_work_time * factor`, clamped to `[MIN_SLEEP, max_sleep]`. /// For folder-level gaps, [`sleep_folder`] uses `MIN_SLEEP * factor`, /// clamped only by `max_sleep`. All sleeps are gated on [`SCANNER_IDLE_MODE`]; +/// the per-disk background I/O budget bound with [`DynamicSleeper::for_disk`] +/// is a hard ceiling that applies regardless of idle mode or preset. #[derive(Clone)] pub struct DynamicSleeper { inner: Arc, + disk_io_budget: Option>, } struct SleeperParams { @@ -128,9 +153,52 @@ impl DynamicSleeper { factor: RwLock::new(speed.sleep_factor()), max_sleep: RwLock::new(speed.max_sleep()), }), + disk_io_budget: None, } } + /// Bind this sleeper to the background I/O budget of one local disk. + /// Walkers that share a disk path share its buckets. + pub fn for_disk(&self, disk_path: &Path) -> Self { + Self { + inner: self.inner.clone(), + disk_io_budget: Some(io_budget::disk_io_budget_for(disk_path)), + } + } + + /// Whether byte charges can change pacing on the bound disk. + pub fn budgets_disk_io_bytes(&self) -> bool { + self.disk_io_budget.as_ref().is_some_and(|budget| budget.budgets_bytes()) + } + + /// Charge the bound disk's background I/O budget and wait it out. + /// + /// This is a hard ceiling layered on the local-disk foreground-priority + /// admission gate: it runs regardless of idle mode or speed preset, and + /// it only ever delays, so it can never admit more than the gate allows. + pub async fn admit_disk_io(&self, ops: u64, bytes: u64) { + if let Some(budget) = self.disk_io_budget.as_ref() { + budget.admit(ops, bytes).await; + } + } + + pub(crate) fn update_disk_io_budget(&self, config: ScannerDiskIoBudgetConfig) { + io_budget::update_disk_io_budget_config(config); + } + + /// Remove the per-disk budget for fixtures that walk large synthetic + /// namespaces through the local-disk path and test checkpoint semantics, + /// not pacing. nextest runs each test in its own process, so this does + /// not leak into other tests. + #[cfg(test)] + pub(crate) fn disable_disk_io_budget_for_tests(&self) { + self.update_disk_io_budget(ScannerDiskIoBudgetConfig { + iops: ScannerIoBudgetLimit::Unlimited, + bytes_per_sec: ScannerIoBudgetLimit::Unlimited, + speed: ScannerSpeed::Default, + }); + } + fn should_sleep() -> bool { SCANNER_IDLE_MODE.load(Ordering::Relaxed) } @@ -141,7 +209,14 @@ impl DynamicSleeper { (factor, max_sleep) } + /// Pace one directory listing: the preset folder gap, then one + /// operation charged to the bound disk's background I/O budget. pub async fn sleep_folder(&self) { + self.sleep_folder_preset().await; + self.admit_disk_io(1, 0).await; + } + + async fn sleep_folder_preset(&self) { if !Self::should_sleep() { return; } @@ -187,10 +262,12 @@ impl DynamicSleeper { self.update_params(speed.sleep_factor(), speed.max_sleep()); } - /// Reload speed and idle-mode settings from the current environment. + /// Reload speed, idle-mode, and disk I/O budget settings from the + /// current environment. pub fn refresh_from_env(&self) { let (speed, idle_mode) = scanner_env_config(); self.update_from_runtime_config(speed.sleep_factor(), speed.max_sleep(), idle_mode, scanner_yield_every_n_objects()); + self.update_disk_io_budget(disk_io_budget_env_config(speed)); } pub(crate) fn update_from_runtime_config( @@ -377,6 +454,73 @@ mod tests { SCANNER_IDLE_MODE.store(prev_mode, Ordering::Relaxed); } + #[tokio::test(start_paused = true)] + #[serial] + async fn sleeper_without_a_disk_binding_never_charges_a_budget() { + let s = DynamicSleeper::new(ScannerSpeed::Fastest); + s.update_disk_io_budget(ScannerDiskIoBudgetConfig { + iops: ScannerIoBudgetLimit::Explicit(0), + bytes_per_sec: ScannerIoBudgetLimit::Explicit(0), + speed: ScannerSpeed::Fastest, + }); + + let start = tokio::time::Instant::now(); + for _ in 0..10 { + s.sleep_folder().await; + s.admit_disk_io(1, 1 << 20).await; + } + assert_eq!(start.elapsed(), Duration::ZERO); + assert!(!s.budgets_disk_io_bytes()); + + s.update_disk_io_budget(ScannerDiskIoBudgetConfig::DEFAULT); + } + + #[tokio::test(start_paused = true)] + #[serial] + async fn sleeper_bound_to_a_disk_charges_folder_and_object_io_as_a_hard_ceiling() { + let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed); + // Sleeps are disabled and the preset is `fastest`: only the explicit + // budget can pace this walk. + SCANNER_IDLE_MODE.store(false, Ordering::Relaxed); + let s = DynamicSleeper::new(ScannerSpeed::Fastest); + s.update_disk_io_budget(ScannerDiskIoBudgetConfig { + iops: ScannerIoBudgetLimit::Explicit(2), + bytes_per_sec: ScannerIoBudgetLimit::Explicit(1_000), + speed: ScannerSpeed::Fastest, + }); + let disk = std::path::PathBuf::from(format!("/rustfs-scanner-sleeper-test-{}", std::process::id())); + let bound = s.for_disk(&disk); + assert!(bound.budgets_disk_io_bytes()); + + // Two listings fit the burst; the third owes one operation at 2/s. + let start = tokio::time::Instant::now(); + bound.sleep_folder().await; + bound.sleep_folder().await; + assert_eq!(start.elapsed(), Duration::ZERO); + bound.sleep_folder().await; + assert_eq!(start.elapsed(), Duration::from_millis(500)); + + // Bytes pace independently: 1500 bytes at 1000 B/s owes 500 ms on + // top of the operation debt already repaid. + tokio::time::advance(Duration::from_secs(5)).await; + let start = tokio::time::Instant::now(); + bound.admit_disk_io(1, 1_500).await; + assert_eq!(start.elapsed(), Duration::from_millis(500)); + + let status = scanner_disk_io_budget_status() + .into_iter() + .find(|entry| entry.disk == disk.to_string_lossy()) + .expect("bound disk reports a budget status"); + assert_eq!(status.iops, Some(2)); + assert_eq!(status.bytes_per_sec, Some(1_000)); + assert_eq!(status.wait_events, 2); + assert_eq!(status.charged_ops, 4); + assert_eq!(status.charged_bytes, 1_500); + + s.update_disk_io_budget(ScannerDiskIoBudgetConfig::DEFAULT); + SCANNER_IDLE_MODE.store(prev_mode, Ordering::Relaxed); + } + #[tokio::test(start_paused = true)] #[serial] async fn test_idle_mode_off_skips_sleep() { diff --git a/crates/scanner/src/sleeper/io_budget.rs b/crates/scanner/src/sleeper/io_budget.rs new file mode 100644 index 000000000..ed4701bc2 --- /dev/null +++ b/crates/scanner/src/sleeper/io_budget.rs @@ -0,0 +1,714 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Per-disk background I/O budget for the data scanner. +//! +//! Each local disk owns two token buckets, one for operations per second and +//! one for bytes per second. Every scanner directory listing and object +//! metadata read charges them before it runs; an exhausted bucket delays the +//! walk until the rate allows the charge again. The budget is a hard ceiling +//! layered on top of the local-disk foreground-priority admission gate: +//! admission decides *when* background I/O may start while foreground +//! requests are active, the budget bounds *how much* background I/O one disk +//! sees per second even while it is idle. A charge can never bypass the gate. +//! +//! A zero or tiny budget still admits one charge per +//! [`SCANNER_DISK_IO_BUDGET_MAX_WAIT`], so a walk keeps making progress +//! instead of starving. + +use std::collections::BTreeMap; +use std::path::Path; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, LazyLock, Mutex, RwLock}; + +use rustfs_config::{ + DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_BYTES_PER_SEC, DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_IOPS, + DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_BYTES_PER_SEC, DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_IOPS, ScannerSpeed, +}; +use rustfs_scanner_metrics::metrics::global_metrics; +use serde::Serialize; +use tokio::time::{Duration, Instant}; +use tracing::info; + +const LOG_COMPONENT_SCANNER: &str = "scanner"; +const LOG_SUBSYSTEM_RUNTIME: &str = "runtime"; +const EVENT_SCANNER_DISK_IO_BUDGET: &str = "scanner_disk_io_budget"; + +const OTEL_SCANNER_DISK_IO_BUDGET_WAIT_SECONDS: &str = "rustfs_scanner_disk_io_budget_wait_seconds"; +const OTEL_SCANNER_DISK_IO_BUDGET_WAITS_TOTAL: &str = "rustfs_scanner_disk_io_budget_waits_total"; + +/// Longest wait one budget charge may impose. +/// +/// This is the minimum service guarantee: whatever the configured rate, a +/// walk is admitted at least once per window. +pub const SCANNER_DISK_IO_BUDGET_MAX_WAIT: Duration = Duration::from_secs(1); + +/// Burst allowance expressed in seconds of the configured rate. One second +/// lets a short directory finish without pacing while a long walk converges +/// on the configured rate. +const BURST_WINDOW_SECS: f64 = 1.0; + +const BUDGET_LIMIT_AUTO: &str = "auto"; +const BUDGET_LIMIT_OFF: &str = "off"; + +/// Media class of a local disk as far as the scanner can tell. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ScannerDiskMedia { + Rotational, + Flash, + /// Media could not be detected; budgets fall back to the rotational + /// defaults because a spindle is the case that needs protection. + Unknown, +} + +impl ScannerDiskMedia { + pub fn as_str(self) -> &'static str { + match self { + Self::Rotational => "rotational", + Self::Flash => "flash", + Self::Unknown => "unknown", + } + } +} + +/// One configured budget axis: media default, no cap, or an explicit rate. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ScannerIoBudgetLimit { + Auto, + Unlimited, + Explicit(u64), +} + +impl ScannerIoBudgetLimit { + /// Parse the `auto` | `off` | `` spelling shared by + /// the persisted keys and the environment variables. + pub fn parse(value: &str) -> Option { + match value.trim().to_ascii_lowercase().as_str() { + BUDGET_LIMIT_AUTO => Some(Self::Auto), + BUDGET_LIMIT_OFF | "disabled" | "unlimited" | "none" => Some(Self::Unlimited), + number => number.parse::().ok().map(Self::Explicit), + } + } +} + +impl std::fmt::Display for ScannerIoBudgetLimit { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Auto => f.write_str(BUDGET_LIMIT_AUTO), + Self::Unlimited => f.write_str(BUDGET_LIMIT_OFF), + Self::Explicit(value) => write!(f, "{value}"), + } + } +} + +/// Configured budget inputs, resolved per disk once its media is known. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct ScannerDiskIoBudgetConfig { + pub iops: ScannerIoBudgetLimit, + pub bytes_per_sec: ScannerIoBudgetLimit, + /// The `fastest` preset means "do not pace"; `auto` budgets resolve to + /// unlimited under it while explicit values still apply. + pub speed: ScannerSpeed, +} + +impl ScannerDiskIoBudgetConfig { + pub const DEFAULT: Self = Self { + iops: ScannerIoBudgetLimit::Auto, + bytes_per_sec: ScannerIoBudgetLimit::Auto, + speed: ScannerSpeed::Default, + }; +} + +impl Default for ScannerDiskIoBudgetConfig { + fn default() -> Self { + Self::DEFAULT + } +} + +/// Effective per-disk rates; `None` means that axis is not capped. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ScannerDiskIoBudgetLimits { + pub iops: Option, + pub bytes_per_sec: Option, +} + +fn resolve_budget_axis(limit: ScannerIoBudgetLimit, speed: ScannerSpeed, media_default: u64) -> Option { + match limit { + ScannerIoBudgetLimit::Explicit(value) => Some(value), + ScannerIoBudgetLimit::Unlimited => None, + ScannerIoBudgetLimit::Auto if speed == ScannerSpeed::Fastest => None, + ScannerIoBudgetLimit::Auto => Some(media_default), + } +} + +/// Resolve the configured budget for one disk's media. Unknown media uses +/// the rotational defaults. +pub fn resolve_disk_io_budget(config: ScannerDiskIoBudgetConfig, media: ScannerDiskMedia) -> ScannerDiskIoBudgetLimits { + let (default_iops, default_bytes) = match media { + ScannerDiskMedia::Flash => ( + DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_IOPS, + DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_BYTES_PER_SEC, + ), + ScannerDiskMedia::Rotational | ScannerDiskMedia::Unknown => ( + DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_IOPS, + DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_BYTES_PER_SEC, + ), + }; + ScannerDiskIoBudgetLimits { + iops: resolve_budget_axis(config.iops, config.speed, default_iops), + bytes_per_sec: resolve_budget_axis(config.bytes_per_sec, config.speed, default_bytes), + } +} + +/// Rates, charges, and wait totals are counts; precision loss above 2^53 +/// is irrelevant for a per-second budget. +fn count_as_f64(value: u64) -> f64 { + value as f64 +} + +#[derive(Debug)] +struct TokenBucket { + rate_per_sec: f64, + capacity: f64, + tokens: f64, + refilled_at: Instant, +} + +impl TokenBucket { + fn new(rate_per_sec: u64, now: Instant) -> Self { + let rate_per_sec = count_as_f64(rate_per_sec); + let capacity = Self::capacity_for(rate_per_sec); + Self { + rate_per_sec, + capacity, + tokens: capacity, + refilled_at: now, + } + } + + fn capacity_for(rate_per_sec: f64) -> f64 { + (rate_per_sec * BURST_WINDOW_SECS).max(1.0) + } + + fn reconfigure(&mut self, rate_per_sec: u64, now: Instant) { + self.refill(now); + self.rate_per_sec = count_as_f64(rate_per_sec); + self.capacity = Self::capacity_for(self.rate_per_sec); + self.tokens = self.tokens.min(self.capacity); + } + + fn refill(&mut self, now: Instant) { + let elapsed = now.saturating_duration_since(self.refilled_at).as_secs_f64(); + self.tokens = (self.tokens + elapsed * self.rate_per_sec).min(self.capacity); + self.refilled_at = now; + } + + /// Charge `amount` and return how long the caller must wait before the + /// bucket is non-negative again, capped at the max wait. Debt is bounded + /// to one burst window so an oversized charge costs at most the max wait + /// plus one window, never a permanent stall. + fn charge(&mut self, amount: f64, now: Instant) -> Duration { + self.refill(now); + self.tokens -= amount; + if self.tokens >= 0.0 { + return Duration::ZERO; + } + let max_wait_secs = SCANNER_DISK_IO_BUDGET_MAX_WAIT.as_secs_f64(); + let wait_secs = if self.rate_per_sec > 0.0 { + (-self.tokens / self.rate_per_sec).min(max_wait_secs) + } else { + max_wait_secs + }; + self.tokens = self.tokens.max(-self.capacity); + // `wait_secs` is finite and within `[0, max_wait_secs]`. + Duration::from_secs_f64(wait_secs) + } +} + +#[derive(Debug)] +struct BudgetState { + limits: ScannerDiskIoBudgetLimits, + iops: Option, + bytes: Option, +} + +impl BudgetState { + fn new(limits: ScannerDiskIoBudgetLimits, now: Instant) -> Self { + Self { + limits, + iops: limits.iops.map(|rate| TokenBucket::new(rate, now)), + bytes: limits.bytes_per_sec.map(|rate| TokenBucket::new(rate, now)), + } + } + + fn reconfigure(&mut self, limits: ScannerDiskIoBudgetLimits, now: Instant) { + Self::reconfigure_axis(&mut self.iops, limits.iops, now); + Self::reconfigure_axis(&mut self.bytes, limits.bytes_per_sec, now); + self.limits = limits; + } + + fn reconfigure_axis(bucket: &mut Option, rate: Option, now: Instant) { + match (bucket.as_mut(), rate) { + (Some(bucket), Some(rate)) => bucket.reconfigure(rate, now), + (None, Some(rate)) => *bucket = Some(TokenBucket::new(rate, now)), + (_, None) => *bucket = None, + } + } +} + +/// The budget of one local disk, shared by every walker on that disk. +#[derive(Debug)] +pub struct ScannerDiskIoBudget { + disk: String, + media: ScannerDiskMedia, + state: Mutex, + wait_events: AtomicU64, + wait_millis: AtomicU64, + charged_ops: AtomicU64, + charged_bytes: AtomicU64, +} + +impl ScannerDiskIoBudget { + fn new(disk: String, media: ScannerDiskMedia, limits: ScannerDiskIoBudgetLimits, now: Instant) -> Self { + Self { + disk, + media, + state: Mutex::new(BudgetState::new(limits, now)), + wait_events: AtomicU64::new(0), + wait_millis: AtomicU64::new(0), + charged_ops: AtomicU64::new(0), + charged_bytes: AtomicU64::new(0), + } + } + + pub fn media(&self) -> ScannerDiskMedia { + self.media + } + + pub fn limits(&self) -> ScannerDiskIoBudgetLimits { + self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner()).limits + } + + /// Whether byte charges can change pacing; callers skip measuring sizes + /// when they cannot. + pub fn budgets_bytes(&self) -> bool { + self.limits().bytes_per_sec.is_some() + } + + fn reconfigure(&self, limits: ScannerDiskIoBudgetLimits) { + let now = Instant::now(); + self.state + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .reconfigure(limits, now); + } + + /// Charge `ops` operations and `bytes` bytes, returning the wait the + /// budget imposes before the caller may proceed. + pub fn charge(&self, ops: u64, bytes: u64) -> Duration { + self.charged_ops.fetch_add(ops, Ordering::Relaxed); + self.charged_bytes.fetch_add(bytes, Ordering::Relaxed); + let now = Instant::now(); + let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + let mut wait = Duration::ZERO; + if ops > 0 + && let Some(bucket) = state.iops.as_mut() + { + wait = wait.max(bucket.charge(count_as_f64(ops), now)); + } + if bytes > 0 + && let Some(bucket) = state.bytes.as_mut() + { + wait = wait.max(bucket.charge(count_as_f64(bytes), now)); + } + wait.min(SCANNER_DISK_IO_BUDGET_MAX_WAIT) + } + + /// Charge and wait out the budget. Runs regardless of idle mode or + /// speed preset: the budget is a ceiling, not a preset sleep. + pub async fn admit(&self, ops: u64, bytes: u64) { + let wait = self.charge(ops, bytes); + if wait.is_zero() { + return; + } + tokio::time::sleep(wait).await; + self.record_wait(wait); + } + + fn record_wait(&self, wait: Duration) { + self.wait_events.fetch_add(1, Ordering::Relaxed); + self.wait_millis + .fetch_add(u64::try_from(wait.as_millis()).unwrap_or(u64::MAX), Ordering::Relaxed); + metrics::histogram!(OTEL_SCANNER_DISK_IO_BUDGET_WAIT_SECONDS, "disk" => self.disk.clone()).record(wait.as_secs_f64()); + metrics::counter!(OTEL_SCANNER_DISK_IO_BUDGET_WAITS_TOTAL, "disk" => self.disk.clone()).increment(1); + global_metrics().record_scanner_disk_io_budget_wait(wait); + } + + pub fn status(&self) -> ScannerDiskIoBudgetStatus { + let limits = self.limits(); + ScannerDiskIoBudgetStatus { + disk: self.disk.clone(), + media: self.media, + iops: limits.iops, + bytes_per_sec: limits.bytes_per_sec, + charged_ops: self.charged_ops.load(Ordering::Relaxed), + charged_bytes: self.charged_bytes.load(Ordering::Relaxed), + wait_events: self.wait_events.load(Ordering::Relaxed), + wait_seconds: count_as_f64(self.wait_millis.load(Ordering::Relaxed)) / 1000.0, + } + } +} + +/// Status of one disk budget as reported by the scanner status endpoint. +#[derive(Clone, Debug, PartialEq, Serialize)] +pub struct ScannerDiskIoBudgetStatus { + pub disk: String, + pub media: ScannerDiskMedia, + /// `null` when that axis is uncapped. + pub iops: Option, + pub bytes_per_sec: Option, + pub charged_ops: u64, + pub charged_bytes: u64, + pub wait_events: u64, + pub wait_seconds: f64, +} + +static DISK_IO_BUDGET_CONFIG: RwLock = RwLock::new(ScannerDiskIoBudgetConfig::DEFAULT); + +static DISK_IO_BUDGETS: LazyLock>>> = + LazyLock::new(|| Mutex::new(BTreeMap::new())); + +pub(crate) fn current_disk_io_budget_config() -> ScannerDiskIoBudgetConfig { + *DISK_IO_BUDGET_CONFIG.read().unwrap_or_else(|poisoned| poisoned.into_inner()) +} + +/// Look up or create the budget for one local disk path. Walkers that share +/// a disk share its buckets; the media is detected once per path. +pub(crate) fn disk_io_budget_for(disk_path: &Path) -> Arc { + let key = disk_path.to_string_lossy().into_owned(); + let mut registry = DISK_IO_BUDGETS.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + if let Some(budget) = registry.get(&key) { + return budget.clone(); + } + let media = detect_disk_media(disk_path); + let config = current_disk_io_budget_config(); + let limits = resolve_disk_io_budget(config, media); + let budget = Arc::new(ScannerDiskIoBudget::new(key.clone(), media, limits, Instant::now())); + info!( + target: "rustfs::scanner", + event = EVENT_SCANNER_DISK_IO_BUDGET, + component = LOG_COMPONENT_SCANNER, + subsystem = LOG_SUBSYSTEM_RUNTIME, + state = "resolved", + disk = %key, + media = media.as_str(), + iops = ?limits.iops, + bytes_per_sec = ?limits.bytes_per_sec, + "Scanner disk I/O budget resolved" + ); + registry.insert(key, budget.clone()); + budget +} + +/// Apply a new configuration to every known disk budget. +pub(crate) fn update_disk_io_budget_config(config: ScannerDiskIoBudgetConfig) { + *DISK_IO_BUDGET_CONFIG.write().unwrap_or_else(|poisoned| poisoned.into_inner()) = config; + let registry = DISK_IO_BUDGETS.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + for budget in registry.values() { + budget.reconfigure(resolve_disk_io_budget(config, budget.media)); + } +} + +/// Per-disk budget status, ordered by disk path. +pub fn scanner_disk_io_budget_status() -> Vec { + let registry = DISK_IO_BUDGETS.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + registry.values().map(|budget| budget.status()).collect() +} + +#[cfg(target_os = "linux")] +fn detect_disk_media(disk_path: &Path) -> ScannerDiskMedia { + let Ok(devices) = rustfs_utils::os::get_physical_device_ids(disk_path.to_string_lossy().as_ref()) else { + return ScannerDiskMedia::Unknown; + }; + let mut detected = ScannerDiskMedia::Unknown; + for device in devices { + let candidates = [ + format!("/sys/block/{device}/queue/rotational"), + format!("/sys/dev/block/{device}/queue/rotational"), + ]; + let Some(content) = candidates + .iter() + .find_map(|candidate| std::fs::read_to_string(candidate).ok()) + else { + continue; + }; + match content.trim() { + // Any spindle in the backing stack makes the path rotational. + "1" => return ScannerDiskMedia::Rotational, + "0" => detected = ScannerDiskMedia::Flash, + _ => {} + } + } + detected +} + +#[cfg(not(target_os = "linux"))] +fn detect_disk_media(_disk_path: &Path) -> ScannerDiskMedia { + ScannerDiskMedia::Unknown +} + +#[cfg(test)] +mod tests { + use super::*; + use serial_test::serial; + + fn config(iops: ScannerIoBudgetLimit, bytes_per_sec: ScannerIoBudgetLimit, speed: ScannerSpeed) -> ScannerDiskIoBudgetConfig { + ScannerDiskIoBudgetConfig { + iops, + bytes_per_sec, + speed, + } + } + + #[test] + fn disk_io_budget_limit_parses_auto_off_and_numbers() { + assert_eq!(ScannerIoBudgetLimit::parse("auto"), Some(ScannerIoBudgetLimit::Auto)); + assert_eq!(ScannerIoBudgetLimit::parse(" AUTO "), Some(ScannerIoBudgetLimit::Auto)); + assert_eq!(ScannerIoBudgetLimit::parse("off"), Some(ScannerIoBudgetLimit::Unlimited)); + assert_eq!(ScannerIoBudgetLimit::parse("unlimited"), Some(ScannerIoBudgetLimit::Unlimited)); + assert_eq!(ScannerIoBudgetLimit::parse("0"), Some(ScannerIoBudgetLimit::Explicit(0))); + assert_eq!(ScannerIoBudgetLimit::parse("4096"), Some(ScannerIoBudgetLimit::Explicit(4096))); + assert_eq!(ScannerIoBudgetLimit::parse("-1"), None); + assert_eq!(ScannerIoBudgetLimit::parse("fast"), None); + assert_eq!(ScannerIoBudgetLimit::Auto.to_string(), "auto"); + assert_eq!(ScannerIoBudgetLimit::Unlimited.to_string(), "off"); + assert_eq!(ScannerIoBudgetLimit::Explicit(7).to_string(), "7"); + } + + #[test] + fn disk_io_budget_resolves_media_defaults_and_overrides() { + let auto = config(ScannerIoBudgetLimit::Auto, ScannerIoBudgetLimit::Auto, ScannerSpeed::Default); + assert_eq!( + resolve_disk_io_budget(auto, ScannerDiskMedia::Rotational), + ScannerDiskIoBudgetLimits { + iops: Some(DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_IOPS), + bytes_per_sec: Some(DEFAULT_SCANNER_DISK_IO_BUDGET_ROTATIONAL_BYTES_PER_SEC), + } + ); + // Unknown media is treated like a spindle. + assert_eq!( + resolve_disk_io_budget(auto, ScannerDiskMedia::Unknown), + resolve_disk_io_budget(auto, ScannerDiskMedia::Rotational) + ); + assert_eq!( + resolve_disk_io_budget(auto, ScannerDiskMedia::Flash), + ScannerDiskIoBudgetLimits { + iops: Some(DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_IOPS), + bytes_per_sec: Some(DEFAULT_SCANNER_DISK_IO_BUDGET_FLASH_BYTES_PER_SEC), + } + ); + // Rotational media is paced tighter than flash on both axes. + let rotational = resolve_disk_io_budget(auto, ScannerDiskMedia::Rotational); + let flash = resolve_disk_io_budget(auto, ScannerDiskMedia::Flash); + assert!(rotational.iops < flash.iops, "{rotational:?} vs {flash:?}"); + assert!(rotational.bytes_per_sec < flash.bytes_per_sec, "{rotational:?} vs {flash:?}"); + + // The fastest preset never paces unless a value is explicit. + let fastest = config(ScannerIoBudgetLimit::Auto, ScannerIoBudgetLimit::Explicit(5), ScannerSpeed::Fastest); + assert_eq!( + resolve_disk_io_budget(fastest, ScannerDiskMedia::Rotational), + ScannerDiskIoBudgetLimits { + iops: None, + bytes_per_sec: Some(5), + } + ); + + let explicit = config(ScannerIoBudgetLimit::Explicit(0), ScannerIoBudgetLimit::Unlimited, ScannerSpeed::Slow); + assert_eq!( + resolve_disk_io_budget(explicit, ScannerDiskMedia::Flash), + ScannerDiskIoBudgetLimits { + iops: Some(0), + bytes_per_sec: None, + } + ); + } + + #[tokio::test(start_paused = true)] + async fn disk_io_budget_waits_once_the_burst_is_spent() { + let budget = ScannerDiskIoBudget::new( + "disk-a".to_string(), + ScannerDiskMedia::Rotational, + ScannerDiskIoBudgetLimits { + iops: Some(10), + bytes_per_sec: None, + }, + Instant::now(), + ); + + // The burst window admits one second of rate without waiting. + for _ in 0..10 { + assert_eq!(budget.charge(1, 0), Duration::ZERO); + } + // The eleventh operation owes one token at 10/s. + assert_eq!(budget.charge(1, 0), Duration::from_millis(100)); + + // After the rate has refilled the deficit, the next charge is free again. + tokio::time::advance(Duration::from_millis(200)).await; + assert_eq!(budget.charge(1, 0), Duration::ZERO); + } + + #[tokio::test(start_paused = true)] + async fn disk_io_budget_bytes_axis_caps_independently_of_iops() { + let budget = ScannerDiskIoBudget::new( + "disk-b".to_string(), + ScannerDiskMedia::Flash, + ScannerDiskIoBudgetLimits { + iops: Some(1_000), + bytes_per_sec: Some(1_000), + }, + Instant::now(), + ); + + assert_eq!(budget.charge(1, 1_000), Duration::ZERO); + assert_eq!(budget.charge(1, 500), Duration::from_millis(500)); + assert!(budget.budgets_bytes()); + } + + #[tokio::test(start_paused = true)] + async fn disk_io_budget_zero_budget_still_admits_one_charge_per_window() { + let budget = ScannerDiskIoBudget::new( + "disk-c".to_string(), + ScannerDiskMedia::Unknown, + ScannerDiskIoBudgetLimits { + iops: Some(0), + bytes_per_sec: Some(0), + }, + Instant::now(), + ); + + // A zero budget starts with one token of burst, then degrades to the + // floor instead of waiting forever. + assert_eq!(budget.charge(1, 1), Duration::ZERO); + for _ in 0..3 { + assert_eq!(budget.charge(1, 4096), SCANNER_DISK_IO_BUDGET_MAX_WAIT); + } + + let started = Instant::now(); + budget.admit(1, 4096).await; + assert_eq!(started.elapsed(), SCANNER_DISK_IO_BUDGET_MAX_WAIT); + let status = budget.status(); + assert_eq!(status.wait_events, 1); + assert_eq!(status.wait_seconds, 1.0); + assert_eq!(status.iops, Some(0)); + assert_eq!(status.bytes_per_sec, Some(0)); + } + + #[tokio::test(start_paused = true)] + async fn disk_io_budget_oversized_charge_owes_at_most_one_extra_window() { + let budget = ScannerDiskIoBudget::new( + "disk-d".to_string(), + ScannerDiskMedia::Rotational, + ScannerDiskIoBudgetLimits { + iops: None, + bytes_per_sec: Some(1_000), + }, + Instant::now(), + ); + + // A charge far above the rate is capped at the max wait ... + assert_eq!(budget.charge(0, 1_000_000), SCANNER_DISK_IO_BUDGET_MAX_WAIT); + // ... and leaves at most one burst window of debt behind. + assert_eq!(budget.charge(0, 0), Duration::ZERO); + assert_eq!(budget.charge(0, 1), SCANNER_DISK_IO_BUDGET_MAX_WAIT); + tokio::time::advance(Duration::from_secs(2)).await; + assert_eq!(budget.charge(0, 1), Duration::ZERO); + } + + #[tokio::test(start_paused = true)] + async fn disk_io_budget_unlimited_axis_never_waits() { + let budget = ScannerDiskIoBudget::new( + "disk-e".to_string(), + ScannerDiskMedia::Flash, + ScannerDiskIoBudgetLimits::default(), + Instant::now(), + ); + + for _ in 0..1_000 { + assert_eq!(budget.charge(1, 1 << 20), Duration::ZERO); + } + assert!(!budget.budgets_bytes()); + let started = Instant::now(); + budget.admit(1, 1 << 20).await; + assert_eq!(started.elapsed(), Duration::ZERO); + assert_eq!(budget.status().wait_events, 0); + } + + #[tokio::test(start_paused = true)] + async fn disk_io_budget_reconfigure_applies_the_new_rate_to_live_buckets() { + let budget = ScannerDiskIoBudget::new( + "disk-f".to_string(), + ScannerDiskMedia::Rotational, + ScannerDiskIoBudgetLimits { + iops: Some(1), + bytes_per_sec: None, + }, + Instant::now(), + ); + assert_eq!(budget.charge(1, 0), Duration::ZERO); + assert_eq!(budget.charge(1, 0), SCANNER_DISK_IO_BUDGET_MAX_WAIT); + + budget.reconfigure(ScannerDiskIoBudgetLimits { + iops: Some(1_000), + bytes_per_sec: Some(10), + }); + // Debt carried over is repaid at the new rate. + assert_eq!(budget.charge(1, 0), Duration::from_millis(2)); + assert!(budget.budgets_bytes()); + + budget.reconfigure(ScannerDiskIoBudgetLimits::default()); + assert_eq!(budget.charge(1, 1 << 30), Duration::ZERO); + assert_eq!(budget.limits(), ScannerDiskIoBudgetLimits::default()); + } + + #[tokio::test(start_paused = true)] + #[serial] + async fn disk_io_budget_registry_shares_one_budget_per_disk_and_follows_config_updates() { + // A path that does not exist resolves to unknown media on every platform. + let path = std::path::PathBuf::from(format!("/rustfs-scanner-budget-test-{}", std::process::id())); + + let first = disk_io_budget_for(&path); + let second = disk_io_budget_for(&path); + assert!(Arc::ptr_eq(&first, &second)); + + let status = scanner_disk_io_budget_status(); + assert!(status.iter().any(|entry| entry.disk == path.to_string_lossy())); + + update_disk_io_budget_config(config( + ScannerIoBudgetLimit::Explicit(3), + ScannerIoBudgetLimit::Unlimited, + ScannerSpeed::Default, + )); + assert_eq!( + first.limits(), + ScannerDiskIoBudgetLimits { + iops: Some(3), + bytes_per_sec: None, + } + ); + + update_disk_io_budget_config(ScannerDiskIoBudgetConfig::DEFAULT); + assert_eq!(first.limits(), resolve_disk_io_budget(ScannerDiskIoBudgetConfig::DEFAULT, first.media())); + } +} diff --git a/crates/scanner/src/storage_api.rs b/crates/scanner/src/storage_api.rs index c0ff85a96..ccfc8a60d 100644 --- a/crates/scanner/src/storage_api.rs +++ b/crates/scanner/src/storage_api.rs @@ -101,6 +101,7 @@ pub(crate) use rustfs_ecstore::api::event::{EventArgs as EcstoreEventArgs, send_ #[cfg(test)] pub(crate) use rustfs_ecstore::api::layout::{ EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints, + SetupType as EcstoreSetupType, }; pub(crate) use rustfs_ecstore::api::notification::NotificationSys as EcstoreNotificationSys; pub(crate) use rustfs_ecstore::api::notification::scanner_peer_transport_error_message_is_retryable; @@ -140,10 +141,21 @@ pub(crate) use rustfs_lifecycle::{ }; use rustfs_storage_api as storage_contracts; +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) mod ecstore_diagnostic { + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::diagnostic::current_read_version_entry_source; + pub(crate) use rustfs_ecstore::api::diagnostic::{ + ReadVersionContextBoundary, ReadVersionEntrySource, observe_read_version_context_boundary, with_read_version_entry, + }; +} + #[cfg(test)] pub(crate) type EcstoreHealResultItem = ::HealResultItem; pub(crate) mod owner { + #[cfg(test)] + pub(crate) use super::storage_contracts::DiskSetSelector; pub(crate) use super::{ MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan, ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner, @@ -158,6 +170,7 @@ pub(crate) mod owner { #[cfg(test)] pub(crate) use rustfs_ecstore::api::set_disk::test_util::{ PutObjectCommitBarrier, PutObjectCommitPause, hold_namespace_commit as ecstore_hold_namespace_commit, + namespace_commits_pending as ecstore_namespace_commits_pending, }; pub(crate) use super::storage_contracts::{ diff --git a/crates/scanner/tests/lifecycle_integration_test.rs b/crates/scanner/tests/lifecycle_integration_test.rs index f913d114a..713f00edc 100644 --- a/crates/scanner/tests/lifecycle_integration_test.rs +++ b/crates/scanner/tests/lifecycle_integration_test.rs @@ -663,8 +663,17 @@ mod serial_tests { ); ExpiryState::resize_workers(1, ecstore.clone()).await; + set_bucket_lifecycle(bucket_name.as_str()) + .await + .expect("install the expiration policy"); let lc_event = LcEvent { action: IlmAction::DeleteAction, + expiry_policy: get_bucket_metadata(bucket_name.as_str()) + .await + .expect("load expiration policy") + .lifecycle_config + .clone() + .map(|configuration| Arc::new(configuration).into()), ..Default::default() }; let bucket_incarnation_id = ecstore @@ -672,6 +681,25 @@ mod serial_tests { .await .expect("read bucket incarnation"); + update_bucket_metadata(bucket_name.as_str(), BUCKET_LIFECYCLE_CONFIG, Vec::new()) + .await + .expect("withdraw the queued expiration policy"); + assert!( + expire_transitioned_object(ecstore.clone(), &oi, &lc_event, &LcEventSrc::Scanner, bucket_incarnation_id) + .await + .is_err(), + "a withdrawn policy must retain both local metadata and remote data" + ); + assert_eq!(backend.remove_count().await, 0); + assert!(backend.contains(&remote_object).await); + ecstore + .get_object_info(bucket_name.as_str(), object_name, &ObjectOptions::default()) + .await + .expect("withdrawn expiration must retain transition metadata"); + set_bucket_lifecycle(bucket_name.as_str()) + .await + .expect("restore the expiration policy"); + // Pause one real tier GET after it has resolved local transition // metadata. The reader still owns the object read lock, so local expiry // cannot commit until this GET finishes. @@ -2451,6 +2479,9 @@ mod serial_tests { // The scanner's restore-expiry action removes only the local restored // copy (same direct-drive pattern as the ilm-2 expiry test). + update_bucket_metadata(bucket_name.as_str(), BUCKET_LIFECYCLE_CONFIG, Vec::new()) + .await + .expect("restore-copy cleanup must not require a lifecycle policy"); let lc_event = LcEvent { action: IlmAction::DeleteRestoredAction, ..Default::default() diff --git a/crates/storage-api/src/error.rs b/crates/storage-api/src/error.rs index b24fedfe4..8c84c8a2c 100644 --- a/crates/storage-api/src/error.rs +++ b/crates/storage-api/src/error.rs @@ -107,6 +107,9 @@ pub enum StorageErrorCode { QuotaExceeded, RemoteClientUnavailable, RemoteNotInitialized, + ConditionalDeleteConflict, + GuardedDeleteUnsupported, + DeleteRecoveryLegacyAmbiguous, } impl StorageErrorCode { @@ -196,6 +199,9 @@ impl StorageErrorCode { Self::QuotaExceeded => 0x53, Self::RemoteClientUnavailable => 0x54, Self::RemoteNotInitialized => 0x55, + Self::ConditionalDeleteConflict => 0x57, + Self::GuardedDeleteUnsupported => 0x58, + Self::DeleteRecoveryLegacyAmbiguous => 0x59, } } @@ -285,6 +291,9 @@ impl StorageErrorCode { 0x53 => Some(Self::QuotaExceeded), 0x54 => Some(Self::RemoteClientUnavailable), 0x55 => Some(Self::RemoteNotInitialized), + 0x57 => Some(Self::ConditionalDeleteConflict), + 0x58 => Some(Self::GuardedDeleteUnsupported), + 0x59 => Some(Self::DeleteRecoveryLegacyAmbiguous), _ => None, } } @@ -382,6 +391,9 @@ mod tests { (StorageErrorCode::SourceStalled, 0x50), (StorageErrorCode::Timeout, 0x51), (StorageErrorCode::InvalidPath, 0x52), + (StorageErrorCode::ConditionalDeleteConflict, 0x57), + (StorageErrorCode::GuardedDeleteUnsupported, 0x58), + (StorageErrorCode::DeleteRecoveryLegacyAmbiguous, 0x59), ]; #[test] diff --git a/crates/storage-api/src/object.rs b/crates/storage-api/src/object.rs index b7aaf28dc..9b1d40315 100644 --- a/crates/storage-api/src/object.rs +++ b/crates/storage-api/src/object.rs @@ -199,10 +199,79 @@ pub struct ObjectToDeleteIdentity { pub delete_marker: bool, } +/// A single ETag condition on the current live object of a bulk delete. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub enum DeleteObjectCondition { + Exists, + MatchEtag(String), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeleteObjectConditionError { + InvalidEtag, +} + +impl fmt::Display for DeleteObjectConditionError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::InvalidEtag => f.write_str("a delete ETag condition must contain one non-empty ETag"), + } + } +} + +impl std::error::Error for DeleteObjectConditionError {} + +impl DeleteObjectCondition { + pub fn parse(value: &str) -> Result { + let value = value.trim(); + let value = match (value.starts_with('"'), value.ends_with('"')) { + (true, true) if value.len() >= 2 => &value[1..value.len() - 1], + (false, false) => value, + _ => return Err(DeleteObjectConditionError::InvalidEtag), + }; + if value.trim().is_empty() || value.contains('"') { + return Err(DeleteObjectConditionError::InvalidEtag); + } + if value == "*" { + Ok(Self::Exists) + } else { + Ok(Self::MatchEtag(value.to_string())) + } + } + + /// Reject malformed values constructed directly or decoded from a peer. + pub fn validate(&self) -> Result<(), DeleteObjectConditionError> { + match self { + Self::Exists => Ok(()), + Self::MatchEtag(value) if !value.trim().is_empty() && value != "*" && !value.contains('"') => Ok(()), + Self::MatchEtag(_) => Err(DeleteObjectConditionError::InvalidEtag), + } + } + + pub fn requires_read_permission(&self) -> bool { + matches!(self, Self::MatchEtag(_)) + } + + /// The caller must separately require a live, existing object. + pub fn matches(&self, object_etag: &str) -> bool { + match self { + Self::Exists => true, + Self::MatchEtag(expected) => { + let object_etag = object_etag + .strip_prefix('"') + .and_then(|value| value.strip_suffix('"')) + .unwrap_or(object_etag); + self.validate().is_ok() && expected == object_etag + } + } + } +} + #[derive(Debug, Default, Clone)] pub struct ObjectToDelete { pub object_name: String, pub version_id: Option, + pub delete_condition: Option, /// RustFS-only precondition checked under the object write lock. #[doc(hidden)] pub expected_identity: Option, @@ -761,6 +830,41 @@ fn is_modified_since(mod_time: &OffsetDateTime, given_time: &OffsetDateTime) -> mod tests { use super::*; + #[test] + fn delete_object_condition_rejects_empty_or_malformed_etags() { + for value in ["", " \t ", "\"\"", "\" \t \"", "\"", "\"etag", "etag\"", "\"\"etag\"\""] { + assert_eq!(DeleteObjectCondition::parse(value), Err(DeleteObjectConditionError::InvalidEtag)); + } + assert!(DeleteObjectCondition::MatchEtag(String::new()).validate().is_err()); + assert!(!DeleteObjectCondition::MatchEtag(String::new()).matches("")); + assert!(DeleteObjectCondition::MatchEtag("*".to_string()).validate().is_err()); + } + + #[test] + fn delete_object_condition_is_a_case_sensitive_opaque_single_etag() { + let condition = DeleteObjectCondition::parse(" \"opaque-ETag-7\" ").expect("quoted ETag should parse"); + assert_eq!(condition, DeleteObjectCondition::MatchEtag("opaque-ETag-7".to_string())); + assert!(condition.matches("opaque-ETag-7")); + assert!(condition.matches("\"opaque-ETag-7\"")); + assert!(!condition.matches("opaque-etag-7")); + assert!(condition.requires_read_permission()); + assert!(condition.validate().is_ok()); + + let condition = DeleteObjectCondition::parse("one,two").expect("ETags are opaque values"); + assert!(condition.matches("one,two")); + assert!(!condition.matches("one")); + assert!(!condition.matches("two")); + } + + #[test] + fn delete_object_condition_exists_does_not_require_read_permission() { + for value in ["*", " \"*\" "] { + let condition = DeleteObjectCondition::parse(value).expect("wildcard condition should parse"); + assert_eq!(condition, DeleteObjectCondition::Exists); + assert!(!condition.requires_read_permission()); + } + } + #[test] fn version_marker_parse_folds_null_and_nil_uuid_into_null() { assert_eq!(VersionMarker::parse("null"), Ok(VersionMarker::Null)); diff --git a/crates/utils/src/dunce.rs b/crates/utils/src/dunce.rs index b6405057c..42133cde1 100644 --- a/crates/utils/src/dunce.rs +++ b/crates/utils/src/dunce.rs @@ -21,6 +21,7 @@ use std::ffi::OsStr; #[cfg(windows)] use std::path::{Component, Prefix}; use std::{ + borrow::Cow, fs, io, path::{Path, PathBuf}, }; @@ -106,6 +107,115 @@ fn canonicalize_win(path: &Path) -> io::Result { Ok(try_simplified(&real_path).map(PathBuf::from).unwrap_or(real_path)) } +/// Returns a form of `path` that Win32 file APIs accept beyond `MAX_PATH`. +/// +/// `std::fs` adds the `\\?\` prefix internally, but crates that call Win32 +/// directly with the caller's path (for example `tempfile`'s `persist`, which +/// uses `MoveFileExW`) fail with `ERROR_PATH_NOT_FOUND` once the path reaches +/// 260 UTF-16 units, unless the process opted into long paths through both the +/// `LongPathsEnabled` policy and an application manifest. +/// +/// The path is first made absolute with `GetFullPathNameW` (via +/// [`std::path::absolute`]), so separators, `.`/`..` and trailing dots or spaces +/// are normalized exactly as a non-verbatim Win32 call would see them, and only +/// then prefixed with `\\?\` (or `\\?\UNC\`). Verbatim and device paths are +/// returned unchanged. +/// +/// On non-Windows this is a no-op. +pub fn extended_length_path(path: &Path) -> io::Result> { + #[cfg(not(windows))] + { + Ok(Cow::Borrowed(path)) + } + #[cfg(windows)] + { + extended_length_path_win(path).map(Cow::Owned) + } +} + +#[cfg(windows)] +fn extended_length_path_win(path: &Path) -> io::Result { + use std::{ + ffi::OsString, + os::windows::ffi::{OsStrExt, OsStringExt}, + }; + + let absolute = std::path::absolute(path)?; + let unc = match absolute.components().next() { + Some(Component::Prefix(prefix)) => match prefix.kind() { + Prefix::Disk(_) => false, + Prefix::UNC(..) => true, + // Verbatim and `\\.\` device paths already bypass Win32 normalization. + _ => return Ok(absolute), + }, + _ => return Ok(absolute), + }; + let wide = absolute.as_os_str().encode_wide().collect::>(); + let (prefix, rest) = if unc { + // `\\server\share\x` becomes `\\?\UNC\server\share\x`. + (r"\\?\UNC", &wide[1..]) + } else { + (r"\\?\", &wide[..]) + }; + let mut extended = prefix.encode_utf16().collect::>(); + extended.extend_from_slice(rest); + Ok(PathBuf::from(OsString::from_wide(&extended))) +} + +/// Like [`simplified`], but without the `MAX_PATH` limit, and also turning +/// `\\?\UNC\server\share\...` into `\\server\share\...`. +/// +/// Use it for a root that callers extend by joining `/`-separated relative +/// paths as strings: inside a verbatim path `/` is not a separator, so every +/// such join fails with `ERROR_INVALID_NAME`. `std::fs` re-adds the +/// extended-length prefix to long paths, and code handing paths to Win32 +/// directly goes through [`extended_length_path`]. A path with a component +/// that Win32 would rewrite (reserved names, trailing dots or spaces, ...) or +/// that is not valid UTF-8 is returned unchanged, as are volume GUID and +/// device paths. +/// +/// On non-Windows this is a no-op. +#[must_use] +pub fn non_verbatim(path: &Path) -> Cow<'_, Path> { + #[cfg(not(windows))] + { + Cow::Borrowed(path) + } + #[cfg(windows)] + { + try_non_verbatim(path).map_or(Cow::Borrowed(path), Cow::Owned) + } +} + +#[cfg(windows)] +fn try_non_verbatim(path: &Path) -> Option { + let mut components = path.components(); + let unc = match components.next() { + Some(Component::Prefix(prefix)) => match prefix.kind() { + Prefix::VerbatimDisk(..) => false, + Prefix::VerbatimUNC(..) => true, + _ => return None, + }, + _ => return None, + }; + for component in components { + match component { + Component::RootDir => {} + Component::Normal(file_name) if is_valid_filename(file_name) && !is_reserved(file_name) => {} + _ => return None, + } + } + + let s = path.to_str()?; + Some(if unc { + // `\\?\UNC\server\share` (8-character prefix) becomes `\\server\share`. + PathBuf::from(format!(r"\\{}", s.get(8..)?)) + } else { + // `\\?\C:\x` (4-character prefix) becomes `C:\x`. + PathBuf::from(s.get(4..)?) + }) +} + /// Returns `true` if the path is relative or starts with a disk prefix (like "C:"). /// /// This checks if the path is in a "simplified" format. On Windows, this means it @@ -134,6 +244,64 @@ pub fn is_simplified(path: &Path) -> bool { #[doc(hidden)] pub use self::canonicalize as realpath; +/// Why a single path segment cannot be used as a regular Win32 file name. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WindowsFileNameError { + /// The segment is empty. + Empty, + /// The segment exceeds 255 UTF-16 code units. + TooLong, + /// The segment contains one of `< > : " / \ | ? *`. + ReservedCharacter, + /// The segment contains a control character (0x00-0x1F). + ControlCharacter, + /// The segment ends with `.` or a space, which Win32 path parsing strips. + TrailingDotOrSpace, + /// The segment names a DOS device (e.g. `NUL`, `COM1`, `CONIN$`), bare or + /// with an extension. + ReservedDeviceName, +} + +/// Checks whether `name` can be created *and* addressed again as a single +/// path segment through the Win32 API. +/// +/// This is the one rule set for Windows file-name validity: it backs +/// [`simplified()`] and the object-key admission checks in ecstore. The rules +/// apply on every platform so callers can evaluate them without a Windows +/// host; whether to enforce them is the caller's decision. +pub fn check_windows_file_name(name: &str) -> Result<(), WindowsFileNameError> { + // Windows filename limit is 255 characters (UTF-16 code units) + // We check byte length first as a fast path optimization. + if name.len() > 255 && name.encode_utf16().count() > 255 { + return Err(WindowsFileNameError::TooLong); + } + + if name.is_empty() { + return Err(WindowsFileNameError::Empty); + } + + // Only ASCII subset is checked, and WTF-8/UTF-8 is safe for that + // Invalid chars: < > : " / \ | ? * and control chars (0-31) + let byte_str = name.as_bytes(); + if byte_str + .iter() + .any(|&c| matches!(c, b'<' | b'>' | b':' | b'"' | b'/' | b'\\' | b'|' | b'?' | b'*')) + { + return Err(WindowsFileNameError::ReservedCharacter); + } + if byte_str.iter().any(|&c| c <= 31) { + return Err(WindowsFileNameError::ControlCharacter); + } + // Filename can't end with . or space + if matches!(byte_str.last(), Some(b' ' | b'.')) { + return Err(WindowsFileNameError::TrailingDotOrSpace); + } + if is_reserved_name(name) { + return Err(WindowsFileNameError::ReservedDeviceName); + } + Ok(()) +} + /// Checks if the filename is valid for a regular Windows path. /// /// This checks for: @@ -149,30 +317,7 @@ fn is_valid_filename(file_name: &OsStr) -> bool { None => return false, }; - // Windows filename limit is 255 characters (UTF-16 code units) - // We check byte length first as a fast path optimization. - if s.len() > 255 && s.encode_utf16().count() > 255 { - return false; - } - - if s.is_empty() { - return false; - } - - // Only ASCII subset is checked, and WTF-8/UTF-8 is safe for that - // Invalid chars: < > : " / \ | ? * and control chars (0-31) - let byte_str = s.as_bytes(); - if byte_str - .iter() - .any(|&c| matches!(c, 0..=31 | b'<' | b'>' | b':' | b'"' | b'/' | b'\\' | b'|' | b'?' | b'*')) - { - return false; - } - // Filename can't end with . or space - if matches!(byte_str.last(), Some(b' ' | b'.')) { - return false; - } - true + matches!(check_windows_file_name(s), Ok(()) | Err(WindowsFileNameError::ReservedDeviceName)) } /// Checks if the filename is a reserved DOS device name (e.g., CON, PRN, COM1). @@ -180,11 +325,13 @@ fn is_valid_filename(file_name: &OsStr) -> bool { /// Reserved names are case-insensitive and apply even if an extension is present (e.g., "CON.txt" is reserved). #[cfg(any(windows, test))] fn is_reserved>(file_name: P) -> bool { - let s = match file_name.as_ref().to_str() { - Some(s) => s, - None => return false, // Reserved names are all ASCII - }; + match file_name.as_ref().to_str() { + Some(s) => is_reserved_name(s), + None => false, // Reserved names are all ASCII + } +} +fn is_reserved_name(s: &str) -> bool { if s.is_empty() { return false; } @@ -200,25 +347,26 @@ fn is_reserved>(file_name: P) -> bool { // Trim trailing spaces (Windows ignores them, so "CON " is "CON") let stem = stem.trim_end(); - // Reserved names are 3 or 4 chars - if stem.len() != 3 && stem.len() != 4 { + if ["CON", "PRN", "AUX", "NUL", "CONIN$", "CONOUT$"] + .iter() + .any(|name| stem.eq_ignore_ascii_case(name)) + { + return true; + } + + // Check COM1-9, LPT1-9, and the ISO 8859-1 superscript digits Windows + // also accepts as port numbers (COM¹, LPT³, ...). + let (Some(base), Some(suffix)) = (stem.get(..3), stem.get(3..)) else { + return false; + }; + if !base.eq_ignore_ascii_case("COM") && !base.eq_ignore_ascii_case("LPT") { return false; } - - if stem.len() == 3 { - return stem.eq_ignore_ascii_case("CON") - || stem.eq_ignore_ascii_case("PRN") - || stem.eq_ignore_ascii_case("AUX") - || stem.eq_ignore_ascii_case("NUL"); + let mut digits = suffix.chars(); + match (digits.next(), digits.next()) { + (Some(digit), None) => matches!(digit, '1'..='9' | '\u{b9}' | '\u{b2}' | '\u{b3}'), + _ => false, } - - // Check COM1-9, LPT1-9 - let (base, suffix) = stem.split_at(3); - if base.eq_ignore_ascii_case("COM") || base.eq_ignore_ascii_case("LPT") { - return suffix.chars().next().map(|c| c.is_ascii_digit() && c != '0').unwrap_or(false); - } - - false } #[cfg(not(windows))] @@ -306,14 +454,39 @@ mod tests { "CON.tar.gz", "con.", "con .", + "COM\u{b9}", + "com\u{b2}.txt", + "LPT\u{b3}", + "CONIN$", + "conout$", + "CONIN$.txt", ]; for name in reserved { assert!(is_reserved(name), "Should be reserved: {}", name); } let not_reserved = [ - "CON0", "COM0", "LPT0", "COM10", "LPT10", "CONsole", "PrNter", "NuLly", "not_con", "con_not", ".CON", "@CON", - "CON。", "foo.con", + "CON0", + "COM0", + "LPT0", + "COM10", + "LPT10", + "CONsole", + "PrNter", + "NuLly", + "not_con", + "con_not", + ".CON", + "@CON", + "CON。", + "foo.con", + "COM\u{b9}0", + "COM\u{bd}", + "CONIN", + "CONIN$$", + "中文名", + "é12", + "COMé", ]; for name in not_reserved { assert!(!is_reserved(name), "Should NOT be reserved: {}", name); @@ -367,6 +540,25 @@ mod tests { assert!(is_valid_filename(OsStr::new(&ok_name))); } + #[test] + fn test_check_windows_file_name_reports_the_violated_rule() { + assert_eq!(check_windows_file_name("file.txt"), Ok(())); + assert_eq!(check_windows_file_name(""), Err(WindowsFileNameError::Empty)); + assert_eq!(check_windows_file_name(&"a".repeat(256)), Err(WindowsFileNameError::TooLong)); + for name in ["a:b", "a\\b", "a/b", "a*b", "a?b", "a\"b", "ab", "a|b"] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::ReservedCharacter), "{name:?}"); + } + for name in ["a\0b", "a\x01b", "tab\there", "a\x1fb"] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::ControlCharacter), "{name:?}"); + } + for name in ["dir.", "file ", "..", "CON."] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::TrailingDotOrSpace), "{name:?}"); + } + for name in ["NUL", "nul.txt", "COM\u{b9}", "CONOUT$"] { + assert_eq!(check_windows_file_name(name), Err(WindowsFileNameError::ReservedDeviceName), "{name:?}"); + } + } + #[test] #[cfg(windows)] fn test_simplified_windows() { @@ -390,6 +582,70 @@ mod tests { assert_eq!(simplified(Path::new(r"\\.\C:\Windows")), Path::new(r"\\.\C:\Windows")); } + #[test] + #[cfg(windows)] + fn extended_length_path_prefixes_normalized_windows_paths() { + assert_eq!( + &*extended_length_path(Path::new(r"C:/data/./bucket/../bucket/object")).unwrap(), + Path::new(r"\\?\C:\data\bucket\object") + ); + assert_eq!( + &*extended_length_path(Path::new(r"\\server\share\data\object")).unwrap(), + Path::new(r"\\?\UNC\server\share\data\object") + ); + for unchanged in [r"\\?\C:\data\object", r"\\?\UNC\server\share\object", r"\\.\C:\data"] { + assert_eq!(&*extended_length_path(Path::new(unchanged)).unwrap(), Path::new(unchanged)); + } + + let long = format!(r"C:\data\{}\xl.meta.bkp", "a".repeat(300)); + let extended = extended_length_path(Path::new(&long)).unwrap(); + assert_eq!(&*extended, Path::new(&format!(r"\\?\{long}"))); + assert_eq!(simplified(&extended), &*extended, "a long path must stay verbatim"); + } + + #[test] + #[cfg(windows)] + fn non_verbatim_strips_verbatim_prefixes_without_the_max_path_limit() { + let long = format!(r"C:\data\{}\root", "a".repeat(250)); + assert_eq!(&*non_verbatim(Path::new(&format!(r"\\?\{long}"))), Path::new(&long)); + assert_eq!( + &*non_verbatim(Path::new(r"\\?\UNC\localhost\C$\rfs\smb")), + Path::new(r"\\localhost\C$\rfs\smb") + ); + let long_unc = format!(r"\\?\UNC\server\share\{}", "b".repeat(250)); + assert_eq!(&*non_verbatim(Path::new(&long_unc)), Path::new(&format!(r"\\{}", &long_unc[8..]))); + + for unchanged in [ + r"C:\data\root", + r"\\server\share\root", + r"\\?\C:\data\CON", + r"\\?\C:\data\trailing.", + r"\\?\C:\data\..\root", + r"\\?\Volume{00000000-0000-0000-0000-000000000000}\root", + r"\\?\GLOBALROOT\Device\HarddiskVolume1\root", + r"\\.\C:\data", + ] { + assert!( + matches!(non_verbatim(Path::new(unchanged)), Cow::Borrowed(p) if p == Path::new(unchanged)), + "{unchanged} must be returned unchanged" + ); + } + } + + #[test] + #[cfg(not(windows))] + fn non_verbatim_is_a_noop_off_windows() { + let path = Path::new("/data/root"); + assert!(matches!(non_verbatim(path), Cow::Borrowed(p) if p == path)); + } + + #[test] + #[cfg(not(windows))] + fn extended_length_path_is_a_noop_off_windows() { + let path = Path::new("/data/./bucket/object"); + assert!(matches!(extended_length_path(path).unwrap(), Cow::Borrowed(p) if p == path)); + } + #[test] #[cfg(not(windows))] fn test_simplified_unix() { diff --git a/crates/utils/src/http/metadata_compat.rs b/crates/utils/src/http/metadata_compat.rs index 305933cd8..8ae681427 100644 --- a/crates/utils/src/http/metadata_compat.rs +++ b/crates/utils/src/http/metadata_compat.rs @@ -223,6 +223,19 @@ pub fn insert_str(map: &mut HashMap, suffix: &str, value: String map.insert(k2, value); } +fn eq_prefix_suffix_ignore_ascii_case(key: &str, prefix: &str, suffix: &str) -> bool { + if key.len() != prefix.len() + suffix.len() { + return false; + } + let (key_prefix, key_suffix) = key.as_bytes().split_at(prefix.len()); + key_prefix.eq_ignore_ascii_case(prefix.as_bytes()) && key_suffix.eq_ignore_ascii_case(suffix.as_bytes()) +} + +fn matches_either_internal_key_ignore_ascii_case(key: &str, suffix: &str) -> bool { + eq_prefix_suffix_ignore_ascii_case(key, RUSTFS_INTERNAL_PREFIX, suffix) + || eq_prefix_suffix_ignore_ascii_case(key, MINIO_INTERNAL_PREFIX, suffix) +} + pub fn get_str(map: &HashMap, suffix: &str) -> Option { if let Some(v) = with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.get(k1).cloned()) { return Some(v); @@ -231,17 +244,15 @@ pub fn get_str(map: &HashMap, suffix: &str) -> Option { return Some(v); } // Rare fallback: case-insensitive scan for non-canonical key casing. - let (k1, k2) = both_keys(suffix); map.iter() - .find(|(key, _)| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2)) + .find(|(key, _)| matches_either_internal_key_ignore_ascii_case(key, suffix)) .map(|(_, value)| value.clone()) } fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap, suffix: &str) -> Option<&'a V> { - let (rustfs_key, minio_key) = both_keys(suffix); let mut value = None; for (key, candidate) in map { - if !key.eq_ignore_ascii_case(&rustfs_key) && !key.eq_ignore_ascii_case(&minio_key) { + if !matches_either_internal_key_ignore_ascii_case(key, suffix) { continue; } if candidate.as_ref().is_empty() || value.is_some_and(|current: &V| current.as_ref() != candidate.as_ref()) { @@ -266,16 +277,14 @@ pub fn contains_key_str(map: &HashMap, suffix: &str) -> bool { if with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.contains_key(k2)) { return true; } - let (k1, k2) = both_keys(suffix); map.keys() - .any(|key| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2)) + .any(|key| matches_either_internal_key_ignore_ascii_case(key, suffix)) } pub fn remove_str(map: &mut HashMap, suffix: &str) { with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.remove(k1)); with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.remove(k2)); - let (k1, k2) = both_keys(suffix); - map.retain(|key, _| !key.eq_ignore_ascii_case(&k1) && !key.eq_ignore_ascii_case(&k2)); + map.retain(|key, _| !matches_either_internal_key_ignore_ascii_case(key, suffix)); } // === Vec type (meta_sys) === @@ -458,6 +467,16 @@ mod tests { assert_eq!(get_consistent_str(&metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID), None); } + #[test] + fn test_consistent_str_rejects_conflicting_mixed_case_aliases() { + let metadata = HashMap::from([ + ("X-RustFS-Internal-transition-tier-destination-id".to_string(), "identity-a".to_string()), + ("X-Minio-Internal-Transition-Tier-Destination-Id".to_string(), "identity-b".to_string()), + ]); + + assert_eq!(get_consistent_str(&metadata, SUFFIX_TRANSITION_TIER_DESTINATION_ID), None); + } + #[test] fn test_consistent_bytes_accepts_single_or_matching_values_and_rejects_conflicts() { let rustfs_key = internal_key_rustfs(SUFFIX_TRANSITION_TIER_DESTINATION_ID); diff --git a/crates/utils/src/os/mod.rs b/crates/utils/src/os/mod.rs index 0d5ed3c22..1708dae36 100644 --- a/crates/utils/src/os/mod.rs +++ b/crates/utils/src/os/mod.rs @@ -64,6 +64,13 @@ pub struct DiskInfo { pub files: u64, pub ffree: u64, pub fstype: String, + /// The volume is reached through a network redirector. + /// + /// Windows names the server's filesystem (usually `NTFS`) in `fstype` for + /// SMB shares, so remote volumes are reported here instead. On Unix the + /// client filesystem type (`nfs`, `cifs`, `smb2`, ...) already identifies + /// network mounts and this stays `false`. + pub remote: bool, pub major: u64, pub minor: u64, pub name: String, diff --git a/crates/utils/src/os/windows.rs b/crates/utils/src/os/windows.rs index 237adbed3..84d1b1023 100644 --- a/crates/utils/src/os/windows.rs +++ b/crates/utils/src/os/windows.rs @@ -13,33 +13,113 @@ // limitations under the License. use crate::os::{DiskInfo, IOStats}; -use std::io::Error; -use std::path::Path; -use windows::Win32::Foundation::MAX_PATH; -use windows::Win32::Storage::FileSystem::{GetDiskFreeSpaceExW, GetDiskFreeSpaceW, GetVolumeInformationW, GetVolumePathNameW}; +use std::io::{Error, ErrorKind}; +use std::os::windows::ffi::OsStrExt; +use std::os::windows::fs::OpenOptionsExt; +use std::os::windows::io::AsRawHandle; +use std::path::{Component, Path, Prefix}; +use windows::Win32::Foundation::{HANDLE, MAX_PATH}; +use windows::Win32::Storage::FileSystem::{ + FILE_FLAG_BACKUP_SEMANTICS, FILE_REMOTE_PROTOCOL_INFO, FileRemoteProtocolInfo, GetDiskFreeSpaceExW, GetDiskFreeSpaceW, + GetDriveTypeW, GetFileInformationByHandleEx, GetFinalPathNameByHandleW, GetVolumeInformationW, GetVolumePathNameW, + VOLUME_NAME_NT, +}; +use windows::core::PCWSTR; + +/// Documented maximum for the `GetVolumeInformationW` name buffers. +const VOLUME_INFORMATION_BUFFER_LEN: usize = MAX_PATH as usize + 1; + +/// `GetDriveTypeW` result for network drives; defined locally because the +/// binding lives behind the unrelated `Win32_System_WindowsProgramming` feature. +const DRIVE_REMOTE: u32 = 4; + +const BACKSLASH: u16 = b'\\' as u16; +const SLASH: u16 = b'/' as u16; + +/// A NUL-terminated UTF-16 path that is safe to hand to Win32 as a `PCWSTR`. +#[derive(Debug, PartialEq, Eq)] +struct WidePath(Vec); + +impl WidePath { + /// Terminates `units` with NUL, rejecting interior NULs that would silently + /// truncate the path seen by Win32. + fn new(mut units: Vec) -> std::io::Result { + if units.contains(&0) { + return Err(Error::new(ErrorKind::InvalidInput, "path contains an interior NUL character")); + } + units.push(0); + Ok(Self(units)) + } + + /// Returns `path` in the extended-length (`\\?\` or `\\?\UNC\`) form, so + /// the volume APIs keep working beyond `MAX_PATH`. + /// + /// The path is first made absolute with `GetFullPathNameW` (via + /// [`std::path::absolute`]) so separators, `.`/`..` and trailing dots or + /// spaces are normalized as a non-verbatim Win32 call would see them. + /// Verbatim and `\\.\` device paths are kept unchanged. + fn extended(path: &Path) -> std::io::Result { + let absolute = std::path::absolute(path)?; + let wide = absolute.as_os_str().encode_wide().collect::>(); + let (prefix, rest) = match absolute.components().next() { + Some(Component::Prefix(prefix)) => match prefix.kind() { + Prefix::Disk(_) => (r"\\?\", &wide[..]), + // `\\server\share\x` becomes `\\?\UNC\server\share\x`. + Prefix::UNC(..) => (r"\\?\UNC", &wide[1..]), + _ => return Self::new(wide), + }, + _ => return Self::new(wide), + }; + let mut extended = prefix.encode_utf16().collect::>(); + extended.extend_from_slice(rest); + Self::new(extended) + } + + /// Like [`WidePath::extended`], with the trailing backslash that + /// directory APIs such as `GetDiskFreeSpaceExW` require for UNC paths. + fn directory(path: &Path) -> std::io::Result { + let mut path = Self::extended(path)?; + path.0.pop(); + if !matches!(path.0.last(), Some(&BACKSLASH | &SLASH)) { + path.0.push(BACKSLASH); + } + path.0.push(0); + Ok(path) + } + + fn units(&self) -> &[u16] { + &self.0[..self.0.len() - 1] + } + + fn as_pcwstr(&self) -> PCWSTR { + PCWSTR::from_raw(self.0.as_ptr()) + } +} /// Returns total and free bytes available in a directory, e.g. `C:\`. -// SAFETY: Windows API calls receive null-terminated UTF-16 paths and valid +// SAFETY: Windows API calls receive NUL-terminated UTF-16 paths and valid // pointers to initialized stack output variables. #[allow(unsafe_code)] pub fn get_info(p: impl AsRef) -> std::io::Result { - let path_wide = to_wide_path(p.as_ref()); + let directory = WidePath::directory(p.as_ref())?; let mut free_bytes_available = 0u64; let mut total_number_of_bytes = 0u64; let mut total_number_of_free_bytes = 0u64; + // The directory itself, not its volume root, is queried so per-directory + // quotas (e.g. FSRM) are reflected in the reported capacity. // SAFETY: - // 1. `path_wide` is a valid null-terminated UTF-16 string. + // 1. `directory` is a NUL-terminated UTF-16 string owned for the whole call. // 2. Pointers to `u64` variables are valid and point to initialized stack memory. unsafe { GetDiskFreeSpaceExW( - windows::core::PCWSTR::from_raw(path_wide.as_ptr()), + directory.as_pcwstr(), Some(&mut free_bytes_available), Some(&mut total_number_of_bytes), Some(&mut total_number_of_free_bytes), ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; + .map_err(win32_error)?; } let total = total_number_of_bytes; @@ -52,23 +132,27 @@ pub fn get_info(p: impl AsRef) -> std::io::Result { ))); } + // `GetDiskFreeSpaceW` and `GetVolumeInformationW` take a volume root, not + // an arbitrary directory. + let volume = get_volume_path(&directory)?; + let mut sectors_per_cluster = 0u32; let mut bytes_per_sector = 0u32; let mut number_of_free_clusters = 0u32; let mut total_number_of_clusters = 0u32; // SAFETY: - // 1. `path_wide` is a valid null-terminated UTF-16 string. + // 1. `volume` is a NUL-terminated UTF-16 string owned for the whole call. // 2. Pointers to `u32` variables are valid and point to initialized stack memory. unsafe { GetDiskFreeSpaceW( - windows::core::PCWSTR::from_raw(path_wide.as_ptr()), + volume.as_pcwstr(), Some(&mut sectors_per_cluster), Some(&mut bytes_per_sector), Some(&mut number_of_free_clusters), Some(&mut total_number_of_clusters), ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; + .map_err(win32_error)?; } Ok(DiskInfo { @@ -77,63 +161,207 @@ pub fn get_info(p: impl AsRef) -> std::io::Result { used: total - free, files: total_number_of_clusters as u64, ffree: number_of_free_clusters as u64, - fstype: get_windows_fs_type(&path_wide).unwrap_or_default(), + fstype: get_volume_information(&volume) + .map(|info| info.file_system_name) + .unwrap_or_default(), + // A failed handle probe is not conclusive; the volume's drive type + // still covers mapped drives and UNC shares. + remote: is_remote_directory(p.as_ref()).unwrap_or(false) || is_remote_volume(&volume)?, ..Default::default() }) } -// SAFETY: Windows volume APIs receive null-terminated UTF-16 paths and fixed -// stack buffers sized for the documented MAX_PATH outputs used here. -#[allow(unsafe_code)] -fn get_windows_fs_type(p: &[u16]) -> std::io::Result { - let path = get_volume_name(p)?; - - let mut volume_serial_number = 0u32; - let mut maximum_component_length = 0u32; - let mut file_system_flags = 0u32; - let mut volume_name_buffer = [0u16; MAX_PATH as usize]; - let mut file_system_name_buffer = [0u16; MAX_PATH as usize]; - - // SAFETY: - // 1. `path` is a valid null-terminated UTF-16 string (volume root path). - // 2. Buffers are allocated with `MAX_PATH` size, which is sufficient for standard Windows paths. - // 3. Pointers to output variables are valid. - unsafe { - GetVolumeInformationW( - windows::core::PCWSTR::from_raw(path.as_ptr()), - Some(&mut volume_name_buffer), - Some(&mut volume_serial_number), - Some(&mut maximum_component_length), - Some(&mut file_system_flags), - Some(&mut file_system_name_buffer), - ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; - } - - Ok(utf16_to_string(&file_system_name_buffer)) +struct VolumeInformation { + serial_number: u32, + file_system_name: String, } -// SAFETY: `v` is a null-terminated UTF-16 path and `volume_name_buffer` is a -// writable MAX_PATH-sized stack buffer for the returned volume path. +// SAFETY: `GetVolumeInformationW` receives a NUL-terminated volume root and a +// slice whose length the bindings pass as the buffer size. #[allow(unsafe_code)] -fn get_volume_name(v: &[u16]) -> std::io::Result> { - let mut volume_name_buffer = [0u16; MAX_PATH as usize]; +fn get_volume_information(volume: &WidePath) -> std::io::Result { + let mut serial_number = 0u32; + let mut file_system_name = [0u16; VOLUME_INFORMATION_BUFFER_LEN]; // SAFETY: - // 1. `v` is a valid null-terminated UTF-16 string. - // 2. `volume_name_buffer` is allocated with `MAX_PATH` size. - // 3. `GetVolumePathNameW` writes to the buffer and respects the buffer size (implicitly MAX_PATH for this API context usually, though explicit length param isn't present, it expects a buffer large enough). - // Note: GetVolumePathNameW documentation says "The buffer should be large enough to hold the path". MAX_PATH is generally safe for volume roots. + // 1. `volume` is a NUL-terminated UTF-16 volume root with a trailing backslash, + // as returned by `GetVolumePathNameW`. + // 2. `file_system_name` holds the documented maximum of `MAX_PATH + 1` units and + // its length is passed alongside the pointer. + // 3. The serial number pointer targets an initialized stack variable. unsafe { - GetVolumePathNameW(windows::core::PCWSTR::from_raw(v.as_ptr()), &mut volume_name_buffer) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; + GetVolumeInformationW( + volume.as_pcwstr(), + None, + Some(&mut serial_number), + None, + None, + Some(&mut file_system_name), + ) + .map_err(win32_error)?; } - let len = volume_name_buffer - .iter() - .position(|&x| x == 0) - .unwrap_or(volume_name_buffer.len()); - Ok(volume_name_buffer[..len].to_vec()) + Ok(VolumeInformation { + serial_number, + file_system_name: utf16_to_string(&file_system_name), + }) +} + +/// NT namespace prefix of paths served by the Multiple UNC Provider, which +/// fronts every network redirector (UNC paths and mapped drive letters). +const MUP_DEVICE_PREFIX: &str = r"\Device\Mup\"; + +/// Opens `path` for metadata queries only, following symlinks and junctions. +fn open_directory(path: &Path) -> std::io::Result { + std::fs::OpenOptions::new() + .access_mode(0) + .custom_flags(FILE_FLAG_BACKUP_SEMANTICS.0) + .open(path) +} + +/// Reports whether `path` is served by a network redirector, following +/// symlinks and junctions to their target. +/// +/// Either signal is sufficient: `FileRemoteProtocolInfo` only succeeds on +/// handles opened through a redirector, and the handle's final NT path lies +/// under `\Device\Mup\` for every UNC or mapped-drive target. +fn is_remote_directory(path: &Path) -> std::io::Result { + let directory = open_directory(path)?; + let handle = HANDLE(directory.as_raw_handle()); + if remote_protocol_info(handle).is_ok() { + return Ok(true); + } + let final_path = final_nt_path(handle)?; + Ok(final_path + .get(..MUP_DEVICE_PREFIX.len()) + .is_some_and(|prefix| prefix.eq_ignore_ascii_case(MUP_DEVICE_PREFIX))) +} + +/// Queries the remote protocol behind `handle`; local volumes fail with +/// `ERROR_INVALID_PARAMETER`. +// SAFETY: `GetFileInformationByHandleEx` receives a caller-owned open handle +// and a pointer/size pair describing a local `FILE_REMOTE_PROTOCOL_INFO`. +#[allow(unsafe_code)] +fn remote_protocol_info(handle: HANDLE) -> std::io::Result { + let mut info = FILE_REMOTE_PROTOCOL_INFO::default(); + // SAFETY: + // 1. The caller keeps `handle` open for the whole call. + // 2. `info` is a writable, initialized `FILE_REMOTE_PROTOCOL_INFO` and its + // exact size is passed as the buffer size. + unsafe { + GetFileInformationByHandleEx( + handle, + FileRemoteProtocolInfo, + (&raw mut info).cast(), + size_of::() as u32, + ) + .map_err(win32_error)?; + } + Ok(info) +} + +/// Returns the NT namespace path of `handle`, e.g. `\Device\HarddiskVolume3\x` +/// or `\Device\Mup\server\share\x`. +// SAFETY: `GetFinalPathNameByHandleW` receives a caller-owned open handle and +// a writable slice whose length the bindings pass as the buffer size. +#[allow(unsafe_code)] +fn final_nt_path(handle: HANDLE) -> std::io::Result { + let mut buffer = vec![0u16; MAX_PATH as usize + 1]; + loop { + // SAFETY: + // 1. The caller keeps `handle` open for the whole call. + // 2. `buffer` is writable and its length is passed as `cchFilePath`. + let len = unsafe { GetFinalPathNameByHandleW(handle, &mut buffer, VOLUME_NAME_NT) } as usize; + if len == 0 { + return Err(Error::last_os_error()); + } + // On success the length excludes the NUL; a length that does not fit + // is the required size including it. + if len < buffer.len() { + return Ok(String::from_utf16_lossy(&buffer[..len])); + } + buffer.resize(len, 0); + } +} + +/// Reports whether `volume`, a root returned by [`get_volume_path`], is a +/// network drive such as a mapped drive letter or a UNC share root. +// SAFETY: `GetDriveTypeW` receives a NUL-terminated volume root. +#[allow(unsafe_code)] +fn is_remote_volume(volume: &WidePath) -> std::io::Result { + // `GetDriveTypeW` documents the conventional `C:\` and `\\server\share\` + // root forms, so the extended-length prefix is removed first. + let root = WidePath::new(simplify_extended(volume.units()).into_owned())?; + // SAFETY: `root` is a NUL-terminated UTF-16 string owned for the whole call. + let drive_type = unsafe { GetDriveTypeW(root.as_pcwstr()) }; + Ok(drive_type == DRIVE_REMOTE) +} + +/// Returns the NUL-terminated root of the volume that contains `path`, e.g. +/// `\\?\C:\` or a mounted folder such as `\\?\C:\mnt\disk1\`. +// SAFETY: `GetVolumePathNameW` receives a NUL-terminated path and a writable +// slice whose length the bindings pass as the buffer size. +#[allow(unsafe_code)] +fn get_volume_path(path: &WidePath) -> std::io::Result { + // The volume path is a prefix of the full input path plus a trailing + // backslash, so the input length (including its NUL) plus one always fits. + let mut buffer = vec![0u16; (path.0.len() + 1).max(MAX_PATH as usize + 1)]; + + // SAFETY: + // 1. `path` is a NUL-terminated UTF-16 string owned for the whole call. + // 2. `buffer` is writable and its length is passed as `cchBufferLength`. + unsafe { + GetVolumePathNameW(path.as_pcwstr(), &mut buffer).map_err(win32_error)?; + } + + let len = buffer.iter().position(|&x| x == 0).unwrap_or(buffer.len()); + buffer.truncate(len); + WidePath::new(buffer) +} + +/// Returns the identity of the volume containing `path` in its conventional +/// form (`C:\`, `\\server\share\`), independent of whether the caller used +/// an extended-length path. +fn volume_id(path: &str) -> std::io::Result { + let volume = get_volume_path(&WidePath::extended(Path::new(path))?)?; + Ok(String::from_utf16_lossy(&simplify_extended(volume.units()))) +} + +/// Strips the `\\?\` prefix from a drive path and maps `\\?\UNC\` back to +/// `\\`; other verbatim paths (e.g. `\\?\Volume{GUID}\`) are returned as is. +fn simplify_extended(units: &[u16]) -> std::borrow::Cow<'_, [u16]> { + const VERBATIM: [u16; 4] = [BACKSLASH, BACKSLASH, b'?' as u16, BACKSLASH]; + let Some(rest) = units.strip_prefix(&VERBATIM[..]) else { + return units.into(); + }; + if let [drive, colon, ..] = rest + && *colon == b':' as u16 + && char::from_u32(u32::from(*drive)).is_some_and(|c| c.is_ascii_alphabetic()) + { + return rest.into(); + } + if let [u, n, c, BACKSLASH, share @ ..] = rest + && String::from_utf16_lossy(&[*u, *n, *c]).eq_ignore_ascii_case("UNC") + { + let mut simplified = vec![BACKSLASH, BACKSLASH]; + simplified.extend_from_slice(share); + return simplified.into(); + } + units.into() +} + +/// Converts a failed `windows` call into an `io::Error` carrying the Win32 +/// error code, so `raw_os_error()` and `kind()` match `std::fs` errors. +fn win32_error(err: windows::core::Error) -> Error { + const FACILITY_WIN32_MASK: u32 = 0xFFFF_0000; + const FACILITY_WIN32_FAILURE: u32 = 0x8007_0000; + + let hresult = err.code().0 as u32; + if hresult & FACILITY_WIN32_MASK == FACILITY_WIN32_FAILURE { + Error::from_raw_os_error((hresult & !FACILITY_WIN32_MASK) as i32) + } else { + Error::other(err) + } } fn utf16_to_string(v: &[u16]) -> String { @@ -141,21 +369,8 @@ fn utf16_to_string(v: &[u16]) -> String { String::from_utf16_lossy(&v[..len]) } -fn to_wide_path(path: &Path) -> Vec { - path.as_os_str().encode_wide().chain(std::iter::once(0)).collect() -} - -// Helper trait to access encode_wide which is only available on Windows -use std::os::windows::ffi::OsStrExt; - pub fn same_disk(disk1: &str, disk2: &str) -> std::io::Result { - let path1_wide = to_wide_path(Path::new(disk1)); - let path2_wide = to_wide_path(Path::new(disk2)); - - let volume1 = get_volume_name(&path1_wide)?; - let volume2 = get_volume_name(&path2_wide)?; - - Ok(volume1 == volume2) + Ok(volume_id(disk1)? == volume_id(disk2)?) } /// Automatic replacement formatting is disabled until Windows has an @@ -165,46 +380,16 @@ pub fn is_mount_point(_path: &std::path::Path) -> std::io::Result { } pub fn get_physical_device_ids(disk: &str) -> std::io::Result> { - let path_wide = to_wide_path(Path::new(disk)); - let volume = get_volume_name(&path_wide)?; - - Ok(vec![String::from_utf16_lossy(&volume)]) + Ok(vec![volume_id(disk)?]) } /// Returns the volume serial number for the filesystem backing `path`. /// /// This is the Windows equivalent of Linux's `st_dev` major:minor pair and is /// useful for diagnostic purposes when validating physical disk independence. -// SAFETY: `path_wide` is a valid null-terminated UTF-16 string and all output -// SAFETY: pointers target initialized stack variables with documented MAX_PATH sizing. -#[allow(unsafe_code)] pub fn get_volume_serial_number(path: &str) -> std::io::Result { - let path_wide = to_wide_path(Path::new(path)); - let volume = get_volume_name(&path_wide)?; - - let mut volume_serial_number = 0u32; - let mut maximum_component_length = 0u32; - let mut file_system_flags = 0u32; - let mut volume_name_buffer = [0u16; MAX_PATH as usize]; - let mut file_system_name_buffer = [0u16; MAX_PATH as usize]; - - // SAFETY: - // 1. `volume` is a valid null-terminated UTF-16 string (volume root path). - // 2. Buffers are allocated with `MAX_PATH` size. - // 3. Pointers to output variables are valid. - unsafe { - GetVolumeInformationW( - windows::core::PCWSTR::from_raw(volume.as_ptr()), - Some(&mut volume_name_buffer), - Some(&mut volume_serial_number), - Some(&mut maximum_component_length), - Some(&mut file_system_flags), - Some(&mut file_system_name_buffer), - ) - .map_err(|e| Error::from_raw_os_error(e.code().0))?; - } - - Ok(volume_serial_number) + let volume = get_volume_path(&WidePath::extended(Path::new(path))?)?; + Ok(get_volume_information(&volume)?.serial_number) } pub fn check_cross_device_mounts(_paths: &[String]) -> std::io::Result<()> { @@ -214,3 +399,231 @@ pub fn check_cross_device_mounts(_paths: &[String]) -> std::io::Result<()> { pub fn get_drive_stats(_major: u32, _minor: u32) -> std::io::Result { Ok(IOStats::default()) } + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + + fn wide(s: &str) -> Vec { + s.encode_utf16().collect() + } + + /// Describes both handle signals for `path`, for assertion messages. + fn handle_probe(path: &Path) -> String { + let directory = match open_directory(path) { + Ok(directory) => directory, + Err(err) => return format!("open failed: {err:?}"), + }; + let handle = HANDLE(directory.as_raw_handle()); + let protocol = remote_protocol_info(handle).map(|info| (info.Protocol, info.ProtocolMajorVersion, info.Flags)); + format!("remote protocol {protocol:?}, final NT path {:?}", final_nt_path(handle)) + } + + /// Creates a directory whose absolute path is well beyond `MAX_PATH`. + fn long_directory(root: &Path) -> PathBuf { + let mut dir = root.to_path_buf(); + for i in 0..6 { + dir.push(format!("{i}-{}", "a".repeat(60))); + } + std::fs::create_dir_all(&dir).expect("std::fs creates long directories itself"); + assert!(dir.as_os_str().len() > MAX_PATH as usize); + dir + } + + #[test] + fn windows_wide_path_is_nul_terminated_extended_length() { + let disk = WidePath::extended(Path::new(r"C:/data/./bucket/../bucket")).unwrap(); + assert_eq!(disk.units(), wide(r"\\?\C:\data\bucket")); + assert_eq!(disk.0.last(), Some(&0)); + + let unc = WidePath::extended(Path::new(r"\\server\share\data")).unwrap(); + assert_eq!(unc.units(), wide(r"\\?\UNC\server\share\data")); + + for unchanged in [r"\\?\C:\data", r"\\?\UNC\server\share\data", r"\\.\C:\data"] { + assert_eq!(WidePath::extended(Path::new(unchanged)).unwrap().units(), wide(unchanged)); + } + + let relative = WidePath::extended(Path::new("relative")).unwrap(); + assert!(relative.units().starts_with(&wide(r"\\?\")), "relative paths are made absolute first"); + } + + #[test] + fn windows_directory_path_has_single_trailing_backslash() { + // A UNC share root without the trailing backslash is rejected by + // `GetDiskFreeSpaceExW`. + let share = WidePath::directory(Path::new(r"\\server\share")).unwrap(); + assert_eq!(share.units(), wide(r"\\?\UNC\server\share\")); + let root = WidePath::directory(Path::new(r"C:\")).unwrap(); + assert_eq!(root.units(), wide(r"\\?\C:\")); + assert_eq!(root.0.last(), Some(&0)); + } + + #[test] + fn windows_wide_path_rejects_interior_nul() { + let err = WidePath::new(wide("C:\\a\0b")).unwrap_err(); + assert_eq!(err.kind(), ErrorKind::InvalidInput); + } + + #[test] + fn windows_simplify_extended_restores_conventional_volume_ids() { + for (input, expected) in [ + (r"\\?\C:\", r"C:\"), + (r"\\?\UNC\server\share\", r"\\server\share\"), + (r"\\?\unc\server\share\", r"\\server\share\"), + ( + r"\\?\Volume{0b6e1f0a-0000-0000-0000-100000000000}\", + r"\\?\Volume{0b6e1f0a-0000-0000-0000-100000000000}\", + ), + (r"C:\", r"C:\"), + ] { + assert_eq!(&*simplify_extended(&wide(input)), wide(expected).as_slice(), "{input}"); + } + } + + #[test] + fn windows_win32_error_keeps_os_error_code() { + // HRESULT_FROM_WIN32(ERROR_PATH_NOT_FOUND) + let err = win32_error(windows::core::Error::from_hresult(windows::core::HRESULT(0x8007_0003_u32 as i32))); + assert_eq!(err.raw_os_error(), Some(3)); + assert_eq!(err.kind(), ErrorKind::NotFound); + + let missing = get_info(r"Z:\rustfs-missing\path").unwrap_err(); + assert!(missing.raw_os_error().is_some_and(|code| code < 0x1_0000), "{missing:?}"); + } + + #[test] + fn windows_get_info_reports_fstype() { + let temp_dir = tempfile::tempdir().unwrap(); + let info = get_info(temp_dir.path()).unwrap(); + assert!(info.total > 0); + assert!(!info.fstype.is_empty(), "GetVolumeInformationW must name the filesystem"); + assert!(!info.remote, "the temp dir is on a local volume"); + assert!( + !is_remote_directory(temp_dir.path()).unwrap(), + "local handle: {}", + handle_probe(temp_dir.path()) + ); + + let ids = get_physical_device_ids(temp_dir.path().to_str().unwrap()).unwrap(); + assert_eq!(ids.len(), 1); + assert!(!ids[0].starts_with(r"\\?\"), "volume ids keep their conventional form: {ids:?}"); + assert!(ids[0].ends_with('\\'), "{ids:?}"); + } + + #[test] + fn windows_volume_apis_accept_paths_beyond_max_path() { + let temp_dir = tempfile::tempdir().unwrap(); + let short = temp_dir.path(); + let long = long_directory(short); + let (short_str, long_str) = (short.to_str().unwrap(), long.to_str().unwrap()); + + let short_info = get_info(short).unwrap(); + let long_info = get_info(&long).unwrap(); + assert!(long_info.total > 0); + assert!(!long_info.fstype.is_empty()); + assert_eq!(long_info.fstype, short_info.fstype); + assert_eq!(long_info.files, short_info.files); + + assert!(same_disk(short_str, long_str).unwrap()); + assert_eq!(get_physical_device_ids(long_str).unwrap(), get_physical_device_ids(short_str).unwrap()); + assert_eq!(get_volume_serial_number(long_str).unwrap(), get_volume_serial_number(short_str).unwrap()); + + let verbatim = format!(r"\\?\{long_str}"); + assert_eq!(get_physical_device_ids(&verbatim).unwrap(), get_physical_device_ids(short_str).unwrap()); + } + + #[test] + fn windows_get_info_accepts_unc_share_root_without_trailing_backslash() { + let temp_dir = tempfile::tempdir().unwrap(); + let absolute = std::path::absolute(temp_dir.path()).unwrap(); + let local = absolute.to_str().unwrap(); + let Some((drive, rest)) = local.split_once(":\\") else { + eprintln!("skipping: temp dir {local} is not on a drive letter"); + return; + }; + let share = format!(r"\\localhost\{drive}$"); + if !Path::new(&share).is_dir() { + eprintln!("skipping: administrative share {share} is not reachable"); + return; + } + + let share_info = get_info(&share).unwrap(); + assert!(share_info.total > 0); + assert!(!share_info.fstype.is_empty()); + // The share reports the server's filesystem, so only the drive type + // reveals that it is reached over SMB. + assert!(share_info.remote, "{share} is a network share: {share_info:?}"); + // The handle probe alone must recognize the share, since it is the + // only signal for links whose volume root is local. + assert!( + is_remote_directory(Path::new(&share)).unwrap(), + "share handle: {}", + handle_probe(Path::new(&share)) + ); + + let long = long_directory(&PathBuf::from(format!(r"{share}\{rest}"))); + let long_info = get_info(&long).unwrap(); + assert!(!long_info.fstype.is_empty()); + assert!(long_info.remote, "{long:?} is below a network share"); + assert!(same_disk(&share, long.to_str().unwrap()).unwrap()); + } + + #[test] + fn windows_get_info_follows_directory_symlink_to_network_share() { + let temp_dir = tempfile::tempdir().unwrap(); + let absolute = std::path::absolute(temp_dir.path()).unwrap(); + let Some((drive, rest)) = absolute.to_str().unwrap().split_once(":\\") else { + eprintln!("skipping: temp dir {absolute:?} is not on a drive letter"); + return; + }; + let target = PathBuf::from(format!(r"\\localhost\{drive}$\{rest}")); + if !target.is_dir() { + eprintln!("skipping: administrative share path {target:?} is not reachable"); + return; + } + let link = temp_dir.path().join("share-link"); + if let Err(err) = std::os::windows::fs::symlink_dir(&target, &link) { + eprintln!("skipping: cannot create directory symlinks: {err}"); + return; + } + + // The link itself lives on a local volume, so its volume root alone + // need not reveal that the target is a share. + let info = get_info(&link).unwrap(); + assert!( + info.remote, + "{link:?} -> {target:?} resolves to a network share: {info:?}; link handle: {}", + handle_probe(&link) + ); + } + + /// Set by the Windows filesystem workflow after mapping a drive letter to + /// the administrative share of the temp dir's drive with `net use`. + const MAPPED_DRIVE_ENV: &str = "RUSTFS_TEST_WINDOWS_MAPPED_DRIVE"; + + #[test] + fn windows_get_info_reports_mapped_network_drive_as_remote() { + let Ok(drive) = std::env::var(MAPPED_DRIVE_ENV) else { + eprintln!("skipping: {MAPPED_DRIVE_ENV} is not set"); + return; + }; + let temp_dir = tempfile::tempdir().unwrap(); + let absolute = std::path::absolute(temp_dir.path()).unwrap(); + let Some((_, rest)) = absolute.to_str().unwrap().split_once(":\\") else { + eprintln!("skipping: temp dir {absolute:?} is not on a drive letter"); + return; + }; + let root = format!(r"{}\", drive.trim_end_matches('\\')); + let mapped = PathBuf::from(format!("{root}{rest}")); + if !mapped.is_dir() { + eprintln!("skipping: {mapped:?} is not reachable through the mapped drive"); + return; + } + + for path in [Path::new(&root), mapped.as_path()] { + let info = get_info(path).unwrap(); + assert!(info.remote, "{path:?} is on a mapped network drive: {info:?}"); + } + } +} diff --git a/crates/utils/src/path.rs b/crates/utils/src/path.rs index 8b53e764b..4f52ffd12 100644 --- a/crates/utils/src/path.rs +++ b/crates/utils/src/path.rs @@ -200,6 +200,29 @@ pub fn path_join_buf(elements: &[&str]) -> String { dst } +/// Object name in the segment form the local disk resolves it to. +/// +/// Windows disks treat `\` as a path separator and `path_join_buf` drops the +/// empty segments that leaves behind, so `a\b`, `a\\b`, `\a\b` and `a\b\` +/// all address the xl.meta of `a/b`. In-memory identities that must agree +/// with that file (namespace locks, metadata caches) fold through this. +/// Elsewhere `\` is an ordinary key byte and the name is returned verbatim. +pub fn object_name_disk_identity(object: &str) -> Cow<'_, str> { + fold_object_name_path_separators(object, cfg!(target_os = "windows")) +} + +/// Splits on `/` and, when `backslash_is_separator`, `\`, dropping empty +/// segments. `.` and `..` segments stay as they are: that can only leave two +/// spellings of one file apart, never merge two different files. +pub fn fold_object_name_path_separators(object: &str, backslash_is_separator: bool) -> Cow<'_, str> { + // Names without `\` already passed the `//` and leading-`/` checks. + if !backslash_is_separator || !object.contains('\\') { + return Cow::Borrowed(object); + } + let segments: Vec<&str> = object.split(['/', '\\']).filter(|segment| !segment.is_empty()).collect(); + Cow::Owned(segments.join(SLASH_SEPARATOR)) +} + /// path_needs_clean returns whether path cleaning may change the path. /// Will detect all cases that will be cleaned, /// but may produce false positives on non-trivial paths. @@ -624,6 +647,27 @@ mod tests { assert_eq!(encode_dir_object("prefix/directory/"), encoded); } + #[test] + fn fold_object_name_path_separators_folds_backslash_only_as_separator() { + assert_eq!(fold_object_name_path_separators("a\\b\\c", true), "a/b/c"); + assert_eq!(fold_object_name_path_separators("a/b\\c", true), "a/b/c"); + // Empty segments collapse on disk, so these address `a/b` as well. + assert_eq!(fold_object_name_path_separators("a\\\\b", true), "a/b"); + assert_eq!(fold_object_name_path_separators("a/\\b", true), "a/b"); + assert_eq!(fold_object_name_path_separators("\\a\\b", true), "a/b"); + assert_eq!(fold_object_name_path_separators("a\\b\\", true), "a/b"); + assert_eq!(fold_object_name_path_separators("a\\b__XLDIR__", true), "a/b__XLDIR__"); + assert!(matches!(fold_object_name_path_separators("a/b", true), Cow::Borrowed("a/b"))); + assert!(matches!(fold_object_name_path_separators("a\\b", false), Cow::Borrowed("a\\b"))); + } + + #[test] + fn object_name_disk_identity_follows_platform_separator() { + let expected = if cfg!(target_os = "windows") { "a/b" } else { "a\\b" }; + assert_eq!(object_name_disk_identity("a\\b"), expected); + assert!(matches!(object_name_disk_identity("a/b"), Cow::Borrowed("a/b"))); + } + #[test] fn test_trim_etag() { // Test with quoted ETag diff --git a/deny.toml b/deny.toml index b810318e7..1a84bdbbb 100644 --- a/deny.toml +++ b/deny.toml @@ -19,9 +19,6 @@ no-default-features = false version = 2 yanked = "deny" ignore = [ - # `paste 1.0.15` — unmaintained. No direct dependency. - # owner: rustfs-maintainers review: 2026-07 - { id = "RUSTSEC-2024-0436", reason = "paste unmaintained; transitive only; tracked for upgrade" }, # `rsa` Marvin timing sidechannel (RUSTSEC-2023-0071). Pulled in via # `openidconnect` (transitive) and historically used directly. No upstream # fix is available yet. Tracked separately for follow-up; remove this diff --git a/docs/README.md b/docs/README.md index d30e3758a..c5b301ab5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,11 @@ collection: ## Operations +- [Table maintenance periodic queueing](operations/s3-tables-maintenance-scheduler.md) — opt-in scheduling, limits, worker handoff, and recovery. + +For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, +see [PUT cleanup isolation](operations/put-cleanup-isolation.md). + - [External certificate issuance and TLS rotation](operations/tls-certificate-rotation.md) — publication, native reload, trust separation and failure handling. - [Architecture guard troubleshooting](operations/architecture-guard-troubleshooting.md) — diagnose repository guard failures. @@ -26,6 +31,9 @@ For legacy protection assessment and bounded protected copies, see [Object integ For encrypted listing compatibility and older stored fingerprints, see [Encrypted object ETags](operations/encrypted-etags.md). +For durable hard quota accounting, node-outage behavior, and fence protocol +upgrades, see [Durable hard quotas](operations/durable-quota-reservation-v2.md). + Operational runbooks live under [`operations/`](operations/). Replication operators should start with: @@ -63,6 +71,9 @@ procedure is required. For storage dashboards, see [Storage metrics and observer selection](operations/storage-metrics.md): drive ownership, snapshot freshness, counter queries, and rolling upgrades. +For per-bucket capacity and object counts, see [Bucket usage metrics](operations/bucket-usage-metrics.md): +OTLP export, cluster deduplication, accounting semantics, and scanner freshness. + For optional shard commitments, see [Independent shard integrity rollout](operations/shard-integrity-rollout.md): activation, legacy repair results, multipart mode changes, and rollback limits. diff --git a/docs/architecture/README.md b/docs/architecture/README.md index b4604a214..f361ea1b8 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -47,6 +47,7 @@ Required headings and strings in these files are asserted by `scripts/check_arch | [background-services-inventory.md](background-services-inventory.md) | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects | | [scanner-usage-publication.md](scanner-usage-publication.md) | changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts | | [scanner-usage-authority-decision.md](scanner-usage-authority-decision.md) | deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary | +| [scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md) | changing what makes a scanner cycle `Complete`, `Superseded`, or converged, adding or re-classifying a dirty-usage producer, deciding whether a background writeback may invalidate a cycle, changing scanner worker concurrency or throttling, or activating scoped dirty acknowledgements | | [config-model-boundary-adr.md](config-model-boundary-adr.md) | touching the server-config model (`Config`, `KV`, `KVS`) or its persistence, or asking which crate owns which part of server configuration | | [admin-route-action-snapshot.md](admin-route-action-snapshot.md) | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → `AdminAction` contract is enforced | | [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md) | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch | diff --git a/docs/architecture/compat-cleanup-register.md b/docs/architecture/compat-cleanup-register.md index 423ca0bd5..fa277d288 100644 --- a/docs/architecture/compat-cleanup-register.md +++ b/docs/architecture/compat-cleanup-register.md @@ -11,6 +11,8 @@ ## Open Items +- `8565` historical rebalance replacement recovery: older schema-seven intents can exhaust their retry budget before format repair when retained rebalance metadata describes removed pools. After validating blank targets, terminal historical topology, and ownership, grant one durable handoff and one rebuild attempt. Remove after supported direct upgrades no longer admit snapshots lacking replacement_preformat_handoff_retry_done and all retained affected intents have been resolved. + - `backlog-2253-metadata-observation-adapter` metadata observations: preserve aligned metadata/error slices for existing layout, shard and exact-version PUT, MPU and delete consumers while read/heal scheduling uses typed disk observations. Pending slots acquire empty placeholders only in this adapter. Remove after those remaining consumers use typed observations; delete the adapter and old facade in a separate cleanup. - `connect-894` pending Connect heartbeats: replay the exact preceding producer-capability list, with or without jobs, when upgrading to the memory-service capability. Preserve request ID, sequence, and persisted body rather than inserting the new capability into a retry. Remove after upgrades from the pre-service-memory capability set are unsupported. @@ -33,6 +35,7 @@ - `table-catalog-migration-fence-v1` durable strong migration fence compatibility: version 1 "PREPARING" fences did not distinguish a known-absent global strong snapshot from an unknown baseline, so retries read them but fail closed if the global snapshot is missing. Version 2 preserves the same JSON shape and records the pre-migration global snapshot ETag in the existing target_snapshot_etag field while the fence is "PREPARING". Remove version 1 reads after every supported direct-upgrade source writes version 2 fences and operators have completed or cancelled every older in-progress backing migration. - `table-catalog-backing-manifest-v1-wire-labels` durable strong backing manifest labels: version 1 published "STRONG_KV_WAL" and "CUT_OVER_LINEARIZABLE_READS" before the implementation was narrowed to the ETag-CAS durable snapshot backing. Internal names and operator documentation describe the implemented semantics, while version 1 responses retain those labels for existing clients. Replace the labels only in a new manifest version with an explicit client migration contract. - `replacement-recovery-status-v1` replacement recovery status capability: new peers treat an unimplemented ReplacementRecoveryStatus RPC as an explicitly non-definitive rolling-upgrade response, so Admin v4 cannot claim distributed replacement completion from old peers. Remove the fallback after the minimum supported RustFS peer version implements ReplacementRecoveryStatus. +- `peer-drive-admission-v1` startup peer re-admission capability: a restarting node treats an unimplemented PeerDriveAdmission RPC as a peer that cannot report, so the startup re-admission gate fails open during rolling upgrades from older releases. Remove the fallback after the minimum supported RustFS peer version implements PeerDriveAdmission. - `table-catalog-dotted-namespace` Iceberg REST namespace path compatibility: existing RustFS clients use dotted namespace paths, while the standard multi-level contract uses the URL-encoded unit separator `%1F`. New servers accept both forms so a rolling upgrade does not invalidate existing catalog configuration. Remove the dotted fallback after the minimum supported RustFS release advertises `%1F` and all supported clients have refreshed their catalog configuration. - `rustfs-5509` FileInfo positional MessagePack decoding: beta.11 serialized 28 fields, while beta.12 inserted transition-version fields in the middle and serialized an incompatible 30-field array. New releases write named maps and retain readers for both shipped array layouts so direct and rolling upgrades can read either release. Remove the positional-array readers after every supported direct-upgrade release writes named maps and no retained RPC payload can contain a pre-map FileInfo array. - `rustfs-5416` Helm distributed startup wait setting: charts that predate explicit local endpoint identity expose startupWaitTimeoutSeconds for their peer DNS/TCP init gate. The new chart keeps the value accepted but ignores it after moving startup convergence into RustFS. Remove the value and its documentation after the minimum supported direct-upgrade chart includes localEndpointHost.autoInject and no longer renders the peer gate. @@ -45,6 +48,8 @@ - `heal-rpc-auth-v2` internode gRPC authentication: servers temporarily accept legacy prefix signatures so old peers remain available during rolling upgrades. Remove the legacy fallback after the minimum supported RustFS peer version sends v2 authentication on every internode gRPC request. - `put-file-auth-epoch-strict` internode put_file epoch compatibility: rc.2 peers can cache a remote put_file capability before that remote node restarts, then continue sending v1 authenticated uploads with the old server epoch; those peers cannot recover from the 409 conflict used by newer clients to trigger a re-probe. Servers temporarily accept signed, non-nil stale put_file epochs while legacy put_file auth remains non-strict so mixed-version rolling upgrades can finish multipart/object writes. Remove the stale-epoch fallback after the minimum supported RustFS peer version re-probes put_file capability after server-epoch conflicts and legacy put_file auth is no longer accepted. - `disk-mutation-body-digest` internode mutating disk RPCs: servers temporarily accept mutating disk RPCs (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes) that carry no signature-bound canonical body digest, so peers from releases that predate body-digest signing remain available during rolling upgrades. Accepted digestless mutations increment the internode body-digest fallback counter; that counter must read zero fleet-wide across a release window before RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT is enabled. Because body-bound requests now consume replay-cache nonces on the receiver, deploy the raised RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY default fleet-wide before enabling strict mode, and watch the internode replay-cache overflow counter for undersized capacity during the rollout. Remove the digestless fallback after the minimum supported RustFS peer version body-binds every mutating disk RPC. +- `backlog-2900-heal-error-display` heal error display boundary: errors that reach the heal crate with their typed identity rendered as text (an older peer's display string, a local format into the TaskExecutionFailed or Other variants, a boxed io::Error) are mapped back to one HealErrorClass by classify_error_display in `crates/heal/src/error.rs`; no other heal code matches error text. Remove after every heal producer and the minimum supported peer version return typed storage errors end-to-end and the needle list has shrunk to empty. + - `heal-status-rpc-v1` node heal status capability: new peers treat an unimplemented BackgroundHealStatus RPC as an explicitly incomplete rolling-upgrade response. Remove the fallback after the minimum supported RustFS peer version implements BackgroundHealStatus. - `backlog-1316` legacy encrypted multipart range seek: the feature is on by default (RUSTFS_ENCRYPTED_RANGE_SEEK, default true) and the switch remains only as a kill switch until every server that can initiate, write, or complete multipart uploads supports the candidate-to-final marker protocol and uploadId commit lock, and pre-upgrade multipart uploads have drained. Remove the RUSTFS_ENCRYPTED_RANGE_SEEK switch after the minimum supported release does so; keep the quorum marker and malformed-layout full-read guards permanently. - `tonic-013-status-render` peer RPC failure classification: internode failures that reach a node only as text (a peer's error_info payload, a status flattened through format!) are classified by matching the rendering of an Unavailable gRPC status. Releases up to 1.0.0-alpha.38 shipped tonic 0.13, which rendered that status as "status: Unavailable, message: ..."; tonic 0.14 renders it as "code: 'The service is currently unavailable', message: ...". Both forms are matched so an older peer's relayed text still marks an unreachable peer offline. Remove the tonic 0.13 form after the minimum supported RustFS peer version ships tonic 0.14 or later. diff --git a/docs/architecture/crate-boundaries.md b/docs/architecture/crate-boundaries.md index 7a37c707f..9d940e37b 100644 --- a/docs/architecture/crate-boundaries.md +++ b/docs/architecture/crate-boundaries.md @@ -85,6 +85,8 @@ The server-config model (`Config`, `KV`, `KVS`) and the global server-config sna `crates/iam/src/oidc/` owns provider configuration, discovery, HTTP transport, login state, and the OIDC runtime. `crates/iam/src/federation/` owns verified identity transactions and authorization mapping; its production code does not import concrete OIDC runtime or provider configuration types. Production `FederatedAuthorization` values are constructed by `CoreFederatedAuthorizationMapper` in `crates/iam/src/federation/mapper.rs`. +`FederatedClaims` is crate-internal. `FederatedAuthorization` has private fields owned by the mapper module and exposes read-only accessors to session binding consumers, including `mapped_iam_groups()` for the IAM groups selected by authorization mapping. Test fixtures are exposed only through the `test-util` feature or unit-test configuration. + `rustfs/src/startup_auth.rs` constructs `StandardOidcAdapter` and publishes the federation service with its OIDC query as one runtime pair. `AppContext` stores and publishes that pair through `FederatedIdentityRuntimeSnapshot`; consumers receive interfaces rather than constructing a second OIDC runtime. `scripts/check_oidc_architecture_boundaries.sh` enforces these boundaries through the existing architecture guard, including the Keycloak workflow path for `crates/iam/src/oidc/`. `crates/iam/src/oidc/config.rs` owns OIDC configuration input, validation, persisted KVS conversion, and `OidcSiteReplicationSnapshot`. `rustfs/src/admin/service/oidc_config.rs` owns stored configuration reads, compare-and-save updates, restart comparison, and provider discovery validation; the OIDC Admin handler translates HTTP requests and errors. `OidcConfigQuery::site_replication_snapshot()` returns only the provider ID, claim name, role policy, client ID, and hashed client secret needed by site replication. `rustfs/src/admin/handlers/site_replication.rs` maps this narrow view to its existing response format without reading raw OIDC credentials. The Keycloak workflow tracks the Admin configuration service path along with IAM OIDC code. diff --git a/docs/architecture/minio-file-format-compat.md b/docs/architecture/minio-file-format-compat.md index 58e9051cf..35df40aba 100644 --- a/docs/architecture/minio-file-format-compat.md +++ b/docs/architecture/minio-file-format-compat.md @@ -114,7 +114,7 @@ Inventory the source first: bucket default encryption means objects can be encry |---|---| | Shipping status | Ships in no default build: absent from `default` and `full` in `rustfs/Cargo.toml`; released binaries and container images never include it. Enable with `--features rio-v2`. | | Pull-request coverage | `test-and-lint-rio-v2` in `.github/workflows/ci.yml`: clippy plus `cargo nextest` for `rustfs` and `rustfs-ecstore` with `--features rio-v2`. This is the cfg-seam guard; it keeps the feature compiling and its unit suite green on every pull request. | -| Full-suite lane | `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` in `ci.yml` run only on the weekly schedule and manual dispatch (`cache-warm.yml` keeps the `ci-feat-rio` cache warm so the scheduled build fits its timeout). | +| Full-suite lane | `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` in `ci.yml` run only on the nightly schedule and manual dispatch (`cache-warm.yml` keeps the `ci-feat-rio` cache warm so the scheduled build fits its timeout). | | Interop evidence | `.github/workflows/minio-interop.yml` (nightly plus manual) regenerates real MinIO backend trees via `crates/rio-v2/tests/minio_fixture_lab/` and runs the `#[ignore]` reader tests in `rustfs/src/storage/minio_generated_read_test.rs` with `--features rio-v2`. Its freshness is tracked in `.github/scheduled-validations.json`. | | Promote-or-delete condition | The variant stays dormant until one of two things happens. **Promote**: a release commits to shipping MinIO SSE migration as a supported capability; then `rio-v2` joins `default`/`full`, the scheduled lanes run on every pull request, and this section is rewritten. **Delete**: no release commits to it and the scheduled lanes are not kept green; then the feature flag, `crates/rio-v2`, the cfg seams in `rustfs/src/storage/sse.rs`, the three `ci.yml` jobs, the `cache-warm.yml` warm step, `minio-interop.yml`, and its `scheduled-validations.json` entry are removed in one change. Either outcome must update `ARCHITECTURE.md` and the `ci.yml` job comments that cite this section. | diff --git a/docs/architecture/placement-repair-invariants.md b/docs/architecture/placement-repair-invariants.md index acdda004d..4fdae2dfb 100644 --- a/docs/architecture/placement-repair-invariants.md +++ b/docs/architecture/placement-repair-invariants.md @@ -72,6 +72,15 @@ No cluster-wide coordinator or second generation token exists for scanner/heal a Rules: cancellation or a local lease alone is not a fence; if a fixture ever demonstrates a stale destructive write, the fix extends the storage-owned generation/admission primitive and validates the token at the final commit rather than adding a coordinator. +### Per-object disposition memory + +A heal task that ends in a terminal verdict is forgotten by the queue as soon as its dedup key is released, so best-effort producers (scanner, read repair, autoheal, MRF hints) resubmit the same object on their own cadence. `HealDispositionCache` in `crates/heal/src/heal/manager/disposition.rs` remembers the last terminal verdict per `(bucket, object, version, pool, set)` and is consulted once, in `submit_heal_request_with_receipt_alias_and_mrf_notice` before `admit_request_to_queue`, for `HealType::Object` requests that are not administrator starts, not forced, not above normal priority, not dry runs, and carry no durable MRF anchor. A verdict applies only to the same pool/set selector; absence in an old pool must not suppress repair in another pool or an unscoped request. The manager retry path never consults it. + +- The verdict comes from `classify_heal_error` in `crates/heal/src/error.rs`, the one classifier shared by the task, the erasure-set healer and the storage adapter. An object-level not-found from the locked storage read is `Absent` for every source and every `recreate_missing` value; an offline disk, a missing volume, a lost quorum, or a lock timeout is never `Absent`. +- `Absent` entries drop resubmissions until the entry expires. `RetryExhausted` and `Permanent` entries re-admit on an exponential schedule (one minute, doubling, capped at one hour), so a failing object is still retried but not on every producer tick. A dropped request receives `Dropped(PolicyDropped)` and the task id that produced the verdict, so the outcome stays observable. +- The memory is bounded (least recently used eviction) and expiring (24 hours); it is process-local and starts empty after a restart. It is not a receipt: it never records or releases a durable responsibility, and durable MRF repairs and `AuthoritativelyAbsent` proofs keep their own storage-owned path. +- Metrics: `rustfs_heal_disposition_cache_hits_total` (by disposition) and `rustfs_heal_disposition_cache_entries`. + ## Behavior Change Gates Any later placement or repair PR must use the following gates: diff --git a/docs/architecture/readiness-matrix.md b/docs/architecture/readiness-matrix.md index 58d98c620..08f62b9f0 100644 --- a/docs/architecture/readiness-matrix.md +++ b/docs/architecture/readiness-matrix.md @@ -22,6 +22,7 @@ | IamReady | Inline IAM bootstrap or deferred IAM recovery publication. | Yes. | Deferred recovery can publish IAM readiness after HTTP has already started. | | Lock quorum | Per-set write quorum readiness. | Yes. | Do not replace the distributed lock quorum check with node count or endpoint count. | | Peer health | `peer_health_ready` runtime status. | Only when `RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled. | The gate is disabled by default; unknown peer health degrades readiness only when enabled. | +| Peer re-admission | A strict majority of the peers in each erasure set this node serves count all of its drives in that set as online (`PeerDriveAdmission` internode RPC). | Startup transition only, while `RUSTFS_STARTUP_PEER_READMISSION_ENABLE` is enabled (default on). | Bounded and fail-open: publishes `FullReady` with a WARN log when `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` elapses, immediately when older peers cannot answer and the majority is out of reach, or after 15 s of unreachable peers keeping it out of reach. Not part of steady-state `/health/ready`. Distinct from peer health, which is this node's view of its peers. | | KMS compatibility | KMS health compatibility readiness. | Only when `RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE` is enabled (default off; `crates/config/src/constants/health.rs`). | When enabled, `/health/ready` additionally requires the KMS service to be running if a global KMS manager exists. KMS startup fatality and health reporting remain separate from pure docs work. | Effective `FullReady` is: @@ -35,6 +36,8 @@ storage_ready && iam_ready && lock_quorum_ready && peer_health_ready an unknown or unsupported peer-health snapshot degrades readiness with `peer_health_unavailable`. +In a distributed deployment, startup publishes `FullReady` only after that local condition holds and the peer re-admission gate has finished. The gate answers how the peers see this node, so a rolling update does not stop the next node while survivors still count this one as unavailable (rustfs/rustfs#8669). It asks over the authenticated internode channel rather than scraping peers' `/health/ready`, so it works before the peers' own HTTP admission opens. The peer answers from the same cached inputs as its `/health/ready` (lock-client reachability and the in-memory drive state), without disk I/O. While the gate waits, `/health/ready` returns 503 with `startup_finalization_pending` and the gated S3 surfaces report `startup_finalization`. + ## Probe Semantics - Liveness reports process availability and must not depend on storage, IAM, @@ -93,6 +96,7 @@ The new fields are additive. Their absence in an older response is not evidence - Do not move peer-health checks into the S3 data hot path. - Do not make peer health affect readiness unless `RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled. +- Keep the peer re-admission gate on the startup `FullReady` transition only. It must stay bounded and fail open, and it must not run in steady-state probes or on the S3 request path. - Do not use this matrix to change the early HTTP listener plus readiness-gate split. - Do not simplify distributed lock quorum, IAM deferred recovery, or KMS fatal diff --git a/docs/architecture/s3-compatibility-matrix.md b/docs/architecture/s3-compatibility-matrix.md index 83ffc522d..a71d1cd5e 100644 --- a/docs/architecture/s3-compatibility-matrix.md +++ b/docs/architecture/s3-compatibility-matrix.md @@ -32,6 +32,7 @@ Counts ignore blank lines and comments; compute them from the files. The lifecyc | Public access block put/get/delete | Supported | `implemented_tests.txt` | | Presigned GET and PUT URLs | Supported | `implemented_tests.txt` | | Range and conditional reads | Supported | `implemented_tests.txt` | +| Conditional bulk deletion of current objects | Supported with the limits below | `crates/e2e_test/src/conditional_delete_objects_test.rs` | | User metadata | Supported | `implemented_tests.txt` | | SSE-C and selected SSE-KMS edge cases | Supported | `implemented_tests.txt` | | Selected versioning, object-lock, checksum, CORS, raw request, and conditional write behavior | Supported | `implemented_tests.txt` | @@ -44,6 +45,16 @@ Counts ignore blank lines and comments; compute them from the files. The lifecyc The MinIO-style `metadata=true` listing extension returns user metadata names without the HTTP `x-amz-meta-` prefix. It is not the standard S3 `ListObjectsV2` response. Clients that expect canonical HTTP header names in `UserMetadata` must normalize the names at that boundary; ordinary HEAD/GET metadata is unaffected. See `rustfs/src/app/bucket_usecase.rs` and its serialization tests. +## Conditional Bulk Deletion + +`DeleteObjects` accepts an optional `ETag` for each object in the XML body. A specific ETag matches the current live object; `*` requires that a current live object exists. Specific ETags require both `s3:DeleteObject` and `s3:GetObject`, while `*` requires only delete permission. Entries without an ETag retain the unconditional delete behavior. + +An ETag mismatch returns a per-object `PreconditionFailed`; a missing object or a current delete marker returns `NoSuchKey`. Explicit empty or weak ETags return `InvalidArgument`. Supplying both ETag and any explicit `VersionId`, including `null`, returns `InvalidRequest`; this combination is outside the supported conditional-delete scope. Version-enabled and version-suspended buckets are supported when the request omits `VersionId`. + +The response can contain successful and failed entries with HTTP 200. `Quiet=true` hides successful entries and preserves errors. Conditions on repeated keys are evaluated independently against one locked snapshot; another successful entry for the same key can still delete that object. Physical deletion, delete-marker creation and successful side effects are counted once per distinct deletion action. + +Conditional deletion requires namespace locking and every participating storage peer to support guarded metadata mutations. Unsupported peers return `NotImplemented` before mutation, and requests never fall back to unconditional deletion. Complete the upgrade of all S3 ingress and storage nodes before clients use this capability. Conditional entries return `SlowDown` during pool rebalance. A generation change during commit returns the per-object concurrent-operation error `OperationAborted`. Bucket-policy enforcement through `s3:if-match` is outside this support statement. + ## Replication Support Boundary Site replication and bucket replication are not the same compatibility claim. diff --git a/docs/architecture/s3-tables-support-matrix.md b/docs/architecture/s3-tables-support-matrix.md index 620f7dae4..05522badb 100644 --- a/docs/architecture/s3-tables-support-matrix.md +++ b/docs/architecture/s3-tables-support-matrix.md @@ -38,7 +38,7 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t | Client or engine | Status | Claim | |---|---|---| -| PyIceberg | Automated | Namespace and table create, append, reload, scan, metadata-location, refs, views, maintenance, diagnostics, optional vended credentials with an exact-prefix data-plane scope check. | +| PyIceberg | Automated | Namespace and table create, append, exact-row reload/scan, metadata-location, refs, views, namespace properties, rename identity and credential negotiation. The smoke discovers the server backing and checks maintenance configuration, scheduler/worker dry-run, diagnostics and export on both backings. Direct metadata maintenance is checked as unsupported on durable-strong. Optional vended credentials include an exact-prefix data-plane scope check. | | Spark Iceberg REST catalog | Manual/live harness | Pinned package inputs, catalog properties, atomic CTAS SQL through staged create, expected `row_count=2`, and a CI opt-in gate for create/refresh/count/cleanup. Live execution and commit-conflict probing remain manual unless enabled in the runner. | | Trino Iceberg REST catalog | Manual/live harness | Catalog properties and a read-only `SELECT COUNT(*)` against a PyIceberg- or Spark-created table. Write compatibility not claimed. | | DuckDB Iceberg 1.5.5 | Automated | `duckdb_smoke.py` covers metadata-location read, single-table create/insert/update/delete/merge, schema evolution, snapshots, concurrent writers, drop, PyIceberg cross-read, and both signing profiles. Purge-on-drop and format v3 are verified fail-closed. Two-table mode runs without claiming cross-table atomicity. AWS `ENDPOINT_TYPE S3_TABLES`, staged-create client integration, and vended-credential integration are not claimed. | @@ -87,6 +87,42 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t ## Maintenance Matrix +Object-backed and durable-strong catalogs support the configuration, scheduler, +worker, job inspection, heartbeat, and quarantine endpoints. Durable-strong +maintenance resolves table identity from the authoritative catalog snapshot; +recoverable job and configuration records remain separate objects. The worker +performs metadata and unreferenced-object cleanup without changing the catalog +pointer. The direct `/maintenance/metadata` endpoint, including snapshot +expiration and compaction plan/commit, remains object-backed only. A configured +`background-enabled` table still requires explicit worker ticks. The standalone +server has opt-in, allowlisted periodic queue admission; see the +[scheduler runbook](../operations/s3-tables-maintenance-scheduler.md). + +Renamed tables retain their original metadata paths. Durable-strong scheduling +and cleanup fail closed with a conflict if another table shares a scanned +metadata or warehouse directory, including a recreated source name after rename. +Such tables require disjoint cleanup roots before maintenance can run; enabling +deletion does not override this ownership check. + +Durable-strong rename preserves maintenance configuration, job identifiers, +current/latest aliases, history, audit events, and quarantine state. A persisted +rename intent protects sidecar migration across interruption: the source remains +authoritative until the catalog CAS succeeds, and the destination records are +ready before publication. Retry the same rename to finish pending recovery. +Backup, restore, register, drop, and subsequent rename operations recover a +pending migration before proceeding. Complete pending rename recovery before +rolling back to a version that does not understand maintenance migration intents. + +Durable-strong scheduler reports read an observational history set, not an atomic +snapshot of all maintenance jobs. They fix at most 4096 history-object names +under the publication read fence, scan those objects outside the fence, and +refresh the selected 10-job audit window, effective configuration, and current +job under a final read fence. Exceeding the history budget fails closed with an +unavailable response; retrying does not remove that size limit. A listed object +that disappears during the scan also returns unavailable; retry after concurrent +rename or maintenance activity has settled. No history object is silently omitted +when a rename removes its old path. + | Capability | Status | Claim | |---|---|---| | Metadata retention dry-run | Supported | Reports retained metadata and deletion candidates without moving the pointer. | @@ -103,7 +139,7 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t | Compaction planning | Preview / controlled | Partition-local and sort-order-local binpack candidates for Parquet; never mixes partitions or sort orders in one group. | | Delete-file or row-level compaction planning | Preview / controlled | Position or equality delete files force machine-readable planning into manual review. | | Compaction commit | Preview / controlled | Commits a safe partition-local Parquet rewrite preserving sort order IDs. | -| Built-in periodic scheduler | Not claimed | Ticks are operator-triggered; no continuous in-process scheduling. | +| Built-in periodic scheduler | Preview / queue-only | Default off; allowlisted, paged admission with jitter, per-pass quotas, current-job checks, and cooperative drain. Workers and failed/expired-job recovery remain operator-triggered; no automatic execution or hard per-table I/O deadline. | | Delete-file or row-level compaction execution | Not claimed | Manual-review maintenance item. | ## Recovery And Strong Backing Matrix @@ -133,7 +169,7 @@ Do not promote a failure case from live probe or load test to an automated claim ## Unsupported Or Not Claimed -Full AWS S3 Tables control-plane parity; full MinIO AIStor private extensions; full Cloudflare R2 Data Catalog or Alibaba OSS Tables interoperability; built-in periodic maintenance scheduling; active-active multi-region writes; multi-table transactions; no-long-term-data-credential bootstrap; online vendor SDK polling; external catalog policy mirroring; delete-file rewrite or row-level compaction execution; built-in SQL execution; Delta Lake or Hudi; end-to-end SQL row-level DML validation through Spark, Trino, or another engine. +Full AWS S3 Tables control-plane parity; full MinIO AIStor private extensions; full Cloudflare R2 Data Catalog or Alibaba OSS Tables interoperability; automatic maintenance execution and failed/expired-job recovery; durable-strong snapshot-expiration and compaction plan/commit; active-active multi-region writes; multi-table transactions; no-long-term-data-credential bootstrap; online vendor SDK polling; external catalog policy mirroring; delete-file rewrite or row-level compaction execution; built-in SQL execution; Delta Lake or Hudi; end-to-end SQL row-level DML validation through Spark or Trino. DuckDB single-table UPDATE/DELETE/MERGE has its own automated smoke coverage as listed above; this does not establish other engines' DML compatibility. ## Verification diff --git a/docs/architecture/scanner-per-bucket-convergence.md b/docs/architecture/scanner-per-bucket-convergence.md new file mode 100644 index 000000000..99f57f829 --- /dev/null +++ b/docs/architecture/scanner-per-bucket-convergence.md @@ -0,0 +1,581 @@ +# Scanner Per-Bucket Convergence Design + +**Use this when:** changing what makes a scanner cycle `Complete`, `Superseded`, or converged; adding or re-classifying a dirty-usage producer; deciding whether a background writeback may invalidate a scanner cycle; changing scanner worker concurrency or throttling; or activating scoped dirty acknowledgements. +**Source of truth:** `classify_nsscanner_cycle`, `scanner_cycle_activity_status`, and `scoped_scan_scope_from_dirty_buckets` in `crates/scanner/src/scanner_io.rs`; `dirty_usage_snapshot_status` and `acknowledge_scoped_dirty_usage` in `crates/scanner/src/scanner_io/dirty_usage.rs`; `scanner_concurrency_limit` in `crates/scanner/src/scanner_io/guards.rs`; the per-disk worker loop and `carry_partial_bucket_values` in `crates/scanner/src/scanner_io/io_cache.rs`; `DynamicSleeper` in `crates/scanner/src/sleeper.rs`; `ScannerNodeActivity` in `crates/scanner/src/scanner/activity.rs`; `persist_replication_state_if_current` in `crates/ecstore/src/bucket/replication/replication_resyncer.rs`; the publication fences in [scanner-usage-publication.md](scanner-usage-publication.md). + +## Decision + +Date: 2026-10-13. Status: accepted design; phases ship in the order given +under [Delivery](#delivery), each as its own change. + +The scanner is a concurrent collector walking a namespace that mutators keep +changing. It is specified here with the vocabulary of concurrent garbage +collection, because the problems and the known-good answers are the same. + +1. **Tri-color buckets with snapshot-at-the-beginning termination.** A cycle + is `Complete` when every bucket in its start snapshot has been walked and + the publication is durable. Buckets that mutators shaded grey during the + walk stay grey for the next cycle and are reported as pending; they do not + turn the cycle into `Superseded`. +2. **The cluster activity probe stays, as a guardrail only.** It vetoes a + publication when the heap it describes is not the heap that was walked: + topology, peer identity, protocol version, or data movement changed. + Namespace and dirty-usage generation movement, and a pending ordinary PUT + commit, never veto; they only shade buckets. +3. **Replication status writebacks are status-class barrier hits.** They + shade a bucket for a scoped rescan but are not namespace activity. They + never uncertify a walk and never block cycle completion. The scanner's own + queued replication heals fall under the same class. +4. **Worker concurrency is paced, not derived from disk count.** The number of + bucket walkers per erasure set and the per-object throttle are set from a + pacing target with CPU and I/O ceilings, with disk type as the default + input. Disk count remains a floor, not a cap. +5. **Guardrail vetoes are bounded too.** A probe that fails for one peer + localises to the buckets that peer dirtied; it does not defer the whole + cycle. A veto is always attributed to a named field, retried on a bounded + backoff, and reported with its reason. The guardrail protects publication + correctness; it is not allowed to become a third way of never converging. + +Every liveness rule above is paired with a boundedness rule in +[Boundedness constraints](#boundedness-constraints). A change that improves +one side must restate its effect on the other. + +Backward compatibility is a hard requirement of this design, not a phase. +Every phase must satisfy [Backward compatibility](#backward-compatibility) +on its own: a cluster can be upgraded node by node, and any node can be +rolled back to the previous release, without a scanner reset and without +losing the authoritative usage baseline. + +## Two failure modes + +Scanner defects fall into two opposite classes and a fix for one can create +the other: + +- **Long tail**: part of the namespace is never reached. A cycle keeps being + invalidated, a budget keeps cutting the same buckets, or a queue keeps + reordering new names ahead of old ones. The symptom is usage that never + converges and lifecycle or heal work that never runs for some buckets. +- **Burst amplification**: a sudden wide failure (a disk loss, a mass delete, + a replication resync) makes the scanner discover a large amount of work at + once and the scanner, heal, and replication paths saturate CPU, disk, and + network together. + +A third class sits beside these two and is easy to mistake for the first: + +- **Guardrail false positives**: the cycle is invalidated not by writes but + by the proof machinery itself. An in-flight PUT counted as a publication + barrier, a journal lock timeout reported as an unavailable activity + baseline, or one unreachable peer in a large cluster defers the whole + cycle, and the dirty map is never cleared. Per-bucket certification does + nothing for this class because the walk never reaches certification. The + guardrail therefore carries its own boundedness rules, attributed per + field and localised per peer, so a misclassification can be fixed without + touching the termination rule. + +The current cycle classifier is in the first class. The pre- and post-walk +proofs require the global dirty-usage generation and the full activity +snapshot to be identical, so one write to one bucket during a walk makes the +whole cycle `Superseded`. Scoped rescans then need a converged baseline from +`complete_scanner_cache_baseline_plan_digest`, which only a quiet full cycle +can produce, so every retry is a full walk and cycle time never shrinks. +Since replication writebacks started marking buckets dirty, a resync is a +standing mutator and the scanner's own queued heals write back during its +walk, so the scanner invalidates itself. This is a stop-the-world +termination rule applied to a concurrent collector; it fails on liveness by +construction, not under load. + +The per-bucket data model that a concurrent collector needs already exists: +per-bucket generations are the card table, `notify_scanner_dirty_usage_mutation` +is the write barrier, `DirtyUsageBucketScope::TopLevelEntries` are finer +cards, and `acknowledge_scoped_dirty_usage` is exact per-card clearing. The +cycle-level decision never consumed it. + +## Collector model + +### Colors + +| Color | Meaning | Where it lives | +|---|---|---| +| white | not walked in this cycle | bucket in the start snapshot without a result | +| grey | walked or not, but a mutator touched it since the cycle's start snapshot | bucket present in the dirty map with a generation newer than its start generation | +| black | walked and certified: its result is current with respect to the start snapshot | bucket acknowledged and cleared from the dirty map | + +The write barrier is the existing committed-mutation observer. It shades a +bucket grey by advancing its generation; it never blocks the mutator and it +runs no scanner code inline. Barrier hits carry a class: + +| Class | Producers | Effect on usage | Effect on color | +|---|---|---|---| +| `Namespace` | PUT, multipart complete, copy, delete, delete-objects, bucket create, lifecycle expiry, tier expiration cleanup, heal rewrites that change size or version count | size, versions, delete markers, histograms | shades grey; the bucket is uncertified if hit during its walk | +| `Status` | replication status writeback (`persist_replication_state_if_current` in `Update` mode), scanner-queued replication heal, any writeback that only changes replication or restore state on an existing version | only `replication_stats` folded in by `apply_scanner_size_summary` | none: never shades, never uncertifies a walk, never counts as pending; refreshed by the full walk that every cycle with an active replication rule performs | + +The class is attached in the mutation observer installed by +`init_background_replication` in `rustfs/src/storage/storage_api.rs`. A +`Namespace` hit advances the bucket's generation; a `Status` hit records only +producer coverage. While the scanner is the replication heal driver, +`ScannerMaintenanceFeatures::needs_regular_cycle` forces a full walk on every +cycle of a deployment with an active replication rule, so a `Status` hit needs +no rescan queue: the walk that would consume it always happens. If a later +phase lifts that forcing, `Status` hits must then join the scoped rescan set, +and that set must travel to the coordinator through the peer dirty-usage +snapshot, because hits land on whichever node commits the writeback. + +A replication maintenance walk refreshes compacted prefixes rather than +sampling their old totals. A completed bucket cache with active replication +starts a new coverage sweep even when the cycle is retried; unfinished +checkpoints retain their existing frontier. Set persistence binds the +observed replication statistics to its execution digest, so same-cycle +reuse cannot restore statistics from an earlier walk. None of these cache +requirements changes the namespace generation or publication fences. + +### Snapshot at the beginning + +A cycle is responsible for the namespace as it was at its start snapshot. +Each bucket walk records its start generation for the local node and for +every peer scoped entry resolved by `resolve_remote_dirty_usage_scope`. +Mutations after that point are the next cycle's work. This is what makes +termination possible under continuous writes: the termination condition is +"no grey bucket in the start snapshot", not "no mutation anywhere". + +Usage is a sum, not a reachability fixed point. A certified bucket's value +plus an uncertified bucket's last certified value is a valid snapshot, so the +collector does not need a globally consistent heap to publish. This is why a +`Complete` cycle with pending buckets is sound as an authoritative baseline. + +### Certification + +After the authoritative publication is durable and the root publication +proof holds, the finalize path acknowledges every walked bucket with the +compare-and-remove semantics of `acknowledge_scoped_dirty_usage`: + +- generation unchanged: black, cleared from the dirty map. `Status` hits + never move it; +- generation moved by a `Namespace` hit: stays grey with the newer + generation and counts as pending; +- walk stopped early (`Partial`): stays grey with its generation, like a + failed walk, and counts as pending; the value published for it is the + last certified one, never the partial walk's running total; +- a sweep that spans two cycles resumes only under the same coverage + identity (`scanner_activity_coverage_digest` with the bucket's own dirty + generation, through `prepare_bucket_checkpoint_with_coverage`). A peer's + namespace and dirty generations stay in that identity while its dirty + buckets are acknowledged per cycle: a write there between the two halves + of a sweep restarts the sweep instead of being acknowledged unwalked. The + local node's generations are left out, because its buckets are certified + one by one; a completed sweep is stored under the reuse identity; +- a peer reported the bucket dirty without a scoped entry the coordinator can + verify: stays grey. This is the mixed-version and partial-peer fallback, + and it is per bucket, not per cycle. + +The whole-cycle generation acknowledgement remains the fallback when no +scoped entries can be built at all. It is never combined with per-bucket +certification in the same cycle. + +### Bounded remark + +Before finalize, the cycle may re-walk grey buckets at most once, within the +remaining cycle budget and with the same per-bucket limits as the main walk. +A bucket still grey after that is handed to the next cycle. There is no loop +until quiet. + +### Cycle classification + +`classify_nsscanner_cycle` keeps its incomplete branches: budget, +cancellation, failed walks, and unverifiable dirty or activity state still +return `Incomplete` or `Deferred`. The final branch changes from a global +equality test to: + +- guardrail vetoed: `Superseded`; +- otherwise: `Complete`, carrying `pending_dirty_buckets` as the count of + buckets left grey by `Namespace` hits or by a walk that stopped early. + +A bucket walk that stops early without exhausting the cycle budget (a sweep +resumed under a newer plan, unresolved objects) is `Partial`. The set scan +stands in for it with the bucket's last certified value from the previous +complete set cache (`carry_partial_bucket_values` in +`crates/scanner/src/scanner_io/io_cache.rs`): usage is a sum, so the +certified buckets plus that value are a valid snapshot. The cycle then +completes with the bucket pending (`ScannerBucketScanStatus::PartialCarried`), +and the next scoped cycle walks it because the pending buckets of the +authoritative baseline join the dirty buckets in +`scoped_scan_scope_from_dirty_buckets`. Only a bucket with no certified +value on a set (its first walk, a changed incarnation, a hard failure) keeps +that set, and with it the cycle, `Incomplete`. + +`usage_snapshot_converged` becomes `pending_dirty_buckets == 0`. A `Complete` +cycle with pending buckets publishes an authoritative snapshot: certified +buckets are current, pending buckets carry the value the walk produced, +which is at least as recent as the previous certified value and may miss +the write that shaded the bucket; the next cycle rescans them. `DataUsageInfo` +in `crates/data-usage/src/data_usage.rs` gains additive fields beside +`usage_snapshot_set_states`: the sorted list `usage_snapshot_pending_buckets`, +its length `usage_snapshot_pending_bucket_count`, and the routing mark +`usage_snapshot_observational` that a vetoed cycle sets so the observed +object is selected by an explicit mark rather than by the convergence flag. +Readers can tell a certified value from an uncertified one without a schema +break. This replaces the behaviour where a superseded result is published +only as an observed snapshot and never becomes a baseline. + +The pending set is read after the guardrail verdict and before publication. +Certification at finalize still compares each bucket's generation against the +start snapshot, so a write that lands between publication and finalize keeps +its bucket dirty even though the published list does not name it; it is the +next cycle's work, exactly as a write after a converged publication is today. + +### Guardrail + +The probe compares the `ScannerNodeActivity` snapshot before and after the +walk. Field roles: + +| Field | Role | +|---|---| +| `instance_id`, `protocol_version`, `topology_digest` | veto: a peer restarted, changed version, or the topology changed | +| `data_movement_active`, `movement_generation` | veto: rebalance or decommission can make the walk describe a different generation | +| `publication_blocked` (a pending ordinary PUT commit) | shade signal only, and part of no plan identity: the write barrier records the bucket after the ACK and certification handles it | +| `maintenance_generation` | veto: no cycle today records whether it published maintenance results, and the field is already part of the structural digest; it becomes a shade signal only once that distinction exists | +| `namespace_generation`, `dirty_usage_generation`, `dirty_usage_pending` | shade signal: consumed per bucket; never a veto on their own | + +The activity snapshot feeds two plan identities with different jobs. The +coverage identity (bucket list and creation times, `topology_digest`, +`protocol_version`, `data_movement_active`, `movement_generation`) decides +whether a resumed bucket checkpoint still describes the same traversal; only +it may mark a resumed sweep as mixed coverage. The reuse identity adds +`instance_id` and `maintenance_generation` and decides whether a completed +cache or a scoped baseline may be reused: a peer restart can lose dirty hints +inside the journal's coalescing window and a maintenance change needs a fresh +discovery walk, so both force a full walk, but neither changes what a +full-walk checkpoint has already covered. `publication_blocked` belongs to +neither identity. The guardrail compares the reuse identity plus the movement +fields. + +The guardrail is bounded in three ways: + +- **Attributed.** A veto names the field that moved. A whole-snapshot + inequality is never a veto on its own, so a misclassification such as an + in-flight PUT reported as data movement is fixed in that field's producer + and nowhere else. +- **Localised.** When the probe cannot be read from one peer, or that peer + answers with a protocol it cannot be verified under, the coordinator keeps + every bucket that peer dirtied grey (through the peer set resolved by + `resolve_remote_dirty_usage_scope`) and publishes the rest. Only a + movement or topology veto, which describes the whole heap, defers the + cycle. `ActivityBaselineUnavailable` therefore becomes a per-peer + condition, not a cycle outcome, unless no peer answered at all. + Phase 2 ships attribution but not localisation: per-bucket certification + covers the local dirty map, and a remote peer whose `dirty_usage_generation` + or `dirty_usage_pending` moved during the walk still supersedes the whole + cycle (`scanner_peer_dirty_usage_moved`), because without a post-walk scoped + snapshot the coordinator cannot name the buckets that peer dirtied and would + otherwise claim convergence over them. The same rule is applied again after + remote leases are granted. Phase 5 lifts it with scoped acknowledgements. +- **Retried on a bound.** A deferred cycle retries on the existing + superseded backoff schedule with a cap, and `usage_freshness` reports the + field and peer that caused the last deferral so the operator can tell a + rebalance from a broken probe. + +### Generations of buckets + +Most buckets are cold and a few are hot. The scanner keeps two populations +with separate cadences, in the sense of a generational collector: + +- **young**: buckets shaded grey within the last `K` cycles. They are walked + every cycle, scoped to their dirty cards where a scope exists, under a + small per-cycle quantum. +- **old**: buckets black for `K` consecutive cycles. They are walked by a + checkpointed full sweep that is guaranteed to visit every old bucket at + least once every `M` cycles, independent of young-generation load. + +Promotion is `K` consecutive certifications; demotion is any `Namespace` +hit. The split exists so that the long tail has its own cadence instead of +competing with hot buckets for one budget. `K` and `M` are scanner runtime +configuration with defaults; the full sweep keeps today's checkpoint and +resume semantics. + +### Baseline bootstrap + +`complete_scanner_cache_baseline_plan_digest` accepts a `Complete` +publication as the structural baseline whether or not it is converged. +Bucket-cache reuse is governed by color: a black bucket reuses its cache +under the structural plan digest, a grey bucket is rescanned. This removes +the dependency where scoped rescans need a converged baseline and a +converged baseline needs a quiet cycle. The cache execution identity in +[scanner-usage-publication.md](scanner-usage-publication.md) is unchanged +for within-cycle reuse. + +## Pacing and worker concurrency + +Before phase 3 the number of bucket walkers per erasure set equalled the +number of disks in the set, capped by `max_concurrent_disk_scans`, and +`scanner_concurrency_limit` took the minimum of the configured value and the +disk count. Buckets queue on one channel per set and each walker takes one +bucket at a time; inside a bucket the folder walk is sequential and +`DynamicSleeper` sleeps after every object for the work time multiplied by +the speed factor. On a single-disk deployment this is one walker, one bucket, +one object at a time, at roughly one third duty cycle under the default +speed. Walker count is therefore a property of topology, not of the work. + +Two later controls bound that walk per disk rather than per object. +rustfs/rustfs#8389 added the local-disk foreground-priority admission gate, +which orders scanner I/O behind foreground requests. rustfs/backlog#2905 added +a per-disk token-bucket budget (operations and bytes per second, defaults by +detected media) that `DynamicSleeper` charges before each directory listing +and metadata read; it is a hard ceiling layered on admission and never lets +the scanner exceed what admission allows. The operator view is in +`docs/operations/scanner-runtime-controls.md`. + +A collector's pacer sets collector work from the mutator's rate under a +resource ceiling. The scanner adopts the same shape: + +- **Target**: the young generation must be walked within one cycle interval + and the old generation within `M` intervals. The pacer derives a required + object rate from the last cycle's object count and wall time. +- **Walkers**: `available` for `scanner_concurrency_limit` becomes the disk + count multiplied by a per-disk walker factor. The factor defaults from disk + type: 1 for rotational media, higher for flash, where xl.meta decode is CPU + bound and there is no seek penalty. Disk count stays the floor so + distributed deployments keep at least today's parallelism. Shipped as + `scanner.walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through + `16`): each local coordinator disk gets that many walkers, which share one + per-disk scanner admission so a peer still sees one scan session per + physical disk; remote disks keep one walker. The default is `1` everywhere + until a local disk can report its media type + (`rustfs_utils::os::Info::rotational` is never populated today); the flash + default above `1` remains the intent. +- **Throttle**: the `DynamicSleeper` factor is the pacer's output, not a + fixed preset. `ScannerSpeed` presets remain as operator overrides and as + the initial value. Foreground workload backoff in `sleeper.rs` remains the + hard ceiling: the pacer never reduces the sleep below what foreground + admission demands. +- **Ceilings**: a CPU share and a per-disk I/O share bound the pacer's + output. When the target cannot be met inside the ceilings the pacer reports + it in `usage_freshness` rather than exceeding them. + +## Boundedness constraints + +Each rule below bounds a liveness rule above. They are part of the design, +not tuning advice. + +| Liveness rule | Bound | +|---|---| +| Grey buckets are rescanned next cycle | A scoped cycle walks at most a configured number of buckets and objects. Grey buckets beyond that are ordered by waiting age, oldest first, so every bucket is walked within a bounded number of cycles. New names never reorder ahead of an older waiter. | +| `Status` hits are not namespace activity | A `Status` hit costs one atomic OR on the writeback path and no scanner work. It never triggers a dirty-usage wake. A replication resync therefore adds nothing to scanner load beyond the full walk the cycle already performs. | +| A `Complete` cycle with pending buckets is a baseline | When the pending set is the whole namespace (a mass write, a disk loss), the next cycle is declared a full sweep explicitly and uses the existing superseded retry backoff schedule. A full sweep is never run under the scoped cycle's wake cadence. | +| Bounded remark | At most one remark pass per cycle, inside the remaining budget. | +| Old-generation sweep every `M` cycles | The sweep is checkpointed and resumes; it does not restart on a `Namespace` hit. A hit only demotes the bucket to young for the following cycle. | +| Pacer raises walkers and lowers sleep | CPU and I/O ceilings are absolute. Foreground admission backoff cannot be undercut. Walker count never exceeds the ceiling-derived maximum even when the target is missed. | +| Faster convergence discovers heal and replication work sooner | Heal admission keeps its queue bounds and `queue_full` and `dropped` accounting in `crates/scanner/src/scanner_folder/ledger.rs`; replication MRF keeps its pending cap. Convergence speed does not change retry intervals. Discovered work is swept lazily under those bounds, in the sense of a lazy sweeper: discovery and execution are decoupled. | +| Guardrail vetoes publication | A veto is attributed to a field and localised to a peer where the field allows it; only movement and topology defer the whole cycle. Deferred retries use the capped superseded backoff. A probe failure on one peer never costs more than that peer's dirtied buckets. | +| One bucket walk may be large | Inside a bucket, the walk takes a per-prefix quantum: a prefix that exhausts it checkpoints and yields to the next prefix or bucket, and resumes under the existing cursor and page-index fences. No single prefix may consume the whole cycle budget. The walker's own bookkeeping must stay linear in entries per prefix. `RawEnumerationProgress::record_entry` in `crates/scanner/src/scanner_folder.rs` once re-normalized the accumulated entry list on every entry (rustfs#8113: 8,000 entries cost about 40 s); since #8114 the page writer hashes each entry once when its page commits, and a regression test pins 20,000 entries in one prefix to that cost so pacing cannot be built on a quadratic walker. | + +## What stays fail-closed + +- An uncertified bucket keeps its last certified value as its reported value. + Live writes do not advance it. Hard quota admission is owned by the + reservation ledger + ([scanner-usage-authority-decision.md](scanner-usage-authority-decision.md)), + so this value is reporting input, not an admission authority. +- The storage publication epoch, leadership claim, CAS revision, publication + lease, and tier registry fences are untouched. This design narrows what a + generation movement means; it removes no fence. +- A peer that cannot produce scoped entries keeps every bucket it dirtied + grey. Rolling upgrades degrade to today's behaviour for those buckets and + nothing else. +- Scoped acknowledgements keep the receiver bounds: entry count, byte size, + concurrency, and deadline. A cycle that would exceed them falls back to + whole-cycle acknowledgement for that cycle. + +## Backward compatibility + +Compatibility means four things here, and each phase must prove all four: +an older release reads every artifact the new release writes; the new +release reads every artifact the older release left behind; a mixed-version +cluster keeps publishing wherever it publishes today; and every behaviour an +operator relies on today is reachable by configuration. Rollback must never +require a scanner reset. + +### Persisted artifacts + +| Artifact | Encoding and tolerance today | Rule in this design | Rollback behaviour | +|---|---|---|---| +| `.usage.v2.json` (`DataUsageInfo`) | JSON; every optional field is `serde(default)` and skipped when empty; unknown fields are ignored | The pending-bucket list, its count, and the observational routing mark are additive optional fields. `usage_snapshot_converged` keeps its meaning. No existing field changes type or semantics. | An older reader ignores the pending list and sees a complete snapshot with `usage_snapshot_converged = false`. Its admin and quota readers accept that as an authoritative publication; its scanner does not use it as a scoped-scan baseline and runs full walks until it publishes a converged snapshot of its own, which is today's behaviour, not a reset. | +| `.usage-cache.bin` (`DataUsageCache`) | MessagePack; `DataUsageScanIdentity`, `DataUsageScanProgress`, and `DataUsageRawEnumerationCursor` deny unknown fields; the map-encoded cache metadata tolerates unknown keys | No field is added to a struct that denies unknown fields. Bucket color and start generation are process state in the dirty map, not cache fields. If a per-bucket value must persist, it goes into the map-encoded metadata as `scan_execution_digest` did. The old-generation sweep reuses the existing checkpoint, resume, and cursor fields unchanged. | An older reader ignores any added map key and treats the cache under its existing identity rules. | +| `.bloomcycle.bin` and recovery markers | Versioned by `schema_version`; a mismatch is quarantined | No schema bump. Pending-bucket state is derived from the dirty map and the published snapshot, not stored in cycle state. | Unchanged. | +| `scanner/durable-dirty-producer-replay/.json` | Strict schema; entries deny unknown fields | The barrier-hit class is not persisted. Replayed entries are treated as `Namespace` class, which can only cause an extra scoped rescan, never a missed one. | Unchanged. | +| `.usage.observed.json` | JSON, same tolerance as `.usage.v2.json` | Still written for guardrail-vetoed cycles. A `Complete` cycle with pending buckets writes the authoritative object, not the observed one. | An older release keeps reading observed objects by its existing rules. | + +### Internode protocol + +- `SCANNER_ACTIVITY_PROTOCOL_VERSION` is matched exactly by + `scanner_cycle_activity_status`; a peer on any other version fails the + probe and the cycle is deferred. Phases 1 to 4 therefore change no + `ScannerNodeActivity` wire field and do not bump the version. The + guardrail reinterprets fields that already exist. +- Phase 5 sends scoped acknowledgements over the existing + `SCOPED_DIRTY_USAGE_PROTOCOL_VERSION` receiver, gated per peer by the + existing `probe_only` capability request. A peer without the capability + keeps its dirtied buckets grey and is acknowledged by the whole-cycle path. + No activity protocol bump is required for scoped acknowledgements. +- If a later change does need an activity protocol bump, it inherits the + existing fail-closed mixed-version window and must say so in its + compatibility note; this design does not widen that window. + +### Admin API and console + +- `ScannerUsageFreshnessSnapshot` and the scanner status contract gain only + additive fields with `serde(default)`. No existing field is renamed, + removed, or changes type. +- `last_cycle_result` keeps its existing string values. A `Complete` cycle + with pending buckets reports `complete`; consumers that do not read + `pending_dirty_buckets` see a completed cycle, which is the truthful + reading for them. +- `usage_snapshot_converged` keeps its meaning on every surface that + exposes it, including account info and cluster usage. +- No contract version carried to the console changes. The console gates on + contract versions elsewhere; this design must not be the change that + trips that gate. + +### Configuration + +- New keys (walker factor, pacer ceilings, `K`, `M`, scoped-cycle quantum) + live in the `scanner` subsystem beside the existing keys. The server + config already preserves unknown keys, so an older release leaves them in + place and ignores them. +- `ScannerSpeed` presets and `max_concurrent_disk_scans` keep their names + and accepted values. Today's throttle is reachable by setting the preset + explicitly and the walker factor to 1; the documented operator procedure + for restoring today's I/O profile is that single pair of settings. +- Every new key has a default that keeps rotational-media behaviour equal to + today's. On flash the defaults deliberately change behaviour: a walker + factor above 1 and a pacer-driven throttle are part of the fix, not a side + effect, so they are on by default and called out in the release note of + the phase that ships them. Compatibility is a property of artifacts, + protocols, APIs, and configuration reachability; it is not a promise that + default scanner throughput stays the same. + +### Metrics + +Existing metric names, labels, and meanings are unchanged. New values are +new series. The `superseded_cycles` counter and the superseded code of the +last-cycle-result gauge in `crates/obs/src/metrics/collectors/scanner.rs` +will fall because superseded becomes rare; that is a semantic consequence, +not a rename. + +### Mixed-version behaviour summary + +| Cluster state | Result | +|---|---| +| All nodes on the new release | Full design in effect. | +| Coordinator new, some peers old (same activity protocol) | Per-bucket certification applies to local and capability-bearing peers; buckets dirtied on old peers stay grey and clear through whole-cycle acknowledgement. Publication continues. | +| Coordinator old, some peers new | Old classifier runs; new peers answer the existing activity and acknowledgement protocols unchanged. A `Status` hit on a new peer no longer moves that peer's dirty generation, so it no longer supersedes the old coordinator's cycle; everything else equals today's. | +| Node rolled back after the new release published | The rolled-back node reads the new `.usage.v2.json` as an unconverged authoritative snapshot and continues with today's rules; no reset. | + +No temporary compatibility path is introduced by this design, so no +`RUSTFS_COMPAT_TODO` entry is required. If an implementation phase finds it +needs one, it registers it in +[compat-cleanup-register.md](compat-cleanup-register.md) with a removal +condition before merging. + +## Surfaces + +- `usage_freshness` in the admin scanner status reports + `dirty_pending_buckets` (`Namespace` class only; after certification the + dirty map holds the shaded pending buckets, and the published pending list + also names buckets whose walk stopped early), the attributed guardrail veto + (`guardrail_veto_total`, `last_guardrail_veto_unix_secs`, + `last_guardrail_veto_reason` as `@`), plus the pacer target, + achieved rate, and whether a ceiling is binding. The console marks usage + stale only for that count or for a guardrail veto. +- `last_cycle_result` reports `success`, its existing label for a completed + cycle, for a cycle with pending buckets; `superseded` is reserved for + guardrail vetoes and the phase-2 peer fallback and should be rare. +- The superseded retry schedule applies only to guardrail vetoes and to + explicit full sweeps. A `Complete` cycle with pending buckets uses the + ordinary dirty-usage wake and the next cycle is scoped to the pending set. +- `rustfs_cluster_usage_objects_snapshot_converged` keeps its meaning. + +## Delivery + +Phases are independent and ship in order; each has an oracle on both failure +modes. + +1. **Status-class barrier hits.** Route replication writebacks, and with + them the scanner-queued heals, off the namespace path so they never + advance the dirty-usage generation, wake the scanner or reach the journal. + Oracle: a single-node site-replication resync reaches a `Complete` cycle + while the resync is running, and `replication_stats` for the bucket + refresh on the following cycle. +2. **Tri-color certification, bounded remark, baseline bootstrap, aging + order, per-prefix quantum, bounded guardrail.** Record start generations, + certify at finalize, publish `Complete` with pending buckets, accept a + non-converged `Complete` publication as the structural baseline, order + grey buckets by waiting age under a per-cycle quantum, give each prefix + inside a bucket its own quantum, and attribute and localise guardrail + vetoes. The first change of this phase ships the termination rule: + certification, `Complete` with pending buckets, the baseline bootstrap + and guardrail attribution. Bounded remark, aging order, the per-prefix + quantum and guardrail localisation follow as their own changes. Oracle: a single-node deployment with one bucket written every + few seconds reaches `Complete` on the first full walk, reports exactly + that bucket pending, and the next cycle walks only that bucket; a + deployment with more buckets than the quantum walks every bucket within + the bounded number of cycles; a bucket holding one flat prefix of roughly + a hundred thousand entries no longer starves the buckets after it; a + multi-node deployment with one peer's probe failing publishes every bucket + that peer did not dirty. +3. **Pacer and walker concurrency.** Pin the entry bookkeeping in + `RawEnumerationProgress::record_entry` to linear cost first (the + clone-per-entry itself was removed by #8114; the regression test and the + in-place page writer are this phase's groundwork), then decouple walker + count from disk count (`walkers_per_disk`, shipped) and drive the throttle + from the pacing target under CPU and I/O ceilings (open). Oracle: on a single flash disk the same namespace + converges in a fraction of today's cycle time with scanner CPU under the + ceiling; on rotational media the default factor keeps today's I/O + profile; a foreground write load at the admission threshold still backs + the scanner off, and latency-sensitive foreground reads on a single NVMe + disk stay within their pre-scanner envelope while a cycle runs. +4. **Generations and the old sweep.** Promote and demote buckets, run the + checkpointed old sweep on its own cadence. Oracle: a cold bucket is walked + at least once every `M` cycles while a hot bucket is written continuously. +5. **Scoped acknowledgement sender.** Activate production of scoped entries + for peers behind the existing receiver so distributed deployments certify + per bucket. Oracle: a multi-node deployment with writes on one node + certifies all buckets written elsewhere and leaves only the hot bucket + grey; a peer on the previous protocol version leaves its dirtied buckets + grey and nothing else. + +Phase 1 lets a replicated single-writer deployment converge; phase 2 makes +continuous business writes converge; phase 3 shortens the cycle itself; +phase 4 protects the long tail; phase 5 extends all of it to distributed +clusters. + +### Validation topologies + +Each phase is validated against the deployment shapes that reported the +defect, not only against synthetic fixtures. The shapes are listed by their +characteristics; the originating reports are the issues referenced from the +tracking discussion. + +| Shape | Exercises | +|---|---| +| single node, single disk, site replication resync in progress | phase 1: `Status`-class hits, self-invalidation by queued heals | +| single node, single disk, parity 0, on the order of 1,500 buckets and 1.4 million objects, continuous writes, no lifecycle or replication | phases 2 and 3: termination rule, baseline bootstrap, aging order, walker count | +| single node, one bucket whose metadata prefix holds on the order of 100,000 keys with noncurrent versions and delete markers under Object Lock | phase 2 per-prefix quantum, phase 3 bookkeeping prerequisite | +| 40 nodes, one pool, 400 drives, continuous log-chunk writes | phase 2 bounded guardrail, phase 5 scoped acknowledgements, journal and lease behaviour under load | +| 4 nodes, 48 drives, on the order of 600 million objects, continuous file-system-backed writes | phase 5 at scale, guardrail attribution under a permanent PUT tail | +| single node, NVMe, latency-sensitive small-object foreground reads | phase 3 ceilings and foreground backoff | + +A phase is not complete until its oracle holds on every shape it names. A +pass on synthetic fixtures alone is evidence for a pull request, not for +release. + +## Non-goals + +- Soft quota or best-effort usage. The authority decision in + [scanner-usage-authority-decision.md](scanner-usage-authority-decision.md) + stands. +- Removing or merging any publication fence. +- Making replication statistics authoritative. They remain observational and + may lag by one cycle after a `Status` hit. +- Parallelising the folder walk inside one bucket. Walker count is raised at + bucket granularity and the walk inside a bucket is bounded by the + per-prefix quantum; intra-bucket parallelism is a separate change with its + own ordering and checkpoint implications. diff --git a/docs/architecture/scanner-usage-authority-decision.md b/docs/architecture/scanner-usage-authority-decision.md index db8714b7e..edd7ed8d9 100644 --- a/docs/architecture/scanner-usage-authority-decision.md +++ b/docs/architecture/scanner-usage-authority-decision.md @@ -1,23 +1,105 @@ # Scanner Usage Authority Decision **Use this when:** deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary. -**Source of truth:** [scanner-usage-publication.md](scanner-usage-publication.md), `crates/ecstore/src/bucket/quota/checker.rs`, and the scanner publication state under `crates/scanner/src/scanner/`. +**Source of truth:** `crates/ecstore/src/bucket/quota/reservation.rs` (`QuotaLedger`, `QuotaContext::reserve`, `exact_bucket_usage`), `crates/ecstore/src/bucket/quota/checker.rs` (`QuotaChecker::get_real_time_usage`), `rustfs/src/app/object/shared.rs` (`map_quota_check_outcome`, `apply_quota_admission`), [scanner-usage-publication.md](scanner-usage-publication.md), and the scanner publication state under `crates/scanner/src/scanner/`. ## Decision -Date: 2026-09-03 +Date: 2026-10-19. Supersedes the 2026-09-03 decision recorded below. + +Hard quota admission is owned by the durable reservation ledger, not by scanner +data usage. `QuotaLedger::accounted_usage` is initialised and reconciled by +`exact_bucket_usage`, a `list_object_versions` walk of the bucket, and +`QuotaContext::reserve` admits a write against `admitted_usage + growth <= +quota_limit`; neither reads a scanner snapshot. Every quota saved through the +admin API carries `reservation_protocol = V2`. + +Scanner data usage is best-effort reporting input: admin info, Console, +metrics, and the degraded fallback of `QuotaChecker::get_real_time_usage` for +buckets whose quota predates the reservation protocol. It must not be able to +fail a write on a bucket the ledger admits. The known violation is +`map_quota_check_outcome`, which turns `QuotaError::UsageUnavailable` into +`ServiceUnavailable` even when `QuotaCheckResult::uses_durable_reservations` +is true and `apply_quota_admission` already admits unconditionally. + +## Rationale + +The ledger already carries the write-path accounting that the 2026-09-03 +decision assumed only the scanner could provide: an exact baseline, per-write +reservations settled at commit, and a reconcile step after a denial. With the +admission decision made there, keeping scanner usage authoritative buys no +enforcement and costs every deployment the proof machinery that makes a +distributed background scan safe to feed a quota: leases that reject pending +PUT commits, observed-versus-authoritative routing, and a usage floor whose +unreadability stops the scanner. Buckets without a configured quota never read +usage at all (`check_quota_with_usage_reporting` returns on the zero-I/O path), +so the authority protected a narrow case while blocking usage statistics, +lifecycle discovery, and heal discovery for everyone. + +## Consequences + +- [scanner-usage-publication.md](scanner-usage-publication.md) keeps the + fences that protect consistency across leadership and data movement: + leadership claim, storage publication epoch, CAS revision, per-set + freshness, tier registry generation. The layers that existed only to make + scanner usage a quota authority are scheduled for staged removal under a + compatibility window: the usage floor as hard-quota input, observed-versus- + authoritative routing as an admission distinction, and strict + publication-lease rejection of pending ordinary namespace commits. +- Buckets with `reservation_protocol = None` are migrated by re-saving their + quota configuration through the existing save path; until migrated they keep + the degraded floor path and fail closed only when neither authoritative usage + nor a floor exists. The re-save is triggered lazily by the first quota read + of such a bucket (`QuotaChecker::get_quota_config`) and in bulk by + `POST /rustfs/admin/v3/quota-migration`; the limit and type are unchanged, + the write keeps the incarnation, quota-revision and fleet-proof checks of + the admin save, and a bucket whose quota changed in between is skipped, + not overwritten. See + [durable-quota-reservation-v2.md](../operations/durable-quota-reservation-v2.md). + Note: the admin default for a new quota is still protocol v1 + (`BucketQuota::new`); v2 is chosen per request or by this migration. Both + are durable, so neither is a migration candidate. +- [scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md) no + longer cites quota authority as a reason to keep uncertified buckets + fail-closed; an uncertified bucket keeps its last certified value as its + reported value. +- The scanner storage boundary (#2219) may be designed against the reporting + contract; it must still expose the CAS key-value, cycle lock, and + recovery-marker capabilities, but the usage-floor and observed-snapshot + capabilities become reporting surfaces rather than proof obligations. + +### Remaining "authority" consumers after D3 (inventory, 2026-10-19) + +The quota path itself no longer reads scanner usage as authority for a +durable bucket: `QuotaChecker::check_quota_with_usage_reporting` reports an +unavailable lookup as `current_usage = None` and `map_quota_check_outcome` +never sees `UsageUnavailable` for it. The surfaces below still carry the +authority shape. Each is listed with its production readers, what it means +now, the condition under which it can be retired, and the compatibility +window. Retirement itself is separate work; none of it is required for +correctness after this decision. + +| Surface | Readers today | Meaning after D3 | Retirement condition | Compatibility window | +|---|---|---|---|---| +| `usage_snapshot_complete` (`DataUsageInfo::is_complete_bucket_usage_snapshot`) | quota-facing loader `discard_incomplete_bucket_usage`, `normalize_loaded_data_usage`, `replace_bucket_usage_memory_from_info_if_generation`, `get_bucket_usage_memory` in `crates/ecstore/src/data_usage/mod.rs`; server/pool info in `crates/ecstore/src/diagnostics/admin_server_info.rs`; `narrow_data_usage_snapshot_to_measured_buckets` in `rustfs/src/app/admin_usecase.rs`; decommission capacity in `crates/ecstore/src/core/pools.rs`; obs metrics, heal progress baseline; scanner `data_usage_info_is_cold`, `complete_scanner_cache_snapshot_plan_digest`, `reject_incomplete_snapshot` | A completeness mark for reporting (admin, Console, metrics, heal progress). For quota it only decides whether a memory value exists, which can no longer fail a durable write. | No bucket with `reservation_protocol = None` remains (lazy trigger plus admin sweep), so the loader's discard is no longer an admission gate; the mark itself stays as the reporting contract. | Until every quota is migrated, plus one release in which the snapshot-protocol admission path is still compiled. | +| `lookup_degraded_bucket_usage_baseline` / `degraded_baseline` (`crates/ecstore/src/data_usage/mod.rs`) | `QuotaChecker::get_real_time_usage` (`crates/ecstore/src/bucket/quota/checker.rs`); `session_capacity_view` in `rustfs/src/protocols/client.rs`; fed by `load_data_usage_from_backend_with_baseline`, `backfill_degraded_baseline_from_observed`, `cache_data_usage_snapshot_result` | Admission input only for snapshot-protocol buckets; for durable buckets it only shapes the reported `current_usage`. | Same as above; afterwards it is a reporting fallback that can be dropped together with the capacity view's choice of source. | Same as above. | +| Usage floor identity (the `scanner_epoch`/`scanner_cycle` identity of `buckets/.usage.v2.json` and `buckets/.usage.json` under `.rustfs.sys`, each with a `.bkp`; there is no separate floor object) | `persisted_usage_floor_for_startup`, `persisted_usage_floor`, `apply_persisted_usage_floor`, `reset_scanner_cycle_recovery_for_intent`, the rc.1–rc.3 `legacy_*_usage_fence` readers in `crates/scanner/src/scanner/cycle_state.rs`; `data_usage_info_has_persisted_baseline_identity` in `crates/scanner/src/scanner.rs`; `usage_snapshot_for_epoch_fence` in `crates/scanner/src/scanner/leadership.rs`; `read_data_usage_persist_baseline` in `crates/scanner/src/scanner/usage_store.rs`; `UsageSnapshotSource` resolution in `crates/ecstore/src/data_usage/mod.rs` | The scanner's own monotonic cycle fence for leadership and cycle recovery. It is no longer a quota proof: an unreadable floor stopping the leader iteration (`usage_floor_load_failed`) is a scanner-availability choice, not a quota requirement. | The quota fallback no longer reads the legacy `.usage.json` baseline; then "floor unreadable stops the scanner" can become "report and rebuild", and the rc.1–rc.3 fences (`RUSTFS_COMPAT_TODO` backlog-2102 / backlog-2122) retire on their own schedule. Cycle monotonicity stays. | One release after the compat markers' own deadline. | +| Observed-versus-authoritative routing (`.usage.observed.json` versus `.usage.v2.json`) | decision in `classify_nsscanner_cycle`, `apply_scanner_cycle_publication_marks`, `publish_observational_snapshot`, `scanner_cycle_activity_status` (`crates/scanner/src/scanner_io.rs`); write routing in `crates/scanner/src/scanner/usage_store.rs`; readers `load_observed_data_usage_snapshot`, `select_admin_data_usage_snapshot`, `merge_partial_observation_for_admin` and the degraded-baseline backfill (`crates/ecstore/src/data_usage/mod.rs`); scanner cold-start and cache reuse; `ScannerCyclePublicationPlan::for_result` | A completeness-marked report. "Never a hard-quota authority" is moot for durable buckets; the distinction remains a reporting mark (`usage_snapshot_observational`, `converged`). | The quota fallback stops distinguishing the two objects; then the pair can collapse into one snapshot carrying the marks, after a reader-first migration of the observed-object consumers (admin, Console, obs). | Readers accept both layouts for one release before the writer changes. | +| Publication-lease strict mode (`purpose == Authoritative && namespace_commits_pending()`) | `acquire_scanner_publication_lease_with_purpose`, `validate_scanner_publication_lease_with_purpose`, `acquire_scanner_publication_lease_guard_with_target`, `scanner_data_usage_publication_blocked` in `crates/ecstore/src/store/mod.rs`; `namespace_commits_pending` in `crates/ecstore/src/runtime/instance.rs`; `scanner_local_publication_defer_reason` in `crates/scanner/src/scanner.rs`; lease RPCs and the lease token on `RenameData`/`Delete` in `crates/protos/src/node.proto` | A pending ordinary PUT commit is a write-barrier event (the cycle is superseded or the bucket pending), not a reason to deny an authoritative lease. Data-movement and generation checks stay. | Drop the pending-commit rejection and its two revalidation twins, and the pending-commit term of the local barrier; keep the movement fence and the lease token on writes. The local barrier's term is handled by the scanner cycle-termination work; the remote lease follows once that lands. | Mixed versions are safe: an old peer still rejects and the coordinator treats that as retryable (`scanner_publication_lease_error_is_retryable`), so the removal rolls out node by node. | + +## Superseded decision (2026-09-03) RustFS keeps scanner data usage as authoritative for quota admission. This selects option A from the backlog decision record: the scanner publication protocol remains necessary while quota admission consumes scanner usage. The cycle epoch, publication CAS, data-movement fence, tier-registry fence, observed snapshot layer, and persisted usage floor are retained and documented as protocol invariants rather than treated as removable compatibility clutter. -## Rationale +### Rationale at the time Quota is a write-path admission decision, so serving quota from best-effort scanner data would turn temporary scanner lag into under-enforcement. The current design therefore needs an availability story for authoritative usage instead of deleting the proof layers that make it authoritative. The scanner usage floor provides that availability story. It is a lower bound used when a complete authoritative snapshot is not available, including cold startup, upgrade recovery, and incomplete-cycle repair. Observed snapshots remain useful for admin and observability, but they do not become quota authority. -## Consequences +### Consequences at the time #2214 is the hard design input for future usage-publication changes. A future proposal may still choose soft quota and MinIO-style best-effort usage, but that would be a product change with its own staged compatibility plan for persisted artifacts. diff --git a/docs/architecture/scanner-usage-publication.md b/docs/architecture/scanner-usage-publication.md index 12c600d21..4985fc2ec 100644 --- a/docs/architecture/scanner-usage-publication.md +++ b/docs/architecture/scanner-usage-publication.md @@ -23,29 +23,38 @@ therefore has three identities: If any identity changes before commit, the result is a candidate for retry or observation, not an authoritative baseline. -Ordinary PUT rename fanouts also track instance-scoped in-flight work. A quorum -ACK does not release it: the actual disk tasks retain ownership until their -rename work ends, including when the request caller is cancelled. Scan admission -remains movement-only so sustained PUTs do not stop namespace walks and -scanner-driven lifecycle discovery. The post-walk local publication check and -remote publication leases reject pending fanouts. Begin/end namespace generations -invalidate scans and cached plans across the fanout; after acquiring remote -leases, the coordinator rechecks the full activity digest before publishing an -authoritative aggregate. This catches a tail that finishes between the scan's -last probe and lease acquisition. +Ordinary PUT rename fanouts track instance-scoped in-flight work +(`NamespaceCommitGuard` in `crates/ecstore/src/runtime/instance.rs`): a quorum +ACK does not release it, the actual disk tasks retain ownership until their +rename work ends. A pending ordinary PUT commit is a write-barrier event for +its bucket, not a usage-publication veto (decision of 2026-10-19). The S3 +handler shades the bucket through `record_dirty_usage_object_from_producer` +after the ACK, the finalize certification compares the bucket's generation, +and a peer whose dirty state moved supersedes the cycle until scoped +acknowledgements exist; together they already cover a tail that lands after +the walk. The guard drops before the handler records the shade, so a +pending-commit veto could never close the window ahead of the shade either. +`scanner_data_usage_publication_blocked` therefore blocks usage publication +only for data movement (`scanner_data_usage_publication_snapshot_blocked`), +and the same rule applies on the post-walk probe, the local defer decision, +the post-lease recheck, the usage-store route probe, and the ECStore +publication admission and remote lease paths. Scan admission remains +movement-only so sustained PUTs do not stop namespace walks and +scanner-driven lifecycle discovery. After acquiring remote leases, the +coordinator still rechecks the guardrail fields of the activity snapshot (the +reuse identity and the movement fields, plus the peers' dirty-usage state while +per-bucket certification is local only) before publishing an authoritative +aggregate. Local namespace and dirty-usage generation movement in that window +is not a veto: its buckets stay in the dirty map and are the next cycle's +work, as described in +[scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md). This adds no namespace or movement lock. An already-verified older snapshot may -still precede a newly started write. Sustained or stalled PUT tails can delay -authoritative usage publication, which resumes through the existing retry -schedule rather than a new immediate-wakeup protocol. Intermediate per-set and -prefix cache readers retain their existing approximate-cache semantics. A -prolonged pending tail with no generation changes can also delay cycle advancement -and fresh rescans of already-current caches; this is not a guarantee of lifecycle -progress under indefinitely stalled storage I/O. - -This PUT-tail protection requires every writer node to be upgraded. It does not -prove that a failed tail replica has healed, and it does not extend the same -in-flight tracking to multipart or other namespace mutation paths. +still precede a newly started write; that write's bucket is the next cycle's +work. Intermediate per-set and prefix cache readers retain their existing +approximate-cache semantics. The in-flight tracking does not extend to +multipart or other namespace mutation paths, and it does not prove that a +failed tail replica has healed. ## Dirty producer replay ownership @@ -93,7 +102,7 @@ They must not be collapsed unless the replacement proves the same exclusions. | CAS revision | backing config object store | lost updates to usage snapshots, scanner caches, or cycle-state objects | | Per-set freshness | scanner aggregation | a merged usage snapshot that combines stale and current set results | | Tier registry generation | scanner tier accounting | bytes classified against a different warm-tier registry | -| Usage floor identity | scanner publication and ECStore quota fallback | empty or legacy values becoming plausible authoritative quota input | +| Usage floor identity | scanner publication and the ECStore quota fallback for buckets without a reservation protocol | empty or legacy values becoming a plausible degraded quota floor | A reader that cannot prove the required fence for its surface must fail closed or use the documented observed path below. It must not synthesize an empty usage @@ -105,8 +114,15 @@ The structural scan-plan digest can remain stable across ordinary bucket writes so a scoped scan can retain unaffected baseline buckets. It is not sufficient proof for reusing a completed result within the same cycle. Bucket work uses an execution digest combining the structural plan and the full activity snapshot, -with the bucket's dirty generation included in its cache identity. Completed set -caches carry the same execution digest separately from their structural plan. +with the bucket's dirty generation included in its cache identity. A bucket +sweep's checkpoint is keyed separately by the coverage identity +(`scanner_activity_coverage_digest`, see +[scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md)), so +the reuse identity can change without restarting a sweep in progress. Completed +set caches carry the same execution digest separately from their structural +plan; a set cache that stands in for buckets whose walk stopped early binds +the carried bucket names into it (`scanner_carried_execution_digest`), so it +is never adopted in place of a full walk of the same cycle, nor the reverse. The persisted set-root fast path requires equal execution identities as well as the existing source, cycle, leader, tier, and cache-structure checks. @@ -121,6 +137,24 @@ reuse without this identity. Older readers can ignore the added map key, but older writers do not enforce its fence; readability is not a mixed-version publication-safety guarantee. +Bucket checkpoints distinguish completed traversal from object health. A failed +object does not stop later completed subtrees from advancing the traversal +frontier. On resume, a sorted index of cached entries with `failed_objects > 0` +excludes their containing subtrees from reuse; healthy covered siblings keep +their exact cached accounting. Failure counters survive retry-cache TTL and +capacity eviction, and any remaining entry failure keeps the bucket partial. +Directory traversal errors and unresolved size reconciliation still stop the +frontier because they do not have the same per-entry coverage proof. + +Coverage follows path-component traversal order, including punctuation in sibling +names. Coverage receipts use digest domain 3 and bind every entry in each covered +subtree using the same predicate as resume, while excluding the unvisited suffix. +No unsigned nested failure can authorize subtree reuse. Current +readers accept and migrate domain 2 and legacy epoch-bound receipts. Older +binaries cannot validate a domain 3 receipt and safely reread the bucket rather +than applying their unconditional covered-subtree skip. The serialized receipt +shape, cache CAS, leader epoch and publication fences are unchanged. + ## Persisted Objects The persisted objects are part of the compatibility contract. Removing one @@ -142,7 +176,11 @@ requires a compatibility window and a dedicated cleanup entry. Observed snapshots are a diagnostic and availability layer. They may be served only when the snapshot explicitly reports that it is partial or observational, and only to consumers that do not make hard quota, durability, or deletion -decisions from it. Admin usage views may expose this state with completeness +decisions from it. The scanner routes a snapshot to the observed object by the +explicit `usage_snapshot_observational` mark that a vetoed or unverifiable cycle +sets, not by `usage_snapshot_converged`: a complete cycle that certified every +bucket but the ones written during its walk publishes an unconverged +authoritative snapshot that names those buckets as pending. Admin usage views may expose this state with completeness flags so operators can see progress while the authoritative publication is blocked. Quota enforcement must not use an observed snapshot as the current usage authority. @@ -155,27 +193,31 @@ available, quota remains unavailable and fails closed. ## Availability Decision -Decision date: 2026-09-03. +Decision date: 2026-10-19, superseding 2026-09-03; the record is +[scanner-usage-authority-decision.md](scanner-usage-authority-decision.md). -RustFS keeps scanner usage as the authority for hard quota admission. The -publication protocol therefore remains necessary: leadership, storage epoch, -lease, CAS, observed snapshot, and usage-floor layers are the proof machinery -that lets a distributed background scan feed a quota decision without accepting -stale or cross-generation usage as current truth. +Hard quota admission is owned by the durable reservation ledger. Scanner usage +is best-effort reporting input and the degraded fallback for quotas that +predate the reservation protocol. The publication protocol keeps the fences +that protect consistency across leadership and data movement: leadership +claim, storage publication epoch, CAS revision, per-set freshness, and tier +registry generation. The layers that existed only to make scanner usage a +quota authority are scheduled for staged removal under a compatibility +window: + +- the usage floor as hard-quota input (it remains the degraded fallback for + `reservation_protocol = None` buckets until they are migrated); +- observed-versus-authoritative routing as an admission distinction (it + remains a completeness mark for readers); +- strict publication-lease rejection of pending ordinary namespace commits. The availability contract is: -- the authoritative fast path reads complete in-memory or persisted scanner - usage; -- during upgrade or publication outage, quota may admit against the persisted - per-bucket usage floor; -- the floor is advisory for the outage window and must converge back to a - complete scanner publication; -- a bucket with neither authoritative usage nor a valid floor fails closed. - -Changing this decision to a soft-quota model would be a product change, not a -scanner refactor. It would need a staged removal of the authority-specific -layers and compatibility handling for the persisted objects above. +- a bucket admitted by the reservation ledger never fails a write because + scanner usage is unavailable; +- a bucket without a reservation protocol keeps the degraded floor path and + fails closed only when neither authoritative usage nor a floor exists; +- scanner publication incompleteness is reported, never converted to zero. ## Deletion And Recovery Rules @@ -195,6 +237,136 @@ Missing, undecodable, or identity-less data is not converted to zero. It is reported as uninitialized, recovery-required, observed-only, or unavailable according to the reader's surface. +Every fail-closed scanner gate has an automatic exit and a bounded dwell time. +None of the exits deletes evidence or relaxes the leadership, CAS, movement, or +usage-floor fences: + +- A retryable recovery observation (a transient cycle-state read failure, a + usage-floor load failure, a pending legacy floor recovery, or a quarantine + marker that could not be written) never becomes terminal. Once the short + retry budget is spent the state reads `backoff`, `retryable` stays true, the + supervisor keeps retrying with an exponential delay capped at ten minutes or + sooner when recovery is explicitly woken, and `next_attempt_at_unix_secs` + plus the `reason` name the next attempt. A successful iteration resets the + count, and an observation older than 24 hours restarts its budget. Followers + still clear only an unclassified `transient` or `backoff` observation on lock + contention. +- A `.bloomcycle.bin` that this reader classifies as `corrupt` is quarantined + by writing the marker, and the next leader iteration rebuilds the counter: it + starts from an uninitialized cycle raised to the usage-floor cycle and lets + the ordinary leadership claim CAS-replace the quarantined revision under a + higher epoch. The marker is retained as evidence. Because its + `primary_revision` no longer matches the primary, older and newer readers + start normally, and the supported cycle-state reset remains the way to remove + it. A `future_schema` counter and a `cleanup-pending` marker stay blocked: the + first belongs to a newer binary and the second to an operator reset in + progress. Only a marker that cannot be written or read keeps the node in + `blocked`, probed every five minutes. +- A pause ledger that cannot be claimed (no replicas, or conflicting or invalid + replica records) runs the leader in `degraded` mode instead of skipping every + cycle. Cycles run untracked against an empty in-memory ledger, publication is + still admitted only through the data-movement fence, `persistence_state` + reports `degraded` with the claim error, and the claim is retried on every + scheduler wake and every movement probe, at most once per five minutes. When + the claim succeeds the durable ledger resumes with its persisted phase and + catch-up semantics. +- The usage-state full-rebuild reset accepts a cluster whose usage slots are + all corrupt and whose cycle state is missing. It starts from the empty + baseline and epoch 1 exactly as a missing floor does, and quota-bearing + buckets keep failing closed until the rebuilt baseline publishes. A corrupt + cycle state must still be rebuilt first. +- A cycle-state save that fails without revealing a newer leadership epoch is + retried under the same epoch: inside the save by reconciling against durable + state (a committed write is confirmed from the read-back, a failed write is + retried with the current revision), and after an exhausted save by reloading + durable state in place. Only a durable state owned by another epoch, or one + that cannot be read back, ends the leader iteration and triggers a new claim, + so peers see a changed `(cycle, epoch)` fence only when ownership changed. + +## Diagnosing Incomplete Census Progress + +Treat bucket coverage, set-worker completion, cycle certification, and usage +publication as separate stages. A successful save can be observational, and a +worker finishing does not certify its namespace coverage. + +| Signal | Meaning and next evidence | +| --- | --- | +| `bucket_checkpoint_prepared` | Checkpoint admission identifies the bucket, pool/set, cycle/leader epoch, and previous cycle/epoch. `resume_position` distinguishes retained traversal position from a fresh sweep; `mixed_coverage` reports whether that retained prefix began under another plan. | +| `bucket_sweep_incomplete` | A bucket reached the end of a sweep but needs another verification pass. `unresolved_objects` reports the complete coverage/health gate, including failed child-cache entries that can be absent from the root counter and diagnostic failure map. Inspect it with `coverage_gap`, `root_failed_objects`, `failed_objects`, `size_reconciliation_entries`, and `mixed_coverage`, using bucket, pool/set, and cycle/leader epoch. `resume_position_on_entry` identifies a sweep entered with traversal position; `resume_position` reports the returned cache position after terminal cleanup. Multiple blockers can coexist. | +| `partial_saved_not_published` | The local or remote worker successfully saved a terminal Partial cache. Correlate its bucket, pool/set, cycle/leader epoch and `resume_position` with admission and sweep events. This confirms persistence, not complete coverage or authoritative publication. | +| `checkpoint_identity_unavailable` | A local worker could not read the persisted bucket identity. `stage=admission` preserves the existing checkpoint without preparing a legacy rebuild; `stage=post_scan` rejects the unverified candidate before persistence. Correlate bucket, pool/set, cycle/epoch and the underlying read error. Remote terminal errors distinguish the same two stages. | +| `checkpoint_identity_changed` | A successful post-scan read no longer matches the admitted identity, including a now-absent sidecar. Rejecting the result protects against bucket replacement; this differs from an unavailable read. | +| `remote_scan_partial` | The coordinator received a Partial terminal response for the named bucket and disk. Correlate the worker's bucket-sweep or checkpoint events; Partial is not a complete bucket root, but the set may still cover the bucket with its last certified value (`partial_buckets_carried`). | +| `partial_buckets_carried` | The set published the last certified value for `carried_buckets` buckets whose walk stopped early; `first_carried_bucket` names one. The cycle can complete with them pending and the next cycle walks them. A partial bucket without a certified value on the set is counted by `set_cache_publish_skipped` instead. | +| `bucket_result_rejected` | A bucket result did not match the cycle's frozen tier-registry generation and was not applied. | +| `set_cache_publish_skipped` | `completed_buckets` counts distinct bucket roots successfully applied to this set cache and `carried_buckets` the partial walks covered by a certified value. Use `first_incomplete_bucket` with pool/set/cycle to locate one actual gap in this set. Inspect `unprocessed_buckets` and budget/cancellation state, then correlate that bucket's worker events. | +| `set_scan_completed` | Set workers have finished. `coverage_complete` reports the structural coverage gate, not durable publication. This event alone does not prove a complete set snapshot. | +| `incomplete_coverage` | The cycle did not obtain complete structural coverage. This does not identify a hot bucket or prove that ordinary writes caused the gap. | +| `publication_proof_pending` | Publication is waiting for a retryable proof. Use cycle/stage, `pending_duration`, the preceding `publication_proof_started` kind, and the peer/purpose in the error. This event does not assert that the walk was Complete. | + +For a report such as 52 completed buckets out of 53, first identify whether +the missing bucket was Partial, failed, never admitted, or rejected at merge. +Only then inspect its worker evidence. Mixed coverage is a resumable sweep +under a newer plan; failed objects and size reconciliation require different +follow-up. A global dirty-generation change alone does not establish which +bucket failed to produce a complete root. + +A mixed tail clears traversal positions before returning Partial. If that +terminal cache is saved, the next fresh sweep binds both coverage plans to its +new request, even if ordinary writes continue between cycles. A periodic +checkpoint saved before terminal cleanup still contains traversal position; +resuming it under a changed plan can legitimately be mixed again. Correlate +admission, sweep completion, and terminal persistence for the same bucket and +pool/set across consecutive cycles. Samples from different sets cannot establish +a repeated mixed loop. Check save failures, remote interruption, and leadership +changes if only periodic progress survives. + +Identity lookup has three outcomes: a verified UUID, a confirmed absent sidecar, +or a read failure. Only confirmed absence may use the existing full-rebuild path +without checkpoint identity. A lock, transport, or corrupt-metadata error must +not become absence and reset a valid frontier. A successful read that proves a +different identity still invalidates the old result. + +The `.bucket-incarnation@latest` resource belongs to a read of the persisted +identity sidecar. Scanner identity checks occur before a walk and again before +saving its result; set inventory preflight also reads identities. A single +remote lock-client warning can coexist with a successful quorum, so it does not +prove that checkpoint persistence failed. Use the scanner's identity stage and +the terminal save result to establish that connection. + +An unknown bucket cannot be quarantined as zero in an authoritative total. +Existing pending-bucket semantics still require complete structural coverage. +A separately labelled subtotal would need proof for each included bucket across +every source, with missing buckets explicitly unavailable or stale; it must not +advance full-census authority or acknowledge their dirty work. + +Use a bounded DEBUG window on the capacity-scanner leader and the workers +serving the affected bucket: + +```text +RUST_LOG=warn,rustfs::scanner=debug,rustfs::scanner::io=debug,rustfs::scanner::folder=debug +``` + +Capture one representative cycle with its effective budget configuration, +source commit on every node, per-set completion summaries, partial-bucket +events, remote errors, publication kind and result, and cycle-state persistence. +The summaries report counts and bounded bucket identifiers, not object-path +inventories. Keep logs access-controlled and redact deployment identifiers +before sharing them. + +An unlimited walk budget can also leave a retryable authoritative publication +waiting for leadership loss or a successful proof. Observational and +authoritative publication both retain the movement fence, on this node and +through the remote publication leases. Do not remove those fences to clear a +status. Verify +that a deployment contains the relevant fixes by commit ancestry; a newer +release name alone is not evidence that it includes current main. + +Keep usage objects, checkpoint data, and recovery material intact during +diagnosis. Acceptance requires usable census values, appropriately advancing +dirty acknowledgements after publication, and progress across later cycles or +leadership changes. Worker completion timestamps alone are insufficient. + ## Existing Fixes As Invariants Several prior scanner fixes are consequences of this contract rather than diff --git a/docs/operations/architecture-guard-troubleshooting.md b/docs/operations/architecture-guard-troubleshooting.md index be23906f8..2aae62e48 100644 --- a/docs/operations/architecture-guard-troubleshooting.md +++ b/docs/operations/architecture-guard-troubleshooting.md @@ -57,7 +57,7 @@ consider adding it to the script's `checked_files` list. ## `check_doc_paths.sh` -Instruction docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`) and every +Instruction docs (`AGENTS.md`, `ARCHITECTURE.md`) and every Markdown file under `docs/` (architecture, operations, testing, index) must not reference repo file paths that no longer exist. If your refactor moved code, update the docs that point at it — the error message lists `doc -> stale-path` diff --git a/docs/operations/bucket-heal-recovery.md b/docs/operations/bucket-heal-recovery.md index 60d10b3ca..b00a5b62b 100644 --- a/docs/operations/bucket-heal-recovery.md +++ b/docs/operations/bucket-heal-recovery.md @@ -1,22 +1,50 @@ # Bucket heal recovery and bucket recreation +**Use this when:** diagnosing incomplete administrator heal coverage, restoring a missing bucket directory, or replaying a bucket heal after restart. + Explicit administrator bucket heals bind the bucket name to its authoritative, non-nil incarnation UUID at admission. The same identity follows the task through queueing, retries, shutdown persistence, and startup replay. Deleting a bucket and creating another with the same name does not transfer the old task to the new bucket. +## Incomplete bucket discovery + +An administrator heal of the cluster or a selected pool/set enumerates bucket candidates from every selected disk, including directories below the ordinary namespace write quorum. Every selected disk must answer before discovery is complete. A pool/set task checks only its selected set; unavailable unrelated pools do not invalidate that scope. Candidate discovery grants no permission to recreate a bucket. Ordinary S3 and scanner listings retain their quorum rules. + +Incomplete discovery uses the existing bounded task retry budget. Once exhausted, the canonical outcome is `execution.state=aborted`, `execution.reason=untraversable`, and `coverage=partial`; the legacy summary is `stopped`. Object counters can remain zero because traversal never started. A genuinely empty, completely enumerated namespace still finishes successfully. Dry runs use the same coverage checks. + +Mutating cluster and pool/set tasks persist their resolved bucket scope only after complete discovery. Resumed checkpoints keep their original frozen scope, including a valid empty scope. A cluster task pins every bucket's incarnation before its first mutation and checkpoints its successful contiguous prefix. Checkpoints are batched into at most 32 intervals per traversal to bound inventory rewriting; an interrupted interval is replayed under the same generations. A fresh task is required to include buckets created after discovery. Cluster dry runs retain their observation-only behavior without requiring bucket incarnation metadata. + +## Restoring a missing bucket directory + +For a bucket that is still intended to exist but has lost its directory on one drive: + +1. Check that the peer drives and authoritative bucket metadata remain available. +2. Start an authenticated administrator heal with `recursive=true`. For the v3 API, use `POST /rustfs/admin/v3/heal/` for the cluster or `POST /rustfs/admin/v3/heal/` for one bucket. A deep scan (`scanMode=2`) additionally checks data integrity. Retain the returned `clientToken` and poll the same endpoint with that query parameter. +3. Inspect both execution and coverage in the terminal outcome. A readable object on a healthy peer alone does not establish that the missing copy was restored. +4. Verify recovered object contents and lengths against known hashes. In an isolated recovery test, stop the original healthy source and read through the repaired target to demonstrate independent recovery. Do not take a production source offline merely to inspect a heal result. + +Default administrator bucket healing can reconstruct missing directories when every selected disk is reachable, each selected set retains a bucket directory, and persisted metadata proves the current generation was committed and has not been retired. The lifecycle fence remains held through reconstruction. Missing, corrupt, uncommitted, retired, or mismatched authority fails closed. Dry runs leave directories unchanged. `recreate=true` remains an explicit recovery option; it is not permission to revive a bucket deleted through S3. Bucket incarnation checks reject stale tasks after deletion and same-name recreation. + +Legacy objects without independent storage commitments can be physically repaired while their canonical dispositions remain `unknown`. Do not infer a verified repair count from progress counters; see [heal terminal reports](../architecture/heal-terminal-reports.md). + ## Recovery records and stale tokens -Pending records use schema 3 in `.rustfs.sys/root-heal-.json` on the existing +Pending records use schema 4 in `.rustfs.sys/root-heal-.json` on the existing coordinator disk. A bucket record includes `bucket_incarnation_id`. Options, task ID, -and the remaining retry budget retain their existing recovery semantics. +and the remaining retry budget retain their existing recovery semantics. A cluster +record additionally carries its frozen bucket generations and completed prefix. +Cluster records with this scope have a 16 MiB limit; other intents and legacy +terminal markers retain the 64 KiB limit. Initial cluster snapshots reserve 1 MiB +for later progress and bounded object outcomes. Oversized inventories fail before repair. A missing bucket, changed incarnation, or legacy bucket record without a usable identity becomes a durable `Failed` terminal result containing `stale_bucket_incarnation`. The original token remains queryable under the existing terminal retention policy. Recovery never fills a legacy task with the identity of -the bucket currently using its name. Existing schema 1 cluster records and schema 2 -non-bucket records keep their recovery behavior. Unsupported, corrupt, and oversized +the bucket currently using its name. Legacy schema 1–3 cluster intents lack a frozen +generation inventory and require a fresh task after upgrade. Other supported legacy +intents and terminal reports remain readable. Unsupported, corrupt, and oversized records remain intact and receive a sibling `quarantined-root-heal-{intent|terminal}-.json` marker. The marker fences the original task ID while allowing an independent compensation heal with a fresh ID. @@ -42,15 +70,16 @@ missing or nil identities and unsigned bodies are rejected. Older nodes return `Unimplemented`; senders do not retry through an unfenced method. Upgrade all participating nodes before expecting these bucket heals to complete. Unscoped RPC bodies retain their prior canonical encoding. Older coordinators do not -understand schema 3 pending records. Finish or cancel pending administrator tasks +understand newer pending record schemas. Finish or cancel pending administrator tasks before downgrading a coordinator; do not rewrite the schema to force replay. The lifecycle fence remains mandatory even when `nolock` disables the optional object namespace lock. Deployments with namespace locking disabled cannot perform incarnation-bound bucket healing. -This contract covers explicit administrator **bucket** tasks. Cluster, prefix, -object, scanner, and replacement tasks retain their existing admission semantics. +The generation-bound repair contract covers explicit administrator **bucket** tasks +and the frozen buckets of mutating administrator cluster tasks. Prefix, object, +scanner, and replacement tasks retain their existing admission semantics. It does not introduce a new object-generation protocol or claim rejection of every already-dispatched native syscall after distributed lock loss; see the existing [heal concurrency model](../architecture/heal-concurrency-model.md). diff --git a/docs/operations/bucket-usage-metrics.md b/docs/operations/bucket-usage-metrics.md new file mode 100644 index 000000000..ed65d05a7 --- /dev/null +++ b/docs/operations/bucket-usage-metrics.md @@ -0,0 +1,122 @@ +# Bucket usage metrics + +Use these metrics to collect every bucket's logical usage through the existing +OpenTelemetry pipeline. RustFS reads the cached scanner usage snapshot once per +collection; exporting metrics does not enumerate objects or run an inline scan. + +## Export and query + +Configure the RustFS nodes with an OTLP/HTTP destination and a stable cluster ID: + +```bash +RUSTFS_OBS_ENDPOINT=http://otel-collector:4318 +RUSTFS_OBS_METRICS_EXPORT_ENABLED=true +OTEL_RESOURCE_ATTRIBUTES=rustfs.cluster.id=production-a +``` + +Append the cluster ID to any existing resource attributes. The shipped +[Collector configuration](../../.docker/observability/otel-collector-config.yaml) +receives OTLP and exposes Prometheus samples on port 8889; its +`resource_to_telemetry_conversion` promotes the cluster ID to +`rustfs_cluster_id`. Prometheus scrapes the Collector. See +[storage metrics](storage-metrics.md#configure-the-pipeline) for pipeline setup +and resource identity. + +The following existing gauges carry a `bucket` label: + +| Metric | Meaning | +|---|---| +| `rustfs_cluster_usage_buckets_total_bytes` | Logical scanner bytes across retained object versions, including noncurrent versions; excludes delete-marker bytes. | +| `rustfs_cluster_usage_buckets_objects_count` | Scanner object-entry/key count, separate from version and delete-marker counts. | + +Logical usage differs from physical drive usage, which includes erasure-coding +overhead and can differ because of compression or remote tiering. Object-entry +count can include keys with only noncurrent versions or delete markers, so it +need not equal the number of currently visible objects from `ListObjectsV2`. + +For monitoring, query all buckets in one cluster: + +```promql +max by (rustfs_cluster_id, job, bucket) ( + rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="production-a"} +) +``` + +```promql +max by (rustfs_cluster_id, job, bucket) ( + rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="production-a"} +) +``` + +Each reporting node exports the whole cluster's logical usage. Summing those +copies would multiply the result by the node count. Keep cluster identity when +grouping: independent clusters can share both a job and a bucket name. The +bundled Bucket Size and Bucket Objects panels use the selected **Storage +cluster** and deduplicate that cluster's reporting nodes. + +Nodes can briefly observe different snapshots. `max` is useful for dashboards +but can retain an older, larger value after deletes. For a consistent accounting +observation, select one reporting node for both bytes and object count: + +```promql +rustfs_cluster_usage_buckets_total_bytes{ + rustfs_cluster_id="production-a", service_instance_id="node1:9000" +} +``` + +Use the same selector with `rustfs_cluster_usage_buckets_objects_count`, and +check the promoted `service_instance_id` in your Collector output. These +floating-point, periodic gauges are usage observations, not an exact billing +ledger or a transactional view of writes and deletes. + +## Freshness and missing values + +Usage collection runs on `RUSTFS_METRICS_CLUSTER_INTERVAL_SEC` (60 seconds by +default), independently of the OTLP export interval +`RUSTFS_OBS_METER_INTERVAL` (30 seconds by default). The separate bucket +collector defaults to 300 seconds. Neither interval forces a scanner cycle. + +Check these gauges on the same reporting node as the bucket values: + +- `rustfs_cluster_usage_objects_since_last_update_seconds`: age of the + persisted usage snapshot when it was collected. A bucket cleanup can advance + the timestamp; it is not necessarily the age of a full object rescan. +- `rustfs_cluster_usage_objects_snapshot_converged`: `1` for a converged + snapshot, `0` when convergence is not confirmed, including observational or + unknown state. Complete observations can still supply usage during sustained writes. + +Before a complete snapshot covers the current bucket namespace, the usage +collector omits its values. A missing series is unknown; do not replace it with +zero for accounting. A bucket confirmed empty in a complete snapshot exports +zero bytes and zero objects. + +A temporary collection failure preserves the last valid export snapshot. +RustFS stops exporting that snapshot once collection stalls beyond three times +the larger collection/export interval. Deleted bucket series receive bounded +zero tombstones and then stop being exported after a valid updated snapshot. +The Collector can retain the last sample after export stops; the shipped +configuration uses a five-minute `metric_expiration`. Prometheus history also +retains past samples. + +If metrics are absent, first check OTLP connectivity and metrics export enablement, +then the scanner leader's `GET /rustfs/admin/v3/scanner/status` and the admin +usage snapshot. A shorter scrape interval cannot repair an incomplete scanner +snapshot. The scanner publication and cache path is shared with the Console's +usage display. + +## Regression checks + +The SDK test uses the production bucket collector and an actual OpenTelemetry +reader to verify two populated buckets, a confirmed-empty bucket, decreasing +usage, and removal from subsequent exports: + +```bash +RUSTUP_TOOLCHAIN=stable cargo test -p rustfs-obs actual_sdk_exports_bucket_size_and_object_count_as_current_gauges --lib +``` + +The PromQL fixture checks same-name buckets in separate clusters and multiple +reporting nodes. Run it from `.docker/observability/tests`: + +```bash +promtool test rules bucket-usage.test.yml +``` diff --git a/docs/operations/cluster-lifecycle-operations.md b/docs/operations/cluster-lifecycle-operations.md index f07056026..87f7c9ab9 100644 --- a/docs/operations/cluster-lifecycle-operations.md +++ b/docs/operations/cluster-lifecycle-operations.md @@ -44,6 +44,14 @@ Rules enforced by `DisksLayout::from_volumes`: with more than one argument every Startup order (`init_startup_storage_foundation` in `rustfs/src/startup_storage.rs`): parse endpoints, validate storage classes against every pool (`ECStore::validate_startup_storage_class`), enforce the unsupported-filesystem policy, initialize local disks, initialize lock clients, then load or create the format quorum and build the store. A storage-class error stops startup before any disk is touched. +### 2.1 Windows endpoints + +Multi-node and multi-pool layouts need URL endpoints, so on Windows the drive path goes after the port with forward slashes: `http://node{1...4}:9000/D:/rustfs/data{1...4}`. `Endpoint::get_file_path` (`crates/ecstore/src/layout/endpoint.rs`) turns the URL path `/D:/rustfs/data1` back into `D:/rustfs/data1` for every scheme; a URL path without a drive, such as `/export1`, stays a URL path. A drive root (`D:\` or `http://node1:9000/D:/`) is accepted in both forms. UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected at parse time: they are not local volumes, and parsing would read the share server as the endpoint host. + +Endpoint parsing then requires distinct physical disks (`validate_local_physical_disk_independence` in `crates/ecstore/src/layout/endpoints.rs`). On Windows the device identity is the volume mount point from `GetVolumePathNameW` (`crates/utils/src/os/windows.rs`), so `D:/rustfs/data1` through `D:/rustfs/data4` are one disk and startup fails with `must use distinct physical disks`. Give each endpoint its own volume (a drive letter or a volume mounted into an NTFS folder). For a single-machine test only, `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` skips the check; it gives no protection against losing that volume. + +A single local path (`rustfs D:\rustfs\data`) is a single-node single-drive deployment and can never gain a pool (§7). If the machine may expand later, start with URL endpoints and at least two drives per pool. + ## 3. Startup configuration | Variable | Effect | Boundary | diff --git a/docs/operations/delete-transaction-recovery.md b/docs/operations/delete-transaction-recovery.md index 175eb84f1..ea44576df 100644 --- a/docs/operations/delete-transaction-recovery.md +++ b/docs/operations/delete-transaction-recovery.md @@ -47,9 +47,46 @@ absence alone cannot certify it. Cross-transaction null-version supersession als predecessor evidence; timestamps alone cannot authorize it. Cleanup respects active snapshots and removes only validated transaction files, with directory removal performed nonrecursively. +A live object upgraded from 1.0.1 can still carry an old +`reserve-delete-data.` inside its data directory. A later certified +deletion may reclaim that regular-file reservation together with the payload +only in the directory owned by the current transaction. The old reservation +never authorizes deletion by itself. Foreign commit markers, malformed marker +names, symlinks, unknown files, live versions, and UUID-prefix child objects +remain protected. Snapshot release uses the same checked cleanup path. + +Ordinary heal applies the same boundary to orphan data directories. A UUID +directory name alone is not evidence that a directory contains disposable +shards: it can be a child object's key. Reclamation requires a complete drive +inventory, readable parent metadata that does not reference the directory, no +pending transaction records, and a leaf containing only recognized payload +files. The final mutation rechecks these conditions under the metadata lease, +including after snapshot deferral. It unlinks only the classified files and +removes the empty directory nonrecursively; strict mode synchronizes both the +leaf and its parent. Missing metadata or an unsupported inspection RPC leaves +the directory intact for later recovery. +The evidence check traverses empty pages until completion and owns its iterator +state for the request, so large multipart directories do not block cleanup and +completed or canceled probes do not consume shared discovery cursors. + ## Entry points -Unclean-shutdown erasure-set healing performs raw transaction recovery before +Every startup, including a clean restart, starts incremental raw discovery of +local transaction records. During normal operation, deletion registers a +bounded, coalesced exact-key retry hint before preparing metadata. Synchronous +completion clears only its own hint generation; an uncertain publication or +unfinished cleanup remains eligible for automatic replay without another +DELETE or administrator request. The on-disk journal and receipt remain the +only durable decision authority; the in-memory hints contain no commit decision. +Returning drives and queue overflow request discovery again, and a low-frequency +bounded reconciliation scan covers records written through remote participants. +Discovery advances by pages and retries failed sources separately, rather than +rescanning every healthy bucket on each maintenance tick. +Corrupt record contents remain discoverable by their physical key; exact-key +inspection rejects them without hiding later records. Unsafe paths, symlinks, +and structural filesystem errors still stop that discovery scope. + +Unclean-shutdown erasure-set healing also performs transaction recovery before ordinary object listing. Administrator bucket, set, prefix, and object heal use the same recovery core. Existing heal selection, lifecycle admission, namespace locking, and movement guards still apply. Prefix recovery settles only matching @@ -68,6 +105,20 @@ configured drive has no current metadata. This is a new authorized decision; it does not infer an old commit from missing causal evidence. Mixed or unrecognized historical layouts remain blocked. +A certified absent-target transaction can coexist with an unrelated old backup +in a different data directory. Automatic recovery finishes the certified +transaction's own cleanup, preserves the ambiguous backup, and retains receipts +until the whole key settles. A fresh unconditional DELETE can discard eligible +legacy residue alongside those receipts under the same checks above. A foreign +transaction marker alone never authorizes removing the backup. + +An isolated `xl.meta.bkp` is not enough to distinguish a committed delete from +an interrupted rollback. Automatic recovery does not guess which outcome was +intended. Preserve a copy of the original volume and use an authoritative +application backup or snapshot for recovery when no transaction decision can be +established. `remove` means explicitly discarding eligible residue; it cannot +restore ambiguous data or bypass versioning and Object Lock protections. + A missing specific-version DELETE first recovers hidden transactions and then reads the current version again. It must not acknowledge an unresolved hidden version as an idempotent no-op. @@ -81,6 +132,8 @@ causes the modern operation to fail before decision publication. Modern nodes also reject legacy forward transactional user-bucket deletes. A cluster still controlled by an older majority retains the older behavior. Do not downgrade while recovery records remain; old binaries cannot interpret the new protocol. +An upgraded initiator reports an unsupported participant as retryable S3 +`ServiceUnavailable` (503), including when the delete aggregates pool results. Keep unavailable drives and transaction directories intact. Bring the full set online, run a scoped heal in dry-run mode, inspect its recovery result, @@ -90,11 +143,37 @@ to be deleted and the bucket meets the restrictions above. Do not recursively remove transaction directories by hand to make a bucket appear empty. Recovery reports discovered, committed, rolled-back, reclaimed historical, -and blocked transaction counts. Unavailable participants return a retryable -quorum error; blocked evidence requires operator investigation. A successful +blocked, pending, and failed outcomes. Pending and failed counts can represent +an incomplete set or a failed discovery/inspection scope rather than a known +number of transactions. Bucket and prefix results +include a structured `deleteRecovery` status: `pending` means unavailable +participants or unfinished work, `blocked` means insufficient safe evidence, +and `failed` means corrupt transaction data. A local corruption or pending +transaction does not prevent other buckets or ordinary object healing from +running. The pass still reports incomplete recovery and does not publish a +completed recovery checkpoint. Cancellation, lifecycle/incarnation changes, +movement fences, invalid paths, and other set-level errors retain their +termination semantics. Exact-object recovery still returns a retryable quorum +error for unavailable participants; blocked evidence requires operator +investigation. A successful S3 delete response does not imply immediate physical cleanup while snapshots or returning participants still retain necessary proof. +If ordinary set healing overlaps unresolved transactions, its resume state +durably records that coverage needs refreshing. Pending retries preserve the +completed object-scan progress. Once recovery resolves, the set is scanned +again before reporting completion, including when a background rollback already +restored and finalized formerly hidden objects. This additional scan is required +on that recovery transition; it is not a periodic full-set heal. + +The upgrade CI retains its pinned rc.5 source and independently tests stable +1.0.1. The legacy-reservation case uses metadata and shards written by the +official 1.0.1 binary and injects an empty reservation while it is stopped; +this is a deterministic interruption fixture, not a hardware power-loss test. +Mixed-version stages verify that modern DELETE fails safely while older +participants are online. They do not promise continuous DELETE availability or +arbitrary downgrade while new recovery records exist. + ## Settlement deadline and durability `RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS` controls the settlement RPC deadline, diff --git a/docs/operations/durable-quota-reservation-v2.md b/docs/operations/durable-quota-reservation-v2.md index 82200dcd3..862ed9f70 100644 --- a/docs/operations/durable-quota-reservation-v2.md +++ b/docs/operations/durable-quota-reservation-v2.md @@ -1,5 +1,7 @@ # Durable quota reservation protocol v2 +**Use this when:** operating durable hard quotas, upgrading quota-enabled clusters, or diagnosing quota writes during a node outage. + Protocol v2 is an opt-in hard-quota protocol for distributed deployments. It keeps the quota limit authoritative in one allocator ledger while moving request reservations into sixteen independently locked shard ledgers. @@ -66,6 +68,43 @@ The default `BucketQuota::new` remains protocol v1. This keeps existing deployments compatible until an operator has completed the rollout and chosen the sharded protocol explicitly. +## Availability and fence protocol upgrades + +Both reservation protocols use the same write-availability rules. Setting a +durable quota still requires the existing whole-fleet capability proof. Once +configured, a quota write does not require that proof to remain live. It needs +the namespace locks, authoritative bucket configuration, quota ledger, and +object mutation fences to satisfy their respective quorums. A failed node +therefore does not stop quota writes when those quorums remain available. + +Object fences use the target erasure set's write quorum. Ledger, allocator, +and shard fencing retains the existing per-pool set checks. An unavailable +set that owns required quota state can still block a write even if its object +data set is healthy; a majority of nodes across the whole cluster is not a +substitute for these quorums. + +Participating disks must also agree with the coordinator's configured +pool/set/disk topology. The dedicated quota RPCs authenticate that topology +identity and reject a different layout, including a different pool order. +An offline node does not change the configured topology, so this check does +not require every node to respond. + +Quota mutations use the dedicated `AcquireQuotaMutationFence`, +`ReleaseQuotaMutationFence`, and `RenameDataWithQuotaFence` RPCs. A fenced +rename requires a valid token for that disk process and object. A restart +discards the token registry, and revocation waits for existing claims to +drain before returning success. Unsupported peers fail closed; clients never +fall back to ordinary rename or lease RPCs for a quota mutation. + +Upgrade every node before relying on degraded quota writes. During a rolling +upgrade, new coordinators cannot count old peers toward quota fence quorums, +so their quota writes can temporarily fail until enough upgraded disks are +available in every required set. Older coordinators retain their whole-fleet +proof requirement. The same limitation applies when rolling back: avoid +routing quota writes to a newer coordinator that lacks enough compatible +peers. The quota and ledger formats are unchanged, including the zero-quota +encoding that makes pre-reservation coordinators reject positive growth. + ## Performance boundary Normal v2 requests take a shared allocator read lock and one shard write lock; @@ -77,3 +116,69 @@ bound, not a best-effort token bucket. Unused credit is intentionally retained until the quota epoch changes or a future reconciliation operation returns it, trading some capacity utilization for a simple crash-safe upper bound. + +## Quotas saved before the reservation protocols + +A quota configuration without `reservation_protocol` (saved by a release +before protocol v1) is still admitted from scanner usage snapshots: its writes +need an authoritative or degraded usage value and fail closed with a +retryable `503` when neither exists. Decision D3 +([scanner-usage-authority-decision.md](../architecture/scanner-usage-authority-decision.md)) +moves such buckets onto the sharded durable protocol by re-saving the same +limit and type with `reservation_protocol: 2`; `created_at` is preserved. + +Two triggers exist and both go through the same guarded save: + +- Lazily, on the first quota read of such a bucket through the instance's + registered bucket metadata system (every PUT, COPY, multipart completion + and admin quota read). The read answers with the configuration it found and + schedules the re-save in the background; it is never blocked or failed by + the attempt. An attempt that does not settle the bucket is retried no + sooner than five minutes later. +- In bulk, with `POST /rustfs/admin/v3/quota-migration` + (`admin:SetBucketQuota`). The sweep visits every bucket and answers with a + per-bucket report: + +```json +{ + "target_reservation_protocol": 2, + "migrated": 1, "already_durable": 3, "no_quota": 10, "skipped": 0, "failed": 0, + "buckets": [ + {"bucket": "photos", "outcome": "migrated", "reservation_protocol": 2, "updated_at": "2026-10-19T02:14:07Z"}, + {"bucket": "logs", "outcome": "already_durable", "reservation_protocol": 1}, + {"bucket": "tmp", "outcome": "no_quota"} + ] +} +``` + +`outcome` is one of `migrated`, `already_durable`, `no_quota`, `skipped` +(`reason`: `unsupported_reservation_protocol`, `quota_revision_changed`, +`bucket_incarnation_changed`) or `failed` (`reason` carries the error). A +failed bucket does not stop the sweep; run the command again after fixing the +cause. + +The save keeps the admin save's preconditions: the whole-fleet capability +proof must be live (a mixed-version fleet fails the attempt rather than +exposing protocol v2 to a node that would fail closed on it), the bucket +incarnation must still match, and the quota revision must be the one that was +read. A quota edited in between is reported as `quota_revision_changed` and +left for the next attempt. A protocol this node does not understand is never +rewritten. Durable quotas (protocol v1 or v2) are not touched; the admin +default for a new quota remains protocol v1. + +Each attempt logs one `quota_reservation_protocol_migration` event +(`trigger` is `lazy` or `admin`, `result` is `migrated`, `noop`, `skipped` or +`failed`) and increments +`rustfs_quota_reservation_protocol_migrations_total` with the same labels. + +## Usage reporting for durable quotas + +For a bucket on protocol v1 or v2 the request-path quota check and the admin +quota endpoints read scanner usage for reporting only. When no usage is +available, `GET /rustfs/admin/v3/quota-stats/{bucket}` and +`POST /rustfs/admin/v3/quota-check/{bucket}` answer `200` with +`current_usage: null` (and no `remaining_quota` or `usage_percentage`) instead +of `503`, and object writes are admitted by the ledger alone. Each such +lookup logs a `quota_usage_report` event at debug level and increments +`rustfs_quota_usage_report_unavailable_total`. A bucket without +`reservation_protocol` keeps the previous behaviour until it is migrated. diff --git a/docs/operations/encrypted-etags.md b/docs/operations/encrypted-etags.md index 463ea42f6..854ee3740 100644 --- a/docs/operations/encrypted-etags.md +++ b/docs/operations/encrypted-etags.md @@ -24,6 +24,44 @@ Clients that require an ETag for these older or multipart objects must issue an authorized HeadObject request, including the SSE-C key when required. GET, HEAD, and conditional requests continue to use the object's stored ETag. +## Replication + +Bucket and site replication forward the stored ETag on `x-rustfs-source-etag` +(with its `x-minio-source-*` twin). A target that adopts it (RustFS, MinIO) +reports the fingerprint for the replica; a target that computes its own ETag +(AWS S3 and most compatible stores) reports the MD5 of the plaintext it +received, which never equals the fingerprint. The post-PUT replica ETag audit +therefore skips encrypted sources. Managed-SSE sources forward no checksum on +the replication PUT either, so a target that transforms their payload is not +detected. See +`docs/operations/replication-outbound-transport.md`. + +The same mismatch affects finding an existing replica on a target that mints +its own version ids (AWS S3, Wasabi), which a heal, MRF retry or +existing-object resync does before writing again: + +- A replica recorded in the source's target-version ledger is addressed by + that id, on a 404 and on an AWS-style 400 for the source id alike. For an + encrypted source the replica counts when its size matches the bytes + replication sent (plaintext for SSE-S3 / SSE-KMS, ciphertext for SSE-C) and + it was written no earlier than the source version. +- Without a ledger entry (for example a PUT whose response was lost), the + replica is located through `ListObjectVersions` by the same size and + write-time rule, after removing versions that other generations of the key + already claim. Only a single remaining match is adopted and recorded in the + ledger. Several matches are never resolved by picking one: convergence writes + the replica again, and a version purge is refused with the usual backoff. + The `null` version of an unversioned target is never matched this way, and + a target that refuses the listing keeps the earlier current-version check, + which writes the replica again. +- Plaintext sources keep exact ETag identity. A target that adopts the source + ETag still matches encrypted sources by ETag first. + +Two same-size generations of a key whose own ledgers were both lost can still +be confused, and a version purge with no ledger and no shape match reports the +replica as already gone. Delete a leftover replica on the target by hand if a +`replication-check` VersionFidelity report shows one. + ## Existing data and migration The upgrade does not rewrite existing object metadata or erase fingerprints diff --git a/docs/operations/object-lock-contention.md b/docs/operations/object-lock-contention.md index a8a10e2df..8ee8e131e 100644 --- a/docs/operations/object-lock-contention.md +++ b/docs/operations/object-lock-contention.md @@ -2,6 +2,14 @@ **Use this when:** an object GET returns `ServiceUnavailable` near the namespace lock acquisition deadline while PUT or Heal is active on the same key. +## Client-visible acquisition timeouts + +The canonical `ApiError` conversion in `rustfs/src/error.rs` retains HTTP 503 and the S3 `ServiceUnavailable` code for typed lock acquisition timeouts, and adds `Retry-After: 1`. The header is a minimum retry delay, not a receipt that the competing owner has released its lock. Use bounded exponential backoff with jitter and keep the client's overall request budget. + +S3 data-plane catalog publication-lock timeouts use the same conversion, typed lock-failure counter, and generic public message; catalog resource and peer details remain in the internal error source. Catalog failures represented only as strings do not inherit lock retry advice. + +This advice does not change the lock acquisition budget or shorten a competing writer's commit tail. It is not added to unrelated disk timeouts, quorum failures, or errors that merely contain timeout text. A 503 near five seconds is insufficient to attribute the request to a namespace lock: disk metadata operations also have a five-second default budget. Correlate the typed lock-failure counters and acquisition events below before changing contention policy. + ## Capture one acquisition and its competing owners Enable the existing diagnostic switch before starting the process and add the diagnostic target to the deployment's log filter: diff --git a/docs/operations/oidc-console-integration.md b/docs/operations/oidc-console-integration.md index 73b3e4b9f..3aa3b7637 100644 --- a/docs/operations/oidc-console-integration.md +++ b/docs/operations/oidc-console-integration.md @@ -16,7 +16,7 @@ Login flow: 3. The IdP redirects back to `/rustfs/admin/v3/oidc/callback/?code=...&state=...`. 4. RustFS exchanges the code at the token endpoint, sending `client_id` and `client_secret` in the request body (`client_secret_post`) together with the PKCE verifier. 5. RustFS validates the ID token signature (JWKS), issuer, audience, expiry, and nonce. -6. RustFS maps claim values to policy names and issues one-hour STS credentials to the Console. +6. RustFS maps claim values to policy names and issues STS credentials to the Console with the lifetime set by `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` (default 3600 seconds, one hour). In-flight `state` and PKCE verifiers are node-local: the authorize and callback requests must reach the same RustFS node. @@ -47,6 +47,7 @@ Process-level settings (environment only, never suffixed per provider): | --- | --- | | `RUSTFS_BROWSER_REDIRECT_URL` | Public browser origin used for callback generation, Console success redirects, and logout fallback. | | `RUSTFS_OUTBOUND_ALLOW_ORIGINS` | Exact `scheme://host[:port]` origins RustFS may contact for discovery, JWKS, and token requests when the IdP resolves to a private, loopback, or container-network address. See [outbound-connection-policy.md](outbound-connection-policy.md). | +| `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` | Credential lifetime in seconds for Console credentials issued by this callback, for every provider. Default `3600`. Values that are missing, not a positive integer, or zero keep the default; values above `604800` (7 days) are clamped to `604800` with a warning. The same value also sets how long the Console logout token stays valid, so RP-initiated logout keeps working for the whole session. Note that `RUSTFS_CONSOLE_AUTH_TIMEOUT` does not control this: it is the Console HTTP request timeout. | Named providers: to use provider id ``, suffix every provider env var with `_` (for example `RUSTFS_IDENTITY_OPENID_CLIENT_ID_keycloak`) and register the callback `/rustfs/admin/v3/oidc/callback/`. Suffix scanning is `parse_single_provider` in `crates/iam/src/oidc/config.rs`. @@ -137,6 +138,7 @@ curl -fsS "/.well-known/openid-configuration" | jq '{issuer, authorizati | Login succeeds, access denied | no claim value matches a policy name | Emit `groups` or `roles` as a flat array equal to policy names; check for `/` prefixes. | | Console redirects to an internal host | `RUSTFS_BROWSER_REDIRECT_URL` unset or proxy headers wrong | Set `RUSTFS_BROWSER_REDIRECT_URL` to the public origin. | | Invalid or expired OIDC state | callback reached a different node | Configure load-balancer session affinity for authorize and callback. | +| Console session expires after about an hour | callback issues credentials with the default lifetime | Set `RUSTFS_CONSOLE_OIDC_SESSION_DURATION` to the desired lifetime in seconds (max `604800`, 7 days) and restart RustFS. `RUSTFS_CONSOLE_AUTH_TIMEOUT` is a request timeout and does not extend the session. | ## Production checklist diff --git a/docs/operations/pool-metadata-recovery.md b/docs/operations/pool-metadata-recovery.md index d4281530a..cdd48cce6 100644 --- a/docs/operations/pool-metadata-recovery.md +++ b/docs/operations/pool-metadata-recovery.md @@ -51,6 +51,10 @@ An unreadable replica, lost fence, or conditional-write conflict leaves the orig - `rustfs_pool_metadata_blocks_total{reason}` and `rustfs_pool_metadata_recoveries_total` count block and recovery transitions. The original cause and phase remain attached to local typed errors; storage/RPC error numbers and on-disk formats are unchanged. - Node and cluster-write readiness include `pool_meta_write_blocked`. Waiting for the metadata save mutex is bounded to 100 ms and reports `pool_metadata_check_timeout`, not a persistent block. Cluster probes retain their existing cache and overall timeout behavior. Liveness and cluster-read quorum checks are unchanged. The authenticated node-local status and safe error fields are described in [S3 write failure diagnostics](s3-write-failure-diagnostics.md). +Pool metadata CAS explicitly selects `TailDrainedWithOwnerRepair`: it waits for the complete rename result and returns repair responsibility to its existing deployment and transaction owner when write quorum commits but a replica fails or signatures diverge. Cache invalidation and the committed cleanup handoff precede that error. Ordinary system metadata retains quorum-success semantics under `Quorum` and `TailDrained`; draining a tail does not require every disk to succeed. A repair-required completion does not enter rollback cleanup or wait for another temporary-data deletion. The committed bytes are not rolled back. Generic MRF and heal-channel requests cannot supply the deployment and transaction authority needed to repair system metadata; pool CAS retains the uncertain outcome and its existing recovery fence. Ordinary object repair keeps its bucket-incarnation fencing. + +An owning identity writer handles `SystemMetadataRepairRequired` before leaving its transaction: it can retry the exact identity up to three times, waiting one, two, then four seconds for disks to reconnect. Each attempt rereads the committed identity, requires byte-for-byte agreement with its intended payload and a usable ETag, and issues a conditional owner-observed write under the original startup or runtime authority. Quorum readback alone does not complete repair. This lets a transient disk failure during pool expansion converge before startup publishes the new topology. Changed or unreadable identity, a rejected repair CAS, lost authority, cancellation, and exhausted repair attempts retain the transaction-unknown gate. Other write errors and `pool.bin` transactions keep their existing recovery behavior. + If a block persists, inspect the first block and subsequent recovery phase, restore disk/peer readability, and verify every metadata and identity copy before restarting. Do not delete metadata to make readiness green. ## Disk replacement and metadata erasure diff --git a/docs/operations/put-cleanup-isolation.md b/docs/operations/put-cleanup-isolation.md new file mode 100644 index 000000000..a9adaca5b --- /dev/null +++ b/docs/operations/put-cleanup-isolation.md @@ -0,0 +1,195 @@ +# PUT and disk cleanup isolation + +Cleanup can use process-wide admission budgets and a dedicated Tokio runtime. +This is opt-in. It preserves cleanup receipts, eligible-data moves to trash, +PUT temporary workspace cleanup, and periodic physical trash removal. It does +not disable healing or skip cleanup. + +## Configuration + +| Variable | Default | Meaning when isolation is enabled | +| --- | --- | --- | +| `RUSTFS_CLEANUP_ISOLATE_ENABLE` | `false` | Enable admission and execution isolation on this node. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_MAX_PENDING` | `1024` | Maximum admitted PUT ownership groups, including surviving commit and cleanup owners. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_WORKERS` | `2` | Maximum active PUT cleanup continuations. Also controls existing deferred queue workers; does not determine OS thread count. | +| `RUSTFS_CLEANUP_ASYNC_THREADS` | up to `2` | Dedicated async worker threads, limited by available parallelism and an explicit CPU mask by default. Independent of logical job concurrency. | +| `RUSTFS_CLEANUP_DISK_MAX_PENDING` | `16 × disk workers` (`64`) | Maximum admitted local disk cleanup operations, including queued and running operations. Shared by local requests, incoming RPC cleanup and periodic GC steps. | +| `RUSTFS_CLEANUP_DISK_WORKERS` | blocking threads (`4`) | Maximum active disk cleanup operations within that budget. | +| `RUSTFS_CLEANUP_GC_WORKERS` | `1` | Maximum periodic disk scans producing GC steps. | +| `RUSTFS_CLEANUP_BLOCKING_THREADS` | `4` | Maximum additional blocking threads on the cleanup runtime. | +| `RUSTFS_CLEANUP_CPUS` | `1-2` on Linux; unbound elsewhere | CPU affinity for cleanup async and blocking threads. Override with a CPU list such as `4-5,8`, or `none` to disable affinity while keeping isolation. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_ENABLE` | `false` | Use the existing bounded deferred queue before coordinator execution. Isolation also works without this queue. | +| `RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_QUEUE_CAPACITY` | `1024` | Capacity of that existing queue; this is not the total admission budget. | + +Counts must be positive. PUT workers must not exceed PUT pending capacity; +GC workers must not exceed disk workers, which must not exceed disk pending +capacity. Async and blocking thread counts must not exceed 1024. CPU IDs must be supported by the affinity implementation and belong +to the initiating thread's allowed CPU set. Non-Linux hosts reject explicit CPU +masks. Configuration is read once at first use, including disk cleanup or GC; +changes require a restart. Invalid configuration rejects new cleanup admission. +PUT checks configuration before staging data. + +Isolation remains disabled by default. When enabling it on Linux, the default +mask requires logical CPU IDs 1 and 2 to be available inside the process/container +CPU set. On smaller or restricted CPU sets, configure permitted CPU IDs or `none` +before enabling isolation. An unavailable default mask is an error; it does not +silently select different CPUs. Configure each node separately. + +Do not combine isolation with cleanup counterfactuals +`RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP`, +`RUSTFS_PUT_RENAME_TAIL_CLEANUP_DEFER_HOLD_WORKER`, or +`RUSTFS_PUT_RENAME_TAIL_CLEANUP_ZERO_TARGET_TMP_DELETE_SKIP`. These combinations +are rejected because they deliberately suppress cleanup or stop queue progress. + +Do not enable `RUSTFS_PUT_RENAME_TAIL_CLEANUP_LANE_ENABLE` together with +`RUSTFS_CLEANUP_ISOLATE_ENABLE`. The legacy per-disk lane adds a second queue +inside the isolated coordinator budget, so a slow lane can hold coordinator +capacity while unrelated cleanup waits. This combination is rejected before +PUT admission; either scheduler can still be enabled on its own. + +## Ownership and execution + +PUT attempts admission before staging. Full admission returns `SlowDown`, so +clients should retry with backoff. Admission does not wait: COPY readers and +data-movement callers can already hold locks needed by admitted PUTs. No body +is ingested or tmp data staged by a rejected PUT. + +The request, detached commit owner, and cleanup continuations share one +reservation. An early ACK or cancelled waiter cannot release it while an owner +still needs cleanup. After the existing rename-convergence and guard-release +handoff, deferred workers, queue overflow and direct continuations all acquire +the same coordinator execution semaphore. Queue overflow cannot bypass that +limit. Awaited failure cleanup and no-tail tmp cleanup use the same budget. +Full-tail receipt persistence remains awaited at its existing handoff point; +old-data reclamation remains after the existing guard release. Incomplete +rollback still preserves staging for recovery. + +Each local disk operation has separate pending and active permits. The wrapper +below RPC authentication uses this gate for low-level `delete`, `delete_data_dir` +and old-data cleanup receipt `write_all`. Local PUT fan-out and incoming peer +RPCs therefore share the same node-local disk budget. Receipt classification +uses the internal receipt filename; ordinary writes retain their current path. +RPC schemas and authentication are unchanged. Enable isolation on every node +to isolate peer execution; settings and CPU masks are not transmitted by RPC. + +Coordinator and receiver permits are deliberately distinct: a coordinator can +wait for peer RPCs without occupying the slots those peers need to service disk +cleanup. Full disk admission returns an I/O `WouldBlock` error through the +existing disk error path. A mutation carrying an external publication or +namespace owner also refuses a full active budget instead of waiting while it +holds a lock. Existing cleanup error handling and receipt recovery still apply. + +Ordinary snapshot-lease release stays on the caller and only updates the in-memory +registry. It does not allocate a cleanup task or wait for the cleanup scheduler. +Invalid cleanup configuration also must not strand a reader token: release still +updates the registry, while any deferred reclamation reports the initialization +error and keeps both its intent and data. It does not fall back to unbudgeted I/O. +If the last release triggers deferred old-data +reclamation, that reclamation tries the same pending and active disk permits. +Only this case dispatches an owned task to the cleanup runtime. +Saturation leaves the token released and the cleanup intent/receipt available +for retry; it does not strand a live reader token or bypass the budget. Quota +fence release keeps its original receiver lifecycle and does not acquire disk +execution permits, since it can wait for mutations that already own them. + +An admitted request waits for disk execution capacity in the caller's existing +future. Cancellation or timeout discards this unstarted operation and releases +pending capacity; it cannot later start deleting a reused path. After execution +capacity is acquired, the worker also checks whether its result receiver closed +before its first poll. A job cancelled while waiting for the runtime to schedule +it is discarded when the worker polls it, releasing its permits without starting +I/O. Once the worker starts the operation, the owned job retains its permits and +disk reference until completion. Cancelling or timing out its waiter does not +free disk capacity while that started job still runs. +This also retains the disk's mount ownership during detached execution. Cleanup +waiter timeouts include queue delay and do not by themselves mark a disk faulty; +the existing active disk-health probe retains its timeout policy. + +Periodic trash and stale-tmp scans have a separate scan limit. Each scan acquires +the shared disk budget to open its directory and then for each entry, releasing +it between entries. The steps borrow the scan's iterator and paths and run in +the same owned task, without a spawn/join or path clones per entry. A scan cannot occupy a disk execution slot while waiting +for another scan slot. GC waits for admission without holding object locks; +request admission is bounded without creating waiting producer tasks. A single +trash subtree is still removed as one operation, so a very large subtree can +hold a slot for its entire removal. Queued scans retain the original mount +handle even when the corresponding disk object is dropped. +Missed periodic ticks are skipped so a slow scan does not cause a burst of +catch-up scans. + +Cleanup runs on `rustfs-cleanup` threads. Nested Tokio tasks, Tokio filesystem +work and ordinary `spawn_blocking` use that runtime. Explicit fsync dispatch +also stays on its blocking pool. File/directory group-commit keys separate +cleanup and foreground producers, preventing batches from moving into the +other class's runtime. Fsync durability settings remain unchanged. + +The blocking limit bounds simultaneous blocking execution, not Tokio's internal +queue or kernel I/O. Disk admission bounds selected operations, not bytes, +syscall count or device IOPS. Existing synchronous path probes and disk-full +delete fallbacks can still run on the dedicated async workers, under the disk +operation budget. All classes still share the filesystem, journal, +page cache and devices with foreground work. + +If CPU binding fails at thread startup, new admissions fail and the failure is +logged; already-admitted cleanup can finish. Affinity does not reserve exclusive +cores, move foreground threads elsewhere, change cgroups, or control kernel +workers and interrupt affinity. + +## Scope and recovery + +The limits are per process, not cluster-wide quotas. This is not a durable work +queue. Existing cleanup receipts and tmp/trash recovery retain their role after +process exit. The option does not bound all storage mutations: metadata/version +DELETE transactions, heal/scanner coordination and +multipart orchestration retain their scheduling. Calls from those paths that +reach the selected disk wrapper entries do share the disk budget; internal +LocalDisk operations that bypass those entries do not. + +## Measurement and tuning + +On Linux, `cargo test -p rustfs-ecstore --test cleanup_isolation_test` exercises +the public disk facade, actual thread affinity, receipt/delete execution and +periodic physical trash removal with temporary disks. It does not benchmark +throughput. + +With PUT stage metrics enabled, observe: + +- `rustfs_cleanup_admitted_puts` and `rustfs_cleanup_admission_rejected_total`; +- `rustfs_cleanup_execution_wait_seconds`; +- `rustfs_cleanup_disk_pending`, `rustfs_cleanup_disk_active`, and + `rustfs_cleanup_disk_admission_rejected_total`; +- `rustfs_cleanup_disk_execution_wait_seconds` and `rustfs_cleanup_gc_active`; +- existing deferred queue depth, age, overflow, completion and service time. + +Compare isolation disabled, budget/pool isolation without CPU affinity, and +isolation with affinity. Keep durability, object sizes, overwrite ratio, +concurrency and cluster placement fixed. Measure PUT/GET throughput and p95/p99 +latency, rejections, queue age, receipt/trash growth, per-disk I/O latency, CPU +usage and context switches. Include a long steady-state interval covering +physical GC, and verify backlog drains afterward. + +Start with small worker and blocking-thread limits, then increase only when +cleanup cannot keep up without harming foreground latency. Too few workers or +CPUs can increase rejections and lower PUT throughput. Larger queues do not +increase sustainable device throughput. Affinity helps only when CPU scheduling +contention matters to the measured workload; no throughput gain is guaranteed. + +The defaults protect a node; they are not a claim of optimal production throughput. +The default disk queue permits sixteen service waves, rather than 256 waves with +1024 pending operations and four workers. For illustration, at a uniform 100 ms +per operation, these bounds correspond to roughly 1.6 s versus 25.6 s to drain a +full queue with no new arrivals. Real service times, stalls and disk fan-out vary; +there is no queue latency guarantee. The ordinary disk waiter timeout defaults to +30 s and does not cancel already-admitted work. + +Size pending capacity for measured arrival bursts and local/peer fan-out, not +only for PUT request concurrency. For example, two coordinator jobs each touching +16 local disks can submit 32 disk operations before incoming peer requests and +GC are counted. Sixty-four slots need not absorb every distributed burst. Increase +the explicit limit when measured rejection rates require it, while keeping queue +age comfortably below waiter deadlines. On many-disk nodes, measure whether four +active operations underutilize devices before increasing disk/blocking workers. +The 1024 PUT admission limit includes uploading requests and detached owners; +it is an object-count bound, not a memory or byte budget. The older deferred queue +remains optional and disabled by default; its overflow path cannot bypass the +new coordinator or disk limits. No unbounded foreground fallback is introduced. diff --git a/docs/operations/replacement-generation-recovery.md b/docs/operations/replacement-generation-recovery.md index f3a93f108..ef92a0da7 100644 --- a/docs/operations/replacement-generation-recovery.md +++ b/docs/operations/replacement-generation-recovery.md @@ -28,6 +28,78 @@ markers can be cleared. Handoff authorities, migration receipts, original legacy records, and completion proofs are retained for diagnosis; automatic retention does not collect a referenced authority. +## Retry budget + +`max_retries` counts rebuild attempts only. A rebuild attempt starts after the +target is formatted and the generation is `rebuilding`; a failure in that phase +consumes one attempt, and a pass that ends with failed or skipped versions +consumes one attempt when the target is ready. Nothing else consumes it: + +- a format fence rejection or any other failure before `format.json` exists; +- a target that is unmounted, restarting, or otherwise not ready; +- a dangling-delete grace wait; +- a mount change and the handoff it causes, including the handoff preparation + and marker transfer; a successor inherits its predecessor's attempt count + unchanged; +- a generation or ownership conflict found while resolving the durable record. + +Each of those keeps the durable error message for status reporting. Only the +handoff lineage bound (32 links) limits repeated interruptions. + +## Retired generations and blocked entries + +A generation that is `abandoned` with a committed handoff, or that was retired +by an approved legacy migration, has transferred its responsibility. It no +longer owns its set: a fresh generation for other target slots can be admitted +while the retired record is retained for diagnosis. Once its successor is +`verified`, `cleanup_pending`, or has published a completion proof, the retired +record is a retention candidate; GC never collects a predecessor whose +successor has not reached that point, and never collects an uncommitted one. +An `abandoned` record without a handoff keeps reserving its target slots until +every slot's `healing.bin` names another generation of the same set. + +Discovery blocks one generation at a time, never a whole set. A record whose +durable copies diverge at the same revision, that cannot be loaded, that binds +to a different set than the disk it was found on, or that cannot be resolved is +recorded as blocked with its reason and the time it was first observed. Every +discovery pass re-reads the record; a block clears as soon as the record merges +and resolves cleanly, and is refreshed otherwise. Sibling generations of the +same set and all other sets keep recovering. Fresh admission for a set is +deferred only while a blocked generation of that set still reserves its +targets. The heal manager exposes the current blocked entries, with reasons, +to the admin status surface. + +## Blocked checkpoints + +`_ahm_checkpoint.blocked` isolates a checkpoint snapshot that the reader +could not verify. The marker is a small JSON record with the reason, the time +it was written, and the earliest time of the next revalidation; older binaries +wrote the literal `blocked` and treat any content as blocked. The current +reader revalidates a blocked snapshot at most once per hour, at the moment a +resume selector or executor consults the marker. If the snapshot verifies with +the current reader (for example, a field order the previous reader could not +digest), the marker is removed and `unblocked_by_revalidation` is logged. +Tampered or genuinely corrupt snapshots stay blocked and the marker's next +revalidation time is advanced. A marker whose snapshot has been removed is +cleared on the next revalidation. Downgrading to an older binary keeps every +marker in place and never clears one. + +## Root heal records + +Administrator root heal records are inventoried one record at a time. A record +with two durable owners, an unreadable quarantine marker, or a quarantine +marker whose source changed is reported as conflicting with its reason; the +remaining accepted responsibilities are still replayed. A conflicting task ID +is refused for new admission, and its status query reports `failed` with the +same reason until an operator repairs the record. + +## Ambiguous legacy delete residue + +Delete recovery that finds a legacy residue it cannot classify no longer +reports request-rate pressure. The heal returns a dedicated storage error that +is not retried automatically; the object remains readable and the residue is +retained for operator inspection. + ## Schema 5/6 maintenance migration A legacy binary does not take the new execution lock. Ordinary startup therefore diff --git a/docs/operations/replication-outbound-transport.md b/docs/operations/replication-outbound-transport.md index 125492735..b99b4bd24 100644 --- a/docs/operations/replication-outbound-transport.md +++ b/docs/operations/replication-outbound-transport.md @@ -9,7 +9,7 @@ - For a single-part object, the checksum the source object was uploaded with, forwarded as its `x-amz-checksum-` header (the value the source verified on upload). A multipart replica is rebuilt through CreateMultipartUpload/UploadPart and carries no object-level checksum header. Managed-SSE objects forward none. - On a PUT that carries Object Lock parameters and no forwarded checksum: `Content-MD5` derived from the source ETag, or an SDK CRC32 checksum when the ETag is not the MD5 of the wire bytes (rustfs#7082). - The source ETag, mtime and version id on `x-rustfs-source-*` headers (with `x-minio-source-*` twins), and the Object Lock mode, retain-until date and legal hold of the source version when present. -- After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED. +- After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s and the source is unencrypted; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED. A managed-SSE (SSE-S3 / SSE-KMS) or SSE-C source is not audited this way: its stored ETag is a keyed fingerprint (new writes, see `docs/operations/encrypted-etags.md`) or a legacy plaintext MD5, and the target re-encrypts what it receives, so the two ETags are never comparable. Managed-SSE sources also forward no checksum (above), so a target that transforms their payload is not detected; an SSE-C ciphertext passthrough keeps its HEAD-back evidence audit. ## Target classes and their known requirements @@ -18,7 +18,8 @@ | Rejects or mis-stores `aws-chunked` bodies (SeaweedFS 3.97) | Handled by the plain-payload default above. | Outbound target matrix, `RejectAwsChunked` mode | | Requires `Content-MD5` or `x-amz-checksum-*` on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) | Satisfied: a locked single PUT carries `Content-MD5` derived from the source ETag (plaintext objects whose ETag is the MD5 of the wire bytes) or an SDK CRC32 checksum (multipart-layout ETags, managed SSE, SSE-C passthrough — this one is an `aws-chunked` trailer, so a target that also rejects that framing cannot take such objects). Releases before this fix (`1.0.0-rc.5`) need `RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true` as a workaround. | Outbound target matrix, `RequireChecksumWithObjectLock` mode | | Stores `x-amz-checksum-*` from a PutObject and returns it on `HEAD ?ChecksumMode=ENABLED` (AWS S3, Wasabi, RustFS) | Satisfied for single-part objects: the replica answers with the source's checksum. Before this fix (`1.0.0-rc.5`) the checksum left the source as `x-amz-meta-` user metadata and no replica carried it (rustfs/backlog#2340). | Outbound target matrix, `Checksummed` shape | -| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands, and later mutations converge through the target-version ledger: the id the target assigned is recorded on the source version (internal key `replication-target-version-`) and version deletes, tag and Object Lock updates address it (tag/retention/legal-hold through the metadata-only APIs, never a re-PUT). A replica written before the ledger existed is located by exact key + ETag, minus the candidates other generations of the key already claim; an ambiguous remainder is refused with a one-hour backoff (`replication_purge_replica_unresolved`). `NoSuchVersion` on a version-addressed DELETE counts as purged. See rustfs/backlog#2340, rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity). | `replication-check`, outbound target matrix, `MintOwnVersionIds` mode (also models Wasabi's 404 `NoSuchVersion` on unknown ids) | +| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands, and later mutations converge through the target-version ledger: the id the target assigned is recorded on the source version (internal key `replication-target-version-`) and version deletes, tag and Object Lock updates address it (tag/retention/legal-hold through the metadata-only APIs, never a re-PUT). A replica the ledger does not name (written before the ledger existed, or a PUT whose response was lost) is located by exact key + ETag, minus the candidates other generations of the key already claim; an ambiguous remainder is refused with a one-hour backoff (`replication_purge_replica_unresolved`). An encrypted source whose ETag the target does not report is matched by size and write time instead, and only a unique match is adopted (`docs/operations/encrypted-etags.md`). `NoSuchVersion` on a version-addressed DELETE counts as purged. See rustfs/backlog#2340, rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity). | `replication-check`, outbound target matrix, `MintOwnVersionIds` mode (also models Wasabi's 404 `NoSuchVersion` on unknown ids); `matrix_mint_own_version_ids_encrypted_redrive_does_not_duplicate` for SSE-S3 on an ETag-computing target | +| Ignores `x-rustfs-source-etag` / `x-minio-source-etag` and reports the MD5 of the received plaintext as the ETag (AWS S3, generic S3 services) | Plaintext single-part replicas are verified against the source ETag. Encrypted sources skip the comparison (see above). In `1.0.2`, the first release with encrypted-ETag fingerprints, every single-part SSE-S3 object (and SSE-KMS where the target does not echo `aws:kms`) failed with `replica etag mismatch` and was retried indefinitely; the only workaround there is `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`, which disables the check for every object. RustFS and MinIO targets adopt the source ETag and were not affected. | Outbound target matrix, `ComputesOwnEtags` mode, `SseS3` shape | | Returns an ETag that is not the content MD5 without announcing SSE | Every single-part object fails ETag verification. Set `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`. | Replication status FAILED with `replica etag mismatch` | ## Environment knobs diff --git a/docs/operations/rolling-restart.md b/docs/operations/rolling-restart.md index a3df35e01..d459afa46 100644 --- a/docs/operations/rolling-restart.md +++ b/docs/operations/rolling-restart.md @@ -2,7 +2,7 @@ **Use this when:** restarting or upgrading nodes of an erasure-coded multi-node cluster, bringing a cluster back after several nodes were down at once, or interpreting `503` degraded-mode responses during startup. -**Source of truth:** `crates/config/src/constants/health.rs` (`DEFAULT_STARTUP_READINESS_MAX_WAIT_SECS`), `rustfs/src/server/readiness.rs` (readiness responses and `Retry-After`), the `iam_bootstrap_retry_failed` log event. +**Source of truth:** `crates/config/src/constants/health.rs` (`DEFAULT_STARTUP_READINESS_MAX_WAIT_SECS`), `rustfs/src/server/readiness.rs` (readiness responses and `Retry-After`), `crates/config/src/constants/drive.rs` (remote-drive recovery thresholds), `helm/rustfs/values.yaml` (`minReadySeconds`), `rustfs/src/server/readiness/peer_readmission.rs` (startup peer re-admission gate), the `iam_bootstrap_retry_failed` and `startup_peer_readmission` log events. Upgrading the binary or container image does not change the on-disk data format unless an explicitly enabled feature documents a version floor. Replacing the executable and restarting does not run a migration step on startup. @@ -19,7 +19,7 @@ Each row is a feature whose activation makes older binaries unable to read what ## TL;DR -- Rolling restart (no downtime): restart one node at a time and wait for the restarted node to report `200` on `/health/ready` before touching the next one. The remaining nodes keep serving traffic. +- Rolling restart (no downtime): restart one node at a time, wait for the restarted node to report `200` on `/health/ready`, and then wait until the other nodes no longer list its drives under `details.storage.unavailableDrives` before touching the next one. The remaining nodes keep serving traffic. - Sequential cold start (several nodes down): nodes started before the cluster has quorum come up in degraded mode. The process stays alive, answers `503` with the blocking reason, and recovers automatically as soon as enough peers are online. Do not restart-loop them; keep starting the remaining nodes. ## Why a single node cannot serve alone @@ -39,9 +39,35 @@ For each node, in any order, one at a time: curl -fsS http://:9000/health/ready ``` -3. Only then move on to the next node. +3. Wait until every other node has re-admitted it. A distributed node holds its startup readiness until a strict majority of the peers in each of its erasure sets count its drives online (see [startup peer re-admission](#startup-peer-re-admission)), but that gate fails open, so check the remaining nodes before moving on. A node's own `/health/ready` says that it can reach the cluster, not that the cluster can reach it again. On each of the other nodes, `details.storage.unavailableDrives` must no longer list the restarted node's drives. `hostOnline: false` means that node cannot dial the restarted peer yet; `runtimeState` of `offline` or `returning` means the dial works and the drive is still passing its recovery probes: -While one node is down, the rest of the cluster keeps quorum and serves all traffic. Taking a second node down before the first is back can cost some erasure sets their write or even read quorum; that is the situation to avoid. + ```bash + curl -s http://:9000/health/ready | jq '.details.storage.unavailableDrives' + ``` + +4. Only then move on to the next node. + +While one node is down, the rest of the cluster keeps quorum and serves all traffic. Taking a second node down before the survivors have re-admitted the first can cost some erasure sets their write or even read quorum; that is the situation to avoid (rustfs/rustfs#8669). + +### Why re-admission lags local readiness + +A survivor dials a peer by its configured host name on every reconnect and keeps no address cache of its own. After the peer's connections drop, its remote drives become `suspect` on the first failed RPC and `offline` on the second (`RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD`, default `2`). The survivor then probes the peer every `RUSTFS_DRIVE_RETURNING_PROBE_INTERVAL_SECS` (default `2`) and needs `RUSTFS_DRIVE_RETURNING_SUCCESS_THRESHOLD` (default `3`) consecutive successes before the drive counts as `online` again, so re-admission trails the first successful dial by about four to six seconds even on a perfect network. + +On Kubernetes the dial itself is the slow part. A replaced pod gets a new IP, but resolvers keep answering with the previous one until the record's TTL expires: kube-dns serves 30-second TTLs, CoreDNS five seconds by default, and NodeLocal DNSCache honours either. Packets to the old IP are usually dropped rather than refused, so every dial runs into the 3-second connect timeout (`RUSTFS_INTERNODE_CONNECT_TIMEOUT_SECS`) and the 1-second budget of the readiness ping (`RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS`) logs `online check timed out while dialing`. Until the stale record expires, survivors count the replaced pod as unreachable even though it already reports Ready. The Helm chart therefore sets `minReadySeconds` (default `45`) on each StatefulSet so the rolling update does not stop the next pod inside that window. Size it to your DNS TTL plus about ten seconds, or set it to `0` when an external procedure performs step 3 itself. + +### Startup peer re-admission + +After a distributed node reaches local readiness, it asks the peers in every erasure set it serves how they currently see its drives. It uses the `PeerDriveAdmission` internode RPC, and each peer answers with the same predicate its own `/health/ready` uses for `unavailableDrives`. Startup readiness is published once a strict majority of each set's peers count all of this node's drives in that set as online. In a three-node set that means both peers. While the node waits, `/health/ready` stays `503` with `startup_finalization_pending` and the log shows `event="startup_peer_readmission" state="waiting"`. + +The wait always ends in publication: + +- `result="admitted"`: the peers re-admitted the node. +- `result="gave_up" reason="deadline_elapsed"`: the startup budget `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` ran out. The gate shares that budget with local readiness. +- `result="gave_up" reason="peers_unreachable"`: for 15 seconds the node could not reach enough peers to make the majority possible. A dead peer does not hold a rollout for the whole budget. +- `result="gave_up" reason="peers_cannot_report"`: too many peers run an older release without the RPC, which is normal for the first nodes of a rolling upgrade. +- `result="skipped"`: the gate is disabled, the deployment is not distributed, or no erasure set shares drives with a peer. + +The gate covers the delay inside RustFS without relying on the orchestrator. The chart's `minReadySeconds` stays useful as a second margin and for older releases. Set `RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false` to return to local-only startup readiness. ## Sequential cold start (multiple nodes down) @@ -68,6 +94,9 @@ When the whole cluster (or several nodes) went down and nodes are brought back o | Variable | Default | Effect | | --- | --- | --- | | `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` | `120` (`DEFAULT_STARTUP_READINESS_MAX_WAIT_SECS`) | How long startup waits for full readiness before continuing in degraded mode with background recovery. Raising it delays the listener during genuinely slow starts; lowering it surfaces degraded mode sooner. Recovery retries continue regardless of this limit. | +| `RUSTFS_STARTUP_PEER_READMISSION_ENABLE` | `true` (`DEFAULT_STARTUP_PEER_READMISSION_ENABLE`) | Hold startup readiness until a strict majority of each erasure set's peers count this node's drives online. The wait is bounded by `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` and fails open. Set `false` to publish on this node's own view only. | +| `RUSTFS_DRIVE_RETURNING_PROBE_INTERVAL_SECS` | `2` (`DEFAULT_DRIVE_RETURNING_PROBE_INTERVAL_SECS`) | Interval between recovery probes to a remote drive that is `offline` or `returning`. | +| `RUSTFS_DRIVE_RETURNING_SUCCESS_THRESHOLD` | `3` (`DEFAULT_DRIVE_RETURNING_SUCCESS_THRESHOLD`) | Consecutive successful probes before an `offline` remote drive is `online` again. Together with the interval this bounds how long survivors keep a restarted peer out after they can dial it. | ## What is not normal diff --git a/docs/operations/s3-tables-maintenance-scheduler.md b/docs/operations/s3-tables-maintenance-scheduler.md new file mode 100644 index 000000000..fe2db9cf8 --- /dev/null +++ b/docs/operations/s3-tables-maintenance-scheduler.md @@ -0,0 +1,111 @@ +# Table Maintenance Planning and Periodic Queueing + +**Use this when:** planning table maintenance, enabling automatic job admission, or recovering a stalled queue. + +The server can periodically **queue** maintenance jobs for explicitly selected table buckets. It does not run workers, delete objects, expire snapshots, or compact data automatically. Workers remain an explicit operator action. Both object-backed and durable-strong catalogs use the existing maintenance configuration and job format. + +## Read-only Metadata Maintenance Planning + +Both backings accept authenticated planning requests at: + +```text +POST /iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/metadata +``` + +For example: + +```json +{ + "retain-recent-metadata-files": 2, + "snapshot-expiration": { + "min-snapshots-to-keep": 1, + "max-snapshot-age-ms": 86400000 + }, + "compaction": { + "target-file-size-bytes": 536870912, + "small-file-threshold-bytes": 67108864, + "min-input-files": 2, + "max-rewrite-bytes-per-job": 1073741824 + } +} +``` + +The optional snapshot-expiration and compaction fields request plans, not commits. An empty JSON object requests ordinary metadata planning. A nonzero retention value overrides the effective maintenance configuration for this request; omission or zero uses the effective configured retention. + +In durable-strong mode, successful planning returns HTTP 200 with a completed `DRY_RUN` job and persists its named, current, and latest reports with audit events. Read-only means that table data, metadata pointers, and catalog generations are unchanged; the request still writes maintenance report sidecars and requires the table-scoped `admin:RunTableMaintenance` permission. Inspect the returned plan and any manual-review state before taking further action. + +This direct endpoint rejects `delete=true`, `commit-snapshot-expiration=true`, or `commit-compaction=true` with HTTP 400 in durable-strong mode, without publishing a job or changing table objects. The existing object-backed mutation behavior is unchanged. These restrictions do not change the separate explicit maintenance worker or quarantine APIs. + +A queued, running, or failed job, active quarantine, or inconsistent current/latest aliases or their named report returns HTTP 409 instead of replacing the existing job or discarding its retry budget. Inspect that state before retrying or taking explicit worker/recovery action. A failed publication that saved only a completed named dry-run report leaves harmless history: no alias advertises a running claim, and replanning is safe because there were no table-object effects. Planning revalidates table identity, metadata pointer, configuration, and the prior report before publication; concurrent changes also conflict. It does not introduce automatic workers, snapshot-expiration commits, or compaction commits on durable-strong backing. + +## Enablement + +The standalone server reads these variables at startup; the embedded server does not start this scheduler. + +| Variable | Default | Accepted value | +|---|---|---| +| `RUSTFS_TABLE_MAINTENANCE_SCHEDULER_ENABLED` | `false` | `true` or `false`; invalid values fail startup | +| `RUSTFS_TABLE_MAINTENANCE_SCHEDULER_BUCKETS` | None | Required when enabled: comma-separated, 1–32 distinct valid bucket names | +| `RUSTFS_TABLE_MAINTENANCE_SCHEDULER_INTERVAL_SECONDS` | `300` | Integer from 30 to 86400 | + +Example: + +```sh +export RUSTFS_TABLE_MAINTENANCE_SCHEDULER_ENABLED=true +export RUSTFS_TABLE_MAINTENANCE_SCHEDULER_BUCKETS=analytics-tables +export RUSTFS_TABLE_MAINTENANCE_SCHEDULER_INTERVAL_SECONDS=300 +``` + +The allowlist does not create or enable table buckets. Each bucket must already have an active catalog. The effective bucket default or table override must set `background-enabled=true` and `worker-paused=false`. Deletion remains disabled by default; queueing honors the existing `delete-enabled` setting but never executes the queued operation. + +## Admission And Limits + +- The first pass is delayed by the configured interval plus 0–10% jitter. Each completed pass waits again; missed ticks do not produce catch-up bursts. +- Each process runs one owned pass at a time, visiting at most eight distinct tables and 64 discovery steps. Bucket turns rotate and namespace/table cursors persist between passes. Restart resumes discovery from the beginning; persisted jobs still prevent duplicate admission. +- A 30-second admission deadline stops starting further store operations. An in-flight store call drains without cancellation or overlap. This is **not** a hard wall-clock, memory, or per-table object-I/O limit: existing planning and history inspection costs still depend on the table. +- The scheduler only admits a first job or a new job after a successful previous job and the configured cooldown. It does not reset failed retry budgets, renew abandoned queues, recover expired workers, or clear quarantine. +- Before publishing, the store rechecks table identity, enabled/unpaused configuration, the observed current job, and complete current/latest report publication under its existing locks. A concurrent manual run, rename, or identity change conflicts rather than overwriting that state. + +Use the same interval on enabled nodes. The existing distributed catalog locks coordinate claims; this is not a cluster-wide throughput quota. A busy or corrupt table is reported as a degraded pass and does not remove other buckets from discovery. + +## Inspection And Recovery + +Inspect the existing endpoint: + +```text +GET /iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler +``` + +Use its current job, quarantine status, and recommended actions. An eligible queue can be consumed through the authenticated `POST .../maintenance/worker/run` endpoint. If a job failed, a queue/worker lease expired, or a previous manual invocation left a Disabled/Paused control report, resolve the underlying condition and explicitly invoke `POST .../maintenance/scheduler/run` or the recommended worker/quarantine operation. Automatic admission deliberately leaves these states unchanged. + +A partial current/latest publication requires recovery before periodic admission; repeated ticks do not repair or discard it. A conditional worker replay can repair the narrowly defined terminal-publication case below. Review retained reports for other cases; do not delete catalog sidecars directly. + +## Conditional Worker Invocation + +An operator can bind a worker invocation to the job and attempt observed through the scheduler or job endpoint: + +```text +POST /iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run +``` + +```json +{ + "worker-id": "maintenance-worker-a", + "expected-job-id": "observed-job-id", + "expected-attempt": 0 +} +``` + +Send both expected fields or neither. Missing one, explicit `null`, invalid types, and unknown fields are rejected. Omitting both retains the existing manual worker behavior; an empty JSON object `{}` still uses the default worker ID. Authentication and the table-scoped `admin:RunTableMaintenance` permission are required in either mode. + +The conditional mode consumes an unexpired `QUEUED` job at attempt zero, or retries a `FAILED` job after its persisted `next-retry-after` when its persisted attempt budget allows. It keeps the same job ID, operation, original candidate lists, cleanup watermark, and total attempt budget. The first attempt is allowed even when that budget is zero; zero disables retries. Subsequent failed attempts calculate their delay from the current valid backoff configuration without replacing the saved budget. + +Stale job/attempt, catalog identity or pointer/generation changes, expired worker/queue leases, active quarantine, and changed recorded maintenance policy conflict without claiming another attempt. The recorded policy includes retention, quarantine settings, and configuration source; deletion must still be enabled for a delete job. The job format does not store a complete historical configuration snapshot. This endpoint does not promise detection of changes to fields that were never recorded. + +If the response is lost, repeat the same worker ID, expected job ID, and expected attempt. When that invocation already claimed the current attempt, the endpoint returns its persisted report without rerunning candidates or consuming another attempt. Inspect `job.status`: a returned `RUNNING` or `FAILED` report is not a successful maintenance outcome. To retry a completed failed attempt after its backoff, explicitly use its new observed attempt number. + +All named/current/latest reports must agree before new work starts. If a completed report was persisted but final alias publication was interrupted, the same invocation can reconcile aliases only when the reports prove the same job, attempt, lease, immutable plan, and audit transition. Incomplete claims, conflicting histories, and expired `RUNNING` jobs fail closed for operator inspection; the endpoint does not take over or cancel an abandoned worker. Catalog commits remain protected by the existing publication locks during execution and reconciliation. + +To pause admission for a table, set `worker-paused=true` in its effective maintenance configuration. To stop the process-wide scheduler, disable the startup flag and restart. Shutdown stops new admissions and waits for owned queueing work; if completion cannot be confirmed within the normal shutdown grace, it retains unclean-shutdown markers and logs `drain_unconfirmed`. A deadline is not proof that a mutation was cancelled. + +This scheduler does not claim automatic worker retry, cross-node worker takeover, bounded history retention, or a fully managed maintenance service. diff --git a/docs/operations/scanner-benchmark-runbook.md b/docs/operations/scanner-benchmark-runbook.md index 46684d268..45bb7ef26 100644 --- a/docs/operations/scanner-benchmark-runbook.md +++ b/docs/operations/scanner-benchmark-runbook.md @@ -248,6 +248,7 @@ Collect at least two runs on the same RustFS commit and the same workload. Keep | Duration budget (when one cycle is too long) | Bound wall-clock time per cycle. | `scanner.cycle_max_duration="1800"`. | | Object budget | Bound objects processed per cycle. | `scanner.cycle_max_objects="1000000"`. | | Directory budget | Bound directories entered per cycle. | `scanner.cycle_max_directories="100000"`. | +| Disk I/O budget (single slow disk) | Cap scanner metadata I/O per local disk regardless of idle time; compare foreground p99 and scan completion against the media default. | `scanner.disk_io_budget_iops="50"` and `scanner.disk_io_budget_bytes_per_sec="8388608"`; `off` removes the cap for an A/B leg. | ## Deployment Matrix diff --git a/docs/operations/scanner-runtime-controls.md b/docs/operations/scanner-runtime-controls.md index 33a182860..ae4915866 100644 --- a/docs/operations/scanner-runtime-controls.md +++ b/docs/operations/scanner-runtime-controls.md @@ -2,7 +2,7 @@ **Use this when:** tuning scanner pacing or heal runtime knobs, reading `/v3/scanner/status`, deciding whether slow lifecycle/replication/heal progress is a scanner problem or a downstream queue problem, or migrating scanner settings from MinIO. -**Source of truth:** `crates/config/src/constants/scanner.rs` and `crates/config/src/constants/heal.rs` (keys, env names, `DEFAULT_*` constants), `crates/scanner/src/runtime_config.rs` (resolution order, parsing, hot update), `crates/scanner/src/scanner_folder.rs` (env-only scanner knobs, alerts), `crates/scanner/src/sleeper.rs` (throttling), `crates/heal/src/heal/manager.rs` (`HealConfig::default`), `crates/utils/src/envs.rs` (`EXTERNAL_COMPATIBLE_SUFFIXES`, MinIO env aliases). +**Source of truth:** `crates/config/src/constants/scanner.rs` and `crates/config/src/constants/heal.rs` (keys, env names, `DEFAULT_*` constants), `crates/scanner/src/runtime_config.rs` (resolution order, parsing, hot update), `crates/scanner/src/scanner_folder.rs` (env-only scanner knobs, alerts), `crates/scanner/src/sleeper.rs` and `crates/scanner/src/sleeper/io_budget.rs` (throttling and per-disk I/O budget), `crates/heal/src/heal/manager.rs` (`HealConfig::default`), `crates/utils/src/envs.rs` (`EXTERNAL_COMPATIBLE_SUFFIXES`, MinIO env aliases). For reproducible scanner-pressure validation and before/after evidence, see [Scanner Benchmark Runbook](scanner-benchmark-runbook.md). Alert thresholds and S3 event names are detailed in [Scanner Excess Alerts](scanner-excess-alerts.md). @@ -43,7 +43,10 @@ Bitrot cycle resolution differs because the canonical persistent key belongs to | `scanner.idle_mode` | `RUSTFS_SCANNER_IDLE_MODE` | boolean | `true` (`DEFAULT_SCANNER_IDLE_MODE`) | Enables preset scanner sleeps and foreground-read backoff. `false` disables those sleeps; per-disk foreground-priority admission remains active. | | `scanner.cache_save_timeout` | `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` | seconds | `14` (`DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS`) | Timeout for saving scanner cache; runtime enforces a minimum of `1` and keeps the default persistence budget within the distributed publication lease. | | `scanner.max_concurrent_set_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS`) | Caps concurrent set-level scanner tasks. `0` keeps topology-derived concurrency. | -| `scanner.max_concurrent_disk_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS`) | Caps concurrent disk bucket walks per set. `0` keeps disk-count-derived concurrency. | +| `scanner.max_concurrent_disk_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS`) | Caps concurrent disk bucket walks per set. `0` keeps walker-count-derived concurrency. | +| `scanner.walkers_per_disk` | `RUSTFS_SCANNER_WALKERS_PER_DISK` | count | `1` (`DEFAULT_SCANNER_WALKERS_PER_DISK`) | Bucket walkers spawned per local disk of a set. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk; remote disks keep one walker. The resulting walker count is what `scanner.max_concurrent_disk_scans` caps, and the disk count stays the floor. Valid range is `1` through `16` (`MAX_SCANNER_WALKERS_PER_DISK`); `1` keeps today's I/O profile and is the right value for rotational media. The `disk_concurrency_budget` log line reports `walkers_per_disk`, `walkers`, the effective `concurrency_limit`, and its `reason` (`available_walkers`, `configured_cap`, or `count_budget`). | +| `scanner.disk_io_budget_iops` | `RUSTFS_SCANNER_DISK_IO_BUDGET_IOPS` | operations/s | `auto` (`DEFAULT_SCANNER_DISK_IO_BUDGET`) | Per-local-disk ceiling on scanner directory listings and object metadata reads. `auto` picks the media default (`100` on rotational or unknown media, `4000` on flash); `off` removes the cap; a number is an explicit rate. See [Per-disk background I/O budget](#per-disk-background-io-budget). | +| `scanner.disk_io_budget_bytes_per_sec` | `RUSTFS_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC` | bytes/s | `auto` (`DEFAULT_SCANNER_DISK_IO_BUDGET`) | Per-local-disk ceiling on the metadata bytes the scanner reads. `auto` picks the media default (`16777216` on rotational or unknown media, `268435456` on flash); `off` removes the cap; a number is an explicit rate. | | `scanner.yield_every_n_objects` | `RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS` | objects | `128` (`DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS`) | Controls how often object loops yield to the async runtime. `0` disables this extra yield. | | `scanner.alert_excess_versions` | `RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS` | versions | `100` (`DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS`) | Version count threshold for scanner alerts. | | `scanner.alert_excess_version_size` | `RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE` | bytes | `1099511627776` (`DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE`) | Retained version byte threshold for scanner alerts. | @@ -61,6 +64,16 @@ Speed presets (`crates/config/src/constants/scanner.rs`) set the base sleep mult Use `scanner.delay`, `scanner.max_wait`, and `scanner.cycle` when the preset is close but one axis needs a precise override. With `idle_mode=true`, directory-level sleep is `1ms x factor` and object-level sleep is `time spent on the object x factor`, both capped at `max_wait`; a foreground-read floor of `FOREGROUND_READ_BACKOFF_PER_REQUEST_MS` (10ms) per concurrent GetObject/streaming read, capped at `FOREGROUND_READ_BACKOFF_MAX_MS` (250ms), is applied on top and can exceed the preset's `max_wait` (`crates/scanner/src/sleeper.rs`). The per-disk foreground-priority gate is independent of `idle_mode` and remains active when sleeps are disabled. +### Walkers per disk + +The scanner walks the buckets of an erasure set with one walker per local disk by default, so a single-disk deployment walks one bucket at a time no matter how `scanner.max_concurrent_disk_scans` is set. `scanner.walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`) multiplies each local disk into that many walkers. + +- Keep the default `1` on rotational media: several walkers on one spindle compete for seeks and slow every walk down. +- On flash, `2` to `4` is the useful range. Raise it when the scanner status shows cycles ending on the cycle budget (`metrics.pacing_pressure.last_cycle_budget_limited`) or when many buckets are never reached within a cycle, and the disk has idle I/O capacity. +- The walkers of one disk share that disk's scanner admission, so a peer still sees one scan session per physical disk; remote disks keep one walker. `scanner.max_concurrent_disk_scans` still caps the resulting walker count, and the disk count stays the floor. +- Confirm the setting took effect from `runtime_config.walkers_per_disk.value` in the admin scanner status and from the `disk_concurrency_budget` log line, which reports `walkers_per_disk`, `walkers`, the effective `concurrency_limit`, and the `reason` it was chosen. +- Helm: `config.rustfs.scanner.walkers_per_disk`; Docker or systemd: the environment variable. The key is additive, so an older binary ignores it. + ### Environment-only scanner knobs These have no persistent key and are read from the environment only. @@ -96,6 +109,10 @@ When a finite deadline expires, RustFS cancels cooperative scanner work and wait An explicit `scanner.cycle` or `RUSTFS_SCANNER_CYCLE` is a minimum inter-cycle cadence: dirty-usage notifications do not bypass that configured interval. The default adaptive policy continues to use dirty-usage notifications to wake the scanner between timer-driven cycles. +Under continuous writes, a structurally complete walk can finish as `superseded`. Its durable observation remains separate from quota authority and leaves dirty work pending. Superseded retries may reuse that complete inventory to refresh dirty buckets, even while authority bootstrap is pending; missing or invalid coverage still forces a full walk. Ordinary maintenance timers, Deep scans, active lifecycle/replication and changed maintenance configuration retain full scans. Server info and datausageinfo can display complete observations, while Last Completed Scan advances only after an authoritative completed cycle. A write-free window is therefore not required to display observed counts, but arbitrary continuous writes do not guarantee authoritative completion. Historically dirty buckets remain eligible for rewalk, so narrower retries do not guarantee shorter cycles. + +Superseded convergence retries retain exponential backoff for usage-only work. Active lifecycle or replication rules, and uncertain maintenance inspection, cap that retry interval at the ordinary configured cadence so complete observations cannot defer regular maintenance. An explicit cycle remains the duty-cycle floor. Convergence mode still suppresses dirty/activity shortcuts and does not grant quota authority; deferred backend/proof retries and pause catch-up limits retain their existing scheduling. Cycle runtime, pacing, and disk admission can still delay completed maintenance. Sustained supersedes with active maintenance may therefore run more full scans than usage-only convergence; use the existing pacing controls to manage that background I/O. + ## Single-disk clean-idle scheduling An erasure single-disk deployment using the built-in cycle and bitrot defaults automatically backs off repeated clean idle scans instead of walking the same unchanged namespace every minute. Each successful timer-driven cycle that finds no dirty usage or unresolved maintenance work doubles the next interval. The status endpoint reports the effective interval and multiplier. @@ -114,6 +131,20 @@ The `rustfs_disk_io_admission_active` foreground gauge is sampled while backgrou This is logical operation admission, not an OS disk scheduler: foreground and background operations may overlap after a background permit is granted, and the gate does not promise a bandwidth or IOPS limit. It is scoped to each `LocalDiskWrapper`; separate paths to one physical device and remote disk RPCs do not share this gate. It also does not observe or schedule unrelated host processes that share the physical device. +## Per-disk background I/O budget + +The admission gate orders scanner I/O behind foreground requests but does not bound how much background I/O an idle disk sees. `scanner.disk_io_budget_iops` and `scanner.disk_io_budget_bytes_per_sec` add that bound. Each local disk owns one token bucket per axis (`crates/scanner/src/sleeper/io_budget.rs`); every directory listing charges one operation, and every object metadata read charges one operation plus the `xl.meta` size. A charge that exceeds the remaining budget delays that walker until the rate allows it. The budget is layered on top of admission: a budgeted operation still queues behind foreground requests, so the budget can only slow the scanner down and never lets it exceed what admission allows. It applies whether or not `idle_mode` is enabled. + +- Media is detected once per disk path from the backing block device's `rotational` flag on Linux, through partitions, `dm-*`, and software RAID; any spindle in the stack counts as rotational. Undetectable media, including non-Linux hosts and overlay filesystems without a bind-mounted volume, uses the rotational defaults because a spindle is the case that needs protection. +- `auto` resolves to `100` operations/s and `16 MiB/s` on rotational or unknown media and to `4000` operations/s and `256 MiB/s` on flash. The rotational values leave roughly half of a 7200 rpm spindle's random reads to foreground traffic; the flash values are a few percent of a SATA SSD. The `fastest` speed preset resolves `auto` to no cap; explicit numbers always apply. +- A budget of `0` or any tiny value never starves a walk: one charge is admitted per second (`SCANNER_DISK_IO_BUDGET_MAX_WAIT`), and an oversized charge owes at most one extra second of debt. +- The walkers of one disk (`scanner.walkers_per_disk`) share that disk's budget. Remote disks are paced by their owning node. +- Changes take effect at the next charge without a restart. + +The status endpoint lists each disk under `disk_io_budget` with its detected `media`, the effective `iops` and `bytes_per_sec` (`null` when uncapped), charged totals, and accumulated `wait_events` / `wait_seconds`. Budget waits are also reported per cycle as `metrics.current_cycle_disk_io_budget_wait_events` / `_duration_seconds` and `metrics.last_cycle_disk_io_budget_wait_events` / `_duration_seconds`, as `metrics.pacing_pressure.last_cycle_disk_io_budget_wait_ratio` with the `disk_io_budget` primary pressure, and in `metrics.usage_freshness.disk_io_budget_wait_events`, `disk_io_budget_wait_millis`, and `last_disk_io_budget_wait_unix_secs`. Prometheus exports `rustfs_scanner_disk_io_budget_waits_total` and `rustfs_scanner_disk_io_budget_wait_seconds` by disk. The `scanner_disk_io_budget` log line reports the resolved media and rates once per disk. + +Like admission, this is in-process pacing: it does not schedule other host processes. Deep bitrot verification reads are issued by the heal subsystem and are paced by heal controls, not by this budget. + ## Status Endpoint ```text @@ -126,6 +157,9 @@ The request must be authenticated with an admin identity that has `ServerInfoAdm |---|---| | `runtime_config` | Effective runtime controls and their value sources. | | `cycle_schedule` | Current effective cycle interval and clean-idle backoff state. | +| `cycle_recovery` | This node's cycle-state recovery observation, retry budget, and reason. | +| `bitrot` | How the current cycle's scan mode was decided: whether Deep is supported on this topology, where the bitrot cadence start came from, and the recorded start. See [Deep scan cadence state](#deep-scan-cadence-state). | +| `disk_io_budget` | One entry per local disk with detected media, effective per-disk I/O budget, charged totals, and accumulated budget waits. | | `metrics` | Scanner work, pressure, checkpoint, lifecycle, replication, heal, bitrot, and alert counters. | | `data_movement_pause` | Global-pause policy, current movement reason, operation epoch, start time, duration, and estimated movement work items. | | `pause_backlog` | Replicated durable pause ledger, post-pause catch-up phase, rate window, retry state, thresholds, and active alert reasons. | @@ -139,11 +173,20 @@ runtime_config.delay.value runtime_config.max_wait_seconds.value runtime_config.cycle_interval_seconds.value runtime_config.bitrot_cycle_seconds.value +runtime_config.walkers_per_disk.value cycle_schedule.effective_interval_seconds cycle_schedule.clean_idle_backoff_enabled cycle_schedule.clean_idle_backoff_multiplier metrics.pacing_pressure.primary_pressure metrics.pacing_pressure.last_cycle_budget_limited +metrics.pacing_pressure.last_cycle_disk_io_budget_wait_ratio +metrics.usage_freshness.disk_io_budget_wait_events +bitrot.deep_scan_supported +bitrot.bitrot_start_kind +bitrot.bitrot_start_source +disk_io_budget[].media +disk_io_budget[].iops +disk_io_budget[].wait_seconds metrics.lifecycle_transition.current_queued metrics.lifecycle_transition.scanner_missed metrics.maintenance_control.primary_control @@ -172,6 +215,20 @@ catch_up_estimate.discovered_expiry_items catch_up_estimate.discovered_transition_items ``` +### Deep scan cadence state + +`bitrot` reports how the scanner decided the scan mode of the running or most recent cycle: + +| Field | Meaning | +|---|---| +| `deep_scan_supported` | `false` on an erasure single-disk deployment, where scanner-originated object heal sampling is unsupported and Deep mode is never selected (rustfs/rustfs#8126). | +| `bitrot_start_kind` | `deep_window` when a recorded Deep window start is in effect, `anchor` when no Deep window is recorded anywhere and the cadence counts from this process's first cycle without one, `none` before the first cycle. | +| `bitrot_start_source` | Where the reference came from: `persisted` (read back from the background-heal object), `memory` (the in-memory anchor, or a window start this process recorded that storage has not returned, for example because the save was skipped without movement admission), or `none`. | +| `bitrot_start_cycle`, `bitrot_start_unix_secs` | The recorded window start, or the anchor time with cycle `0`; both `0` when none. | +| `current_scan_mode` | `normal` or `deep`. | + +A cycle that finds no recorded Deep window runs Normal and anchors the cadence in process memory; nothing is persisted until the configured bitrot cycle has elapsed since that anchor (or on every cycle when the cycle is `0`), at which point the Deep cycle records the window start exactly as before. An anchor never opens the Deep selection window: a missing record used to be read as a window that started at cycle `0`, which made every cycle below the selection window run Deep whenever the start could not be read back (rustfs/rustfs#8015). The window start also stays in process memory across a skipped or failed save, so the next cycle continues the window instead of reopening one; when a persisted and an in-memory start both exist, the newer one wins and ties go to the persisted record. `RUSTFS_SCANNER_BITROT_CYCLE_SECS=off` still never selects Deep, deployments whose background-heal state persists normally keep their existing cadence, and a restart resets the anchor. + ### Periodic checkpoint persistence `metrics.scan_checkpoint` is the in-memory resume hint; it does not confirm a durable save. `metrics.scan_checkpoints_persisted` lists the latest confirmed periodic save for each `(pool_index, set_index)` handled by this process, with its bucket, `saved_unix_secs`, and optional checkpoint cursor. It retains one observation per set, is reset on process restart, and does not include the final bucket save. Query the peer executing the walk for remote-worker observations. `metrics.scan_checkpoint_save_failures` counts failed periodic saves, including failures to refresh CAS revisions after a write; failed attempts do not advance the confirmed save observation. @@ -186,6 +243,12 @@ Shared compacted accumulators defer periodic emission until the owning subtree c A confirmed periodic save does not prove restart resumability. Adoption still validates bucket/set identity and, for non-empty caches, connected cache scope. Skipping completed subtrees additionally requires a validated coverage frontier; raw enumeration state alone does not prove completed subtree coverage. +### Cycle recovery on followers + +Cycle recovery status is local to each node. A transient cycle-state read failure can exhaust the short retry budget and enter `backoff`; the observation stays retryable, the enabled scanner keeps retrying leadership with an exponential delay capped at ten minutes or sooner when recovery is explicitly woken, and `next_attempt_at_unix_secs` together with `reason` name the next attempt. If the next lock attempt is contended or times out waiting for `leader.lock`, the node records a follower iteration and clears only its unclassified `transient` or `backoff` observation. Normal follower retry cadence resumes without restarting the process. + +This clears local retry bookkeeping, not persisted cycle state or recovery markers. Classified recovery states, cleanup-pending resets, and usage-floor failures remain unchanged. Any later leadership claim revalidates durable cycle state before scanning. A healthy follower recovery status does not prove that leader-dispatched scans are progressing; inspect the work counters separately. + ## Usage State Reset The supported break-glass route for rebuilding scanner usage state is `POST /v3/scanner/usage-state/reset` with body `{"mode":"full-rebuild"}`, authenticated as an admin identity holding `ConfigUpdateAdminAction` (route registered in `rustfs/src/admin/route_registration_test.rs`). Use it only after the scanner status shows a usage-floor load failure, a conflicting persisted usage floor, or an operator decision to discard the durable usage baseline and rebuild it from a full scanner pass. @@ -227,7 +290,7 @@ When a running scanner leader already holds `leader.lock`, submit a durable asyn The async form returns HTTP 202 when a new or replayable intent is accepted. Reusing the same `idempotency_key` is safe for retries of the same operator action; a different payload for the same key is rejected as a conflict. Poll `GET /v3/scanner/usage-state/recovery-intents/{intent_id}` until the intent reaches a terminal state. -A successful reset leaves usage state in `bootstrap-pending` only as a fenced rebuild marker. With the scanner enabled, the leader must treat that marker as pending full-rebuild work: clean-idle backoff must not extend the wait, an empty pause-backlog catch-up ledger must not rate-limit the rebuild, and the next admitted scanner cycle must run as a full scan until an authoritative usage snapshot replaces the marker. +A successful reset leaves usage state in `bootstrap-pending` only as a fenced rebuild marker. With the scanner enabled, the leader must treat that marker as pending full-rebuild work: clean-idle backoff must not extend the wait, an empty pause-backlog catch-up ledger must not rate-limit the rebuild, and the next admitted scanner cycle must rebuild full inventory. After a complete observation is durably saved, superseded retries may refresh dirty buckets using its validated coverage; bootstrap remains pending until authoritative publication replaces the marker. ## Cleanup With The Scanner Disabled @@ -246,7 +309,7 @@ RustFS uses a `global_pause` policy while pool decommission or rebalance can hid | `data_movement_pause.reasons` | In-process decommission worker state combined with durable pool and rebalance operation metadata. Exhausted operation epochs or movement generations fail closed and appear as explicit reasons. | | `data_movement_pause.duration_seconds`, start time, `movement_backlog_work_items` | From durable metadata; a worker-only or exhausted-counter snapshot can report `paused=true` with zero start time and backlog. `movement_backlog_work_items` counts remaining movement bucket work units, not expired objects. | | `catch_up_estimate` | Movement estimate plus dirty-usage buckets and lifecycle items discovered before or during the pause. `undiscovered_ilm_items_known=false` because a global pause cannot count newly expired objects without scanning; use `usage_baseline_unix_secs` to judge the estimate's age. | -| `pause_backlog.persistence_state` | `persistence_unavailable` when the `.scanner-pause-backlog.json` ledger cannot be read or updated; scanner cycles stay gated and persistence is retried every five minutes. | +| `pause_backlog.persistence_state` | `degraded` when the `.scanner-pause-backlog.json` ledger cannot be claimed or updated; `error` carries the reason. Scanner cycles keep running untracked behind the data-movement fence, and the ledger claim is retried on each scheduler wake at most every five minutes. `unavailable` means no scanner runtime on this node has attempted a claim. | | `pause_backlog.phase` | `idle`, `paused`, `catching_up`, or `retry_exhausted`. The ledger returns to `idle` only after one successful full namespace scan and zero known dirty-usage, expiry, and transition queues. | | `membership_repair_pending` | A rejoining decommission source is being re-seeded from the last committed surviving-set ledger before a new full-membership commit is allowed. | | `pause_backlog.thresholds` | Exact pause-duration, deferred-cycle, backlog-size, rate, and failure limits used by the running binary. | @@ -288,11 +351,12 @@ Unlabeled Prometheus gauges: |---|---|---| | `queued_scans` | Set or disk scan queues are backing up. | Lower scanner concurrency or increase pacing delay if user traffic is affected. | | `cycle_budget` | The last cycle stopped because a runtime/object/directory budget was reached. | Check `last_cycle_partial_reason` and `last_cycle_partial_source`; increase the specific budget if scans need to finish sooner. | +| `disk_io_budget` | The last cycle waited on the per-disk background I/O budget. | Expected on rotational media; raise `scanner.disk_io_budget_iops` or `scanner.disk_io_budget_bytes_per_sec` only when foreground latency has headroom. | | `throttle_pause` | Scanner sleeps or cooperative yields were observed. | Expected when `idle_mode` is enabled; inspect pause ratios before tuning. | | `active_scans` | Scanner work is active but not currently queued or budget-limited. | Usually healthy; correlate with CPU/disk metrics. | | `none` | No current scanner pressure was observed. | No scanner pacing action needed. | -The ratio fields `last_cycle_throttle_sleep_ratio`, `last_cycle_yield_ratio`, and `last_cycle_total_pause_ratio` are fractions of the last cycle duration. If CPU is high but pause ratios are already high, increasing `scanner.delay` or `scanner.max_wait` may have limited value; check active paths, source work, and disk activity before changing the cycle interval. +The ratio fields `last_cycle_throttle_sleep_ratio`, `last_cycle_disk_io_budget_wait_ratio`, `last_cycle_yield_ratio`, and `last_cycle_total_pause_ratio` are fractions of the last cycle duration. If CPU is high but pause ratios are already high, increasing `scanner.delay` or `scanner.max_wait` may have limited value; check active paths, source work, and disk activity before changing the cycle interval. ## Reading Source Work @@ -489,7 +553,7 @@ Stale-upload cleanup differs in one crash-recovery detail. RustFS's stale multip Treat replacement recovery as verified only after the repair task has completed for the exact replacement instance and an operator has confirmed the target disk contains the expected `xl.meta` and data parts for every relevant object version. A replacement that is not mounted, is unsafe to format, loses its marker, or returns a partial target outcome must be treated as deferred or incomplete. Do not automate destructive replacement actions from an `idle` observation alone. A new node must not infer replacement completion from an old or unavailable peer; regard that information as unknown or degraded until every required peer can report the same replacement instance and verified completion. -`GET /rustfs/admin/v4/heal/replacement-recovery` reports durable automatic replacement records from survivor disks. `local.records[]` entries distinguish `waiting_for_replacement`, `running`, `incomplete`, `unrecoverable`, `cleanup_pending`, `completed`, and `unknown`; `local.definitive=false` or any `unknown` record means the node could not prove a local replacement state. The `cluster` section queries the replacement-recovery peer RPC and sets `cluster.definitive=true` only when the expected peer topology is complete, every peer supports the RPC, every peer snapshot is locally definitive, and all peers report the same replacement records. Old peers, unavailable peers, malformed peer payloads, topology gaps, and generation disagreements are reported as degraded or unknown rather than complete. +`GET /rustfs/admin/v4/heal/replacement-recovery` reports durable automatic replacement records from survivor disks. `local.records[]` entries distinguish `waiting_for_replacement`, `running`, `incomplete`, `unrecoverable`, `cleanup_pending`, `completed`, and `unknown`; `local.definitive=false` or any `unknown` record means the node could not prove a local replacement state. The `cluster` section queries the replacement-recovery peer RPC and sets `cluster.definitive=true` only when the expected peer topology is complete, every peer supports the RPC, every peer snapshot is locally definitive, and copies of each task agree. Different anchors can report different tasks, and a definitive node may report an empty list or a subset of another node's history. Disjoint target generations merge directly. Every pair of generations sharing a target slot must appear together in at least one definitive local snapshot; this preserves retained completion proofs and committed handoff histories while keeping overlapping records found only on separate peers non-definitive. Shared history is not transitive: observing A with B and B with C does not establish that A and C are compatible. A definitive history can include retired or unfinished generations, so consumers must inspect each record's state rather than treating `cluster.definitive` as proof that every generation completed. Old peers, unavailable peers, malformed peer payloads, topology gaps, and task disagreements are reported as degraded or unknown rather than complete. Replacement resume and checkpoint files use an independent on-disk schema. A newer reader rejects a future schema rather than continuing with data it cannot interpret, while an older binary cannot safely enforce the new generation fence. Do not roll a cluster back after a replacement generation has started; complete that recovery with the current-or-newer release, and if it cannot complete, keep that version for diagnosis rather than deleting its durable records or continuing with an older binary. @@ -632,6 +696,8 @@ config: idle_mode: "true" yield_every_n_objects: "128" bitrot_cycle_secs: "2592000" + disk_io_budget_iops: "100" + disk_io_budget_bytes_per_sec: "16777216" ``` Use `extraEnv` for environment variables that are not represented by chart values, including every heal knob above. diff --git a/docs/testing/README.md b/docs/testing/README.md index 47492e7fc..037d3ee53 100644 --- a/docs/testing/README.md +++ b/docs/testing/README.md @@ -12,7 +12,7 @@ Pick the lowest layer that can prove the change; add a higher-layer test only wh | Unit & crate integration | Per-crate logic and in-process integration tests | `cargo nextest run --all --exclude e2e_test` (or `-p `); `make test` wraps it | Every PR, required (`Test and Lint`, `ci` profile) | | ecstore black-box | Erasure-coded read/write/recovery validation; profiles `quick` / `full` / `destructive` / `fuzz` | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local and release validation only; not wired into any workflow. Contract: [ecstore-validation-suite-design.md](ecstore-validation-suite-design.md) | | e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `python3 scripts/e2e_binary.py build --features e2e-test-hooks`, then `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test` | PR: `e2e-smoke` (report-only); merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`, `e2e-distributed`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md); 4-node 4-disk map: [distributed-e2e.md](distributed-e2e.md) | -| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `python3 scripts/e2e_binary.py build`, then `python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) | +| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `python3 scripts/e2e_binary.py build`, then `python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) | | s3s-e2e conformance | External S3 conformance tool against a live server | `./scripts/e2e-run.sh ./target/debug/rustfs ` | PR, report-only (second half of the `End-to-End Tests` job) | | S3 compatibility | `ceph/s3-tests` (boto3; allow-list `scripts/s3-tests/implemented_tests.txt`) and MinIO `mint` | `scripts/s3-tests/run.sh`; mint via `.github/workflows/mint.yml` | s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only | | Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) plus the 4-node kill/fresh-drive/blackhole cases in `crates/e2e_test/src/distributed/chaos_test.rs` | Part of the e2e crate (`e2e-reliability` and `e2e-distributed`) | Reliability cases with `e2e-full`; 4-node chaos on storage-sensitive PRs and nightly via `e2e-distributed` | diff --git a/docs/testing/ci-gates.md b/docs/testing/ci-gates.md index 8ea7e09f2..cbff6cca4 100644 --- a/docs/testing/ci-gates.md +++ b/docs/testing/ci-gates.md @@ -1,7 +1,7 @@ # CI gate matrix **Use this when:** a check is red and you need to know whether it blocks the merge, which workflow and job produced it, and how to reproduce it locally. -**Source of truth:** the live `main` ruleset (command below) for required status; `.github/workflows/.yml` for triggers, `paths`, `timeout-minutes`, and cron; `.config/nextest.toml` for e2e profile filters; `.github/scheduled-validations.json` for the freshness-watchdog list. +**Source of truth:** the live `main` ruleset (command below) for required status; `.github/workflows/.yml` for triggers, `paths`, `timeout-minutes`, and cron; `.config/nextest.toml` for e2e profile filters; `.github/scheduled-validations.json` for the freshness-watchdog list; `.github/workflows/README.md` for runner placement and billing. A job blocks a merge when its exact check name is required by the live `main` ruleset, or when its result is required by the `Test and Lint` aggregate. A workflow name, a `merge_group` trigger, or an unrelated red PR check does not make a job required by itself. @@ -19,9 +19,9 @@ Every PR enters `ci.yml`. The `classify-changes` job uses the base revision of ` PRs changing only Rust sources under `crates/e2e_test/src/` and known E2E selection digests (optionally with those documentation paths) use the `e2e` scope. They keep workspace validation, the server build, E2E smoke, and both S3 lanes; independent workflows retain their path selection. Connect boundary, serial ILM, optional protocol/rio-v2, and real io_uring jobs are skipped because their sources and test inputs are unchanged. Manifests, fixtures, shared nextest configuration, scripts, production sources, and unknown paths still select `full`. A dependency on `e2e_test` from another workspace member or reachable local dependency disables this shortcut. All non-PR events retain their full selection. -Alternate-feature unit tests and Clippy for rio-v2, Swift, SFTP, and HTTP/3 run on main pushes, merge groups, and scheduled/manual validation. PRs run the default workspace and retain ILM, S3, real io_uring, and Connect security boundaries in their applicable scopes. The aggregate requires the alternate-feature jobs to be skipped on PRs and successful on non-PR events. This reduces repeated builds across PR revisions; feature-specific regressions can be detected after merging when the merge queue is not used. For changes to these features, dispatch `ci.yml` against the candidate branch before merging when that signal is needed, including PRs targeting `gateway/integration`, which has no automatic push trigger in this workflow. Scheduled results supplement the main baseline and do not replace it. +Alternate-feature unit tests and Clippy for rio-v2, Swift, SFTP, and HTTP/3 run on merge groups and on scheduled (nightly) or manual validation. PRs and main pushes run the default workspace and retain ILM, S3, real io_uring, and Connect security boundaries in their applicable scopes. The aggregate requires the alternate-feature jobs to be skipped on PRs and main pushes and successful on merge-group, scheduled, and manual events. This avoids repeating four feature builds on every PR revision and on every merge; feature-specific regressions are detected by the nightly run when the merge queue is not used. For changes to these features, dispatch `ci.yml` against the candidate branch before merging when that signal is needed, including PRs targeting `gateway/integration`, which has no automatic push trigger in this workflow. Scheduled results supplement the main baseline and do not replace it. -Workspace Clippy uses the same non-PR cadence. Every code PR still builds and runs the complete default workspace test suite, documentation tests, and migration proofs; Quick Checks retain formatting and script/workflow linting. Without a merge queue, Clippy diagnostics may first fail main after merge. A manual `ci.yml` dispatch against the candidate branch runs Clippy before merging when needed. +Workspace Clippy runs on every event, pull requests included, ahead of the workspace tests in the same job. `cache-warm.yml` warms Clippy's check artifacts under `ci-dev` with the same invocation, so a pull request re-checks only the crates its diff touches. `make pre-commit` does not run Clippy; run `cargo clippy --all-targets -- -D warnings` locally before pushing. Serial ILM tests run at the end of `test-and-lint` for the `full` scope, reusing the workspace's Cargo package and feature selection. The existing ignored-test filter and `-j1` remain; `--profile default` keeps the no-retry execution and preserves the separate core CI JUnit report. Any ILM failure fails the workspace job and required aggregate. Logs remain in the `ilm-integration` artifact. There is no separate ILM runner or cache restore. @@ -51,13 +51,11 @@ The four site-replication state-writer concurrency proofs reserve the available | Event | Check name | Workflow / job | Merge status | Reproduce | |---|---|---|---|---| | PR, non-doc change | `Quick Checks` | `ci.yml` `quick-checks` | Required | `make pre-commit` | -| PR, non-doc change | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate | `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --exclude e2e_test --doc`; migration evidence check described below | -| Main push, merge group, schedule, manual dispatch | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate on these events | workspace tests plus `cargo clippy --all-targets -- -D warnings` | +| PR, non-doc change, and every non-PR event | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate | `cargo clippy --all-targets -- -D warnings`; `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --exclude e2e_test --doc`; migration evidence check described below | | PR, non-doc change | `Typos` | `ci.yml` `typos` | Via aggregate | `typos` | | Full selection | `Workspace Test and Lint` (serial ILM step) | `ci.yml` `test-and-lint` | Via aggregate | exact `Run ignored ILM integration tests serially` command in the job | -| Main push, merge group, schedule, manual dispatch | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)`, `Test and Lint (http3)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate on these events; skipped on PRs | `cargo nextest run` with the job's `--features` | -| PR, full selection | `Connect Short Credential Boundary` | `ci.yml` `connect-short-credential-boundary` | Via aggregate | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `cargo check -p rustfs --release --features connect-e2e-short-credentials` must fail | -| PR, full selection | `Offline Enrollment E2E Root Boundary` | `ci.yml` `offline-enrollment-e2e` | Via aggregate | `scripts/check_offline_enrollment_e2e.sh` | +| Merge group, nightly schedule, manual dispatch | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)`, `Test and Lint (http3)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate on these events; skipped on PRs and main pushes | `cargo nextest run` with the job's `--features` | +| PR, full selection | `Connect Feature Boundaries` | `ci.yml` `connect-feature-boundaries` | Via aggregate | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `cargo check -p rustfs --release --features connect-e2e-short-credentials` must fail; `scripts/check_offline_enrollment_e2e.sh` | | PR, non-doc change | `Build RustFS Debug Binary` | `ci.yml` `build-rustfs-debug-binary` | Via aggregate; prerequisite for black-box jobs | `python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks` (binary plus its `rustfs.e2e.json` sidecar) | | PR, full selection | `io_uring Integration (real)` | `ci.yml` `uring-integration` | Via aggregate | `cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture` | | PR, non-doc change | `End-to-End Tests` | `ci.yml` `e2e-tests` | Via aggregate | `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test`, then `./scripts/e2e-run.sh ./target/debug/rustfs ` under the same wrapper; membership guards `scripts/check_test_wiring.py --check-profile e2e-smoke ` and `scripts/check_security_smoke_count.sh check ` | @@ -66,7 +64,7 @@ The four site-replication state-writer concurrency proofs reserve the available | PR touching `paths` in `audit.yml` | `Cargo Deny`, `Workflow Pin Report`, `Dependency Review` | `audit.yml` `cargo-deny`, `workflow-pin-report`, `dependency-review` | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` | | PR touching `paths` in `nix.yml` | `Nix Build & Check` | `nix.yml` `nix-validation` | Report-only | `nix flake check` | | PR touching `paths` in `fuzz.yml` | `Build Fuzz Harness`, `Smoke / ` | `fuzz.yml` `fuzz-build`, `pr-fuzz-smoke` | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` | -| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety` | `windows-filesystem.yml` `rename-safety` | Report-only | the `cargo test -p rustfs-ecstore --lib ` commands in the job, on Windows | +| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety`, `Startup Smoke` | `windows-filesystem.yml` `rename-safety`, `startup-smoke` | Report-only | the `cargo test -p rustfs-ecstore --lib ` commands in the job, on Windows; `cargo build --locked -p rustfs --bin rustfs`, then `rustfs.exe --version` and a single-volume server polled on `/health/ready` | | PR touching `paths` in `coverage.yml` | `Workspace line coverage` | `coverage.yml` `coverage` | Report-only | `make coverage`; `python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json` | | PR touching `paths` in `e2e-upgrade.yml` | `Upgrade compatibility` | `e2e-upgrade.yml` `upgrade` | Report-only | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` commands with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) | | PR touching `paths` in `oidc-keycloak.yml` | `OIDC Keycloak live gate` | `oidc-keycloak.yml` `oidc-keycloak-live` | Report-only | `cargo build --locked -p rustfs --bin rustfs`, then `bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs` | @@ -92,8 +90,9 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run, sched | Workflow (cadence) | Jobs | Verdict and artifacts | In freshness list | Reproduce | |---|---|---|---|---| -| `ci.yml` (weekly) | full matrix, including the schedule/dispatch-only rio-v2 jobs `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` | strict aggregate; the full E2E lane runs on dispatch, merge groups, and main pushes | yes | dispatch `ci.yml` | -| `build.yml` (weekly) | `build-rustfs` over the six-target platform matrix in `prepare-platform-matrix` (four Linux, macOS aarch64, Windows x86_64) | build/package integrity | yes | dispatch `build.yml` with an exact platform set | +| `ci.yml` (nightly) | full matrix, including the schedule/dispatch-only rio-v2 jobs `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` | strict aggregate; the full E2E lane runs on dispatch, merge groups, and main pushes | yes | dispatch `ci.yml` | +| `cache-warm.yml` (every six hours, plus main pushes that change `Cargo.lock`, `rust-toolchain.toml` or the setup action) | `warm-ci-dev`, `warm-ci-feat-rio`, `warm-ci-feat-proto`, `warm-ci-uring` | no verdict; sole writer of the `rust-cache` keys that `ci.yml` restores | no | dispatch `cache-warm.yml` | +| `build.yml` (nightly) | `build-rustfs` over the six-target platform matrix in `prepare-platform-matrix` (four Linux, macOS aarch64, Windows x86_64) | build/package integrity | yes | dispatch `build.yml` with an exact platform set | | `e2e-replication-nightly.yml` (nightly) | `repl-nightly`, `cluster-nightly`, `protocols-nightly` | three independent gates; JUnit, membership listing, server logs | yes | `python3 scripts/e2e_binary.py build --bins`, then `python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--features e2e-test-hooks` on both steps with `--profile e2e-nightly`; `--features ftps,webdav,sftp` on both steps with `-j 1 --profile e2e-protocols` | | `e2e-distributed.yml` (storage-sensitive PRs + nightly) | `distributed` | fail-closed 4-node 4-disk S3, durability, replication, movement, fault, and direct/rolling upgrade gate; JUnit, membership listing, per-node server logs | yes, with `never_ran_grace_until` | download the pinned previous release as in the workflow, export `RUSTFS_UPGRADE_SOURCE_BINARY`, run `python3 scripts/e2e_binary.py build --bins`, then `python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test` | | `e2e-s3tests.yml` (weekly) | `s3tests` (single and distributed, four shards each), `upstream-head-canary` | compatibility gate; report, JUnit, node IDs, server logs | yes | `scripts/s3-tests/run.sh` against an existing single or distributed target | @@ -126,7 +125,7 @@ Post-merge and tag-driven; not a substitute for a PR gate. | Trigger | Workflow / job | Result | |---|---|---| -| Push to `main`, weekly schedule, dispatch | `build.yml` `build-rustfs` (a development build on a main push restricts the matrix to the Linux targets) | build artifacts; no release publication | +| Nightly schedule, dispatch | `build.yml` `build-rustfs` (development build over the full platform matrix) | build artifacts and the dev download channel; no release publication | | Valid release or preview tag | `build.yml` `build-rustfs`, `create-release`, `upload-release-assets`, `publish-release` | draft release, checksummed assets, publish | | Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image` | multi-architecture images and vulnerability report | | Successful release-tag build (`workflow_run`) | `package.yml` `package` | DEB/RPM packages and checksums uploaded to the release | diff --git a/docs/testing/security-regressions.md b/docs/testing/security-regressions.md index 7b3a2588b..01badc82c 100644 --- a/docs/testing/security-regressions.md +++ b/docs/testing/security-regressions.md @@ -12,7 +12,7 @@ Every fixed RustFS GitHub Security Advisory maps to at least one named regressio | [GHSA-3p3x-734c-h5vx](https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx) | Constant-time secret comparison on WebDAV/FTPS password login | rustfs/rustfs#4403 | `assert_ftps_ghsa_3p3x_wrong_credentials_rejected` (`crates/e2e_test/src/protocols/ftps_core.rs`); `GHSA-3p3x` auth-failure block in `test_webdav_core_operations` (`crates/e2e_test/src/protocols/webdav_core.rs`) | e2e (`e2e-protocols`) | | [GHSA-r5qv-rc46-hv8q](https://github.com/rustfs/rustfs/security/advisories/GHSA-r5qv-rc46-hv8q) | Internode RPC authentication must fail closed | rustfs/rustfs#4402 | `ghsa_r5qv_resolve_shared_secret_rejects_default_fallback`, `ghsa_r5qv_verify_rpc_signature_fails_closed_on_missing_or_invalid_auth` (`crates/ecstore/src/cluster/rpc/http_auth.rs`) | unit | | [GHSA-m77q-r63m-pj89](https://github.com/rustfs/rustfs/security/advisories/GHSA-m77q-r63m-pj89) | STS JWTs signed with the shared root secret (intentionally unfixed) | n/a; tests pin the by-design behaviour and must flip red -> green when m77q is fixed | `test_ghsa_m77q_sts_session_token_signed_with_root_secret`, `test_created_sts_credentials_authorize_with_session_token_claims` (`crates/iam/src/sys.rs`); `token_signing_key` doc (`crates/iam/src/root_credentials.rs`) | unit | -| [GHSA-5354-r3w2-34m8](https://github.com/rustfs/rustfs/security/advisories/GHSA-5354-r3w2-34m8) | Service-account parent must stay within caller scope; a non-owner with `CreateServiceAccountAdminAction` could parent a service account to root | rustfs/rustfs#5141 | `ghsa_5354_non_owner_service_account_parent_confined_to_scope`, `ghsa_5354_scope_guard_matches_owner_or_self_scope_for_derived_credentials`, and the `add_service_account_parent_within_scope` invariant they pin (`rustfs/src/admin/handlers/service_account.rs`) | unit | +| [GHSA-5354-r3w2-34m8](https://github.com/rustfs/rustfs/security/advisories/GHSA-5354-r3w2-34m8) | Service-account parent must stay within caller scope; a non-owner with `CreateServiceAccountAdminAction` could parent a service account to root | rustfs/rustfs#5141 | `ghsa_5354_non_owner_service_account_parent_confined_to_scope`, `ghsa_5354_scope_guard_preserves_non_root_self_scope`, and the `add_service_account_parent_within_scope` invariant they pin (`rustfs/src/admin/handlers/service_account.rs`) | unit | | [GHSA-3ppv-fx5m-m749](https://github.com/rustfs/rustfs/security/advisories/GHSA-3ppv-fx5m-m749) | Versioned reads (`get_object`, CopyObject source, UploadPartCopy source) authorize against `s3:GetObjectVersion`, not `s3:GetObject` | rustfs/rustfs#5142 | `ghsa_3ppv_versioned_read_selects_get_object_version_action` and the `versioned_read_action` helper it pins (`rustfs/src/storage/access.rs`) | unit | | [GHSA-v9cp-qfw9-9pfp](https://github.com/rustfs/rustfs/security/advisories/GHSA-v9cp-qfw9-9pfp) | `ForAllValues:`/`ForAnyValue:` negated string operators applied negation to the aggregate instead of the per-value predicate | fixed, GHSA private-fork merge | `ghsa_v9cp_for_all_values_not_equals_partial_overlap`, `ghsa_v9cp_for_any_value_not_equals_partial_overlap` and the absent-key/positive-quantifier cases beside them (`crates/policy/tests/quantified_negation.rs`); the value set must partially overlap the policy set, since contained or disjoint sets cannot tell the quantifiers apart | crate test | | [GHSA-6r96-hmgc-726c](https://github.com/rustfs/rustfs/security/advisories/GHSA-6r96-hmgc-726c) | Request headers must not populate server-derived IAM condition keys (`userid`, `groups`, `jwt:`/`ldap:` claims) | fixed, GHSA private-fork merge | `ghsa_6r96_identity_condition_keys_ignore_spoofed_headers`, `ghsa_6r96_claim_condition_keys_ignore_spoofed_headers`, and `test_request_headers_still_reach_conditions`, which keeps the reserved set from growing too broad (`rustfs/src/auth.rs`) | unit | diff --git a/flake.nix b/flake.nix index a58fcd85f..fd9381e4e 100644 --- a/flake.nix +++ b/flake.nix @@ -77,7 +77,7 @@ rustfs = rustPlatform.buildRustPackage { pname = "rustfs"; - version = "1.0.1"; + version = "1.0.3"; src = ./.; diff --git a/helm/README.md b/helm/README.md index b4cec75c0..ae920fa09 100644 --- a/helm/README.md +++ b/helm/README.md @@ -32,6 +32,42 @@ You can override the inference by setting `drivesPerNode` explicitly, e.g. (with `--cascade=orphan` to keep pods and PVCs) and recreate it, or perform a full reinstall. +## Bring-your-own drives (JBOD) + +By default every drive is a PersistentVolumeClaim sized by +`storageclass.dataStorageSize`. To use disks that you provide yourself, such as +JBOD `hostPath` or local volumes, set `storageclass.dataStorageSize` to an empty +string. The chart then renders no data PVCs (no data `volumeClaimTemplates` in +distributed mode, no data PVC in standalone mode) and no data `volumeMounts`, +and you supply the drives through `extraVolumes` and `extraVolumeMounts`: + +```yaml +replicaCount: 3 +drivesPerNode: 4 +storageclass: + dataStorageSize: "" +extraVolumes: + - name: disk0 + hostPath: {path: /mnt/disk0, type: Directory} + # ... disk1 to disk3 +extraVolumeMounts: + - name: disk0 + mountPath: /data/rustfs0 + # ... /data/rustfs1 to /data/rustfs3 +``` + +Mount the drives at `/data/rustfs0` … `/data/rustfs` when +`drivesPerNode` is greater than 1, otherwise at `/data`, so they match the +generated `RUSTFS_VOLUMES`. The logs PVC is unaffected; set +`config.rustfs.obs_log_directory` to an empty string to drop it as well. Each +entry of `pools.list` may override `storageclass.dataStorageSize` +independently. In standalone mode, `mode.standalone.existingClaim.dataClaim` +still mounts the named claim as `data`. + +Switching `storageclass.dataStorageSize` between empty and non-empty changes +`volumeClaimTemplates`, so the same StatefulSet recreation rules as for +`drivesPerNode` apply. + --- ## Upgrade notes @@ -140,6 +176,9 @@ behavior. | config.rustfs.scanner.cache_save_timeout_secs | string | `""` | Override scanner cache save timeout in seconds with `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (minimum `1`). | | config.rustfs.scanner.max_concurrent_set_scans | string | `""` | Cap concurrent scanner set tasks with `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (`0` keeps topology-derived concurrency). | | config.rustfs.scanner.max_concurrent_disk_scans | string | `""` | Cap concurrent scanner disk bucket walks per set with `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (`0` keeps disk-count-derived concurrency). | +| config.rustfs.scanner.walkers_per_disk | string | `""` | Bucket walkers per local disk with `RUSTFS_SCANNER_WALKERS_PER_DISK` (`1` to `16`; the default `1` keeps one bucket walk per disk, raise it on flash so one disk walks several buckets concurrently). | +| config.rustfs.scanner.disk_io_budget_iops | string | `""` | Per-local-disk ceiling on scanner metadata operations per second with `RUSTFS_SCANNER_DISK_IO_BUDGET_IOPS` (`auto` picks the media default, `off` removes the cap, a number is an explicit rate). | +| config.rustfs.scanner.disk_io_budget_bytes_per_sec | string | `""` | Per-local-disk ceiling on scanner metadata bytes per second with `RUSTFS_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC` (`auto` picks the media default, `off` removes the cap, a number is an explicit rate). | | config.rustfs.scanner.yield_every_n_objects | string | `""` | Yield to the async runtime every N scanned objects with `RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS` (`0` disables extra yield). | | config.rustfs.scanner.alert_excess_versions | string | `""` | Set version count threshold for scanner alerts with `RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS`. | | config.rustfs.scanner.alert_excess_version_size | string | `""` | Set retained version byte threshold for scanner alerts with `RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE`. | @@ -208,6 +247,7 @@ behavior. | livenessProbe.periodSeconds | int | `5` | | | livenessProbe.successThreshold | int | `1` | | | livenessProbe.timeoutSeconds | int | `3` | | +| minReadySeconds | int | `45` | Seconds a replaced pod must stay Ready before the rolling update moves to the next pod. Covers the platform DNS TTL and survivor re-admission window after a restart (see `docs/operations/rolling-restart.md`). `0` disables. | | mode.distributed.enabled | bool | `true` | RustFS distributed mode support, namely multiple pod multiple pvc. | | mode.standalone.enabled | bool | `false` | RustFS standalone mode support, namely one pod one pvc. | | mode.standalone.existingClaim.dataClaim |string |`""` |Whether to use existing pvc claim for data storage. | diff --git a/helm/rustfs/Chart.yaml b/helm/rustfs/Chart.yaml index 72ad9abc8..ae3dd7e33 100644 --- a/helm/rustfs/Chart.yaml +++ b/helm/rustfs/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: rustfs description: RustFS helm chart to deploy RustFS on kubernetes cluster. type: application -version: "1.0.1" -appVersion: "1.0.1" +version: "1.0.3" +appVersion: "1.0.3" home: https://rustfs.com icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg maintainers: diff --git a/helm/rustfs/templates/configmap.yaml b/helm/rustfs/templates/configmap.yaml index e5aaa352f..09e79ba27 100644 --- a/helm/rustfs/templates/configmap.yaml +++ b/helm/rustfs/templates/configmap.yaml @@ -123,6 +123,15 @@ data: {{- end }} {{- if ne (toString .max_concurrent_disk_scans) "" }} RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS: {{ .max_concurrent_disk_scans | quote }} + {{- end }} + {{- if ne (toString .walkers_per_disk) "" }} + RUSTFS_SCANNER_WALKERS_PER_DISK: {{ .walkers_per_disk | quote }} + {{- end }} + {{- if ne (toString .disk_io_budget_iops) "" }} + RUSTFS_SCANNER_DISK_IO_BUDGET_IOPS: {{ .disk_io_budget_iops | quote }} + {{- end }} + {{- if ne (toString .disk_io_budget_bytes_per_sec) "" }} + RUSTFS_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC: {{ .disk_io_budget_bytes_per_sec | quote }} {{- end }} {{- if ne (toString .yield_every_n_objects) "" }} RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS: {{ .yield_every_n_objects | quote }} diff --git a/helm/rustfs/templates/deployment.yaml b/helm/rustfs/templates/deployment.yaml index 97b663626..22284b5a7 100644 --- a/helm/rustfs/templates/deployment.yaml +++ b/helm/rustfs/templates/deployment.yaml @@ -2,6 +2,9 @@ {{- $tlsEnabled := $tls.enabled | default false -}} {{- $logDir := .Values.config.rustfs.obs_log_directory }} {{- $logDirEnabled := ne $logDir "" }} +{{- /* With an empty storageclass.dataStorageSize and no existingClaim.dataClaim, + the data volume is supplied through extraVolumes/extraVolumeMounts. */}} +{{- $dataVolumeEnabled := or .Values.mode.standalone.existingClaim.dataClaim (not (empty .Values.storageclass.dataStorageSize)) }} {{- if .Values.mode.standalone.enabled }} apiVersion: apps/v1 @@ -74,6 +77,7 @@ spec: priorityClassName: {{ .Values.priorityClassName }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} + {{- if or $dataVolumeEnabled $logDirEnabled }} initContainers: - name: init-step image: "{{ .Values.image.initImage.repository }}:{{ .Values.image.initImage.tag }}" @@ -84,18 +88,23 @@ spec: - sh - -c - | + {{- if $dataVolumeEnabled }} mkdir -p /data + {{- end }} {{- if $logDirEnabled }} mkdir -p /mnt/rustfs/logs chmod 755 /mnt/rustfs/logs {{- end }} volumeMounts: + {{- if $dataVolumeEnabled }} - name: data mountPath: /data + {{- end }} {{- if $logDirEnabled }} - name: logs mountPath: /mnt/rustfs {{- end }} + {{- end }} containers: - name: {{ .Chart.Name }} image: "{{ .Values.image.rustfs.repository }}:{{ .Values.image.rustfs.tag | default .Chart.AppVersion }}" @@ -138,8 +147,10 @@ spec: mountPath: {{ $logDir }} subPath: logs {{- end }} + {{- if $dataVolumeEnabled }} - name: data mountPath: /data + {{- end }} volumes: {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} @@ -188,6 +199,7 @@ spec: claimName: {{ include "rustfs.fullname" $ }}-logs {{- end }} {{- end }} + {{- if $dataVolumeEnabled }} - name: data persistentVolumeClaim: {{- if .Values.mode.standalone.existingClaim.dataClaim }} @@ -195,4 +207,5 @@ spec: {{- else }} claimName: {{ include "rustfs.fullname" $ }}-data {{- end }} + {{- end }} {{- end }} diff --git a/helm/rustfs/templates/pvc.yaml b/helm/rustfs/templates/pvc.yaml index d2ddbd95a..ca7f49466 100644 --- a/helm/rustfs/templates/pvc.yaml +++ b/helm/rustfs/templates/pvc.yaml @@ -1,4 +1,4 @@ -{{- if and .Values.mode.standalone.enabled (not .Values.mode.standalone.existingClaim.dataClaim) }} +{{- if and .Values.mode.standalone.enabled (not .Values.mode.standalone.existingClaim.dataClaim) (not (empty .Values.storageclass.dataStorageSize)) }} {{- with .Values.storageclass }} apiVersion: v1 kind: PersistentVolumeClaim diff --git a/helm/rustfs/templates/statefulset.yaml b/helm/rustfs/templates/statefulset.yaml index 54de97694..bfd3cf3db 100644 --- a/helm/rustfs/templates/statefulset.yaml +++ b/helm/rustfs/templates/statefulset.yaml @@ -39,6 +39,10 @@ {{- $pools := include "rustfs.pools" . | fromJsonArray }} {{- range $pool := $pools }} {{- $drivesPerNode := int $pool.drives }} +{{- /* An empty storageclass.dataStorageSize means the data volumes are supplied + through extraVolumes/extraVolumeMounts (for example JBOD hostPath disks), + so the chart renders no data PVC templates and no data mounts. */}} +{{- $dataPvcEnabled := not (empty $pool.storageclass.dataStorageSize) }} --- apiVersion: apps/v1 kind: StatefulSet @@ -57,6 +61,13 @@ spec: serviceName: {{ include "rustfs.fullname" $ }}-headless replicas: {{ $pool.replicaCount }} podManagementPolicy: Parallel + {{- if gt (int $.Values.minReadySeconds) 0 }} + {{- /* A replaced pod reports Ready from its own view before surviving + peers have re-dialed its new IP (platform DNS TTL) and re-admitted + its drives. Hold the rollout so the next pod is not stopped while + survivors still count the previous one as unavailable (#8669). */}} + minReadySeconds: {{ int $.Values.minReadySeconds }} + {{- end }} selector: matchLabels: {{- include "rustfs.selectorLabels" $ | nindent 6 }} @@ -142,19 +153,23 @@ spec: priorityClassName: {{ $.Values.priorityClassName }} securityContext: {{- toYaml $.Values.podSecurityContext | nindent 8 }} + {{- if or $dataPvcEnabled $logDirEnabled }} initContainers: - name: init-step image: "{{ $.Values.image.initImage.repository }}:{{ $.Values.image.initImage.tag }}" imagePullPolicy: {{ $.Values.image.initImage.pullPolicy }} securityContext: {{- toYaml $.Values.containerSecurityContext | nindent 12 }} + {{- if $dataPvcEnabled }} env: - name: DRIVES_PER_NODE value: {{ $drivesPerNode | quote }} + {{- end }} command: - sh - -c - | + {{- if $dataPvcEnabled }} if [ "$DRIVES_PER_NODE" -gt 1 ]; then for i in $(seq 0 $(($DRIVES_PER_NODE - 1))); do mkdir -p /data/rustfs$i @@ -162,11 +177,13 @@ spec: else mkdir -p /data fi + {{- end }} {{- if $logDirEnabled }} mkdir -p /mnt/rustfs/logs chmod 755 /mnt/rustfs/logs {{- end }} volumeMounts: + {{- if $dataPvcEnabled }} {{- if gt $drivesPerNode 1 }} {{- range $i := until $drivesPerNode }} - name: data-rustfs-{{ $i }} @@ -176,10 +193,12 @@ spec: - name: data mountPath: /data {{- end }} + {{- end }} {{- if $logDirEnabled }} - name: logs mountPath: /mnt/rustfs {{- end }} + {{- end }} containers: - name: {{ $.Chart.Name }} image: "{{ $.Values.image.rustfs.repository }}:{{ $.Values.image.rustfs.tag | default $.Chart.AppVersion }}" @@ -232,6 +251,7 @@ spec: mountPath: {{ $logDir }} subPath: logs {{- end }} + {{- if $dataPvcEnabled }} {{- if gt $drivesPerNode 1 }} {{- range $i := until $drivesPerNode }} - name: data-rustfs-{{ $i }} @@ -241,6 +261,7 @@ spec: - name: data mountPath: /data {{- end }} + {{- end }} {{- if or $.Values.mtls.enabled $tlsEnabled $.Values.extraVolumes }} volumes: {{- if or $.Values.mtls.enabled $tlsEnabled }} @@ -282,6 +303,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} {{- end }} + {{- if or $logDirEnabled $dataPvcEnabled }} volumeClaimTemplates: {{- if $logDirEnabled }} - apiVersion: v1 @@ -302,6 +324,7 @@ spec: requests: storage: {{ $pool.storageclass.logStorageSize }} {{- end }} + {{- if $dataPvcEnabled }} {{- if gt $drivesPerNode 1 }} {{- range $i := until $drivesPerNode }} - apiVersion: v1 @@ -341,5 +364,7 @@ spec: requests: storage: {{ $pool.storageclass.dataStorageSize }} {{- end }} + {{- end }} + {{- end }} {{- end }} {{- end }} diff --git a/helm/rustfs/values.yaml b/helm/rustfs/values.yaml index de0c4b7f1..14ce3b648 100644 --- a/helm/rustfs/values.yaml +++ b/helm/rustfs/values.yaml @@ -8,7 +8,9 @@ replicaCount: 4 # Number of data drives (PVCs) per pod. Combined with replicaCount this -# determines the total drive count seen by rustfs. +# determines the total drive count seen by rustfs. When +# storageclass.dataStorageSize is empty no PVCs are created and the drives come +# from extraVolumes/extraVolumeMounts instead. # - drivesPerNode > 1: each pod gets N volumes mounted at /data/rustfs0.../data/rustfsN-1 # - drivesPerNode = 1: each pod gets a single volume mounted at /data # @@ -185,6 +187,12 @@ config: max_concurrent_set_scans: "" # Cap concurrent scanner disk bucket walks per set (0 keeps disk-count-derived concurrency) max_concurrent_disk_scans: "" + # Bucket walkers per local disk (1-16; default 1 keeps one bucket walk per disk, raise on flash) + walkers_per_disk: "" + # Per-local-disk scanner I/O ceiling in operations per second (auto, off, or a number) + disk_io_budget_iops: "" + # Per-local-disk scanner I/O ceiling in bytes per second (auto, off, or a number) + disk_io_budget_bytes_per_sec: "" # Yield to the async runtime every N scanned objects (0 disables extra yield) yield_every_n_objects: "" # Version count threshold for scanner alerts @@ -443,6 +451,15 @@ readinessProbe: successThreshold: 1 failureThreshold: 3 +# Seconds a replaced pod must stay Ready before the StatefulSet rolling update +# moves to the next pod. A restarted node is Ready as soon as it can reach the +# cluster, but surviving nodes keep dialing its previous IP until the platform +# DNS record expires (kube-dns serves 30s TTLs, CoreDNS 5s by default) and then +# need a few more probes before they count its drives again. Stopping the next +# pod inside that window takes two members of an erasure set away from the +# survivors (#8669). Set to 0 to disable. +minReadySeconds: 45 + nodeSelector: {} tolerations: [] @@ -465,6 +482,22 @@ topologySpreadConstraints: storageclass: name: local-path + # Size of each data PVC. Set it to an empty string ("") to stop the chart from + # creating data PersistentVolumeClaims: the StatefulSet then renders no data + # volumeClaimTemplates (standalone mode renders no data PVC) and no data + # volumeMounts, so the drives must come from extraVolumes/extraVolumeMounts, + # for example JBOD hostPath or local disks. Mount them at the paths RustFS + # expects: /data/rustfs0 ... /data/rustfs when drivesPerNode + # is greater than 1, otherwise /data. Keep drivesPerNode and replicaCount + # accurate because they still drive the generated RUSTFS_VOLUMES. A pool + # under pools.list can override this value, including with "". Standalone + # mode keeps the data volume when mode.standalone.existingClaim.dataClaim is + # set. logStorageSize is independent: the logs PVC is still created unless + # config.rustfs.obs_log_directory is empty. + # WARNING: switching between an empty and a non-empty value changes the + # StatefulSet volumeClaimTemplates, which Kubernetes forbids updating in + # place. Delete the StatefulSet with --cascade=orphan and recreate it, or + # reinstall. dataStorageSize: 256Mi logStorageSize: 256Mi pvcAnnotations: diff --git a/rustfs.spec b/rustfs.spec index c3b00b3c2..5b4bef67b 100644 --- a/rustfs.spec +++ b/rustfs.spec @@ -1,7 +1,7 @@ %global _enable_debug_packages 0 %global _empty_manifest_terminate_build 0 Name: rustfs -Version: 1.0.1 +Version: 1.0.3 Release: 1 Summary: High-performance distributed object storage for MinIO alternative @@ -57,6 +57,12 @@ install %_builddir/%{name}-%{version}/target/%_arch/%_arch-unknown-linux-gnu/rel %_bindir/rustfs %changelog +* Sun Oct 18 2026 overtrue +- Update RPM package to RustFS 1.0.3 + +* Fri Oct 16 2026 overtrue +- Update RPM package to RustFS 1.0.2 + * Sat Oct 03 2026 overtrue - Update RPM package to RustFS 1.0.1 diff --git a/rustfs/Cargo.toml b/rustfs/Cargo.toml index 04680f2cb..7338ed8b3 100644 --- a/rustfs/Cargo.toml +++ b/rustfs/Cargo.toml @@ -77,6 +77,10 @@ rio-v2 = ["rustfs-ecstore/rio-v2"] pyroscope = ["rustfs-obs/pyroscope", "dep:pyroscope", "dep:object", "dep:symbolic-demangle"] # Tokio runtime telemetry. Requires `--cfg tokio_unstable`; use `make build-profiling`. dial9 = ["rustfs-obs/dial9"] +# Diagnostic-only Rust GlobalAlloc live-byte accounting; adds per-allocation atomic updates. +diagnostic-allocator-live = [] +# Diagnostic-only S3-operation attribution for internode ReadVersion and list-walker metrics. +diagnostic-read-version-caller = ["rustfs-ecstore/diagnostic-read-version-caller", "rustfs-io-metrics/diagnostic-read-version-caller", "rustfs-scanner/diagnostic-read-version-caller"] http3 = ["dep:h3", "dep:h3-quinn", "dep:quinn"] hotpath = [ "hotpath/hotpath", diff --git a/rustfs/build.rs b/rustfs/build.rs index a302de37d..2f109366f 100644 --- a/rustfs/build.rs +++ b/rustfs/build.rs @@ -12,6 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::io::Write; + fn main() -> shadow_rs::SdResult<()> { println!("cargo:rerun-if-env-changed=RUSTFS_BUILD_VERSION"); if let Ok(version) = std::env::var("RUSTFS_BUILD_VERSION") @@ -24,6 +26,38 @@ fn main() -> shadow_rs::SdResult<()> { println!("cargo:rustc-env=RUSTFS_BUILD_VERSION={version}"); } - shadow_rs::ShadowBuilder::builder().build()?; + println!("cargo:rerun-if-env-changed=RUSTC"); + let target = std::env::var("TARGET")?; + let target_os = std::env::var("CARGO_CFG_TARGET_OS")?; + let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH")?; + let output = std::process::Command::new(std::env::var("RUSTC")?).arg("-vV").output()?; + if !output.status.success() { + return Err(std::io::Error::other("failed to query the build compiler").into()); + } + let output = String::from_utf8(output.stdout)?; + let release = output + .lines() + .find_map(|line| line.strip_prefix("release: ")) + .ok_or_else(|| std::io::Error::other("build compiler did not report its release"))?; + let channel = match release.split_once('-') { + Some((_, channel)) if matches!(channel.split('.').next(), Some("beta" | "nightly" | "dev")) => { + channel.split('.').next().unwrap() + } + None if release.split('.').count() == 3 && release.split('.').all(|part| part.parse::().is_ok()) => "stable", + _ => return Err(std::io::Error::other("unrecognized build compiler release channel").into()), + }; + + // Generate target metadata once so every shadow-rs consumer agrees, even + // when the build script and compiler run on a different host platform. + let mut deny = shadow_rs::default_deny(); + deny.extend([shadow_rs::BUILD_OS, shadow_rs::RUST_CHANNEL]); + shadow_rs::ShadowBuilder::builder() + .deny_const(deny) + .hook(|file: &std::fs::File| { + writeln!(&*file, "pub const BUILD_OS: &str = {:?};", format!("{target_os}-{target_arch}"))?; + writeln!(&*file, "pub const RUST_CHANNEL: &str = {:?};", format!("{channel}-{target}"))?; + Ok(()) + }) + .build()?; Ok(()) } diff --git a/rustfs/src/admin/console.rs b/rustfs/src/admin/console.rs index 4019c9876..f03a3f3ee 100644 --- a/rustfs/src/admin/console.rs +++ b/rustfs/src/admin/console.rs @@ -1051,10 +1051,12 @@ mod tests { .uri(format!("{CONSOLE_PREFIX}/index.html")) .body(Body::empty()) .expect("failed to build request"); - request.extensions_mut().insert(crate::server::RemoteAddr(SocketAddr::new( - IpAddr::from([198, 51, 100, last_octet]), - 50000, - ))); + request + .extensions_mut() + .insert(Some(crate::server::RemoteAddr(SocketAddr::new( + IpAddr::from([198, 51, 100, last_octet]), + 50000, + )))); request }; diff --git a/rustfs/src/admin/handlers/bucket_meta.rs b/rustfs/src/admin/handlers/bucket_meta.rs index 1a23d64dd..8641d2c05 100644 --- a/rustfs/src/admin/handlers/bucket_meta.rs +++ b/rustfs/src/admin/handlers/bucket_meta.rs @@ -525,10 +525,13 @@ impl Operation for ImportBucketMetadata { // First pass: read all file contents into memory let mut file_contents = Vec::new(); for i in 0..zip_reader.len() { - let mut file = zip_reader + let (mut file, file_path) = zip_reader .by_index(i) + .and_then(|file| { + let path = file.name()?.into_owned(); + Ok((file, path)) + }) .map_err(|e| s3_error!(InternalError, "failed to read archive entry: {e}"))?; - let file_path = file.name().to_string(); let mut content = Vec::new(); file.read_to_end(&mut content) @@ -1564,7 +1567,7 @@ mod backup_zip_compatibility_tests { let mut content = Vec::new(); file.read_to_end(&mut content).expect("read diagnostic config"); assert!(!content.windows(RAW_SECRET.len()).any(|window| window == RAW_SECRET)); - files.insert(file.name().to_string(), content); + files.insert(file.name().expect("diagnostic entry name").into_owned(), content); } assert!(!files.contains_key(&format!("_diagnostic/{BUCKET}/{config_file}"))); assert_eq!(files[&format!("_diagnostic/{BUCKET}/{BUCKET_VERSIONING_CONFIG}")], VERSIONING_XML); @@ -1701,8 +1704,9 @@ mod backup_zip_compatibility_tests { let mut content = Vec::new(); file.read_to_end(&mut content).expect("read diagnostic config"); assert!(!content.windows(SECRET.len()).any(|window| window == SECRET.as_bytes())); - assert!(file.name() == DIAGNOSTIC_EXPORT_MANIFEST || file.name().starts_with("_diagnostic/")); - files.insert(file.name().to_string(), content); + let file_path = file.name().expect("diagnostic entry name").into_owned(); + assert!(file_path == DIAGNOSTIC_EXPORT_MANIFEST || file_path.starts_with("_diagnostic/")); + files.insert(file_path, content); } assert_eq!(files[&format!("_diagnostic/{HEALTHY}/{BUCKET_VERSIONING_CONFIG}")], VERSIONING_XML); assert!(files.contains_key(&format!("_diagnostic/{UNREADABLE}/{BUCKET_VERSIONING_CONFIG}"))); diff --git a/rustfs/src/admin/handlers/config_admin.rs b/rustfs/src/admin/handlers/config_admin.rs index 15bcd7763..f8d99aaee 100644 --- a/rustfs/src/admin/handlers/config_admin.rs +++ b/rustfs/src/admin/handlers/config_admin.rs @@ -66,17 +66,19 @@ use rustfs_config::{ BASE_DSN_STRING, COMMENT_KEY, DEFAULT_DELIMITER, ENABLE_KEY, ENV_PREFIX, ENV_SCANNER_ALERT_EXCESS_FOLDERS, ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, ENV_SCANNER_ALERT_EXCESS_VERSIONS, ENV_SCANNER_BITROT_CYCLE_SECS, ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, ENV_SCANNER_CYCLE, ENV_SCANNER_CYCLE_MAX_DIRECTORIES, - ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, ENV_SCANNER_IDLE_MODE, + ENV_SCANNER_CYCLE_MAX_DURATION_SECS, ENV_SCANNER_CYCLE_MAX_OBJECTS, ENV_SCANNER_DELAY, + ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, ENV_SCANNER_DISK_IO_BUDGET_IOPS, ENV_SCANNER_IDLE_MODE, ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, ENV_SCANNER_MAX_WAIT_SECS, ENV_SCANNER_SPEED, - ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, HEAL_SUB_SYS, - MAX_ADMIN_REQUEST_BODY_SIZE, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, + ENV_SCANNER_START_DELAY_SECS, ENV_SCANNER_WALKERS_PER_DISK, ENV_SCANNER_YIELD_EVERY_N_OBJECTS, HEAL_BITROT_CYCLE, + HEAL_SUB_SYS, MAX_ADMIN_REQUEST_BODY_SIZE, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TOPIC, MQTT_USERNAME, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE, SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, - SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, - SCANNER_START_DELAY, SCANNER_SUB_SYS, SCANNER_YIELD_EVERY_N_OBJECTS, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, - WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, + SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, SCANNER_DISK_IO_BUDGET_IOPS, SCANNER_IDLE_MODE, SCANNER_MAX_CONCURRENT_DISK_SCANS, + SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY, SCANNER_SUB_SYS, + SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CERT, + WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, + WEBHOOK_RETRY_INTERVAL, }; use rustfs_credentials::Credentials; use rustfs_policy::policy::action::{Action, AdminAction}; @@ -286,6 +288,24 @@ const SCANNER_HELP_KEYS: &[HelpKeyMetadata] = &[ description: "cap concurrent disk bucket walks per set, 0 uses disk-count defaults", optional: true, }, + HelpKeyMetadata { + key: SCANNER_WALKERS_PER_DISK, + type_name: "number", + description: "bucket walkers per local disk, 1 through 16; remote disks keep one walker", + optional: true, + }, + HelpKeyMetadata { + key: SCANNER_DISK_IO_BUDGET_IOPS, + type_name: "string", + description: "per-local-disk ceiling on scanner metadata operations per second: auto picks the media default, off removes the cap, a number is an explicit rate", + optional: true, + }, + HelpKeyMetadata { + key: SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC, + type_name: "string", + description: "per-local-disk ceiling on scanner metadata bytes per second: auto picks the media default, off removes the cap, a number is an explicit rate", + optional: true, + }, HelpKeyMetadata { key: SCANNER_YIELD_EVERY_N_OBJECTS, type_name: "number", @@ -1619,6 +1639,9 @@ fn env_help_key(sub_system: &str, key: &str) -> String { (SCANNER_SUB_SYS, SCANNER_CACHE_SAVE_TIMEOUT) => ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS.to_string(), (SCANNER_SUB_SYS, SCANNER_MAX_CONCURRENT_SET_SCANS) => ENV_SCANNER_MAX_CONCURRENT_SET_SCANS.to_string(), (SCANNER_SUB_SYS, SCANNER_MAX_CONCURRENT_DISK_SCANS) => ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(), + (SCANNER_SUB_SYS, SCANNER_WALKERS_PER_DISK) => ENV_SCANNER_WALKERS_PER_DISK.to_string(), + (SCANNER_SUB_SYS, SCANNER_DISK_IO_BUDGET_IOPS) => ENV_SCANNER_DISK_IO_BUDGET_IOPS.to_string(), + (SCANNER_SUB_SYS, SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC) => ENV_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC.to_string(), (SCANNER_SUB_SYS, SCANNER_YIELD_EVERY_N_OBJECTS) => ENV_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(), (SCANNER_SUB_SYS, SCANNER_ALERT_EXCESS_VERSIONS) => ENV_SCANNER_ALERT_EXCESS_VERSIONS.to_string(), (SCANNER_SUB_SYS, SCANNER_ALERT_EXCESS_VERSION_SIZE) => ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE.to_string(), diff --git a/rustfs/src/admin/handlers/heal.rs b/rustfs/src/admin/handlers/heal.rs index f6c02aa0c..f0ef8c491 100644 --- a/rustfs/src/admin/handlers/heal.rs +++ b/rustfs/src/admin/handlers/heal.rs @@ -673,63 +673,80 @@ where } } -fn canonical_replacement_records( - records: &[rustfs_heal::ReplacementRecoveryRecord], -) -> Vec { - let mut records = records.to_vec(); - for record in &mut records { - record.target_slots.sort(); +fn reconcile_replacement_recovery_records( + snapshots: Vec, +) -> (Vec, bool) { + struct Observation { + record: rustfs_heal::ReplacementRecoveryRecord, + sources: Vec, } - records.sort_by(|left, right| { - ( - &left.task_id, - &left.generation, - &left.set_disk_id, - left.state, - &left.target_slots, - &left.verified_at, - &left.reason, - ) - .cmp(&( - &right.task_id, - &right.generation, - &right.set_disk_id, - right.state, - &right.target_slots, - &right.verified_at, - &right.reason, - )) - }); - records -} -fn merge_replacement_recovery_records( - snapshots: &[rustfs_heal::ReplacementRecoverySnapshot], -) -> Vec { - let mut records = BTreeMap::::new(); - for snapshot in snapshots { - for record in canonical_replacement_records(&snapshot.records) { - match records.entry(record.task_id.clone()) { - std::collections::btree_map::Entry::Vacant(entry) => { - entry.insert(record); - } - std::collections::btree_map::Entry::Occupied(entry) if entry.get() == &record => {} + let mut observations = BTreeMap::::new(); + for (source, snapshot) in snapshots.into_iter().enumerate() { + for mut record in snapshot.records { + record.target_slots.sort(); + let observation = match observations.entry(record.task_id.clone()) { + std::collections::btree_map::Entry::Vacant(entry) => entry.insert(Observation { + record, + sources: Vec::new(), + }), std::collections::btree_map::Entry::Occupied(mut entry) => { - let task_id = entry.key().clone(); - entry.insert(rustfs_heal::ReplacementRecoveryRecord { - task_id, - state: rustfs_heal::ReplacementRecoveryState::Unknown, - generation: None, - set_disk_id: None, - target_slots: Vec::new(), - reason: Some("peer replacement recovery records disagree".to_string()), - verified_at: None, - }); + if entry.get().record != record { + entry.get_mut().record = rustfs_heal::ReplacementRecoveryRecord { + task_id: entry.key().clone(), + state: rustfs_heal::ReplacementRecoveryState::Unknown, + generation: None, + set_disk_id: None, + target_slots: Vec::new(), + reason: Some("peer replacement recovery records disagree".to_string()), + verified_at: None, + }; + } + entry.into_mut() } + }; + // Snapshot order keeps sources sorted; repeated copies within one + // snapshot must not duplicate a generation's provenance. + if snapshot.definitive && observation.sources.last() != Some(&source) { + observation.sources.push(source); } } } - records.into_values().collect() + + let conflict = if observations + .values() + .any(|observation| observation.record.state == rustfs_heal::ReplacementRecoveryState::Unknown) + { + true + } else { + let mut target_sources = BTreeMap::<&str, Vec<&[usize]>>::new(); + for observation in observations.values() { + for target in &observation.record.target_slots { + target_sources.entry(target).or_default().push(&observation.sources); + } + } + target_sources.into_values().any(|sources| { + if sources.len() < 2 { + return false; + } + // A complete retained history normally has one common anchor. + // Checking that witness avoids comparing every historical pair. + if sources[0] + .iter() + .any(|source| sources[1..].iter().all(|other| other.binary_search(source).is_ok())) + { + return false; + } + // Pairwise shared history is sufficient even without one anchor + // containing the whole group. A transitive chain alone is not. + sources.iter().enumerate().any(|(index, left)| { + sources[index + 1..] + .iter() + .any(|right| !left.iter().any(|source| right.binary_search(source).is_ok())) + }) + }) + }; + (observations.into_values().map(|observation| observation.record).collect(), conflict) } fn aggregate_replacement_recovery_cluster_status( @@ -762,24 +779,15 @@ fn aggregate_replacement_recovery_cluster_status( reason.get_or_insert("peer_replacement_recovery_status_not_definitive"); } - // Replacement recovery state is anchored on one surviving disk. Nodes - // without that disk legitimately report a definitive empty snapshot, so - // only non-empty snapshots must agree with each other. - let mut non_empty_records = snapshots - .iter() - .filter(|snapshot| !snapshot.records.is_empty()) - .map(|snapshot| canonical_replacement_records(&snapshot.records)); - if let Some(first_records) = non_empty_records.next() - && non_empty_records.any(|records| records != first_records) - { - reason.get_or_insert("peer_replacement_recovery_status_conflict"); - } - - let records = merge_replacement_recovery_records(&snapshots); - if records - .iter() - .any(|record| matches!(record.state, rustfs_heal::ReplacementRecoveryState::Unknown)) - { + // Replacement completion proofs are anchored per generation on one + // surviving disk, so sequential automatic generations legitimately leave + // different completed tasks on different peers, and peers without an + // anchor report definitive empty snapshots. Peers conflict only when they + // disagree about one task (the merge marks those copies `Unknown`) or + // claim overlapping targets without a shared definitive local history. + // Retained histories and disjoint generations can be merged. + let (records, conflict) = reconcile_replacement_recovery_records(snapshots); + if conflict { reason.get_or_insert("peer_replacement_recovery_status_conflict"); } @@ -1390,6 +1398,9 @@ pub(crate) struct ReplacementRecoveryStatusResponse { pub path: String, pub scope: &'static str, pub local: rustfs_heal::ReplacementRecoverySnapshot, + /// Generations this node's discovery could not verify or resolve, with + /// reasons. Node-local and additive; it is not part of the peer merge. + pub local_blocked: Vec, pub cluster: ReplacementRecoveryClusterStatus, } @@ -1403,6 +1414,7 @@ pub(crate) struct ReplacementRecoveryClusterStatus { fn build_replacement_recovery_status_response( local: rustfs_heal::ReplacementRecoverySnapshot, + local_blocked: Vec, cluster: ReplacementRecoveryClusterStatus, ) -> ReplacementRecoveryStatusResponse { ReplacementRecoveryStatusResponse { @@ -1410,6 +1422,7 @@ fn build_replacement_recovery_status_response( path: format!("{}{}", ADMIN_PREFIX, REPLACEMENT_RECOVERY_STATUS_ROUTE_SUFFIX), scope: "cluster_survivor_disks", local, + local_blocked, cluster, } } @@ -1910,7 +1923,8 @@ impl Operation for ReplacementRecoveryStatusHandler { let local = rustfs_heal::current_replacement_recovery_snapshot().await; let cluster = read_cluster_replacement_recovery_status(local.clone(), notification_system.as_deref(), expected_nodes).await?; - let response = build_replacement_recovery_status_response(local, cluster); + let local_blocked = rustfs_heal::current_replacement_recovery_blocks(); + let response = build_replacement_recovery_status_response(local, local_blocked, cluster); let body = encode_replacement_recovery_status(&response)?; info!( event = EVENT_ADMIN_RESPONSE_EMITTED, @@ -2195,6 +2209,14 @@ mod tests { } } + fn replacement_snapshot_with_retained_history() -> rustfs_heal::ReplacementRecoverySnapshot { + let mut snapshot = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + snapshot + .records + .push(replacement_record("22222222-2222-4222-8222-222222222222")); + snapshot + } + #[tokio::test] async fn heal_selector_preflight_rejects_before_probe_and_token_creation() { use crate::admin::storage_api::runtime::PoolEndpoints; @@ -2345,7 +2367,7 @@ mod tests { fn replacement_recovery_status_response_reports_cluster_proof() { let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], Vec::new(), 1, true); - let response = build_replacement_recovery_status_response(local, cluster); + let response = build_replacement_recovery_status_response(local, Vec::new(), cluster); let value = serde_json::to_value(response).expect("replacement status response should serialize"); assert_eq!(value["contractVersion"], 2); @@ -2353,6 +2375,7 @@ mod tests { assert_eq!(value["scope"], "cluster_survivor_disks"); assert_eq!(value["local"]["definitive"], true); assert_eq!(value["local"]["records"][0]["state"], "completed"); + assert_eq!(value["localBlocked"], serde_json::json!([])); assert_eq!(value["cluster"]["definitive"], true); assert!(value["cluster"]["reason"].is_null()); assert_eq!(value["cluster"]["records"][0]["state"], "completed"); @@ -2369,7 +2392,9 @@ mod tests { } #[test] - fn replacement_recovery_cluster_requires_matching_peer_records() { + fn replacement_recovery_cluster_rejects_overlapping_targets_without_shared_history() { + // Separate anchors cannot establish that these overlapping records + // describe compatible historical replacement instances. let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); let peer = replacement_snapshot("22222222-2222-4222-8222-222222222222"); let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); @@ -2379,6 +2404,247 @@ mod tests { assert_eq!(cluster.records.len(), 2); } + #[test] + fn replacement_recovery_cluster_merges_disjoint_generations_across_anchors() { + // rustfs/rustfs#8366: sequential automatic generations anchor their + // completion proofs on different surviving disks, so peers hold + // disjoint completed tasks and the remaining peers hold none at all. + let mut rebuilt_record = replacement_record("11111111-1111-4111-8111-111111111111"); + rebuilt_record.target_slots = vec!["http://node-a:9000/mnt/disk2".to_string()]; + let mut anchor_record = replacement_record("22222222-2222-4222-8222-222222222222"); + anchor_record.target_slots = vec!["http://node-a:9000/mnt/disk1".to_string()]; + let rebuilt = rustfs_heal::ReplacementRecoverySnapshot { + records: vec![rebuilt_record], + definitive: true, + reason: None, + }; + let anchor = rustfs_heal::ReplacementRecoverySnapshot { + records: vec![anchor_record], + definitive: true, + reason: None, + }; + let empty = rustfs_heal::ReplacementRecoverySnapshot { + records: Vec::new(), + definitive: true, + reason: None, + }; + let cluster = + aggregate_replacement_recovery_cluster_status(vec![rebuilt], vec![Ok(Some(anchor)), Ok(Some(empty))], 3, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records.len(), 2); + assert!( + cluster + .records + .iter() + .all(|record| matches!(record.state, rustfs_heal::ReplacementRecoveryState::Completed)) + ); + } + + #[test] + fn replacement_recovery_cluster_rejects_peer_disagreement_about_one_task() { + let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + let mut disagreeing = replacement_record("11111111-1111-4111-8111-111111111111"); + disagreeing.set_disk_id = Some("pool_0_set_1".to_string()); + let peer = rustfs_heal::ReplacementRecoverySnapshot { + records: vec![disagreeing], + definitive: true, + reason: None, + }; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records[0].state, rustfs_heal::ReplacementRecoveryState::Unknown); + } + + #[test] + fn replacement_recovery_cluster_accepts_retained_history_on_one_anchor() { + let local = replacement_snapshot_with_retained_history(); + let empty = rustfs_heal::ReplacementRecoverySnapshot { + records: Vec::new(), + definitive: true, + reason: None, + }; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], vec![Ok(Some(empty))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, local.records); + } + + #[test] + fn replacement_recovery_cluster_accepts_matching_retained_histories() { + let local = replacement_snapshot_with_retained_history(); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], vec![Ok(Some(local.clone()))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, local.records); + } + + #[test] + fn replacement_recovery_cluster_accepts_committed_handoff_history() { + let mut local = replacement_snapshot_with_retained_history(); + let successor = local.records[1].task_id.clone(); + let predecessor = &mut local.records[0]; + predecessor.state = rustfs_heal::ReplacementRecoveryState::Unrecoverable; + predecessor.reason = Some(format!("replacement responsibility transferred to {successor}")); + predecessor.verified_at = None; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local.clone()], Vec::new(), 1, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, local.records); + } + + #[test] + fn replacement_recovery_cluster_merges_retained_history_with_disjoint_peer_generation() { + let local = replacement_snapshot_with_retained_history(); + let mut peer = replacement_snapshot("33333333-3333-4333-8333-333333333333"); + peer.records[0].target_slots = vec!["http://node-a:9000/mnt/disk2".to_string()]; + let mut expected = local.records.clone(); + expected.extend(peer.records.clone()); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + + #[test] + fn replacement_recovery_cluster_accepts_retained_history_reported_only_by_peer() { + let local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + let peer = replacement_snapshot_with_retained_history(); + let expected = peer.records.clone(); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + + #[test] + fn replacement_recovery_cluster_rejects_overlap_outside_retained_history() { + let local = replacement_snapshot_with_retained_history(); + let peer = replacement_snapshot("33333333-3333-4333-8333-333333333333"); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records.len(), 3); + } + + #[test] + fn replacement_recovery_cluster_rejects_disagreeing_copies_in_retained_history() { + let local = replacement_snapshot_with_retained_history(); + let mut peer = local.clone(); + peer.records[0].verified_at = Some(43); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records.len(), 2); + assert_eq!(cluster.records[0].state, rustfs_heal::ReplacementRecoveryState::Unknown); + assert_eq!(cluster.records[1].state, rustfs_heal::ReplacementRecoveryState::Completed); + } + + #[test] + fn replacement_recovery_cluster_reconciles_target_and_source_matrix() { + let task_ids = [ + "11111111-1111-4111-8111-111111111111", + "22222222-2222-4222-8222-222222222222", + "33333333-3333-4333-8333-333333333333", + ]; + for targets in 0u8..8 { + let mut records = task_ids.map(replacement_record); + for (task, record) in records.iter_mut().enumerate() { + record.target_slots = vec![format!("http://node-a:9000/mnt/disk{}", (targets >> task) & 1)]; + } + // All non-empty source subsets cover pairwise histories without + // one common anchor, transitive-only histories, and empty peers. + for a in 1u8..8 { + for b in 1u8..8 { + for c in 1u8..8 { + let masks = [a, b, c]; + let mut snapshots = (0..3).map(|source| rustfs_heal::ReplacementRecoverySnapshot { + records: records + .iter() + .enumerate() + .filter(|(task, _)| masks[*task] & (1 << source) != 0) + .map(|(_, record)| record.clone()) + .collect(), + definitive: true, + reason: None, + }); + let expected = (0..3).all(|left| { + (left + 1..3).all(|right| { + records[left].target_slots != records[right].target_slots || masks[left] & masks[right] != 0 + }) + }); + let local = snapshots.next().expect("three source snapshots"); + let cluster = aggregate_replacement_recovery_cluster_status( + vec![local], + snapshots.map(|snapshot| Ok(Some(snapshot))).collect(), + 3, + true, + ); + + assert_eq!(cluster.definitive, expected, "targets={targets}, sources={masks:?}"); + assert_eq!(cluster.reason.is_none(), expected); + assert_eq!(cluster.records, records); + } + } + } + } + } + + #[test] + fn replacement_recovery_cluster_rejects_overlap_on_a_secondary_target() { + let mut local = replacement_snapshot_with_retained_history(); + local.records[0].target_slots.push("http://node-a:9000/mnt/disk2".to_string()); + let mut peer = replacement_snapshot("33333333-3333-4333-8333-333333333333"); + peer.records[0].target_slots = vec!["http://node-a:9000/mnt/disk2".to_string()]; + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(!cluster.definitive); + assert_eq!(cluster.reason.as_deref(), Some("peer_replacement_recovery_status_conflict")); + assert_eq!(cluster.records.len(), 3); + } + + #[test] + fn replacement_recovery_cluster_deduplicates_reordered_targets() { + let mut local = replacement_snapshot("11111111-1111-4111-8111-111111111111"); + local.records[0].target_slots.push("http://node-a:9000/mnt/disk2".to_string()); + let mut peer = local.clone(); + peer.records[0].target_slots.reverse(); + let expected = local.records.clone(); + let cluster = aggregate_replacement_recovery_cluster_status(vec![local], vec![Ok(Some(peer))], 2, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + + #[test] + fn replacement_recovery_cluster_accepts_history_from_a_high_index_peer() { + let empty = rustfs_heal::ReplacementRecoverySnapshot { + records: Vec::new(), + definitive: true, + reason: None, + }; + let mut peers = vec![Ok(Some(empty.clone())); 80]; + let history = replacement_snapshot_with_retained_history(); + let expected = history.records.clone(); + peers.push(Ok(Some(history))); + let cluster = aggregate_replacement_recovery_cluster_status(vec![empty], peers, 82, true); + + assert!(cluster.definitive); + assert!(cluster.reason.is_none()); + assert_eq!(cluster.records, expected); + } + #[test] fn replacement_recovery_cluster_accepts_definitive_empty_peers() { let local = rustfs_heal::ReplacementRecoverySnapshot { diff --git a/rustfs/src/admin/handlers/oidc.rs b/rustfs/src/admin/handlers/oidc.rs index 8a095d2e1..c7151b839 100644 --- a/rustfs/src/admin/handlers/oidc.rs +++ b/rustfs/src/admin/handlers/oidc.rs @@ -24,7 +24,7 @@ use hyper::Method; use matchit::Params; use rustfs_config::{ENV_RUSTFS_BROWSER_REDIRECT_URL, MAX_ADMIN_REQUEST_BODY_SIZE}; use rustfs_iam::federation::{FederatedSessionBindingError, FederationError}; -use rustfs_iam::oidc::{OidcProviderConfigInput, OidcProviderValidationInput}; +use rustfs_iam::oidc::{OidcProviderConfigInput, OidcProviderValidationInput, console_session_duration_secs}; use rustfs_policy::policy::action::{Action, AdminAction}; use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error}; use serde::Serialize; @@ -450,8 +450,9 @@ impl Operation for OidcCallbackHandler { let redirect_uri = derive_callback_uri(&req, provider_id)?; + let session_duration_seconds = console_session_duration_secs(); let login = federation - .complete_authorization_code(&state, &code, &redirect_uri, 3600, &DefaultFederatedSessionBinding) + .complete_authorization_code(&state, &code, &redirect_uri, session_duration_seconds, &DefaultFederatedSessionBinding) .await .map_err(|error| { if let FederationError::CodeExchange(message) = &error { @@ -477,7 +478,7 @@ impl Operation for OidcCallbackHandler { callback_federation_error(error) })?; let authorization = &login.session.authorization; - let actual_provider_id = &authorization.provider_id; + let actual_provider_id = authorization.provider_id(); debug!( event = EVENT_ADMIN_OIDC_STATE, @@ -493,10 +494,10 @@ impl Operation for OidcCallbackHandler { component = LOG_COMPONENT_ADMIN, subsystem = LOG_SUBSYSTEM_OIDC, provider_id = %actual_provider_id, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), - policies = ?authorization.policies, - groups = ?authorization.groups, + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), + policies = ?authorization.policies(), + groups = ?authorization.mapped_iam_groups(), state = "claims_mapped", "admin oidc state" ); @@ -1375,4 +1376,30 @@ mod tests { ); } } + + #[test] + fn console_callback_uses_the_configured_session_duration() { + let production = include_str!("oidc.rs") + .split("\n#[cfg(test)]\n") + .next() + .expect("production source must precede tests"); + let callback = production + .split_once("impl Operation for OidcCallbackHandler") + .expect("the OIDC callback handler must exist") + .1; + let callback = callback.split_whitespace().collect::(); + + assert!( + callback.contains("session_duration_seconds=console_session_duration_secs()"), + "the callback must resolve the Console session duration from the environment" + ); + assert!( + callback.contains(".complete_authorization_code(&state,&code,&redirect_uri,session_duration_seconds,"), + "the callback must hand the resolved session duration to the code exchange" + ); + assert!( + !callback.contains(".complete_authorization_code(&state,&code,&redirect_uri,3600,"), + "the callback must not hard-code the session duration" + ); + } } diff --git a/rustfs/src/admin/handlers/quota.rs b/rustfs/src/admin/handlers/quota.rs index 8c6f2e18b..892517d8e 100644 --- a/rustfs/src/admin/handlers/quota.rs +++ b/rustfs/src/admin/handlers/quota.rs @@ -19,8 +19,11 @@ use crate::admin::handlers::site_replication::site_replication_bucket_meta_hook; use crate::admin::router::{AdminOperation, Operation, S3Router}; use crate::admin::runtime_sources::{current_bucket_metadata_handle, current_object_store_handle}; use crate::admin::storage_api::bucket::metadata_sys::{self, BucketMetadataSys}; -use crate::admin::storage_api::bucket::quota::checker::QuotaChecker; -use crate::admin::storage_api::bucket::quota::{BucketQuota, QuotaError, QuotaOperation}; +use crate::admin::storage_api::bucket::quota::checker::{QuotaChecker, record_reservation_protocol_migration}; +use crate::admin::storage_api::bucket::quota::{ + BucketQuota, QUOTA_RESERVATION_PROTOCOL_V2, QuotaError, QuotaOperation, QuotaProtocolMigration, +}; +use crate::admin::storage_api::contract::bucket::{BucketOperations as _, BucketOptions}; use crate::auth::{check_key_valid, get_session_token}; use crate::error::ApiError; use crate::server::ADMIN_PREFIX; @@ -33,6 +36,7 @@ use s3s::{Body, S3Error, S3Request, S3Response, S3Result, s3_error}; use serde::{Deserialize, Serialize}; use serde_json; use std::sync::Arc; +use time::format_description::well_known::Rfc3339; use tokio::sync::RwLock; use tracing::{debug, info, warn}; use url::form_urlencoded; @@ -158,7 +162,9 @@ pub struct BucketQuotaResponse { pub struct BucketQuotaStats { pub bucket: String, pub quota_limit: Option, - pub current_usage: u64, + /// `null` when the bucket is admitted by the reservation ledger and no + /// scanner usage is available; never reported as zero. + pub current_usage: Option, pub remaining_quota: Option, pub usage_percentage: Option, } @@ -175,17 +181,107 @@ pub struct CheckQuotaResponse { pub operation_type: String, pub operation_size: u64, pub allowed: bool, - pub current_usage: u64, + /// `null` when the bucket is admitted by the reservation ledger and no + /// scanner usage is available; never reported as zero. + pub current_usage: Option, pub quota_limit: Option, pub remaining_quota: Option, } +/// One bucket of the reservation protocol migration report. +#[derive(Debug, Serialize)] +pub struct BucketQuotaProtocolMigrationEntry { + pub bucket: String, + /// `migrated`, `already_durable`, `no_quota`, `skipped` or `failed`. + pub outcome: &'static str, + #[serde(skip_serializing_if = "Option::is_none")] + pub reservation_protocol: Option, + /// The skip reason or the failure message. + #[serde(skip_serializing_if = "Option::is_none")] + pub reason: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub updated_at: Option, +} + +#[derive(Debug, Default, Serialize)] +pub struct BucketQuotaProtocolMigrationReport { + pub target_reservation_protocol: u32, + pub migrated: usize, + pub already_durable: usize, + pub no_quota: usize, + pub skipped: usize, + pub failed: usize, + pub buckets: Vec, +} + +impl BucketQuotaProtocolMigrationReport { + fn record(&mut self, bucket: String, outcome: &Result) { + let entry = match outcome { + Ok(QuotaProtocolMigration::Migrated { + reservation_protocol, + updated_at, + }) => { + self.migrated += 1; + BucketQuotaProtocolMigrationEntry { + bucket, + outcome: "migrated", + reservation_protocol: Some(*reservation_protocol), + reason: None, + updated_at: updated_at.format(&Rfc3339).ok(), + } + } + Ok(QuotaProtocolMigration::AlreadyDurable { reservation_protocol }) => { + self.already_durable += 1; + BucketQuotaProtocolMigrationEntry { + bucket, + outcome: "already_durable", + reservation_protocol: Some(*reservation_protocol), + reason: None, + updated_at: None, + } + } + Ok(QuotaProtocolMigration::NoQuota) => { + self.no_quota += 1; + BucketQuotaProtocolMigrationEntry { + bucket, + outcome: "no_quota", + reservation_protocol: None, + reason: None, + updated_at: None, + } + } + Ok(QuotaProtocolMigration::Skipped { reason }) => { + self.skipped += 1; + BucketQuotaProtocolMigrationEntry { + bucket, + outcome: "skipped", + reservation_protocol: None, + reason: Some(reason.as_str().to_string()), + updated_at: None, + } + } + Err(err) => { + self.failed += 1; + BucketQuotaProtocolMigrationEntry { + bucket, + outcome: "failed", + reservation_protocol: None, + reason: Some(err.to_string()), + updated_at: None, + } + } + }; + self.buckets.push(entry); + } +} + /// Quota management handlers pub struct SetBucketQuotaHandler; pub struct GetBucketQuotaHandler; pub struct ClearBucketQuotaHandler; pub struct GetBucketQuotaStatsHandler; pub struct CheckBucketQuotaHandler; +pub struct MigrateBucketQuotaProtocolHandler; fn bucket_metadata_from_context() -> Option>> { current_bucket_metadata_handle() @@ -249,6 +345,12 @@ pub fn register_quota_route(r: &mut S3Router) -> std::io::Result AdminOperation(&CheckBucketQuotaHandler {}), )?; + r.insert( + Method::POST, + format!("{}{}", ADMIN_PREFIX, "/v3/quota-migration").as_str(), + AdminOperation(&MigrateBucketQuotaProtocolHandler {}), + )?; + Ok(()) } @@ -614,21 +716,17 @@ impl Operation for GetBucketQuotaStatsHandler { let quota_checker = QuotaChecker::new(metadata_sys_lock.clone()); - let (quota, current_usage_opt) = quota_checker + let (quota, current_usage) = quota_checker .get_quota_stats(&bucket) .await .map_err(|error| map_quota_query_error(error, &bucket, "get quota stats"))?; - let current_usage = current_usage_opt.unwrap_or(0); - let usage_percentage = quota.quota.and_then(|limit| { - if limit == 0 { - None - } else { - Some((current_usage as f64 / limit as f64) * 100.0) - } - }); + let usage_percentage = match (quota.quota, current_usage) { + (Some(limit), Some(current_usage)) if limit > 0 => Some((current_usage as f64 / limit as f64) * 100.0), + _ => None, + }; - let remaining_quota = quota.get_remaining_quota(current_usage); + let remaining_quota = current_usage.and_then(|current_usage| quota.get_remaining_quota(current_usage)); let response = BucketQuotaStats { bucket, @@ -729,7 +827,7 @@ impl Operation for CheckBucketQuotaHandler { operation_type: request.operation_type, operation_size: request.operation_size, allowed: result.allowed, - current_usage: result.current_usage.unwrap_or(0), + current_usage: result.current_usage, quota_limit: result.quota_limit, remaining_quota: result.remaining, }; @@ -741,6 +839,77 @@ impl Operation for CheckBucketQuotaHandler { } } +/// `POST /rustfs/admin/v3/quota-migration`: re-save every quota that predates +/// the reservation protocols under the sharded durable protocol and report +/// each bucket (rustfs/backlog#2904). The limit and type are unchanged; a +/// bucket that fails is reported, not retried, and never stops the sweep. +#[async_trait::async_trait] +impl Operation for MigrateBucketQuotaProtocolHandler { + #[tracing::instrument(skip_all)] + async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { + warn!( + event = EVENT_ADMIN_QUOTA_STATE, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_QUOTA, + action = "migrate_bucket_quota_protocol", + state = "requested", + "admin quota state" + ); + + let Some(ref cred) = req.credentials else { + return Err(s3_error!(InvalidRequest, "authentication required")); + }; + + let (cred, owner) = + check_key_valid(get_session_token(&req.uri, &req.headers).unwrap_or_default(), &cred.access_key).await?; + + let remote_addr = req.extensions.get::>().and_then(|opt| opt.map(|a| a.0)); + validate_admin_request( + &req.headers, + &cred, + owner, + false, + vec![Action::AdminAction(AdminAction::SetBucketQuotaAdminAction)], + remote_addr, + ) + .await?; + + let store = current_object_store_handle().ok_or_else(|| s3_error!(InternalError, "object store is not initialized"))?; + let metadata_sys_lock = bucket_metadata_from_context() + .ok_or_else(|| s3_error!(InternalError, "{}", rustfs_config::QUOTA_METADATA_SYSTEM_ERROR_MSG))?; + let buckets = store.list_bucket(&BucketOptions::default()).await.map_err(ApiError::from)?; + + let quota_checker = QuotaChecker::new(metadata_sys_lock); + let mut report = BucketQuotaProtocolMigrationReport { + target_reservation_protocol: QUOTA_RESERVATION_PROTOCOL_V2, + ..Default::default() + }; + for bucket in buckets { + let outcome = quota_checker.migrate_reservation_protocol(&bucket.name).await; + record_reservation_protocol_migration(&bucket.name, "admin", &outcome); + report.record(bucket.name, &outcome); + } + + info!( + event = EVENT_ADMIN_QUOTA_STATE, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_QUOTA, + action = "migrate_bucket_quota_protocol", + state = "completed", + migrated = report.migrated, + already_durable = report.already_durable, + no_quota = report.no_quota, + skipped = report.skipped, + failed = report.failed, + "admin quota state" + ); + + let json = serde_json::to_string(&report).map_err(|e| s3_error!(InternalError, "failed to serialize response: {}", e))?; + + Ok(S3Response::new((StatusCode::OK, Body::from(json)))) + } +} + #[cfg(test)] mod tests { use super::*; @@ -801,6 +970,80 @@ mod tests { assert_eq!(error.code(), &s3s::S3ErrorCode::ServiceUnavailable); } + /// rustfs/backlog#2904: unknown usage on a ledger-admitted bucket is + /// reported as `null`, never as zero. + #[test] + fn quota_stats_report_unknown_usage_as_null() { + let stats = BucketQuotaStats { + bucket: "test-bucket".to_string(), + quota_limit: Some(2048), + current_usage: None, + remaining_quota: None, + usage_percentage: None, + }; + let value = serde_json::to_value(&stats).expect("stats should serialize"); + assert!(value["current_usage"].is_null()); + assert!(value["remaining_quota"].is_null()); + + let check = CheckQuotaResponse { + bucket: "test-bucket".to_string(), + operation_type: "PUT".to_string(), + operation_size: 512, + allowed: true, + current_usage: None, + quota_limit: Some(2048), + remaining_quota: None, + }; + let value = serde_json::to_value(&check).expect("check response should serialize"); + assert!(value["current_usage"].is_null()); + assert_eq!(value["allowed"], serde_json::Value::Bool(true)); + } + + #[test] + fn quota_protocol_migration_report_counts_every_outcome() { + use crate::admin::storage_api::bucket::quota::QuotaMigrationSkipReason; + + let mut report = BucketQuotaProtocolMigrationReport { + target_reservation_protocol: QUOTA_RESERVATION_PROTOCOL_V2, + ..Default::default() + }; + report.record( + "a".to_string(), + &Ok(QuotaProtocolMigration::Migrated { + reservation_protocol: QUOTA_RESERVATION_PROTOCOL_V2, + updated_at: time::OffsetDateTime::UNIX_EPOCH, + }), + ); + report.record("b".to_string(), &Ok(QuotaProtocolMigration::AlreadyDurable { reservation_protocol: 1 })); + report.record("c".to_string(), &Ok(QuotaProtocolMigration::NoQuota)); + report.record( + "d".to_string(), + &Ok(QuotaProtocolMigration::Skipped { + reason: QuotaMigrationSkipReason::QuotaRevisionChanged, + }), + ); + report.record("e".to_string(), &Err(QuotaError::UsageUnavailable { bucket: "e".to_string() })); + + assert_eq!( + (report.migrated, report.already_durable, report.no_quota, report.skipped, report.failed), + (1, 1, 1, 1, 1) + ); + let value = serde_json::to_value(&report).expect("report should serialize"); + assert_eq!(value["target_reservation_protocol"], 2); + assert_eq!(value["buckets"][0]["outcome"], "migrated"); + assert_eq!(value["buckets"][0]["reservation_protocol"], 2); + assert_eq!(value["buckets"][0]["updated_at"], "1970-01-01T00:00:00Z"); + assert_eq!(value["buckets"][1]["reservation_protocol"], 1); + assert!(value["buckets"][2].get("reason").is_none()); + assert_eq!(value["buckets"][3]["reason"], "quota_revision_changed"); + assert_eq!(value["buckets"][4]["outcome"], "failed"); + assert!( + value["buckets"][4]["reason"] + .as_str() + .is_some_and(|reason| reason.contains("e")) + ); + } + #[test] fn test_quota_operation_parsing() { let parse_operation = |operation: &str| match operation.to_uppercase().as_str() { diff --git a/rustfs/src/admin/handlers/rebalance.rs b/rustfs/src/admin/handlers/rebalance.rs index 14dc61b64..40868a91a 100644 --- a/rustfs/src/admin/handlers/rebalance.rs +++ b/rustfs/src/admin/handlers/rebalance.rs @@ -42,17 +42,15 @@ use s3s::{ s3_error, }; use serde::{Deserialize, Serialize}; -use std::{ - future::Future, - sync::Arc, - time::{Duration, Instant}, -}; +use std::{future::Future, sync::Arc, time::Duration}; use time::OffsetDateTime; -use tracing::{error, info, warn}; +use tokio::time::Instant; +use tracing::{debug, error, info, warn}; const LOG_COMPONENT_ADMIN: &str = "admin"; const LOG_SUBSYSTEM_REBALANCE: &str = "rebalance"; const EVENT_ADMIN_REBALANCE_STATE: &str = "admin_rebalance_state"; +const REBALANCE_NOT_STARTED_MESSAGE: &str = "pool rebalance is not started"; const REBALANCE_START_FLEET_PROOF_MARKER: &str = "pool activation requires a live fleet capability proof"; const REBALANCE_START_FLEET_PROOF_RETRY_BUDGET: Duration = Duration::from_secs(30); const REBALANCE_START_FLEET_PROOF_RETRY_DELAY: Duration = Duration::from_secs(5); @@ -128,6 +126,13 @@ fn is_rebalance_start_fleet_proof_retryable(err: &StorageError) -> bool { && err.to_string().contains(REBALANCE_START_FLEET_PROOF_MARKER) } +fn rebalance_init_failure_error(err: &StorageError) -> S3Error { + if is_rebalance_start_fleet_proof_retryable(err) { + return rebalance_fleet_proof_unavailable_error(); + } + s3_error!(InternalError, "failed to initialize rebalance: {}", err) +} + fn rebalance_start_failure_error(start_err: &StorageError, rollback_result: &Result<(), String>) -> S3Error { if crate::storage_api::capacity::is_pool_activation_fleet_proof_error(start_err) { if rollback_result.is_ok() { @@ -611,7 +616,20 @@ impl Operation for RebalanceStart { let buckets: Vec = bucket_infos.into_iter().map(|bucket| bucket.name).collect(); - let id = match store.init_rebalance_start(buckets).await { + let fleet_proof_deadline = Instant::now() + REBALANCE_START_FLEET_PROOF_RETRY_BUDGET; + // A missing proof fails before init publishes metadata. Retrying the full + // operation reacquires its gates and revalidates competing starts. + let id = match retry_rebalance_start_operation( + "init_rebalance_start", + "", + &request_id, + &actor, + &remote_addr, + fleet_proof_deadline, + || store.init_rebalance_start(buckets.clone()), + ) + .await + { Ok(id) => id, Err(StorageError::DecommissionAlreadyRunning) => { log_rebalance_request_rejected("start", "decommission_in_progress", &request_id, &actor, &remote_addr); @@ -622,7 +640,7 @@ impl Operation for RebalanceStart { return Err(s3_error!(OperationAborted, "rebalance is already in progress")); } Err(e) => { - return Err(s3_error!(InternalError, "failed to initialize rebalance: {}", e)); + return Err(rebalance_init_failure_error(&e)); } }; @@ -639,7 +657,6 @@ impl Operation for RebalanceStart { "admin rebalance state" ); let notification_sys = current_notification_system(); - let fleet_proof_deadline = Instant::now() + REBALANCE_START_FLEET_PROOF_RETRY_BUDGET; for step in rebalance_start_steps(notification_sys.is_some()) { match step { RebalanceStartStep::PropagateFence => { @@ -901,8 +918,24 @@ impl Operation for RebalanceStatus { let mut meta = RebalanceMeta::new(); if let Err(err) = meta.load(first_pool).await { if err == StorageError::ConfigNotFound { - log_rebalance_request_rejected("status", "rebalance_not_started", &request_id, &actor, &remote_addr); - return Err(s3_error!(NoSuchResource, "pool rebalance is not started")); + debug!( + event = EVENT_ADMIN_REBALANCE_STATE, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_REBALANCE, + state = "not_started", + action = "status", + request_id = %request_id, + actor = %actor, + remote_addr = %remote_addr, + "admin rebalance state" + ); + // Preserve the idle 404 contract without triggering the custom-route error logger. + let (parts, body) = S3Error::with_message(S3ErrorCode::NoSuchResource, REBALANCE_NOT_STARTED_MESSAGE) + .to_http_response()? + .into_parts(); + let mut response = S3Response::with_headers((parts.status, body), parts.headers); + response.extensions = parts.extensions; + return Ok(response); } return Err(s3_error!(InternalError, "failed to load rebalance metadata from pool 0: {}", err)); @@ -1028,7 +1061,7 @@ impl Operation for RebalanceStop { let Some(expected_rebalance_id) = rebalance_stop_target_id(&store).await? else { log_rebalance_request_rejected("stop", "rebalance_not_started", &request_id, &actor, &remote_addr); - return Err(s3_error!(NoSuchResource, "pool rebalance is not started")); + return Err(S3Error::with_message(S3ErrorCode::NoSuchResource, REBALANCE_NOT_STARTED_MESSAGE)); }; let notification_sys = current_notification_system(); @@ -1170,10 +1203,10 @@ mod rebalance_handler_tests { Body, HeaderMap, Method, Operation, Params, RebalPoolProgress, RebalanceAdminStatus, RebalancePoolStatus, RebalanceStart, RebalanceStartStep, RebalanceStatus, RebalanceStop, RebalanceStopPropagationStatus, S3ErrorCode, S3Request, StatusCode, Uri, build_rebalance_admin_status, build_rebalance_pool_statuses, build_rebalance_stop_propagation_status, - is_rebalance_start_fleet_proof_retryable, rebalance_pool_used, rebalance_query_present, rebalance_remaining_buckets, - rebalance_rollback_failure_message, rebalance_rollback_stop_failure_message, rebalance_start_failure_error, - rebalance_start_rollback_error, rebalance_start_steps, rebalance_stop_target_id, rebalance_used_pct, - retry_rebalance_start_operation, rollback_result_label, stop_rebalance_admission_first, + is_rebalance_start_fleet_proof_retryable, rebalance_init_failure_error, rebalance_pool_used, rebalance_query_present, + rebalance_remaining_buckets, rebalance_rollback_failure_message, rebalance_rollback_stop_failure_message, + rebalance_start_failure_error, rebalance_start_rollback_error, rebalance_start_steps, rebalance_stop_target_id, + rebalance_used_pct, retry_rebalance_start_operation, rollback_result_label, stop_rebalance_admission_first, }; use crate::admin::storage_api::error::StorageError; use crate::admin::storage_api::rebalance::{ @@ -1533,7 +1566,7 @@ mod rebalance_handler_tests { } #[tokio::test(start_paused = true)] - async fn test_rebalance_start_retry_waits_for_fleet_proof() { + async fn test_rebalance_init_retry_waits_for_fleet_proof() { use std::sync::{ Arc, atomic::{AtomicUsize, Ordering}, @@ -1544,12 +1577,12 @@ mod rebalance_handler_tests { let attempts = Arc::clone(&attempts); async move { retry_rebalance_start_operation( - "load_rebalance_meta(start=true)", - "rebalance-id", + "init_rebalance_start", + "", "request-id", "actor", "remote", - std::time::Instant::now() + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET, + super::Instant::now() + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET, move || { let attempts = Arc::clone(&attempts); async move { @@ -1557,7 +1590,7 @@ mod rebalance_handler_tests { if attempt == 0 { Err(StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER)) } else { - Ok(()) + Ok("initialized-once") } } }, @@ -1570,12 +1603,78 @@ mod rebalance_handler_tests { assert_eq!(attempts.load(Ordering::SeqCst), 1); tokio::time::advance(super::REBALANCE_START_FLEET_PROOF_RETRY_DELAY).await; - task.await + let id = task + .await .expect("retry task should not panic") .expect("fleet proof retry should eventually succeed"); + assert_eq!(id, "initialized-once"); assert_eq!(attempts.load(Ordering::SeqCst), 2); } + #[tokio::test(start_paused = true)] + async fn test_rebalance_init_retry_exhaustion_returns_503_with_retry_after() { + let mut attempts = 0; + let started = super::Instant::now(); + let deadline = started + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET; + let err = retry_rebalance_start_operation::<(), _, _>( + "init_rebalance_start", + "", + "request-id", + "actor", + "remote", + deadline, + || { + attempts += 1; + std::future::ready(Err(StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER))) + }, + ) + .await + .expect_err("a missing proof must exhaust the bounded retry budget"); + + assert_eq!(super::Instant::now() - started, super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET); + assert_eq!(attempts, 7, "one immediate attempt plus six bounded readiness retries"); + let response = rebalance_init_failure_error(&err); + assert_eq!(response.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); + assert_eq!( + response + .headers() + .and_then(|headers| headers.get(http::header::RETRY_AFTER)) + .and_then(|value| value.to_str().ok()), + Some(super::REBALANCE_START_RETRY_AFTER_SECS) + ); + } + + #[tokio::test(start_paused = true)] + async fn test_rebalance_init_does_not_retry_other_failures() { + for source in [ + StorageError::other("disk read failed"), + StorageError::other("pool activation fleet capability proof expired before commit"), + StorageError::RebalanceAlreadyRunning, + ] { + let original = source.to_string(); + let mut source = Some(source); + let started = super::Instant::now(); + let err = retry_rebalance_start_operation::<(), _, _>( + "init_rebalance_start", + "", + "request-id", + "actor", + "remote", + started + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET, + || std::future::ready(Err(source.take().expect("non-readiness errors must not be retried"))), + ) + .await + .expect_err("non-readiness errors must be returned unchanged"); + assert_eq!(err.to_string(), original); + assert_eq!(super::Instant::now(), started, "non-readiness failures must not wait"); + if !matches!(err, StorageError::RebalanceAlreadyRunning) { + let response = rebalance_init_failure_error(&err); + assert_eq!(response.code(), &S3ErrorCode::InternalError); + assert!(response.message().is_some_and(|message| message.contains(&original))); + } + } + } + #[test] fn test_rebalance_rollback_stop_failure_message_lists_failures() { let failures = vec![ diff --git a/rustfs/src/admin/handlers/scanner.rs b/rustfs/src/admin/handlers/scanner.rs index f73ff4337..a90e4f3c1 100644 --- a/rustfs/src/admin/handlers/scanner.rs +++ b/rustfs/src/admin/handlers/scanner.rs @@ -47,6 +47,8 @@ struct ScannerStatusResponse { cycle_schedule: rustfs_scanner::ScannerCycleScheduleStatus, runtime_config: rustfs_scanner::runtime_config::ScannerRuntimeConfigStatus, cycle_recovery: rustfs_scanner::ScannerCycleRecoveryStatus, + bitrot: rustfs_scanner::ScannerBitrotStatus, + disk_io_budget: Vec, data_movement_pause: ScannerDataMovementPauseStatus, pause_backlog: rustfs_scanner::ScannerPauseBacklogStatus, catch_up_estimate: ScannerCatchUpEstimate, @@ -220,6 +222,8 @@ fn scanner_status_response( cycle_schedule, runtime_config, cycle_recovery: rustfs_scanner::scanner::scanner_cycle_recovery_status(), + bitrot: rustfs_scanner::scanner_bitrot_status(), + disk_io_budget: rustfs_scanner::scanner_disk_io_budget_status(), data_movement_pause, pause_backlog, catch_up_estimate, @@ -839,6 +843,15 @@ mod tests { assert_eq!(encoded["data_movement_pause"]["paused"], false); assert_eq!(encoded["catch_up_estimate"]["estimated"], false); assert_eq!(encoded["catch_up_estimate"]["undiscovered_ilm_items_known"], true); + assert!( + ["none", "memory", "persisted"].contains(&encoded["bitrot"]["bitrot_start_source"].as_str().unwrap_or("")), + "bitrot start source must name where the Deep cadence start came from: {}", + encoded["bitrot"] + ); + assert!(encoded["bitrot"]["deep_scan_supported"].is_boolean()); + assert!(encoded["bitrot"]["bitrot_start_cycle"].is_u64()); + assert!(encoded["bitrot"]["bitrot_start_unix_secs"].is_u64()); + assert!(encoded["disk_io_budget"].is_array()); } #[test] diff --git a/rustfs/src/admin/handlers/service_account.rs b/rustfs/src/admin/handlers/service_account.rs index 938ba7836..f046cfff7 100644 --- a/rustfs/src/admin/handlers/service_account.rs +++ b/rustfs/src/admin/handlers/service_account.rs @@ -125,14 +125,25 @@ fn is_service_account_owner_of(caller: &StoredCredentials, target_parent_user: & /// /// The `CreateServiceAccountAdminAction` permission gates *whether* a caller may /// create service accounts, not *for whom*. A non-owner is therefore confined to -/// its own scope; only an owner may target another user — including the root +/// its own non-root scope; only an owner may target another user — including the root /// credential, whose existence check is deliberately lenient. Without this a /// caller holding only that action could pass `target_user = ` /// and mint a root-parented service account that authenticates as owner via /// `prepare_service_account_auth`, a full-takeover privilege escalation. Mirrors /// `imported_service_account_parent_allowed` on the ImportIam path (GHSA-566f). -fn add_service_account_parent_within_scope(target_user: &str, req_user: &str, req_parent_user: &str, owner: bool) -> bool { - owner || target_user == req_user || target_user == req_parent_user +fn add_service_account_parent_within_scope( + root_access_key: &str, + target_user: &str, + req_user: &str, + req_parent_user: &str, + owner: bool, +) -> bool { + let effective_parent = if target_user == req_user { + req_parent_user + } else { + target_user + }; + owner || (effective_parent != root_access_key && (target_user == req_user || target_user == req_parent_user)) } /// Fail-closed ownership check used by DeleteServiceAccount for a caller that @@ -401,9 +412,8 @@ impl Operation for AddServiceAccount { // a root-parented (owner) service account. Evaluated on the original // `target_user` the caller submitted, before the derived-credential rewrite // to `req_parent_user` below, so the guard is bound to the parent actually - // requested. Its allow set is exactly `owner || is_svc_acc`. See the helper's - // doc comment. - if !add_service_account_parent_within_scope(&target_user, &req_user, &req_parent_user, owner) { + // requested and to the effective parent after that rewrite. + if !add_service_account_parent_within_scope(&sys_cred.access_key, &target_user, &req_user, &req_parent_user, owner) { return Err(s3_error!(AccessDenied, "service account parent is outside requester scope")); } @@ -1470,76 +1480,88 @@ mod tests { // Attack: non-owner "alice" targets root as the parent -> DENY. assert!( - !add_service_account_parent_within_scope(root, "alice", "alice", false), + !add_service_account_parent_within_scope(root, root, "alice", "alice", false), "non-owner must not parent a service account to root" ); // Non-owner targeting an unrelated user -> DENY. assert!( - !add_service_account_parent_within_scope("bob", "alice", "alice", false), + !add_service_account_parent_within_scope(root, "bob", "alice", "alice", false), "non-owner must not parent a service account to another user" ); // Self-scoped creation (default target == own key) -> ALLOW. assert!( - add_service_account_parent_within_scope("alice", "alice", "alice", false), + add_service_account_parent_within_scope(root, "alice", "alice", "alice", false), "non-owner may create a service account for itself" ); // Derived credential targeting its own parent -> ALLOW. assert!( - add_service_account_parent_within_scope("parent-user", "svc-key", "parent-user", false), + add_service_account_parent_within_scope(root, "parent-user", "svc-key", "parent-user", false), "derived credential may create a service account under its own parent" ); // Owner may target any parent, including root (cross-user creation intact). assert!( - add_service_account_parent_within_scope(root, root, root, true), + add_service_account_parent_within_scope(root, root, root, root, true), "owner retains cross-user / root-parent creation" ); assert!( - add_service_account_parent_within_scope("bob", root, root, true), + add_service_account_parent_within_scope(root, "bob", root, root, true), "owner may create a service account for another user" ); } #[test] - fn ghsa_5354_scope_guard_matches_owner_or_self_scope_for_derived_credentials() { - // The handler evaluates `add_service_account_parent_within_scope` on the - // original `target_user`, before the `target_user = req_parent_user` rewrite - // taken inside the `is_svc_acc` branch. The guard's allow set must therefore - // stay exactly `owner || is_svc_acc`, where - // `is_svc_acc == (target_user == req_user || target_user == req_parent_user)`. - // - // The named-boundary test above only exercises the self-scoped case with - // `req_user == req_parent_user`. A *derived* credential has `req_user` - // (its own key) distinct from `req_parent_user` (its parent), which is the - // realistic attacker shape: a service account holding - // CreateServiceAccountAdminAction. Pin the equivalence across that shape so a - // later change to either the guard or the rewrite cannot silently let a - // derived non-owner credential escape its own parent's scope. + fn ghsa_5354_scope_guard_preserves_non_root_self_scope() { let root = rustfs_credentials::DEFAULT_ACCESS_KEY; - let cases: &[(&str, &str, &str, bool)] = &[ + let cases: &[(&str, &str, &str, bool, bool)] = &[ // Derived non-owner (own key != parent) aiming at root -> deny. - (root, "attacker-sa", "alice", false), + (root, "attacker-sa", "alice", false, false), // Derived non-owner aiming at an unrelated user -> deny. - ("bob", "attacker-sa", "alice", false), + ("bob", "attacker-sa", "alice", false, false), // Derived non-owner aiming at its own parent -> allow. - ("alice", "attacker-sa", "alice", false), + ("alice", "attacker-sa", "alice", false, true), // Derived non-owner aiming at its own key -> allow. - ("attacker-sa", "attacker-sa", "alice", false), + ("attacker-sa", "attacker-sa", "alice", false, true), // Top-level non-owner aiming at itself -> allow. - ("alice", "alice", "alice", false), + ("alice", "alice", "alice", false, true), // Owner is unrestricted, including root and a derived shape. - (root, "attacker-sa", "alice", true), - ("bob", root, root, true), + (root, "attacker-sa", "alice", true, true), + ("bob", root, root, true, true), ]; - for &(target_user, req_user, req_parent_user, owner) in cases { - let is_svc_acc = target_user == req_user || target_user == req_parent_user; + for &(target_user, req_user, req_parent_user, owner, expected) in cases { assert_eq!( - add_service_account_parent_within_scope(target_user, req_user, req_parent_user, owner), - owner || is_svc_acc, - "scope guard must equal `owner || is_svc_acc` for (target={target_user}, req_user={req_user}, req_parent={req_parent_user}, owner={owner})" + add_service_account_parent_within_scope(root, target_user, req_user, req_parent_user, owner), + expected, + "scope guard mismatch for (target={target_user}, req_user={req_user}, req_parent={req_parent_user}, owner={owner})" ); } } + #[test] + fn credential_boundary_root_derived_non_owner_cannot_create_owner_service_account() { + let root = "deployment-root"; + for target in [root, "restricted-service-account", "other-user"] { + assert!( + !add_service_account_parent_within_scope(root, target, "restricted-service-account", root, false), + "a root-derived non-owner must be denied even when targeting its own key: {target}" + ); + } + assert!(add_service_account_parent_within_scope(root, root, root, root, true)); + assert!(add_service_account_parent_within_scope( + root, + "alice", + "alice-service-account", + "alice", + false + )); + assert!(add_service_account_parent_within_scope( + root, + "alice-service-account", + "alice-service-account", + "alice", + false + )); + } + #[test] fn guess_user_provider_detects_builtin_accounts() { let credentials = StoredCredentials { diff --git a/rustfs/src/admin/handlers/sts.rs b/rustfs/src/admin/handlers/sts.rs index 054a1d11c..75211130c 100644 --- a/rustfs/src/admin/handlers/sts.rs +++ b/rustfs/src/admin/handlers/sts.rs @@ -432,7 +432,7 @@ async fn handle_assume_role_with_web_identity(body: AssumeRoleRequest) -> S3Resu })?; let authorization = &session.authorization; let new_cred = &session.credentials; - let subject = authorization.claims.session_identity(); + let subject = authorization.session_identity(); let xml = assume_role_with_web_identity_response_xml(new_cred, &subject); diff --git a/rustfs/src/admin/handlers/table_catalog/maintenance.rs b/rustfs/src/admin/handlers/table_catalog/maintenance.rs index c0dc5fb61..38f3ab525 100644 --- a/rustfs/src/admin/handlers/table_catalog/maintenance.rs +++ b/rustfs/src/admin/handlers/table_catalog/maintenance.rs @@ -27,13 +27,51 @@ impl Operation for RestTableMetadataMaintenanceHandler { ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?; let request = read_json_body::(req.input).await?; let metadata_backend = table_catalog_backend_from_extensions(&req.extensions)?; - let store = table_catalog_object_store_from_extensions(&req.extensions)?; + let store = table_catalog_store_from_extensions(&req.extensions)?; let response = - table_metadata_maintenance_response(&store, &metadata_backend, &warehouse, &namespace, &table, request).await?; + configured_table_metadata_maintenance_response(&store, &metadata_backend, &warehouse, &namespace, &table, request) + .await?; build_json_response(StatusCode::OK, &response) } } +async fn configured_table_metadata_maintenance_response( + store: &crate::table_catalog::ConfiguredTableCatalogStore, + metadata_backend: &B, + bucket: &str, + namespace: &crate::table_catalog::Namespace, + table: &str, + request: TableMetadataMaintenanceRequest, +) -> S3Result +where + B: crate::table_catalog::TableCatalogObjectBackend + Clone, +{ + match store { + crate::table_catalog::ConfiguredTableCatalogStore::ObjectBacked(store) => { + table_metadata_maintenance_response(store, metadata_backend, bucket, namespace, table, request).await + } + crate::table_catalog::ConfiguredTableCatalogStore::DurableStrong(store) => { + if request.delete || request.commit_snapshot_expiration || request.commit_compaction { + return Err(s3_error!( + InvalidRequest, + "metadata maintenance deletion and snapshot expiration or compaction commits are not supported with durable-strong table catalog backing" + )); + } + store + .run_table_metadata_maintenance_planning( + bucket, + &namespace.public_name(), + table, + request.retain_recent_metadata_files, + request.snapshot_expiration, + request.compaction, + ) + .await + .map_err(catalog_store_error) + } + } +} + pub struct GetTableMaintenanceConfigHandler {} #[async_trait::async_trait] @@ -157,16 +195,32 @@ impl Operation for RunTableMaintenanceWorkerHandler { authorize_table_catalog_resource_request(&req, &resource, AdminAction::RunTableMaintenanceAction).await?; ensure_table_bucket_enabled_from_extensions(&req.extensions, &warehouse).await?; let request = read_json_body::(req.input).await?; + let condition = request.condition()?; let store = table_catalog_store_from_extensions(&req.extensions)?; - let response = store - .run_table_metadata_maintenance_worker_once( - &warehouse, - &namespace.public_name(), - &table, - request.worker_id().to_string(), - ) - .await - .map_err(catalog_store_error)?; + let response = match condition { + Some(condition) => { + store + .run_table_metadata_maintenance_worker_once_conditionally( + &warehouse, + &namespace.public_name(), + &table, + request.worker_id().to_string(), + &condition, + ) + .await + } + None => { + store + .run_table_metadata_maintenance_worker_once( + &warehouse, + &namespace.public_name(), + &table, + request.worker_id().to_string(), + ) + .await + } + } + .map_err(catalog_store_error)?; build_json_response(StatusCode::OK, &response) } } diff --git a/rustfs/src/admin/handlers/table_catalog/mod.rs b/rustfs/src/admin/handlers/table_catalog/mod.rs index 976d70532..130cea31d 100644 --- a/rustfs/src/admin/handlers/table_catalog/mod.rs +++ b/rustfs/src/admin/handlers/table_catalog/mod.rs @@ -427,12 +427,52 @@ impl TableMaintenanceSchedulerRunRequest { struct TableMaintenanceWorkerRunRequest { #[serde(default, rename = "worker-id")] worker_id: Option, + #[serde( + default, + rename = "expected-job-id", + deserialize_with = "deserialize_present_worker_condition_field" + )] + expected_job_id: Option, + #[serde( + default, + rename = "expected-attempt", + deserialize_with = "deserialize_present_worker_condition_field" + )] + expected_attempt: Option, +} + +// Explicit null must not turn a conditional request into an unconditional run. +fn deserialize_present_worker_condition_field<'de, D, T>(deserializer: D) -> Result, D::Error> +where + D: serde::Deserializer<'de>, + T: Deserialize<'de>, +{ + T::deserialize(deserializer).map(Some) } impl TableMaintenanceWorkerRunRequest { fn worker_id(&self) -> &str { self.worker_id.as_deref().unwrap_or(DEFAULT_TABLE_MAINTENANCE_WORKER_ID) } + + fn condition(&self) -> S3Result> { + match (&self.expected_job_id, self.expected_attempt) { + (None, None) => Ok(None), + (Some(job_id), Some(attempt)) => { + if job_id.trim().is_empty() || self.worker_id().trim().is_empty() { + return Err(s3_error!(InvalidRequest, "conditional maintenance requires non-empty job and worker ids")); + } + Ok(Some(crate::table_catalog::TableMaintenanceWorkerCondition { + expected_job_id: job_id.clone(), + expected_attempt: attempt, + })) + } + _ => Err(s3_error!( + InvalidRequest, + "expected-job-id and expected-attempt must be supplied together" + )), + } + } } #[derive(Debug, Deserialize)] @@ -1862,6 +1902,16 @@ where .then(|| publication.guards.iter().all(|guard| !guard.is_lock_lost())) } + fn table_commit_object_mutation_fence(&self) -> Option { + let publication = self.publication.lock(); + Some(crate::table_catalog::TableCatalogObjectMutationFence::from_signals( + publication + .guards + .iter() + .map(crate::table_catalog::TableCatalogLockGuard::lock_lost_signal), + )) + } + fn complete_table_commit_publication(&self) { self.complete_publication(); } @@ -5497,7 +5547,8 @@ fn catalog_store_error(err: crate::table_catalog::TableCatalogStoreError) -> S3E crate::table_catalog::TableCatalogStoreError::Unsupported(message) => { iceberg_rest_error(ICEBERG_ERROR_UNSUPPORTED_OPERATION, StatusCode::NOT_ACCEPTABLE, message) } - crate::table_catalog::TableCatalogStoreError::Unavailable(_) => iceberg_rest_error( + crate::table_catalog::TableCatalogStoreError::Unavailable(_) + | crate::table_catalog::TableCatalogStoreError::LockUnavailable { .. } => iceberg_rest_error( ICEBERG_ERROR_REST, StatusCode::SERVICE_UNAVAILABLE, "table catalog is temporarily unavailable", diff --git a/rustfs/src/admin/handlers/table_catalog/tests.rs b/rustfs/src/admin/handlers/table_catalog/tests.rs index 9ce007d95..7580f3ac5 100644 --- a/rustfs/src/admin/handlers/table_catalog/tests.rs +++ b/rustfs/src/admin/handlers/table_catalog/tests.rs @@ -87,6 +87,26 @@ async fn call_load_table_handler(request: S3Request) -> S3Result) -> S3Result> { + let mut router = matchit::Router::new(); + router + .insert("/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run", ()) + .expect("worker test route should insert"); + let path = request.uri.path().to_string(); + let matched = router.at(&path).expect("worker test route should match"); + RunTableMaintenanceWorkerHandler {}.call(request, matched.params).await +} + +async fn call_metadata_maintenance_handler(request: S3Request) -> S3Result> { + let mut router = matchit::Router::new(); + router + .insert("/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/metadata", ()) + .expect("maintenance test route should insert"); + let path = request.uri.path().to_string(); + let matched = router.at(&path).expect("maintenance test route should match"); + RestTableMetadataMaintenanceHandler {}.call(request, matched.params).await +} + #[tokio::test] async fn commit_backend_borrowed_put_preserves_owner_authorization_and_precondition() { use crate::admin::storage_api::object::WriteCommitGuard; @@ -246,6 +266,539 @@ async fn table_catalog_authentication_and_credentials_use_the_request_context() let resolved_store = runtime_sources::object_store_from_extensions(&request.extensions).expect("request object store should resolve"); assert!(Arc::ptr_eq(&resolved_store, &store)); + + let mut request = request; + request.method = Method::POST; + request.uri = "/iceberg/v1/request-warehouse/namespaces/analytics/tables/events/maintenance/worker/run" + .parse() + .expect("worker URI"); + request.input = Body::from(br#"{"expected-job-id":"job-a","expected-attempt":0}"#.to_vec()); + let err = call_maintenance_worker_handler(request) + .await + .expect_err("table read permission must not authorize a conditional maintenance worker"); + assert_eq!(err.code(), &S3ErrorCode::AccessDenied); +} + +#[tokio::test] +#[serial_test::serial] +async fn table_maintenance_worker_handler_enforces_conditions_after_authorization() { + use crate::admin::storage_api::RUSTFS_META_BUCKET; + use crate::table_catalog::{ + TableCatalogObject, TableMaintenanceConfig, TableMetadataMaintenanceJobStatus, TableMetadataMaintenanceReport, + }; + + async fn inventory(backend: &B) -> BTreeMap<(String, String), TableCatalogObject> { + let mut objects = BTreeMap::new(); + for bucket in ["warehouse", RUSTFS_META_BUCKET] { + for object in backend.list_objects(bucket, "").await.expect("list isolated fixture objects") { + let stored = backend + .read_object(bucket, &object) + .await + .expect("read fixture object") + .expect("listed object exists"); + objects.insert((bucket.to_string(), object), stored); + } + } + objects + } + + temp_env::async_with_vars( + [( + crate::table_catalog::ENV_TABLE_CATALOG_BACKING, + Some(crate::table_catalog::TABLE_CATALOG_BACKING_OBJECT), + )], + async { + let (_temp_dir, _disk_paths, object_store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + object_store + .make_bucket("warehouse", &MakeBucketOptions::default()) + .await + .expect("create table bucket"); + rustfs_iam::store::object::ObjectStore::new(object_store.clone()) + .save_iam_config( + serde_json::json!({"version": 1}), + format!("{}/format.json", *rustfs_iam::store::object::IAM_CONFIG_PREFIX), + ) + .await + .expect("seed request IAM format"); + let iam = rustfs_iam::build_iam_sys(object_store.clone()) + .await + .expect("initialize request IAM"); + let context = Arc::new(AppContext::new( + object_store.clone(), + Arc::new(RequestIam { handle: iam }), + Arc::new(RequestKms), + )); + let access_key = "maintenance-handler-root"; + let secret_key = "maintenance-handler-root-secret"; + assert!(context.publish_action_credentials(rustfs_credentials::Credentials { + access_key: access_key.to_string(), + secret_key: secret_key.to_string(), + status: "on".to_string(), + ..Default::default() + })); + let request = |body: serde_json::Value| { + let mut request = table_catalog_handler_request(context.clone(), access_key, secret_key); + request.method = Method::POST; + request.uri = "/iceberg/v1/warehouse/namespaces/analytics/tables/events/maintenance/worker/run" + .parse() + .expect("worker URI"); + request.input = Body::from(serde_json::to_vec(&body).expect("worker request JSON")); + request + }; + let setup = request(serde_json::json!({})); + let backend = table_catalog_backend_from_extensions(&setup.extensions).expect("request catalog backend"); + let catalog = table_catalog_store_from_backend(backend.clone()).expect("object-backed catalog"); + enable_table_bucket_marker(object_store.as_ref(), "warehouse") + .await + .expect("enable table bucket"); + ensure_table_bucket_entry(&catalog, "warehouse", true) + .await + .expect("create table bucket entry"); + let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace"); + create_namespace_response( + &catalog, + "warehouse", + CreateNamespaceRequest { + namespace: vec!["analytics".to_string()], + properties: BTreeMap::new(), + }, + true, + ) + .await + .expect("create namespace"); + let create_request = serde_json::from_value::(serde_json::json!({ + "name": "events", + "schema": { + "type": "struct", + "schema-id": 0, + "fields": [{"id": 1, "name": "id", "required": true, "type": "long"}] + } + })) + .expect("create table request"); + create_table_response( + &catalog, + &TableCommitObjectBackend::trusted(backend.clone()), + "warehouse", + &namespace, + create_request, + true, + ) + .await + .expect("create table and real metadata"); + catalog + .put_table_maintenance_config( + "warehouse", + "analytics", + "events", + TableMaintenanceConfig { + background_enabled: true, + ..Default::default() + }, + ) + .await + .expect("enable background maintenance"); + let queued = catalog + .run_table_maintenance_scheduler_once("warehouse", "analytics", "events", "handler-scheduler".to_string()) + .await + .expect("queue maintenance") + .report; + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.job.attempt, 0); + let entry = catalog + .load_table("warehouse", "analytics", "events") + .await + .expect("load table") + .expect("table exists"); + let before = inventory(&backend).await; + + for body in [ + serde_json::json!({"worker-id":"worker-a", "expected-job-id":"missing-job", "expected-attempt":0}), + serde_json::json!({"worker-id":"worker-a", "expected-job-id":queued.job.job_id, "expected-attempt":1}), + ] { + let error = call_maintenance_worker_handler(request(body)) + .await + .expect_err("stale condition must not dispatch an unconditional worker"); + assert_eq!(error.status_code(), Some(StatusCode::CONFLICT)); + for id in [queued.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + catalog + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(queued.clone()), + ); + } + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry.clone())); + assert_eq!(inventory(&backend).await, before); + } + + let claim = serde_json::json!({ + "worker-id":"worker-a", "expected-job-id":queued.job.job_id, "expected-attempt":0 + }); + let response = call_maintenance_worker_handler(request(claim.clone())) + .await + .expect("authorized conditional claim"); + assert_eq!(response.output.0, StatusCode::OK); + let completed: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered worker response")).expect("worker report"); + assert_eq!(completed.job.job_id, queued.job.job_id); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.worker_id.as_deref(), Some("worker-a")); + for id in [queued.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + catalog + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(completed.clone()), + ); + } + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry.clone())); + let committed = inventory(&backend).await; + let response = call_maintenance_worker_handler(request(claim)) + .await + .expect("same-worker replay"); + assert_eq!(response.output.0, StatusCode::OK); + let replay: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered replay response")).expect("replayed report"); + assert_eq!(replay, completed); + assert_eq!(inventory(&backend).await, committed); + + let error = call_maintenance_worker_handler(request(serde_json::json!({ + "worker-id":"worker-b", "expected-job-id":queued.job.job_id, "expected-attempt":0 + }))) + .await + .expect_err("another worker cannot replay an old condition"); + assert_eq!(error.status_code(), Some(StatusCode::CONFLICT)); + for id in [queued.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + catalog + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(completed.clone()), + ); + } + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry.clone())); + assert_eq!(inventory(&backend).await, committed); + + let response = call_maintenance_worker_handler(request(serde_json::json!({}))) + .await + .expect("legacy unconditional request remains supported"); + assert_eq!(response.output.0, StatusCode::OK); + let legacy: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered legacy response")) + .expect("legacy worker report"); + assert_ne!(legacy.job.job_id, completed.job.job_id); + assert_eq!(legacy.job.worker_id.as_deref(), Some(DEFAULT_TABLE_MAINTENANCE_WORKER_ID)); + assert_eq!(legacy.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry)); + }, + ) + .await; +} + +#[tokio::test] +#[serial_test::serial] +async fn strong_metadata_maintenance_handler_plans_without_changing_table_objects() { + use crate::admin::storage_api::RUSTFS_META_BUCKET; + use crate::table_catalog::{ + TableCatalogObject, TableCatalogObjectPaths, TableMaintenanceAuditAction, TableMetadataMaintenanceJobStatus, + TableMetadataMaintenanceOperation, TableMetadataMaintenanceReport, + }; + + async fn inventory(backend: &B) -> BTreeMap<(String, String), TableCatalogObject> { + let mut objects = BTreeMap::new(); + for bucket in ["warehouse", RUSTFS_META_BUCKET] { + for object in backend.list_objects(bucket, "").await.expect("list isolated fixture objects") { + let stored = backend + .read_object(bucket, &object) + .await + .expect("read fixture object") + .expect("listed object exists"); + objects.insert((bucket.to_string(), object), stored); + } + } + objects + } + + temp_env::async_with_vars( + [( + crate::table_catalog::ENV_TABLE_CATALOG_BACKING, + Some(crate::table_catalog::TABLE_CATALOG_BACKING_DURABLE_STRONG), + )], + async { + let (_temp_dir, _disk_paths, object_store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + object_store + .make_bucket("warehouse", &MakeBucketOptions::default()) + .await + .expect("create table bucket"); + rustfs_iam::store::object::ObjectStore::new(object_store.clone()) + .save_iam_config( + serde_json::json!({"version": 1}), + format!("{}/format.json", *rustfs_iam::store::object::IAM_CONFIG_PREFIX), + ) + .await + .expect("seed request IAM format"); + let iam = rustfs_iam::build_iam_sys(object_store.clone()) + .await + .expect("initialize request IAM"); + let reader_access_key = "planning-handler-reader"; + let reader_secret_key = "planning-handler-reader-secret"; + iam.create_user( + reader_access_key, + &AddOrUpdateUserReq { + secret_key: reader_secret_key.to_string(), + policy: None, + status: AccountStatus::Enabled, + }, + ) + .await + .expect("create read-only user"); + iam.set_policy( + "planning-handler-read", + Policy::parse_config(br#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:GetTable"]}]}"#) + .expect("read-only policy"), + ) + .await + .expect("save read-only policy"); + iam.policy_db_set(reader_access_key, UserType::Reg, false, "planning-handler-read") + .await + .expect("attach read-only policy"); + let context = Arc::new(AppContext::new( + object_store.clone(), + Arc::new(RequestIam { handle: iam }), + Arc::new(RequestKms), + )); + let access_key = "planning-handler-root"; + let secret_key = "planning-handler-root-secret"; + assert!(context.publish_action_credentials(rustfs_credentials::Credentials { + access_key: access_key.to_string(), + secret_key: secret_key.to_string(), + status: "on".to_string(), + ..Default::default() + })); + let request = |body: &str| { + let mut request = table_catalog_handler_request(context.clone(), access_key, secret_key); + request.method = Method::POST; + request.uri = "/iceberg/v1/warehouse/namespaces/analytics/tables/events/maintenance/metadata" + .parse() + .expect("maintenance URI"); + request.input = Body::from(body.as_bytes().to_vec()); + request + }; + let setup = request("{}"); + let backend = table_catalog_backend_from_extensions(&setup.extensions).expect("request catalog backend"); + enable_table_bucket_marker(object_store.as_ref(), "warehouse") + .await + .expect("enable table bucket"); + // The isolated ECStore does not run catalog startup. Seed its snapshot + // before using the production request path, which requires it to exist. + let bootstrap = crate::table_catalog::StrongTableCatalogStore::new(backend.clone()); + ensure_table_bucket_entry(&bootstrap, "warehouse", true) + .await + .expect("seed durable snapshot and bucket entry"); + let catalog = table_catalog_store_from_backend(backend.clone()).expect("strong catalog"); + assert!(matches!(&catalog, crate::table_catalog::ConfiguredTableCatalogStore::DurableStrong(_))); + ensure_table_bucket_entry(&catalog, "warehouse", true) + .await + .expect("read bootstrapped bucket through configured catalog"); + let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace"); + let table = crate::table_catalog::IdentifierSegment::parse("events").expect("table"); + create_namespace_response( + &catalog, + "warehouse", + CreateNamespaceRequest { + namespace: vec!["analytics".to_string()], + properties: BTreeMap::new(), + }, + true, + ) + .await + .expect("create namespace"); + let create_request = serde_json::from_value::(serde_json::json!({ + "name": "events", + "schema": {"type": "struct", "schema-id": 0, + "fields": [{"id": 1, "name": "id", "required": true, "type": "long"}]} + })) + .expect("create request"); + create_table_response( + &catalog, + &TableCommitObjectBackend::trusted(backend.clone()), + "warehouse", + &namespace, + create_request, + true, + ) + .await + .expect("create table and metadata"); + let entry = catalog.load_table("warehouse", "analytics", "events").await.unwrap().unwrap(); + let before = inventory(&backend).await; + + for body in [ + r#"{"delete":true}"#, + r#"{"commit-snapshot-expiration":true}"#, + r#"{"commit-compaction":true}"#, + ] { + let error = call_metadata_maintenance_handler(request(body)) + .await + .expect_err("strong mutation must be rejected"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert!( + error + .message() + .unwrap_or_default() + .contains("not supported with durable-strong") + ); + assert_eq!(inventory(&backend).await, before, "{body}"); + } + for body in [ + r#"{"delete":null}"#, + r#"{"retain-recent-metadata-files":-1}"#, + r#"{"snapshot-expiration":{"min-snapshots-to-keep":"1","max-snapshot-age-ms":1000}}"#, + r#"{"compaction":true}"#, + r#"{"force":true}"#, + r#"{"delete":false,"delete":true}"#, + ] { + let error = call_metadata_maintenance_handler(request(body)) + .await + .expect_err("malformed request must be rejected"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert_eq!(inventory(&backend).await, before, "{body}"); + } + let mut anonymous = request("{}"); + anonymous.credentials = None; + let error = call_metadata_maintenance_handler(anonymous) + .await + .expect_err("planning requires authentication"); + assert_eq!(error.code(), &S3ErrorCode::InvalidRequest); + assert!(error.message().unwrap_or_default().contains("authentication required")); + let mut reader = request("not-json"); + reader.credentials = Some(s3s::auth::Credentials { + access_key: reader_access_key.to_string(), + secret_key: s3s::auth::SecretKey::from(reader_secret_key.to_string()), + }); + let error = call_metadata_maintenance_handler(reader) + .await + .expect_err("read-only permission must not authorize planning"); + assert_eq!(error.code(), &S3ErrorCode::AccessDenied); + assert_eq!(inventory(&backend).await, before); + + for body in [ + serde_json::json!({}), + serde_json::json!({ + "retain-recent-metadata-files": 2, + "snapshot-expiration": {"min-snapshots-to-keep": 1, "max-snapshot-age-ms": 1000}, + "compaction": {"target-file-size-bytes": 1024, "small-file-threshold-bytes": 512, + "min-input-files": 2, "max-rewrite-bytes-per-job": 4096} + }), + ] { + let before = inventory(&backend).await; + let response = call_metadata_maintenance_handler(request(&body.to_string())) + .await + .expect("authorized strong planning"); + assert_eq!(response.output.0, StatusCode::OK); + let report: TableMetadataMaintenanceReport = + serde_json::from_slice(&response.output.1.bytes().expect("buffered planning response")) + .expect("planning report"); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(report.job.table_id, entry.table_id); + assert_eq!(report.current_metadata_location, entry.metadata_location); + assert_eq!(report.job.current_generation, entry.generation); + assert_eq!(report.job.deleted_metadata_file_count, 0); + assert_eq!(report.job.deleted_object_count, 0); + assert_eq!( + report.job.retain_recent_metadata_files, + if body.get("compaction").is_some() { 2 } else { 0 } + ); + assert_eq!(report.snapshot_expiration.is_some(), body.get("snapshot-expiration").is_some()); + assert_eq!(report.compaction.is_some(), body.get("compaction").is_some()); + if let Some(plan) = &report.snapshot_expiration { + assert_eq!(plan.config.min_snapshots_to_keep, 1); + assert_eq!(plan.config.max_snapshot_age_ms, 1000); + assert!(plan.committed_metadata_location.is_none()); + } + if let Some(plan) = &report.compaction { + assert_eq!(plan.config.target_file_size_bytes, 1024); + assert_eq!(plan.config.max_rewrite_bytes_per_job, 4096); + assert!(plan.committed_metadata_location.is_none()); + } + assert!( + report + .audit_events + .iter() + .any(|event| event.action == TableMaintenanceAuditAction::WorkerStarted) + ); + assert!( + report + .audit_events + .iter() + .any(|event| event.action == TableMaintenanceAuditAction::WorkerSucceeded) + ); + let reloaded = table_catalog_store_from_backend(backend.clone()).expect("reload strong catalog"); + for id in [report.job.job_id.as_str(), "latest", "current"] { + assert_eq!( + reloaded + .get_table_metadata_maintenance_report("warehouse", "analytics", "events", id) + .await + .unwrap(), + Some(report.clone()) + ); + } + assert_eq!( + reloaded.load_table("warehouse", "analytics", "events").await.unwrap(), + Some(entry.clone()) + ); + let paths = TableCatalogObjectPaths::default(); + let report_paths = [ + paths.table_maintenance_job_path("warehouse", &namespace, &table, &entry.table_id, &report.job.job_id), + paths.table_maintenance_latest_job_path("warehouse", &namespace, &table, &entry.table_id), + paths.table_maintenance_current_job_path("warehouse", &namespace, &table, &entry.table_id), + ]; + let outside_report = |((bucket, key), _): &((String, String), TableCatalogObject)| { + bucket != RUSTFS_META_BUCKET || !report_paths.contains(key) + }; + assert_eq!( + inventory(&backend) + .await + .into_iter() + .filter(outside_report) + .collect::>(), + before.into_iter().filter(outside_report).collect::>(), + "planning may only publish its named, latest, and current audit reports", + ); + } + + catalog + .put_table_maintenance_config( + "warehouse", + "analytics", + "events", + crate::table_catalog::TableMaintenanceConfig { + background_enabled: true, + ..Default::default() + }, + ) + .await + .expect("enable queue admission"); + let queued = catalog + .run_table_maintenance_scheduler_once("warehouse", "analytics", "events", "planning-test-scheduler".to_string()) + .await + .expect("queue a maintenance job") + .report; + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + let before = inventory(&backend).await; + let error = call_metadata_maintenance_handler(request("{}")) + .await + .expect_err("planning must not replace a queued job"); + assert_eq!(error.status_code(), Some(StatusCode::CONFLICT)); + assert_eq!(inventory(&backend).await, before); + assert_eq!(catalog.load_table("warehouse", "analytics", "events").await.unwrap(), Some(entry)); + }, + ) + .await; } #[tokio::test] @@ -646,17 +1199,25 @@ fn catalog_conflicts_use_operation_specific_iceberg_errors() { #[test] fn catalog_unavailable_errors_use_iceberg_503() { - let unavailable = catalog_store_error(crate::table_catalog::TableCatalogStoreError::Unavailable( - "failed to acquire catalog table lock: quorum required 3, achieved 1".to_string(), - )); - assert_eq!(unavailable.code(), &S3ErrorCode::Custom(ICEBERG_ERROR_REST.into())); - assert_eq!(unavailable.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); - assert_eq!(unavailable.message(), Some("table catalog is temporarily unavailable")); - assert!( - unavailable - .headers() - .is_none_or(|headers| !headers.contains_key(rustfs_utils::http::RETRY_AFTER)) - ); + for error in [ + crate::table_catalog::TableCatalogStoreError::Unavailable( + "failed to acquire catalog table lock: quorum required 3, achieved 1".to_string(), + ), + crate::table_catalog::TableCatalogStoreError::LockUnavailable { + message: "private publication fence".to_string(), + source: Box::new(rustfs_lock::LockError::timeout("private/lock", std::time::Duration::from_secs(5))), + }, + ] { + let unavailable = catalog_store_error(error); + assert_eq!(unavailable.code(), &S3ErrorCode::Custom(ICEBERG_ERROR_REST.into())); + assert_eq!(unavailable.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE)); + assert_eq!(unavailable.message(), Some("table catalog is temporarily unavailable")); + assert!( + unavailable + .headers() + .is_none_or(|headers| !headers.contains_key(rustfs_utils::http::RETRY_AFTER)) + ); + } let internal = catalog_store_error(crate::table_catalog::TableCatalogStoreError::Internal( "commit state is unknown".to_string(), @@ -724,6 +1285,7 @@ fn table_catalog_handlers_require_table_admin_actions() { ("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"), ("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"), ("RunTableMaintenanceSchedulerHandler", "AdminAction::RunTableMaintenanceAction"), + ("RunTableMaintenanceWorkerHandler", "AdminAction::RunTableMaintenanceAction"), ("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"), ("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"), ("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"), @@ -810,6 +1372,7 @@ fn table_catalog_handlers_require_table_admin_actions() { ("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"), ("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"), ("RunTableMaintenanceSchedulerHandler", "AdminAction::RunTableMaintenanceAction"), + ("RunTableMaintenanceWorkerHandler", "AdminAction::RunTableMaintenanceAction"), ("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"), ("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"), ("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"), @@ -1026,6 +1589,99 @@ fn table_commit_publication_acquires_migration_permits_before_publication_lock() ); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn table_commit_publication_forwards_lease_loss_to_storage_mutation_fence() { + use crate::admin::storage_api::RUSTFS_META_BUCKET; + use crate::table_catalog::test_support::RecordingLeaseClient; + use crate::table_catalog::{TableCatalogLockGuard, TableCatalogObjectPaths, TableCommitPublication}; + use rustfs_lock::LockClient; + + let metadata_backend = TestTableCatalogObjectBackend::content_addressed(); + let commit_backend = TableCommitObjectBackend::trusted(metadata_backend.clone()); + TableCommitPublication::begin_table_bucket(&commit_backend, "analytics") + .await + .unwrap(); + TableCommitPublication::prepare(&commit_backend, "analytics", "sales", "orders") + .await + .unwrap(); + let client = Arc::new(RecordingLeaseClient { + inner: rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new( + rustfs_lock::FastObjectLockManager::new(), + )))), + acquired: std::sync::Mutex::new(None), + }); + let lock = rustfs_lock::NamespaceLockWrapper::new( + rustfs_lock::NamespaceLock::with_clients_and_quorum( + "rest-publication-fence-probe".to_string(), + vec![client.clone() as Arc], + 1, + ), + rustfs_lock::ObjectKey::new( + RUSTFS_META_BUCKET, + TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(), + ), + "rest-publication-migration".to_string(), + ); + let guard = lock.get_read_lock(StdDuration::from_secs(5)).await.unwrap(); + // The guard exposes an aggregate ID; LocalClient owns the recorded response ID. + let lock_id = client + .acquired + .lock() + .unwrap() + .clone() + .expect("local migration lease has an ID"); + assert!(client.check_status(&lock_id).await.unwrap().is_some()); + { + let mut publication = commit_backend.publication.lock(); + publication.guards.push(TableCatalogLockGuard::test_namespace(guard)); + publication.catalog_migration_read_permit = true; + } + let fence = TableCommitPublication::object_mutation_fence(&commit_backend) + .expect("REST publication must forward its held guards to the storage fence"); + fence.ensure_held().unwrap(); + assert!(TableCommitPublication::holds_table_bucket(&commit_backend, "analytics")); + assert!(TableCommitPublication::holds_table(&commit_backend, "analytics", "sales", "orders")); + assert_eq!(TableCommitPublication::catalog_migration_read_permit_status(&commit_backend), Some(true)); + + let object = "rest-publication-fence-probe.json"; + metadata_backend + .seed_object(RUSTFS_META_BUCKET, object, b"before".to_vec()) + .await; + let before = metadata_backend + .read_object(RUSTFS_META_BUCKET, object) + .await + .unwrap() + .unwrap(); + let pause = metadata_backend.pause_next_put(RUSTFS_META_BUCKET, object).await; + let operation = metadata_backend.put_object_fenced( + RUSTFS_META_BUCKET, + object, + b"after".to_vec(), + crate::table_catalog::TableCatalogPutPrecondition::IfMatch(before.etag.clone().unwrap()), + &fence, + ); + let interfere = async { + tokio::time::timeout(StdDuration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + assert!(client.force_release(&lock_id).await.unwrap()); + tokio::time::timeout(StdDuration::from_secs(15), async { + while fence.ensure_held().is_ok() { + tokio::time::sleep(StdDuration::from_millis(10)).await; + } + }) + .await + .expect("REST publication fence observes migration lease loss during backend await"); + assert!(!TableCommitPublication::holds_table_bucket(&commit_backend, "analytics")); + assert!(!TableCommitPublication::holds_table(&commit_backend, "analytics", "sales", "orders")); + assert_eq!(TableCommitPublication::catalog_migration_read_permit_status(&commit_backend), Some(false)); + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(metadata_backend.read_object(RUSTFS_META_BUCKET, object).await.unwrap(), Some(before)); +} + #[test] fn table_catalog_resource_builds_policy_object_scope() { let namespace = crate::table_catalog::Namespace::parse("analytics.daily_events").expect("namespace should parse"); @@ -1271,6 +1927,7 @@ fn table_maintenance_worker_run_request_uses_stable_default_worker_id() { serde_json::from_value(serde_json::json!({})).expect("worker run request should parse"); assert_eq!(request.worker_id(), "rustfs-maintenance-worker"); + assert!(request.condition().expect("legacy request").is_none()); } #[test] @@ -1281,6 +1938,84 @@ fn table_maintenance_worker_run_request_accepts_worker_id() { .expect("worker run request should parse worker id"); assert_eq!(request.worker_id(), "worker-a"); + assert!(request.condition().expect("legacy request").is_none()); +} + +#[test] +fn table_maintenance_worker_condition_accepts_paired_fields() { + for attempt in [0, 1, u16::MAX] { + let request: TableMaintenanceWorkerRunRequest = serde_json::from_value(serde_json::json!({ + "expected-job-id": "job-a", "expected-attempt": attempt + })) + .expect("paired condition parses"); + let condition = request + .condition() + .expect("valid condition") + .expect("condition remains present"); + assert_eq!(condition.expected_job_id, "job-a"); + assert_eq!(condition.expected_attempt, attempt); + } +} + +#[test] +fn table_maintenance_worker_condition_rejects_incomplete_or_empty_fields() { + for value in [ + serde_json::json!({"expected-job-id":"job-a"}), + serde_json::json!({"expected-attempt":0}), + serde_json::json!({"expected-job-id":"", "expected-attempt":0}), + serde_json::json!({"expected-job-id":" ", "expected-attempt":0}), + serde_json::json!({"worker-id":" ", "expected-job-id":"job-a", "expected-attempt":0}), + ] { + let request: TableMaintenanceWorkerRunRequest = serde_json::from_value(value).expect("well-typed request"); + let err = request + .condition() + .expect_err("partial or empty condition must not run unconditionally"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + } +} + +#[test] +fn table_maintenance_worker_condition_rejects_null_malformed_duplicate_and_unknown_fields() { + for body in [ + r#"{"expected-job-id":null,"expected-attempt":null}"#, + r#"{"expected-job-id":null}"#, + r#"{"expected-attempt":null}"#, + r#"{"expected-job-id":"job-a","expected-attempt":-1}"#, + r#"{"expected-job-id":"job-a","expected-attempt":65536}"#, + r#"{"expected-job-id":"job-a","expected-attempt":0.5}"#, + r#"{"expected-job-id":"job-a","expected-attempt":"0"}"#, + r#"{"expected-job-id":"job-a","expected-attempt":false}"#, + r#"{"expected-job-id":1,"expected-attempt":0}"#, + r#"{"expected-job-id":"job-a","expected-job-id":"job-b","expected-attempt":0}"#, + r#"{"expected-job-id":"job-a","expected-attempt":0,"expected-attempt":1}"#, + r#"{"expected-job-id":"job-a","expected-attempt":0,"force":true}"#, + ] { + assert!(serde_json::from_str::(body).is_err(), "{body}"); + } +} + +#[tokio::test] +async fn table_maintenance_worker_handler_requires_authentication_for_both_request_forms() { + for body in [r#"{}"#, r#"{"expected-job-id":"job-a","expected-attempt":0}"#] { + let request = S3Request { + input: Body::from(body.as_bytes().to_vec()), + method: Method::POST, + uri: "/iceberg/v1/warehouse/namespaces/analytics/tables/events/maintenance/worker/run" + .parse() + .expect("worker URI"), + headers: HeaderMap::new(), + extensions: http::Extensions::new(), + credentials: None, + region: None, + service: None, + trailing_headers: None, + }; + let err = call_maintenance_worker_handler(request) + .await + .expect_err("worker requires authentication"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + assert!(err.message().unwrap_or_default().contains("authentication required")); + } } #[test] @@ -4211,24 +4946,46 @@ fn table_metadata_file_name_scoping_is_bounded_and_identity_sensitive() { #[tokio::test] async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metadata_files() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::DurableStrong, "curated").await; +} + +#[tokio::test] +async fn object_catalog_cross_namespace_rename_accepts_isolated_standard_commits() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::ObjectBacked, "curated").await; +} + +#[tokio::test] +async fn object_catalog_same_namespace_rename_accepts_isolated_standard_commits() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::ObjectBacked, "analytics").await; +} + +#[tokio::test] +async fn strong_catalog_same_namespace_rename_accepts_isolated_standard_commits() { + assert_renamed_and_recreated_table_commits(crate::table_catalog::TableCatalogBackingMode::DurableStrong, "analytics").await; +} + +async fn assert_renamed_and_recreated_table_commits(mode: crate::table_catalog::TableCatalogBackingMode, destination: &str) { let metadata_backend = TestTableCatalogObjectBackend::content_addressed(); - let store = crate::table_catalog::StrongTableCatalogStore::new(metadata_backend.clone()); + let store = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend.clone(), mode); let source_namespace = crate::table_catalog::Namespace::parse("analytics").expect("source namespace should parse"); - let destination_namespace = crate::table_catalog::Namespace::parse("curated").expect("destination namespace should parse"); + let destination_namespace = crate::table_catalog::Namespace::parse(destination).expect("destination namespace should parse"); create_standard_events_table(&store, &metadata_backend, &source_namespace).await; - create_namespace_response( - &store, - "warehouse", - CreateNamespaceRequest { - namespace: vec!["curated".to_string()], - properties: BTreeMap::new(), - }, - true, - ) - .await - .expect("destination namespace should be created"); + let original = store.load_table("warehouse", "analytics", "events").await.unwrap().unwrap(); + if source_namespace != destination_namespace { + create_namespace_response( + &store, + "warehouse", + CreateNamespaceRequest { + namespace: vec![destination.to_string()], + properties: BTreeMap::new(), + }, + true, + ) + .await + .expect("destination namespace should be created"); + } store - .rename_table("warehouse", "analytics", "events", "curated", "events_v2") + .rename_table("warehouse", "analytics", "events", destination, "events_v2") .await .expect("table should rename"); @@ -4249,7 +5006,7 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad .expect("source identifier should be reusable"); let renamed = store - .load_table("warehouse", "curated", "events_v2") + .load_table("warehouse", destination, "events_v2") .await .expect("renamed table lookup should succeed") .expect("renamed table should exist"); @@ -4259,18 +5016,196 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad .expect("recreated table lookup should succeed") .expect("recreated table should exist"); assert_ne!(renamed.table_id, recreated.table_id); + assert_ne!(renamed.table_uuid, recreated.table_uuid); + assert_ne!(renamed.warehouse_location, recreated.warehouse_location); + assert_eq!(renamed.table_id, original.table_id); + assert_eq!(renamed.table_uuid, original.table_uuid); + assert_eq!(renamed.warehouse_location, original.warehouse_location); + assert_eq!(renamed.metadata_location, original.metadata_location); + assert_eq!(renamed.version_token, original.version_token); + assert_eq!(renamed.generation, original.generation); let commit_id = "11111111-1111-4111-8111-111111111111"; + let renamed_request = || { + let mut request = standard_property_commit_request(commit_id, &renamed.table_uuid, "curated"); + request.idempotency_key = Some("renamed-request".to_string()); + request + }; let renamed_commit = standard_commit_table_response( &store, &trusted_table_commit_backend(&metadata_backend), "warehouse", &destination_namespace, "events_v2", - standard_property_commit_request(commit_id, &renamed.table_uuid, "curated"), + renamed_request(), ) .await .expect("renamed table commit should succeed"); + let committed = store + .load_table("warehouse", destination, "events_v2") + .await + .unwrap() + .unwrap(); + assert_eq!(committed.table_id, renamed.table_id); + assert_eq!(committed.table_uuid, renamed.table_uuid); + assert_eq!(committed.warehouse_location, renamed.warehouse_location); + assert_eq!(committed.generation, renamed.generation + 1); + assert_ne!(committed.version_token, renamed.version_token); + assert_ne!(committed.metadata_location, renamed.metadata_location); + assert_eq!(renamed_commit.metadata["table-uuid"], renamed.table_uuid); + assert_eq!(renamed_commit.metadata["location"], renamed.warehouse_location); + assert_eq!(renamed_commit.metadata["properties"]["owner"], "curated"); + let receipt = store + .get_commit_by_id("warehouse", &renamed.table_id, commit_id) + .await + .unwrap() + .unwrap(); + assert_eq!(receipt.status, crate::table_catalog::CommitLogStatus::Committed); + assert_eq!(receipt.new_version_token, committed.version_token); + assert_eq!(receipt.new_metadata_location, committed.metadata_location); + assert_eq!( + store + .get_commit_by_idempotency_key("warehouse", &renamed.table_id, receipt.idempotency_key.as_deref().unwrap()) + .await + .unwrap(), + Some(receipt.clone()) + ); + let replay = standard_commit_table_response( + &store, + &trusted_table_commit_backend(&metadata_backend), + "warehouse", + &destination_namespace, + "events_v2", + renamed_request(), + ) + .await + .expect("same commit ID should replay after rename"); + assert_eq!(replay.metadata_location, renamed_commit.metadata_location); + assert_eq!(replay.metadata, renamed_commit.metadata); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); + + let stale_request = crate::table_catalog::TableCommitRequest { + table_bucket: "warehouse".to_string(), + namespace: destination.to_string(), + table: "events_v2".to_string(), + commit_id: "stale-after-rename".to_string(), + idempotency_key: Some("stale-after-rename".to_string()), + operation: "append".to_string(), + expected_version_token: renamed.version_token.clone(), + expected_metadata_location: renamed.metadata_location.clone(), + new_metadata_location: committed.metadata_location.clone(), + requirements: Vec::new(), + writer: None, + }; + for metadata_only in [false, true] { + let mut request = stale_request.clone(); + if metadata_only { + request.expected_version_token = committed.version_token.clone(); + request.commit_id = "stale-metadata-after-rename".to_string(); + request.idempotency_key = Some(request.commit_id.clone()); + } + assert!(matches!( + store.commit_table(request.clone()).await, + Err(crate::table_catalog::TableCatalogStoreError::Conflict(_)) + )); + assert!( + store + .get_commit_by_id("warehouse", &renamed.table_id, &request.commit_id) + .await + .unwrap() + .is_none() + ); + assert!( + store + .get_commit_by_idempotency_key("warehouse", &renamed.table_id, request.idempotency_key.as_deref().unwrap()) + .await + .unwrap() + .is_none() + ); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); + } + let current_namespace_path = crate::table_catalog::default_table_metadata_dir_path( + &destination_namespace, + &crate::table_catalog::IdentifierSegment::parse("events_v2").unwrap(), + ); + let foreign_namespace_path = crate::table_catalog::default_table_metadata_dir_path( + &source_namespace, + &crate::table_catalog::IdentifierSegment::parse("foreign").unwrap(), + ); + let retained_directory = committed.metadata_location.rsplit_once('/').unwrap().0; + let persisted_metadata = read_table_metadata_json(&metadata_backend, "warehouse", &committed.metadata_location) + .await + .unwrap(); + for (index, location) in [ + format!("{current_namespace_path}/00003.metadata.json"), + format!("{foreign_namespace_path}/00003.metadata.json"), + format!("s3://other-warehouse/{foreign_namespace_path}/00003.metadata.json"), + format!("{retained_directory}/../00003.metadata.json"), + format!("{retained_directory}/child/00003.metadata.json"), + format!("{retained_directory}/%2e%2e.json"), + ] + .into_iter() + .enumerate() + { + if index < 2 { + metadata_backend + .put_json("warehouse", &location, persisted_metadata.clone()) + .await; + } + let mut request = stale_request.clone(); + request.commit_id = format!("invalid-renamed-path-{index}"); + request.idempotency_key = Some(request.commit_id.clone()); + request.expected_version_token = committed.version_token.clone(); + request.expected_metadata_location = committed.metadata_location.clone(); + request.new_metadata_location = location; + assert!(matches!( + store.commit_table(request.clone()).await, + Err(crate::table_catalog::TableCatalogStoreError::Invalid(_)) + )); + assert!( + store + .get_commit_by_id("warehouse", &renamed.table_id, &request.commit_id) + .await + .unwrap() + .is_none() + ); + assert!( + store + .get_commit_by_idempotency_key("warehouse", &renamed.table_id, request.idempotency_key.as_deref().unwrap()) + .await + .unwrap() + .is_none() + ); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); + assert_eq!( + store.load_table("warehouse", "analytics", "events").await.unwrap(), + Some(recreated.clone()) + ); + } + let wrong_identity = standard_commit_table_response( + &store, + &trusted_table_commit_backend(&metadata_backend), + "warehouse", + &destination_namespace, + "events_v2", + standard_property_commit_request("22222222-2222-4222-8222-222222222222", &recreated.table_uuid, "wrong-identity"), + ) + .await + .expect_err("recreated source identity cannot commit to the renamed table"); + assert_eq!(wrong_identity.status_code(), Some(StatusCode::CONFLICT)); + assert_eq!( + store.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed.clone()) + ); let recreated_commit = standard_commit_table_response( &store, &trusted_table_commit_backend(&metadata_backend), @@ -4300,7 +5235,7 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad assert!( !renamed_commit .metadata_location - .contains("/namespaces/curated/tables/events_v2/") + .contains(&format!("/namespaces/{destination}/tables/events_v2/")) ); assert!( recreated_commit @@ -4309,8 +5244,8 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad ); assert_eq!(recreated_commit.metadata["table-uuid"], recreated.table_uuid); assert_eq!(recreated_commit.metadata["properties"]["owner"], "analytics"); - for metadata_location in [renamed_commit.metadata_location, recreated_commit.metadata_location] { - let object_key = test_snapshot_object_key("warehouse", &metadata_location); + for metadata_location in [&renamed_commit.metadata_location, &recreated_commit.metadata_location] { + let object_key = test_snapshot_object_key("warehouse", metadata_location); assert!( metadata_backend .object_exists("warehouse", &object_key) @@ -4318,6 +5253,27 @@ async fn renamed_and_recreated_tables_with_the_same_commit_id_use_disjoint_metad .expect("metadata lookup should succeed") ); } + let restarted = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend, mode); + assert_eq!( + restarted.load_table("warehouse", destination, "events_v2").await.unwrap(), + Some(committed) + ); + let recreated_after = restarted + .load_table("warehouse", "analytics", "events") + .await + .unwrap() + .unwrap(); + assert_eq!(recreated_after.table_id, recreated.table_id); + assert_eq!(recreated_after.table_uuid, recreated.table_uuid); + assert_eq!(recreated_after.warehouse_location, recreated.warehouse_location); + assert_eq!(recreated_after.generation, recreated.generation + 1); + assert_eq!( + restarted + .get_commit_by_id("warehouse", &renamed.table_id, commit_id) + .await + .unwrap(), + Some(receipt) + ); } #[tokio::test] diff --git a/rustfs/src/admin/handlers/user.rs b/rustfs/src/admin/handlers/user.rs index 5d58382bf..1d01339d2 100644 --- a/rustfs/src/admin/handlers/user.rs +++ b/rustfs/src/admin/handlers/user.rs @@ -163,7 +163,7 @@ fn imported_service_account_status(status: &str) -> Option { const SERVICE_ACCOUNT_PARENT_SCOPE_ERROR: &str = "service account parent is outside requester scope"; const SERVICE_ACCOUNT_ACCESS_KEY_MISMATCH_ERROR: &str = "service account access key does not match import entry"; -fn imported_service_account_parent_allowed(parent: &str, requester: &Credentials, owner: bool) -> bool { +fn imported_service_account_parent_allowed(root_access_key: &str, parent: &str, requester: &Credentials, owner: bool) -> bool { if parent.is_empty() { return false; } @@ -172,6 +172,10 @@ fn imported_service_account_parent_allowed(parent: &str, requester: &Credentials return true; } + if parent == root_access_key { + return false; + } + if requester.is_temp() || requester.is_service_account() { return temp_identity_parent(requester).is_some_and(|requester_parent| requester_parent == parent); } @@ -180,12 +184,13 @@ fn imported_service_account_parent_allowed(parent: &str, requester: &Credentials } fn imported_service_account_parent_scope_failure( + root_access_key: &str, access_key: &str, parent: &str, requester: &Credentials, owner: bool, ) -> Option { - (!imported_service_account_parent_allowed(parent, requester, owner)).then(|| IAMErrEntity { + (!imported_service_account_parent_allowed(root_access_key, parent, requester, owner)).then(|| IAMErrEntity { name: access_key.to_string(), error: SERVICE_ACCOUNT_PARENT_SCOPE_ERROR.to_string(), }) @@ -1089,7 +1094,9 @@ impl Operation for ImportIam { continue; } - if let Some(err) = imported_service_account_parent_scope_failure(&ak, &req.parent, &cred, owner) { + if let Some(err) = + imported_service_account_parent_scope_failure(&sys_cred.access_key, &ak, &req.parent, &cred, owner) + { failed.service_accounts.push(err); continue; } @@ -1657,7 +1664,12 @@ mod tests { ..Default::default() }; - assert!(!imported_service_account_parent_allowed("root-access-key", &requester, false)); + assert!(!imported_service_account_parent_allowed( + "root-access-key", + "root-access-key", + &requester, + false + )); } #[test] @@ -1666,13 +1678,26 @@ mod tests { access_key: "delegated-importer".to_string(), ..Default::default() }; - let err = imported_service_account_parent_scope_failure("svc-access-key", "root-access-key", &requester, false) - .expect("non-owner must not import a service account for another parent"); + let err = imported_service_account_parent_scope_failure( + "root-access-key", + "svc-access-key", + "root-access-key", + &requester, + false, + ) + .expect("non-owner must not import a service account for another parent"); assert_eq!(err.name, "svc-access-key"); assert_eq!(err.error, SERVICE_ACCOUNT_PARENT_SCOPE_ERROR); assert!( - imported_service_account_parent_scope_failure("svc-access-key", "delegated-importer", &requester, false).is_none() + imported_service_account_parent_scope_failure( + "root-access-key", + "svc-access-key", + "delegated-importer", + &requester, + false + ) + .is_none() ); } @@ -1703,6 +1728,33 @@ mod tests { assert!(imported_service_account_access_key_failure("svcalpha", "svcalpha").is_none()); } + #[test] + fn credential_boundary_root_derived_non_owner_import_rejects_owner_service_account() { + let root = "deployment-root"; + for temporary in [false, true] { + let requester = Credentials { + access_key: "restricted-account".to_string(), + parent_user: root.to_string(), + session_token: "session-token".to_string(), + expiration: temporary.then(|| time::OffsetDateTime::now_utc() + time::Duration::hours(1)), + claims: (!temporary).then(|| { + HashMap::from([( + rustfs_credentials::IAM_POLICY_CLAIM_NAME_SA.to_string(), + serde_json::json!("inherited-policy"), + )]) + }), + ..Default::default() + }; + assert_eq!(requester.is_service_account(), !temporary); + assert!(requester.is_temp()); + let error = imported_service_account_parent_scope_failure(root, "backdoor", root, &requester, false) + .expect("a restricted root-derived caller must not import an owner account"); + assert_eq!(error.name, "backdoor"); + assert_eq!(error.error, SERVICE_ACCOUNT_PARENT_SCOPE_ERROR); + assert!(imported_service_account_parent_allowed(root, root, &requester, true)); + } + } + #[test] fn test_import_service_account_parent_allows_owner_restore() { let requester = Credentials { @@ -1710,7 +1762,12 @@ mod tests { ..Default::default() }; - assert!(imported_service_account_parent_allowed("any-imported-parent", &requester, true)); + assert!(imported_service_account_parent_allowed( + "root-access-key", + "any-imported-parent", + &requester, + true + )); } #[test] @@ -1720,7 +1777,12 @@ mod tests { ..Default::default() }; - assert!(imported_service_account_parent_allowed("delegated-importer", &requester, false)); + assert!(imported_service_account_parent_allowed( + "root-access-key", + "delegated-importer", + &requester, + false + )); } #[test] @@ -1732,8 +1794,18 @@ mod tests { ..Default::default() }; - assert!(imported_service_account_parent_allowed("parent-user", &requester, false)); - assert!(!imported_service_account_parent_allowed("other-parent", &requester, false)); + assert!(imported_service_account_parent_allowed( + "root-access-key", + "parent-user", + &requester, + false + )); + assert!(!imported_service_account_parent_allowed( + "root-access-key", + "other-parent", + &requester, + false + )); } #[test] diff --git a/rustfs/src/admin/route_policy.rs b/rustfs/src/admin/route_policy.rs index 8acd5f313..be233427d 100644 --- a/rustfs/src/admin/route_policy.rs +++ b/rustfs/src/admin/route_policy.rs @@ -395,6 +395,12 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[ SET_BUCKET_QUOTA, RouteRiskLevel::High, ), + admin( + HttpMethod::Post, + "/rustfs/admin/v3/quota-migration", + SET_BUCKET_QUOTA, + RouteRiskLevel::High, + ), admin( HttpMethod::Put, "/rustfs/admin/v3/bucket-durability/{bucket}", diff --git a/rustfs/src/admin/route_registration_test.rs b/rustfs/src/admin/route_registration_test.rs index ac24a3d6f..130098d91 100644 --- a/rustfs/src/admin/route_registration_test.rs +++ b/rustfs/src/admin/route_registration_test.rs @@ -263,6 +263,7 @@ fn expected_admin_route_matrix() -> Vec { admin_route_sample(Method::DELETE, "/v3/quota/{bucket}", "/v3/quota/test-bucket"), admin_route_sample(Method::GET, "/v3/quota-stats/{bucket}", "/v3/quota-stats/test-bucket"), admin_route_sample(Method::POST, "/v3/quota-check/{bucket}", "/v3/quota-check/test-bucket"), + admin_route(Method::POST, "/v3/quota-migration"), admin_route_sample(Method::PUT, "/v3/bucket-durability/{bucket}", "/v3/bucket-durability/test-bucket"), admin_route_sample(Method::GET, "/v3/bucket-durability/{bucket}", "/v3/bucket-durability/test-bucket"), admin_route_sample(Method::DELETE, "/v3/bucket-durability/{bucket}", "/v3/bucket-durability/test-bucket"), @@ -1382,6 +1383,7 @@ fn test_register_routes_cover_representative_admin_paths() { assert_route(&router, Method::GET, &admin_path("/v3/get-bucket-quota")); assert_route(&router, Method::PUT, &admin_path("/v3/quota/test-bucket")); assert_route(&router, Method::GET, &admin_path("/v3/quota-stats/test-bucket")); + assert_route(&router, Method::POST, &admin_path("/v3/quota-migration")); assert_route(&router, Method::PUT, &admin_path("/v3/bucket-durability/test-bucket")); assert_route(&router, Method::GET, &admin_path("/v3/bucket-durability/test-bucket")); assert_route(&router, Method::DELETE, &admin_path("/v3/bucket-durability/test-bucket")); diff --git a/rustfs/src/admin/service/federated_identity.rs b/rustfs/src/admin/service/federated_identity.rs index 7417a7322..1641a9447 100644 --- a/rustfs/src/admin/service/federated_identity.rs +++ b/rustfs/src/admin/service/federated_identity.rs @@ -30,6 +30,10 @@ use std::collections::{BTreeMap, HashMap}; use time::{Duration, OffsetDateTime}; use tracing::{debug, warn}; +const LOG_COMPONENT_ADMIN: &str = "admin"; +const LOG_SUBSYSTEM_OIDC: &str = "oidc"; +const EVENT_ADMIN_OIDC_POLICY: &str = "admin_oidc_policy"; + pub(crate) struct DefaultFederatedSessionBinding; // Legacy receivers reject an empty parsed mapping; current receivers ignore it when the virtual-parent marker is present. @@ -104,34 +108,33 @@ fn select_oidc_policies( fn build_oidc_token_claims(transaction: &FederatedSessionTransaction) -> HashMap { let authorization = &transaction.authorization; - let claims = &authorization.claims; let mut token_claims = HashMap::new(); - token_claims.insert("sub".to_string(), Value::String(claims.sub.clone())); + token_claims.insert("sub".to_string(), Value::String(authorization.subject().to_string())); token_claims.insert("iss".to_string(), Value::String("rustfs-oidc".to_string())); - token_claims.insert("oidc_provider".to_string(), Value::String(authorization.provider_id.clone())); + token_claims.insert("oidc_provider".to_string(), Value::String(authorization.provider_id().to_string())); - if !claims.email.is_empty() { - token_claims.insert("email".to_string(), Value::String(claims.email.clone())); + if !authorization.email().is_empty() { + token_claims.insert("email".to_string(), Value::String(authorization.email().to_string())); } - if !claims.username.is_empty() { - token_claims.insert("preferred_username".to_string(), Value::String(claims.username.clone())); + if !authorization.username().is_empty() { + token_claims.insert("preferred_username".to_string(), Value::String(authorization.username().to_string())); } - if !authorization.groups.is_empty() { + if !authorization.mapped_iam_groups().is_empty() { token_claims.insert( "groups".to_string(), Value::Array( authorization - .groups + .mapped_iam_groups() .iter() .map(|group| Value::String(group.clone())) .collect(), ), ); } - if !authorization.roles.is_empty() { + if !authorization.roles().is_empty() { token_claims.insert( "roles".to_string(), - Value::Array(authorization.roles.iter().cloned().map(Value::String).collect()), + Value::Array(authorization.roles().iter().cloned().map(Value::String).collect()), ); } @@ -242,7 +245,6 @@ fn issue_credentials( secret: Option<&str>, ) -> Result { let authorization = &transaction.authorization; - let claims = &authorization.claims; let parent_user = authorization.oidc_virtual_parent().ok_or_else(|| { FederatedSessionBindingError::InvalidRequest("verified OIDC identity is missing issuer or subject".to_string()) })?; @@ -262,16 +264,19 @@ fn issue_credentials( )); } debug!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, - email = %claims.email, - username = %claims.username, - sub = %claims.sub, - policy_count = authorization.policies.len(), - group_count = authorization.groups.len(), - policies = ?authorization.policies, - groups = ?authorization.groups, - roles_claim_key = ?authorization.roles_claim_key, + email = %authorization.email(), + username = %authorization.username(), + sub = %authorization.subject(), + policy_count = authorization.policies().len(), + group_count = authorization.mapped_iam_groups().len(), + policies = ?authorization.policies(), + groups = ?authorization.mapped_iam_groups(), + roles_claim_key = ?authorization.roles_claim_key(), has_session_policy = transaction.session_policy.is_some(), "OIDC STS credential claims prepared" ); @@ -291,7 +296,7 @@ fn issue_credentials( let mut credentials = get_new_credentials_with_metadata(&token_claims, secret) .map_err(|error| FederatedSessionBindingError::Internal(format!("credential generation failed: {error}")))?; credentials.parent_user = parent_user; - credentials.groups = Some(authorization.groups.clone()); + credentials.groups = Some(authorization.mapped_iam_groups().to_vec()); Ok(credentials) } @@ -325,12 +330,12 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { FederatedSessionBindingError::InvalidRequest("verified OIDC identity is missing issuer or subject".to_string()) })?; let (selected_policy_names, ignorable_policy_names) = match iam_store - .sts_policy_db_get(&parent_user, &Some(authorization.groups.clone())) + .sts_policy_db_get(&parent_user, &Some(authorization.mapped_iam_groups().to_vec())) .await .map_err(|_| FederatedSessionBindingError::Internal("failed to resolve OIDC policy mapping".to_string()))? { mapped_policy_names if !mapped_policy_names.is_empty() => (mapped_policy_names, &[][..]), - _ => (authorization.policies.clone(), authorization.group_claim_policies.as_slice()), + _ => (authorization.policies().to_vec(), authorization.group_claim_policies()), }; // A name may be skipped only after storage confirms the policy is absent. A failed lookup // must not be mistaken for absence, or an unreadable Deny policy would be dropped from the @@ -339,7 +344,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { for policy_name in &selected_policy_names { let exists = iam_store.policy_exists(policy_name).await.map_err(|err| { warn!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, policy = %policy_name, error = %err, @@ -367,7 +375,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { .map_err(|rejection| match rejection { OidcPolicyRejection::Unsupported(policy_names) => { warn!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, unsupported_policies = ?policy_names, "OIDC STS rejected existing policies whose names are not allowed in session claims" @@ -376,7 +387,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { } OidcPolicyRejection::Unresolved(policy_names) => { debug!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, unresolved_policies = ?policy_names, "OIDC STS rejected required policies without a current policy" @@ -389,7 +403,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { })?; if !selection.ignored.is_empty() { debug!( - provider_id = %authorization.provider_id, + event = EVENT_ADMIN_OIDC_POLICY, + component = LOG_COMPONENT_ADMIN, + subsystem = LOG_SUBSYSTEM_OIDC, + provider_id = %authorization.provider_id(), parent_user = %parent_user, ignored_group_policies = ?selection.ignored, "OIDC STS ignoring group claim values without a matching policy" @@ -402,10 +419,10 @@ impl FederatedSessionBinding for DefaultFederatedSessionBinding { if tracing::enabled!(tracing::Level::DEBUG) { log_oidc_policy_diagnostics( &iam_store, - &authorization.provider_id, + authorization.provider_id(), &credentials.parent_user, &selected_policy_names, - &authorization.groups, + authorization.mapped_iam_groups(), ) .await; } @@ -428,7 +445,7 @@ mod tests { use super::*; use crate::admin::runtime_sources::{AppContext, current_object_store_handle, publish_test_app_context}; use hmac::{Hmac, KeyInit, Mac}; - use rustfs_iam::federation::{FederatedAuthorization, FederatedClaims}; + use rustfs_iam::federation::FederatedAuthorizationFixture; use rustfs_iam::store::{ Store, UserType, object::{IAM_CONFIG_POLICIES_PREFIX, IAM_CONFIG_PREFIX}, @@ -443,31 +460,37 @@ mod tests { use sha2::Sha512; use std::sync::Arc; - fn transaction() -> FederatedSessionTransaction { + fn authorization_fixture() -> FederatedAuthorizationFixture { + FederatedAuthorizationFixture { + provider_id: "default".to_string(), + sub: "subject".to_string(), + email: "user@example.com".to_string(), + username: "user".to_string(), + source_groups: vec!["source-group".to_string()], + attributes: HashMap::from([ + ("iss".to_string(), serde_json::json!("https://idp.example.test")), + ("raw_poison".to_string(), serde_json::json!("must-not-leak")), + ]), + policies: vec!["readwrite".to_string()], + group_claim_policies: Vec::new(), + mapped_iam_groups: vec!["devs".to_string()], + roles_claim_key: Some("roles".to_string()), + roles: vec!["admin".to_string(), "reader".to_string()], + } + } + + fn transaction_from_fixture(authorization: FederatedAuthorizationFixture) -> FederatedSessionTransaction { FederatedSessionTransaction { - authorization: FederatedAuthorization { - provider_id: "default".to_string(), - claims: FederatedClaims { - sub: "subject".to_string(), - email: "user@example.com".to_string(), - username: "user".to_string(), - groups: vec!["source-group".to_string()], - raw: HashMap::from([ - ("iss".to_string(), serde_json::json!("https://idp.example.test")), - ("raw_poison".to_string(), serde_json::json!("must-not-leak")), - ]), - }, - policies: vec!["readwrite".to_string()], - group_claim_policies: Vec::new(), - groups: vec!["devs".to_string()], - roles_claim_key: Some("roles".to_string()), - roles: vec!["admin".to_string(), "reader".to_string()], - }, + authorization: authorization.into_authorization(), duration_seconds: 3600, session_policy: None, } } + fn transaction() -> FederatedSessionTransaction { + transaction_from_fixture(authorization_fixture()) + } + async fn ready_test_iam() -> Arc> { let _ = rustfs_credentials::init_global_action_credentials( Some("TESTROOTACCESSKEY".to_string()), @@ -531,11 +554,12 @@ mod tests { #[test] fn token_claims_omit_empty_optional_oidc_fields() { - let mut transaction = transaction(); - transaction.authorization.claims.email.clear(); - transaction.authorization.claims.username.clear(); - transaction.authorization.groups.clear(); - transaction.authorization.roles.clear(); + let mut authorization = authorization_fixture(); + authorization.email.clear(); + authorization.username.clear(); + authorization.mapped_iam_groups.clear(); + authorization.roles.clear(); + let transaction = transaction_from_fixture(authorization); assert_eq!( build_oidc_token_claims(&transaction), @@ -547,6 +571,36 @@ mod tests { ); } + #[test] + #[serial] + fn console_session_duration_bounds_signed_credential_expiration() { + let secret = "console-session-expiration-test-secret"; + for (configured, expected_seconds) in [(None, 3600), (Some("7200"), 7200), (Some("604801"), 604_800)] { + temp_env::with_var(rustfs_config::ENV_CONSOLE_OIDC_SESSION_DURATION, configured, || { + let mut transaction = transaction(); + transaction.duration_seconds = rustfs_iam::oidc::console_session_duration_secs(); + let before = OffsetDateTime::now_utc().unix_timestamp(); + let credentials = issue_credentials(&transaction, &["readonly".to_string()], Some(secret)) + .expect("configured Console session should issue credentials"); + let after = OffsetDateTime::now_utc().unix_timestamp(); + let claims = rustfs_iam::sys::get_claims_from_token_with_secret(&credentials.session_token, secret) + .expect("Console credential signature should verify"); + let expiration = claims["exp"].as_i64().expect("signed expiration should be an integer"); + assert!( + (before + expected_seconds..=after + expected_seconds).contains(&expiration), + "signed expiration must reflect the bounded Console session for {configured:?}" + ); + assert_eq!( + credentials + .expiration + .expect("Console credential must expire") + .unix_timestamp(), + expiration + ); + }); + } + } + #[test] fn issued_credentials_and_replication_item_use_minio_parent_shape() { let transaction = transaction(); @@ -634,9 +688,10 @@ mod tests { #[serial] async fn binding_uses_sts_policy_when_regular_mapping_collides() { let iam = ready_test_iam().await; - let mut transaction = transaction(); - transaction.authorization.claims.sub = "binding-sts-policy-subject".to_string(); - transaction.authorization.policies = vec!["readwrite".to_string()]; + let mut authorization = authorization_fixture(); + authorization.sub = "binding-sts-policy-subject".to_string(); + authorization.policies = vec!["readwrite".to_string()]; + let transaction = transaction_from_fixture(authorization); let parent = transaction .authorization .oidc_virtual_parent() @@ -750,7 +805,7 @@ mod tests { let mut transaction = transaction(); transaction.session_policy = Some("not-json".to_string()); - let error = issue_credentials(&transaction, &transaction.authorization.policies, None) + let error = issue_credentials(&transaction, transaction.authorization.policies(), None) .expect_err("invalid policy should fail first"); assert!(matches!(error, FederatedSessionBindingError::InvalidRequest(_))); } @@ -782,10 +837,11 @@ mod tests { #[test] fn missing_oidc_issuer_fails_closed_before_credential_issuance() { - let mut transaction = transaction(); - transaction.authorization.claims.raw.clear(); + let mut authorization = authorization_fixture(); + authorization.attributes.clear(); + let transaction = transaction_from_fixture(authorization); - let error = issue_credentials(&transaction, &transaction.authorization.policies, Some("signing-secret")) + let error = issue_credentials(&transaction, transaction.authorization.policies(), Some("signing-secret")) .expect_err("credential issuance should reject a missing issuer"); assert!(matches!(error, FederatedSessionBindingError::InvalidRequest(_))); } @@ -865,12 +921,12 @@ mod tests { } fn group_claim_transaction(subject: &str, groups: &[&str]) -> FederatedSessionTransaction { - let mut transaction = transaction(); - transaction.authorization.claims.sub = subject.to_string(); - transaction.authorization.groups = names(groups); - transaction.authorization.policies = names(groups); - transaction.authorization.group_claim_policies = names(groups); - transaction + let mut authorization = authorization_fixture(); + authorization.sub = subject.to_string(); + authorization.mapped_iam_groups = names(groups); + authorization.policies = names(groups); + authorization.group_claim_policies = names(groups); + transaction_from_fixture(authorization) } fn assert_binding_rejected(error: FederatedSessionBindingError, expected_message: &str) { @@ -898,7 +954,7 @@ mod tests { .expect("issued session token should verify"); assert_eq!(claims.get("policy"), Some(&serde_json::json!("readonly"))); - assert_eq!(credentials.groups.as_deref(), Some(transaction.authorization.groups.as_slice())); + assert_eq!(credentials.groups.as_deref(), Some(transaction.authorization.mapped_iam_groups())); let groups = credentials.groups.clone(); let conditions = HashMap::new(); @@ -987,10 +1043,11 @@ mod tests { #[serial] async fn binding_keeps_role_policy_strict() { ready_test_iam().await; - let mut transaction = transaction(); - transaction.authorization.claims.sub = "binding-strict-role-policy-subject".to_string(); - transaction.authorization.policies = names(&["readonly", "missing-role-policy"]); - transaction.authorization.group_claim_policies = Vec::new(); + let mut authorization = authorization_fixture(); + authorization.sub = "binding-strict-role-policy-subject".to_string(); + authorization.policies = names(&["readonly", "missing-role-policy"]); + authorization.group_claim_policies = Vec::new(); + let transaction = transaction_from_fixture(authorization); let error = DefaultFederatedSessionBinding .bind(&transaction) diff --git a/rustfs/src/admin/storage_api.rs b/rustfs/src/admin/storage_api.rs index 08ea72948..352357a4e 100644 --- a/rustfs/src/admin/storage_api.rs +++ b/rustfs/src/admin/storage_api.rs @@ -532,11 +532,15 @@ pub(crate) mod metadata_sys { pub(crate) mod quota { pub(crate) mod checker { pub(crate) type QuotaChecker = super::super::ecstore_bucket::quota::checker::QuotaChecker; + pub(crate) use super::super::ecstore_bucket::quota::checker::record_reservation_protocol_migration; } pub(crate) type BucketQuota = super::ecstore_bucket::quota::BucketQuota; pub(crate) type QuotaError = super::ecstore_bucket::quota::QuotaError; pub(crate) type QuotaOperation = super::ecstore_bucket::quota::QuotaOperation; + pub(crate) type QuotaProtocolMigration = super::ecstore_bucket::quota::QuotaProtocolMigration; + #[cfg(test)] + pub(crate) type QuotaMigrationSkipReason = super::ecstore_bucket::quota::QuotaMigrationSkipReason; pub(crate) use super::ecstore_bucket::quota::{QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2}; } diff --git a/rustfs/src/app/bucket_usecase.rs b/rustfs/src/app/bucket_usecase.rs index 445978a09..69d06c455 100644 --- a/rustfs/src/app/bucket_usecase.rs +++ b/rustfs/src/app/bucket_usecase.rs @@ -1257,15 +1257,19 @@ fn build_list_objects_v2_metadata_output( } } -fn create_bucket_exists_response(is_owner: bool) -> S3Result> { - if is_owner { +fn create_bucket_exists_response(is_owner: bool, region: &str) -> S3Result> { + if is_owner && region == RUSTFS_REGION { + // AWS preserves the legacy idempotent response for buckets recreated in + // us-east-1 (the default RustFS region). return Ok(S3Response::new(CreateBucketOutput::default())); } - Err(s3_error!( - BucketAlreadyExists, - "The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again." - )) + let error_code = if is_owner { + S3ErrorCode::BucketAlreadyOwnedByYou + } else { + S3ErrorCode::BucketAlreadyExists + }; + Err(S3Error::with_message(error_code, "The requested bucket already exists.")) } fn resolve_notification_region(global_region: Option, request_region: Option) -> String { @@ -1526,9 +1530,13 @@ impl DefaultBucketUsecase { .await; } Err(StorageError::BucketExists(_)) => { - // Per S3 spec: bucket namespace is global. Owner recreating returns 200 OK; - // non-owner gets 409 BucketAlreadyExists. - let result = create_bucket_exists_response(requester_is_owner); + // Match AWS CreateBucket semantics: an owned bucket is idempotent in + // us-east-1 and returns BucketAlreadyOwnedByYou elsewhere. + let region = self + .global_region() + .map(|region| region.to_string()) + .unwrap_or_else(|| RUSTFS_REGION.to_string()); + let result = create_bucket_exists_response(requester_is_owner, ®ion); let _ = helper.complete(&result); return result; } @@ -4299,16 +4307,22 @@ mod tests { #[test] fn create_bucket_exists_response_returns_ok_for_owner() { - let response = create_bucket_exists_response(true).expect("owner recreate should succeed"); + let response = create_bucket_exists_response(true, RUSTFS_REGION).expect("owner recreate in us-east-1 should succeed"); assert_eq!(response.output.location, None); } #[test] fn create_bucket_exists_response_returns_bucket_already_exists_for_non_owner() { - let err = create_bucket_exists_response(false).expect_err("non-owner recreate should fail"); + let err = create_bucket_exists_response(false, RUSTFS_REGION).expect_err("non-owner recreate should fail"); assert_eq!(err.code(), &S3ErrorCode::BucketAlreadyExists); } + #[test] + fn create_bucket_exists_response_returns_owned_by_you_outside_us_east_1() { + let err = create_bucket_exists_response(true, "eu-west-1").expect_err("owner recreate outside us-east-1 should fail"); + assert_eq!(err.code(), &S3ErrorCode::BucketAlreadyOwnedByYou); + } + #[test] fn build_request_with_req_info_preserves_owner_state() { let input = CreateBucketInput::builder() diff --git a/rustfs/src/app/lifecycle_transition_api_test.rs b/rustfs/src/app/lifecycle_transition_api_test.rs index c546cee72..854ff88c5 100644 --- a/rustfs/src/app/lifecycle_transition_api_test.rs +++ b/rustfs/src/app/lifecycle_transition_api_test.rs @@ -589,26 +589,47 @@ async fn put_object_if_none_match_existing_object_returns_precondition_failed() .await .expect("Failed to fetch existing object info"); let existing_etag = existing_info.etag.expect("existing object should have an ETag"); + let initial_versions = live_object_version_count(&ecstore, bucket.as_str(), object).await; - let replacement_input = PutObjectInput::builder() - .bucket(bucket.clone()) - .key(object.to_string()) - .body(Some(streaming_blob_from_bytes(replacement_payload))) - .content_length(Some(replacement_payload.len() as i64)) - .build() - .unwrap(); - let mut req = build_request(replacement_input, Method::PUT); - req.headers - .insert(IF_NONE_MATCH, HeaderValue::from_str(existing_etag.as_str()).unwrap()); + for (condition, expected_code) in [ + (ETagCondition::Any, s3s::S3ErrorCode::PreconditionFailed), + (ETagCondition::ETag(ETag::Strong(existing_etag)), s3s::S3ErrorCode::NotImplemented), + ( + ETagCondition::ETag(ETag::Strong("different-etag".to_string())), + s3s::S3ErrorCode::NotImplemented, + ), + ] { + let header = condition.to_http_header().expect("conditional PUT header should encode"); + // Cover header-only callers and the DTO-plus-header shape of parsed HTTP requests. + for if_none_match in [None, Some(condition)] { + let has_dto = if_none_match.is_some(); + let replacement_input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .body(Some(streaming_blob_from_bytes(replacement_payload))) + .content_length(Some(replacement_payload.len() as i64)) + .if_none_match(if_none_match) + .build() + .expect("conditional replacement PUT input should build"); + let mut req = build_request(replacement_input, Method::PUT); + req.headers.insert(IF_NONE_MATCH, header.clone()); - let err = Box::pin(usecase.execute_put_object(&fs, req)).await.unwrap_err(); - - assert_eq!(err.code(), &s3s::S3ErrorCode::PreconditionFailed); - assert_eq!( - read_object_bytes(&ecstore, bucket.as_str(), object).await, - initial_payload, - "failed conditional PutObject must not overwrite the current object" - ); + let err = Box::pin(usecase.execute_put_object(&fs, req)) + .await + .expect_err("conditional PUT must reject the replacement"); + assert_eq!(err.code(), &expected_code, "header={header:?}, has_dto={has_dto}"); + assert_eq!( + read_object_bytes(&ecstore, bucket.as_str(), object).await, + initial_payload, + "failed conditional PutObject must not overwrite the current object" + ); + assert_eq!( + live_object_version_count(&ecstore, bucket.as_str(), object).await, + initial_versions, + "failed conditional PutObject must not create a version" + ); + } + } } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] @@ -635,29 +656,50 @@ async fn copy_object_if_none_match_existing_destination_returns_precondition_fai .await .expect("Failed to fetch destination object info"); let dst_etag = dst_info.etag.expect("destination object should have an ETag"); + let initial_versions = live_object_version_count(&ecstore, dst_bucket.as_str(), dst_object).await; - let copy_input = CopyObjectInput::builder() - .copy_source(CopySource::Bucket { - bucket: src_bucket.clone().into(), - key: src_object.to_string().into(), - version_id: None, - }) - .bucket(dst_bucket.clone()) - .key(dst_object.to_string()) - .build() - .unwrap(); - let mut req = build_request(copy_input, Method::PUT); - req.headers - .insert(IF_NONE_MATCH, HeaderValue::from_str(dst_etag.as_str()).unwrap()); + for (condition, expected_code) in [ + (ETagCondition::Any, s3s::S3ErrorCode::PreconditionFailed), + (ETagCondition::ETag(ETag::Strong(dst_etag)), s3s::S3ErrorCode::NotImplemented), + ( + ETagCondition::ETag(ETag::Strong("different-etag".to_string())), + s3s::S3ErrorCode::NotImplemented, + ), + ] { + let header = condition.to_http_header().expect("conditional COPY header should encode"); + // Cover header-only callers and the DTO-plus-header shape of parsed HTTP requests. + for if_none_match in [None, Some(condition)] { + let has_dto = if_none_match.is_some(); + let copy_input = CopyObjectInput::builder() + .copy_source(CopySource::Bucket { + bucket: src_bucket.clone().into(), + key: src_object.to_string().into(), + version_id: None, + }) + .bucket(dst_bucket.clone()) + .key(dst_object.to_string()) + .if_none_match(if_none_match) + .build() + .expect("conditional COPY input should build"); + let mut req = build_request(copy_input, Method::PUT); + req.headers.insert(IF_NONE_MATCH, header.clone()); - let err = Box::pin(usecase.execute_copy_object(req)).await.unwrap_err(); - - assert_eq!(err.code(), &s3s::S3ErrorCode::PreconditionFailed); - assert_eq!( - read_object_bytes(&ecstore, dst_bucket.as_str(), dst_object).await, - dst_payload, - "failed conditional CopyObject must not overwrite the current destination object" - ); + let err = Box::pin(usecase.execute_copy_object(req)) + .await + .expect_err("conditional COPY must reject the replacement"); + assert_eq!(err.code(), &expected_code, "header={header:?}, has_dto={has_dto}"); + assert_eq!( + read_object_bytes(&ecstore, dst_bucket.as_str(), dst_object).await, + dst_payload, + "failed conditional CopyObject must not overwrite the current destination object" + ); + assert_eq!( + live_object_version_count(&ecstore, dst_bucket.as_str(), dst_object).await, + initial_versions, + "failed conditional CopyObject must not create a version" + ); + } + } } #[tokio::test(flavor = "multi_thread", worker_threads = 1)] diff --git a/rustfs/src/app/multipart_usecase.rs b/rustfs/src/app/multipart_usecase.rs index b65921624..a2c9136e6 100644 --- a/rustfs/src/app/multipart_usecase.rs +++ b/rustfs/src/app/multipart_usecase.rs @@ -72,7 +72,7 @@ use super::storage_api::multipart_usecase::{ use super::trailer_adapter::trailer_source; use crate::app::object::{ ConcurrencyManager, ForegroundWriteAdmission, get_concurrency_manager, guard_put_object_body_read_timeout, - put_object_body_read_timeout, reject_oversize_single_upload, + put_object_body_read_timeout, reject_non_wildcard_write_if_none_match, reject_oversize_single_upload, }; use crate::app::object_data_cache::{ ObjectDataCacheAdapter, invalidate_object_data_cache_after_complete_multipart_success, @@ -548,7 +548,7 @@ impl DefaultMultipartUsecase { #[hotpath::measure(impl_type = "MultipartUsecase")] pub async fn execute_complete_multipart_upload( &self, - req: S3Request, + mut req: S3Request, ) -> S3Result> { reject_presigned_multipart_max_total_object_size_for_other_operation( &req.headers, @@ -566,7 +566,7 @@ impl DefaultMultipartUsecase { S3Operation::CompleteMultipartUpload, ); let replication_authorized = replication_request_authorized(&req); - let input = req.input.clone(); + let input = std::mem::take(&mut req.input); let CompleteMultipartUploadInput { multipart_upload, bucket, @@ -580,6 +580,7 @@ impl DefaultMultipartUsecase { .. } = input; + reject_non_wildcard_write_if_none_match(if_none_match.as_ref(), &req.headers)?; validate_table_catalog_object_mutation(&bucket, &key).await?; if if_match.is_some() || if_none_match.is_some() { @@ -589,23 +590,22 @@ impl DefaultMultipartUsecase { match store.get_object_info(&bucket, &key, &ObjectOptions::default()).await { Ok(info) => { - if !info.delete_marker { - if let Some(ifmatch) = if_match - && let Some(strong_etag) = ifmatch.into_etag() - && info - .etag - .as_ref() - .is_some_and(|etag| ETag::Strong(etag.clone()) != strong_etag) + if info.delete_marker { + if if_match.is_some() { + return Err(ApiError::from(StorageError::ObjectNotFound(bucket.clone(), key.clone())).into()); + } + } else { + if let Some(ifmatch) = &if_match + && !ifmatch.is_any() + && ifmatch + .as_etag() + .and_then(ETag::as_strong) + .is_none_or(|expected| info.etag.as_deref() != Some(expected)) { return Err(s3_error!(PreconditionFailed)); } - if let Some(ifnonematch) = if_none_match - && let Some(strong_etag) = ifnonematch.into_etag() - && info - .etag - .as_ref() - .is_some_and(|etag| ETag::Strong(etag.clone()) == strong_etag) - { + // The request guard permits only the wildcard here. + if if_none_match.is_some() { return Err(s3_error!(PreconditionFailed)); } } @@ -1124,9 +1124,11 @@ impl DefaultMultipartUsecase { ); } - let current_opts: ObjectOptions = get_opts(&bucket, &key, opts.version_id.clone(), None, &req.headers) - .await - .map_err(ApiError::from)?; + let current_opts: ObjectOptions = internal_object_info_lookup_opts( + get_opts(&bucket, &key, opts.version_id.clone(), None, &req.headers) + .await + .map_err(ApiError::from)?, + ); match store.get_object_info(&bucket, &key, ¤t_opts).await { Ok(existing_obj_info) => { validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &opts)? @@ -2892,6 +2894,351 @@ mod tests { ); } + #[test] + #[serial_test::serial] + fn conditional_multipart_upload_if_none_match_returns_precondition_failed_when_object_exists() { + crate::app::gating_test_env::run_large_stack_test( + "conditional-mpu-if-none-match", + conditional_multipart_upload_if_none_match_returns_precondition_failed_when_object_exists_inner, + ); + } + + async fn conditional_multipart_upload_if_none_match_returns_precondition_failed_when_object_exists_inner() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + use crate::app::storage_api::test::contract::object::ObjectOperations as _; + + let (store, bucket) = + crate::app::gating_test_env::durable_quota_test_bucket("conditional-mpu-if-none-match", 16_384).await; + let object = "conditional-object"; + let usecase = DefaultMultipartUsecase::from_global(); + let initial_body = b"initial multipart body"; + + let create_input = CreateMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .build() + .expect("create multipart input should build"); + let mut initial_create_req = build_request(create_input.clone(), Method::POST); + initial_create_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let first_upload_id = usecase + .execute_create_multipart_upload(initial_create_req) + .await + .expect("initial conditional create multipart upload should succeed") + .output + .upload_id + .expect("create response should contain upload id"); + + let mut reader = PutObjReader::from_vec(initial_body.to_vec()); + let first_part = store + .put_object_part(&bucket, object, &first_upload_id, 1, &mut reader, &ObjectOptions::default()) + .await + .expect("initial multipart part should be staged"); + let first_complete_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(first_upload_id) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(CompletedMultipartUpload { + parts: Some(vec![CompletedPart { + part_number: Some(1), + e_tag: first_part.etag.map(|etag| to_s3s_etag(&etag)), + ..Default::default() + }]), + })) + .build() + .expect("initial complete multipart input should build"); + let mut first_complete_req = build_request(first_complete_input, Method::POST); + first_complete_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + usecase + .execute_complete_multipart_upload(first_complete_req) + .await + .expect("initial conditional complete multipart upload should succeed"); + + let committed_info = store + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("committed object info should be readable"); + let committed_etag = committed_info.etag.clone().expect("committed object should have an ETag"); + assert_eq!(committed_info.size, initial_body.len() as i64); + + let mut second_create_req = build_request(create_input.clone(), Method::POST); + second_create_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let create_err = usecase + .execute_create_multipart_upload(second_create_req) + .await + .expect_err("create multipart upload with If-None-Match: * on existing key must fail"); + assert_eq!(create_err.code(), &S3ErrorCode::PreconditionFailed); + + let second_upload_id = usecase + .execute_create_multipart_upload(build_request(create_input, Method::POST)) + .await + .expect("unconditional create multipart upload should succeed") + .output + .upload_id + .expect("second create response should contain upload id"); + let mut second_reader = PutObjReader::from_vec(b"replacement multipart body".to_vec()); + let second_part = store + .put_object_part(&bucket, object, &second_upload_id, 1, &mut second_reader, &ObjectOptions::default()) + .await + .expect("second multipart part should be staged"); + let completed_parts = CompletedMultipartUpload { + parts: Some(vec![CompletedPart { + part_number: Some(1), + e_tag: second_part.etag.map(|etag| to_s3s_etag(&etag)), + ..Default::default() + }]), + }; + + // DTO-only If-None-Match: * exercises the early precheck in execute_complete_multipart_upload. + let dto_only_complete_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id.clone()) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(completed_parts.clone())) + .build() + .expect("DTO-only complete multipart input should build"); + let dto_complete_err = usecase + .execute_complete_multipart_upload(build_request(dto_only_complete_input, Method::POST)) + .await + .expect_err("DTO-only If-None-Match: * on existing key must fail"); + assert_eq!(dto_complete_err.code(), &S3ErrorCode::PreconditionFailed); + + for condition in [ETag::Weak(committed_etag.clone()), ETag::Strong("different-etag".to_string())] { + let input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id.clone()) + .if_match(Some(s3s::dto::ETagCondition::ETag(condition))) + .multipart_upload(Some(completed_parts.clone())) + .build() + .expect("If-Match complete input should build"); + let err = usecase + .execute_complete_multipart_upload(build_request(input, Method::POST)) + .await + .expect_err("weak or nonmatching If-Match must reject the overwrite"); + assert_eq!(err.code(), &S3ErrorCode::PreconditionFailed); + } + + // Write requests reject ETag-valued If-None-Match before checking object state. + let weak_if_none_match_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id.clone()) + .if_none_match(Some(s3s::dto::ETagCondition::ETag(ETag::Weak(committed_etag.clone())))) + .multipart_upload(Some(completed_parts.clone())) + .build() + .expect("weak If-None-Match complete multipart input should build"); + let weak_complete_err = usecase + .execute_complete_multipart_upload(build_request(weak_if_none_match_input, Method::POST)) + .await + .expect_err("weak If-None-Match ETag must be rejected as unsupported"); + assert_eq!(weak_complete_err.code(), &S3ErrorCode::NotImplemented); + + let second_complete_input = CompleteMultipartUploadInput::builder() + .bucket(bucket.clone()) + .key(object.to_string()) + .upload_id(second_upload_id) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(completed_parts)) + .build() + .expect("second complete multipart input should build"); + let mut second_complete_req = build_request(second_complete_input, Method::POST); + second_complete_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let complete_err = usecase + .execute_complete_multipart_upload(second_complete_req) + .await + .expect_err("complete multipart upload with If-None-Match: * on existing key must fail"); + assert_eq!(complete_err.code(), &S3ErrorCode::PreconditionFailed); + + let after_info = store + .get_object_info(&bucket, object, &ObjectOptions::default()) + .await + .expect("original object must remain readable after rejected conditional completions"); + assert_eq!(after_info.etag.as_deref(), Some(committed_etag.as_str())); + assert_eq!(after_info.size, initial_body.len() as i64); + + // Two-pool regression: when Pool 0 holds an incomplete multipart upload and Pool 1 + // holds the committed object, conditional new_multipart_upload and complete_multipart_upload + // must still reject If-None-Match: * with PreconditionFailed. + let (_multi_temp_dir, _pool_paths, multi_store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + let multi_bucket = format!("multi-pool-cond-mpu-{}", Uuid::new_v4().simple()); + multi_store + .make_bucket(&multi_bucket, &MakeBucketOptions::default()) + .await + .expect("create multi-pool test bucket"); + let multi_object = "cross-pool-conditional-object"; + let staged_pool_0 = multi_store.pools[0] + .new_multipart_upload(&multi_bucket, multi_object, &ObjectOptions::default()) + .await + .expect("stage incomplete upload in pool 0"); + let mut pool0_part_reader = PutObjReader::from_vec(b"pool-0-staged-part".to_vec()); + let pool0_part = multi_store.pools[0] + .put_object_part( + &multi_bucket, + multi_object, + &staged_pool_0.upload_id, + 1, + &mut pool0_part_reader, + &ObjectOptions::default(), + ) + .await + .expect("write staged part in pool 0"); + let mut pool1_reader = PutObjReader::from_vec(b"pool-1-committed-object".to_vec()); + multi_store.pools[1] + .put_object(&multi_bucket, multi_object, &mut pool1_reader, &ObjectOptions::default()) + .await + .expect("commit object in pool 1"); + + let multi_cond_opts = ObjectOptions { + http_preconditions: Some(HTTPPreconditions { + if_none_match: Some("*".to_string()), + ..Default::default() + }), + ..Default::default() + }; + let multi_create_err = multi_store + .new_multipart_upload(&multi_bucket, multi_object, &multi_cond_opts) + .await + .expect_err("multi-pool new_multipart_upload with If-None-Match: * must fail when object is in pool 1"); + assert_eq!(multi_create_err, StorageError::PreconditionFailed); + + let multi_complete_err = Arc::clone(&multi_store) + .complete_multipart_upload( + &multi_bucket, + multi_object, + &staged_pool_0.upload_id, + vec![CompletePart { + part_num: 1, + etag: pool0_part.etag, + ..Default::default() + }], + &multi_cond_opts, + ) + .await + .expect_err("multi-pool complete_multipart_upload with If-None-Match: * must fail when object is in pool 1"); + assert_eq!(multi_complete_err, StorageError::PreconditionFailed); + metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + + // Delete-marker regression: when the current version in a versioned bucket is a delete + // marker, If-Match must fail with NoSuchKey while If-None-Match: * succeeds. + let versioned_bucket = format!("versioned-cond-mpu-{}", Uuid::new_v4().simple()); + store + .make_bucket( + &versioned_bucket, + &MakeBucketOptions { + versioning_enabled: true, + ..Default::default() + }, + ) + .await + .expect("create versioned bucket"); + let mut v_reader = PutObjReader::from_vec(b"v1-body".to_vec()); + store + .put_object( + &versioned_bucket, + object, + &mut v_reader, + &ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("write initial versioned object"); + let marker = store + .delete_object( + &versioned_bucket, + object, + ObjectOptions { + versioned: true, + ..Default::default() + }, + ) + .await + .expect("delete versioned object to create delete marker"); + assert!(marker.delete_marker); + + let dm_create_input = CreateMultipartUploadInput::builder() + .bucket(versioned_bucket.clone()) + .key(object.to_string()) + .build() + .expect("delete-marker create input"); + let mut dm_create_req = build_request(dm_create_input, Method::POST); + dm_create_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + let dm_upload_id = usecase + .execute_create_multipart_upload(dm_create_req) + .await + .expect("If-None-Match: * create must succeed past a delete marker") + .output + .upload_id + .expect("upload id"); + let mut dm_part_reader = PutObjReader::from_vec(b"after-delete-marker".to_vec()); + let dm_part = store + .put_object_part( + &versioned_bucket, + object, + &dm_upload_id, + 1, + &mut dm_part_reader, + &ObjectOptions::default(), + ) + .await + .expect("stage part past delete marker"); + let dm_parts = CompletedMultipartUpload { + parts: Some(vec![CompletedPart { + part_number: Some(1), + e_tag: dm_part.etag.map(|etag| to_s3s_etag(&etag)), + ..Default::default() + }]), + }; + + let dm_if_match_input = CompleteMultipartUploadInput::builder() + .bucket(versioned_bucket.clone()) + .key(object.to_string()) + .upload_id(dm_upload_id.clone()) + .if_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(dm_parts.clone())) + .build() + .expect("delete-marker If-Match complete input"); + let mut dm_if_match_req = build_request(dm_if_match_input, Method::POST); + dm_if_match_req + .headers + .insert(http::header::IF_MATCH, HeaderValue::from_static("*")); + let dm_if_match_err = usecase + .execute_complete_multipart_upload(dm_if_match_req) + .await + .expect_err("If-Match against a current delete marker must fail with NoSuchKey"); + assert_eq!(dm_if_match_err.code(), &S3ErrorCode::NoSuchKey); + + let dm_if_none_match_input = CompleteMultipartUploadInput::builder() + .bucket(versioned_bucket) + .key(object.to_string()) + .upload_id(dm_upload_id) + .if_none_match(Some(s3s::dto::ETagCondition::Any)) + .multipart_upload(Some(dm_parts)) + .build() + .expect("delete-marker If-None-Match complete input"); + let mut dm_if_none_match_req = build_request(dm_if_none_match_input, Method::POST); + dm_if_none_match_req + .headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*")); + usecase + .execute_complete_multipart_upload(dm_if_none_match_req) + .await + .expect("If-None-Match: * completion must succeed past a delete marker"); + } + #[tokio::test] #[serial_test::serial] async fn create_multipart_rejects_ciphertext_replication_before_parts_are_staged() { @@ -2966,7 +3313,7 @@ mod tests { #[tokio::test] #[serial_test::serial] - async fn multipart_completion_rejects_rotated_quota_capability_before_rename() { + async fn multipart_completion_commits_after_quota_fleet_proof_loss_before_rename() { use crate::app::storage_api::test::set_disk::{MultipartCommitBarrier, MultipartCommitPause}; let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("rotated-proof-mpu-quota", 4096).await; @@ -3000,27 +3347,19 @@ mod tests { .await }); barrier.wait_until_paused().await; - assert!( - crate::storage::storage_api::ecstore_notification::rotate_cross_pool_fence_fleet_proof_for_test(), - "the gating environment must have a current fleet proof" - ); + let _without_proof = crate::storage::storage_api::ecstore_notification::without_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::storage::storage_api::ecstore_notification::acquire_cross_pool_fence_fleet_proof().is_none()); barrier.release(); - let err = complete + complete .await .expect("completion task should not panic") - .expect_err("a replaced fleet proof must fence multipart rename"); - assert!(matches!( - err, - StorageError::NamespaceLockQuorumUnavailable { - mode: "quota_reservation", - .. - } - )); - store - .get_multipart_info(&bucket, object, &upload.upload_id, &ObjectOptions::default()) + .expect("fleet proof loss must not invalidate multipart quota reservation and disk fences"); + let stored = store + .get_object_info(&bucket, object, &ObjectOptions::default()) .await - .expect("proof rotation must preserve the multipart upload for retry"); + .expect("completed multipart object must remain readable after fleet proof loss"); + assert_eq!(stored.size, 4096); } #[tokio::test] diff --git a/rustfs/src/app/object/copy.rs b/rustfs/src/app/object/copy.rs index 0b3cb0526..760e854b3 100644 --- a/rustfs/src/app/object/copy.rs +++ b/rustfs/src/app/object/copy.rs @@ -15,6 +15,7 @@ //! CopyObject path. use super::*; +use rustfs_utils::path::fold_object_name_path_separators; use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation}; use crate::error::ServerSideSourceReadError; @@ -96,6 +97,16 @@ fn copy_namespace_lock_error(bucket: &str, object: &str, mode: &'static str, err } } +/// Object name the namespace lock of a copy endpoint is keyed by: the disk +/// identity of the stored name. On Windows that folds spellings +/// `path_join_buf` keeps apart (`a\b\` vs `a/b`), so a copy that is not a +/// same-object copy can still have its source read guard and destination +/// write lock on one lock, and must take the self-copy lock to not wait on +/// itself. +fn copy_namespace_lock_object(key: &str, backslash_is_separator: bool) -> String { + fold_object_name_path_separators(&encode_dir_object(key), backslash_is_separator).into_owned() +} + async fn acquire_self_copy_namespace_lock( store: &S, bucket: &str, @@ -168,6 +179,7 @@ impl DefaultObjectUsecase { req.uri.query(), req.extensions.get::().is_some(), )?; + reject_non_wildcard_write_if_none_match(req.input.if_none_match.as_ref(), &req.headers)?; if let Some(context) = &self.context { let _ = context.object_store(); } @@ -339,6 +351,10 @@ impl DefaultObjectUsecase { apply_bucket_generation_guard(&req, &bucket, &mut dst_opts)?; let cp_src_dst_same = path_join_buf(&[&src_bucket, &src_key]) == path_join_buf(&[&bucket, &key]); + let backslash_is_separator = cfg!(target_os = "windows"); + let dst_lock_object = copy_namespace_lock_object(&key, backslash_is_separator); + let self_copy_lock = cp_src_dst_same + || (src_bucket == bucket && copy_namespace_lock_object(&src_key, backslash_is_separator) == dst_lock_object); let expected_current_version_id = expected_current_version_id(&req.headers)?; if expected_current_version_id.is_some() && (!cp_src_dst_same || version_id.is_none() || dest_version_id.is_some() || !dst_opts.versioned) @@ -400,7 +416,7 @@ impl DefaultObjectUsecase { // with that key (for example, copying `bucket/bucket` onto itself). let bucket_sse_config = load_bucket_default_sse_config(&bucket).await?; let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?; - if cp_src_dst_same && key == bucket { + if self_copy_lock && dst_lock_object == bucket { dst_opts.object_lock_config_snapshot = Some(store.object_lock_config_snapshot(&bucket).await.map_err(ApiError::from)?); } @@ -414,7 +430,7 @@ impl DefaultObjectUsecase { // expected-current historical copies. With lock optimization disabled, the source // stream retains its read guard until EOF; taking the write lock later in ECStore // would self-deadlock before the copy can consume that stream. - let _self_copy_lock_guard = if cp_src_dst_same { + let _self_copy_lock_guard = if self_copy_lock { let guard = acquire_self_copy_namespace_lock(store.as_ref(), &bucket, &key).await?; src_opts.no_lock = true; src_get_opts.no_lock = true; @@ -930,6 +946,21 @@ mod tests { use std::sync::Arc; use tokio::io::AsyncReadExt; + #[test] + fn copy_namespace_lock_object_folds_backslash_spellings_only_as_separators() { + // `a\b\` is not a same-object copy of `a/b` by `path_join_buf` (the + // trailing separator survives), yet on Windows both are the xl.meta + // of `a/b` and share one namespace lock. + assert_eq!(copy_namespace_lock_object("a\\b\\", true), copy_namespace_lock_object("a/b", true)); + assert_eq!(copy_namespace_lock_object("\\a\\b", true), "a/b"); + assert_ne!(copy_namespace_lock_object("a\\b\\", false), copy_namespace_lock_object("a/b", false)); + // Directory objects keep their encoded identity. + assert_eq!(copy_namespace_lock_object("a/b/", true), "a/b__XLDIR__"); + assert_ne!(copy_namespace_lock_object("a/b/", true), copy_namespace_lock_object("a/b", true)); + assert_eq!(copy_namespace_lock_object("bucket", true), "bucket"); + assert_eq!(copy_namespace_lock_object("\\bucket", true), "bucket"); + } + #[test] fn local_copy_does_not_inherit_source_replication_generation() { let mut metadata = HashMap::from([ diff --git a/rustfs/src/app/object/delete.rs b/rustfs/src/app/object/delete.rs index f231fc904..c0202e388 100644 --- a/rustfs/src/app/object/delete.rs +++ b/rustfs/src/app/object/delete.rs @@ -15,6 +15,9 @@ //! DeleteObject / DeleteObjects path. use super::*; +use crate::app::storage_api::object_usecase::DeleteObjectCondition; +use s3s::dto::ObjectIdentifier; +use std::collections::HashSet; async fn authorize_recursive_delete( req: &mut S3Request, @@ -110,6 +113,48 @@ fn normalize_delete_objects_version_id( } } +fn parse_delete_objects_condition(object: &ObjectIdentifier) -> Result, s3s::dto::Error> { + let Some(etag) = object.e_tag.as_ref() else { + return Ok(None); + }; + let (code, message) = if object.version_id.is_some() { + ( + S3ErrorCode::InvalidRequest, + "An ETag condition cannot be combined with an explicit VersionId".to_string(), + ) + } else if let Some(etag) = etag.as_strong() { + match DeleteObjectCondition::parse(etag) { + Ok(condition) => return Ok(Some(condition)), + Err(error) => (S3ErrorCode::InvalidArgument, error.to_string()), + } + } else { + (S3ErrorCode::InvalidArgument, "A delete ETag condition must use a strong ETag".to_string()) + }; + Err(s3s::dto::Error { + code: Some(code.as_str().to_string()), + key: Some(object.key.clone()), + message: Some(message), + version_id: object.version_id.clone(), + }) +} + +fn delete_objects_action_identity( + deleted: &StorageDeletedObject, + requested_version_id: Option, +) -> (String, Option, bool) { + let creates_delete_marker = deleted.delete_marker && requested_version_id.is_none(); + let target_version = if creates_delete_marker { + deleted.delete_marker_version_id + } else { + deleted.version_id.or(requested_version_id) + }; + ( + deleted.object_name.clone(), + target_version.filter(|version| !version.is_nil()), + creates_delete_marker, + ) +} + #[cfg(test)] type DeleteSnapshotTestHook = (String, Arc, Arc); @@ -391,6 +436,17 @@ fn delete_response_version_id(version_id: Option, synthetic_version_id: bo } } +fn delete_marker_response_version_id( + committed_version_id: Option, + creates_null_delete_marker: bool, + synthetic_version_id: bool, +) -> Option { + // The suspended-marker write shape uses None on disk. Only its response + // identity is null; changing the write shape would change delete semantics. + let version_id = committed_version_id.or_else(|| creates_null_delete_marker.then(Uuid::nil)); + delete_response_version_id(version_id, synthetic_version_id) +} + fn project_delete_objects_pre_stat_error( object: ObjectToDelete, synthetic_version_id: bool, @@ -437,7 +493,7 @@ fn reduce_delete_objects_result<'a>( ) -> Result<&'a StorageDeletedObject, s3s::dto::Error> { match error { None => Ok(deleted), - Some(error) if is_delete_objects_not_found(error) => Ok(deleted), + Some(error) if object.delete_condition.is_none() && is_delete_objects_not_found(error) => Ok(deleted), Some(error) => { let api_error = ApiError::from(error.clone()); Err(s3s::dto::Error { @@ -490,7 +546,9 @@ impl DefaultObjectUsecase { struct DeleteResult { delete_object: Option, error: Option, + emit_side_effects: bool, synthetic_version_id: bool, + version_suspended: bool, // The version identity the request addressed, before any // synthetic-directory synthesis. `None` means the request carried // no version id, i.e. a committed delete marker was created, not @@ -520,6 +578,13 @@ impl DefaultObjectUsecase { }); continue; } + let delete_condition = match parse_delete_objects_condition(obj_id) { + Ok(condition) => condition, + Err(error) => { + delete_results[idx].error = Some(error); + continue; + } + }; let raw_version_id = obj_id.version_id.clone(); let (version_id, version_uuid) = match normalize_delete_objects_version_id(raw_version_id.clone()) { Ok(parsed) => parsed, @@ -541,7 +606,14 @@ impl DefaultObjectUsecase { req_info.version_id = version_id.clone(); } - let auth_res = authorize_request(&mut req, delete_object_authorize_action(version_id.as_deref())).await; + let mut auth_res = authorize_request(&mut req, delete_object_authorize_action(version_id.as_deref())).await; + if auth_res.is_ok() + && delete_condition + .as_ref() + .is_some_and(DeleteObjectCondition::requires_read_permission) + { + auth_res = authorize_request(&mut req, Action::S3Action(S3Action::GetObjectAction)).await; + } if auth_res.is_err() { if !bulk_denial_logged { bulk_denial_logged = true; @@ -583,10 +655,11 @@ impl DefaultObjectUsecase { continue; } - let synthetic_version_id = version_id.is_none() && is_dir_object(&obj_id.key); + let synthetic_version_id = delete_condition.is_none() && version_id.is_none() && is_dir_object(&obj_id.key); let object = ObjectToDelete { object_name: obj_id.key.clone(), version_id: version_uuid, + delete_condition, synthetic_version_id, ..Default::default() }; @@ -598,7 +671,7 @@ impl DefaultObjectUsecase { if authorized_deletes.is_empty() { let output = DeleteObjectsOutput { - deleted: Some(Vec::new()), + deleted: (!delete.quiet.unwrap_or_default()).then(Vec::new), errors: Some(delete_results.into_iter().filter_map(|result| result.error).collect()), ..Default::default() }; @@ -687,7 +760,7 @@ impl DefaultObjectUsecase { opts, skip_stat, } = prepared; - let synthetic_version_id = object.version_id.is_none() && is_dir_object(&object.object_name); + let synthetic_version_id = object.synthetic_version_id; let error = if skip_stat { None } else { @@ -763,6 +836,7 @@ impl DefaultObjectUsecase { return result; } + let mut committed_actions = HashSet::new(); for (i, err) in errs.iter().enumerate() { let didx = object_to_delete_idx[i]; @@ -773,8 +847,15 @@ impl DefaultObjectUsecase { delete_results[didx].synthetic_version_id, ) { Ok(deleted_object) => { - delete_results[didx].delete_object = Some(deleted_object.clone()); let (versioned, version_suspended) = object_versioning[i]; + delete_results[didx].delete_object = Some(deleted_object.clone()); + delete_results[didx].version_suspended = version_suspended; + if !committed_actions + .insert(delete_objects_action_identity(deleted_object, delete_results[didx].requested_version_id)) + { + continue; + } + delete_results[didx].emit_side_effects = true; let creates_delete_marker = object_to_delete[i].version_id.is_none() && versioned && !version_suspended; let committed_delete_marker = dobjs[i].delete_marker; let delete_accounting = accounting.get(i).and_then(Option::as_ref); @@ -804,8 +885,9 @@ impl DefaultObjectUsecase { .filter_map(|result| result.delete_object.as_ref().map(|object| (result, object))) .map(|(result, object)| DeletedObject { delete_marker: { if object.delete_marker { Some(true) } else { None } }, - delete_marker_version_id: delete_response_version_id( + delete_marker_version_id: delete_marker_response_version_id( object.delete_marker_version_id, + object.delete_marker && result.version_suspended && result.requested_version_id.is_none(), result.synthetic_version_id, ), key: Some(object.object_name.clone()), @@ -828,7 +910,7 @@ impl DefaultObjectUsecase { .filter_map(|v| v.error.clone()) .collect::>(); let output = DeleteObjectsOutput { - deleted: Some(deleted), + deleted: (!delete.quiet.unwrap_or_default()).then_some(deleted), errors: Some(errors), ..Default::default() }; @@ -843,6 +925,7 @@ impl DefaultObjectUsecase { let replication_deletes = if replicate_deletes { delete_results .iter() + .filter(|result| result.emit_side_effects) .filter_map(|result| result.delete_object.as_ref()) .filter(|dobj| deleted_object_has_pending_replication_delete(dobj)) .cloned() @@ -870,6 +953,7 @@ impl DefaultObjectUsecase { let deleted_any = delete_results.iter().any(|result| result.delete_object.is_some()); let delete_producers = delete_results .iter() + .filter(|result| result.emit_side_effects) .filter_map(|result| { result.delete_object.as_ref().map(|deleted_object| { if deleted_object.delete_marker && result.requested_version_id.is_none() { @@ -884,6 +968,9 @@ impl DefaultObjectUsecase { spawn_background_with_context(Some(request_context), async move { let _activity_guard = DeleteTailActivityGuard::new(DeleteTailStage::Notify); for res in delete_results { + if !res.emit_side_effects { + continue; + } if let Some(dobj) = res.delete_object { // `DeleteMarkerCreated` is a creation event; a delete // marker removed by an explicit version id is a plain @@ -924,8 +1011,10 @@ impl DefaultObjectUsecase { rustfs_scanner::record_dirty_usage_bucket_from_producers(&bucket, delete_producers); } // Record write operation for capacity management (inline to avoid per-request tokio::spawn overhead) - let manager = get_capacity_manager(); - manager.record_write_operation().await; + if deleted_any { + let manager = get_capacity_manager(); + manager.record_write_operation().await; + } result } @@ -1951,6 +2040,26 @@ mod tests { assert_eq!(delete_response_version_id(None, false), None); } + #[test] + fn delete_objects_created_null_marker_response_uses_the_committed_marker_identity() { + let uuid = Uuid::new_v4(); + assert_eq!(delete_marker_response_version_id(None, true, false), Some("null".to_string())); + assert_eq!(delete_marker_response_version_id(None, false, false), None); + assert_eq!(delete_marker_response_version_id(None, true, true), None); + assert_eq!(delete_marker_response_version_id(Some(uuid), false, false), Some(uuid.to_string())); + assert_eq!( + delete_marker_response_version_id(Some(uuid), true, false), + Some(uuid.to_string()), + "the committed identity takes precedence over any null fallback" + ); + assert_eq!( + delete_marker_response_version_id(Some(Uuid::nil()), false, false), + Some("null".to_string()), + "explicit null-marker removal keeps its existing response identity" + ); + assert_eq!(delete_marker_response_version_id(Some(Uuid::nil()), true, true), None); + } + #[test] fn delete_entry_response_version_id_echoes_requested_identity_for_marker_removal() { let version_id = Uuid::new_v4(); @@ -2290,6 +2399,84 @@ mod tests { assert_eq!(internal_version_id, None); } + #[test] + fn delete_objects_condition_parser_preserves_presence_and_rejects_version_combinations() { + let object = ObjectIdentifier { + key: "key".to_string(), + ..Default::default() + }; + assert_eq!(parse_delete_objects_condition(&object).expect("missing ETag is unconditional"), None); + for etag in ["", "\"\"", " \t "] { + let error = parse_delete_objects_condition(&ObjectIdentifier { + e_tag: Some(ETag::Strong(etag.to_string())), + ..object.clone() + }) + .expect_err("an explicit empty ETag must never become an unconditional delete"); + assert_eq!(error.code.as_deref(), Some("InvalidArgument")); + assert_eq!(error.key.as_deref(), Some("key")); + } + for version in ["null", "", "bad-version"] { + let error = parse_delete_objects_condition(&ObjectIdentifier { + e_tag: Some(ETag::Strong("*".to_string())), + version_id: Some(version.to_string()), + ..object.clone() + }) + .expect_err("any explicit VersionId is unsupported with a condition"); + assert_eq!(error.code.as_deref(), Some("InvalidRequest")); + assert_eq!(error.version_id.as_deref(), Some(version)); + } + } + + #[test] + fn delete_objects_action_identity_deduplicates_actions_without_losing_historical_targets() { + let current = StorageDeletedObject { + object_name: "key".to_string(), + ..Default::default() + }; + assert_eq!( + delete_objects_action_identity(¤t, None), + delete_objects_action_identity(¤t, Some(Uuid::nil())) + ); + let historical = Uuid::new_v4(); + assert_ne!( + delete_objects_action_identity(¤t, None), + delete_objects_action_identity(¤t, Some(historical)) + ); + let marker = StorageDeletedObject { + delete_marker: true, + delete_marker_version_id: Some(historical), + ..current + }; + assert_ne!( + delete_objects_action_identity(&marker, None), + delete_objects_action_identity(&marker, Some(historical)) + ); + } + + #[test] + fn delete_objects_condition_parser_rejects_weak_validators_without_dropping_the_condition() { + for value in ["current-etag", "*", ""] { + let object = ObjectIdentifier { + key: "key".to_string(), + e_tag: Some(ETag::Weak(value.to_string())), + ..Default::default() + }; + let error = parse_delete_objects_condition(&object) + .expect_err("a weak validator must fail instead of falling back to an unconditional delete"); + assert_eq!(error.code.as_deref(), Some("InvalidArgument")); + assert_eq!(error.key.as_deref(), Some("key")); + } + let object = ObjectIdentifier { + key: "key".to_string(), + e_tag: Some(ETag::Strong("current-etag".to_string())), + ..Default::default() + }; + assert_eq!( + parse_delete_objects_condition(&object).expect("strong validator retains its condition"), + Some(DeleteObjectCondition::MatchEtag("current-etag".to_string())) + ); + } + #[test] fn delete_objects_treats_raw_io_not_found_as_idempotent() { assert!(is_delete_objects_not_found(&StorageError::FileNotFound)); @@ -2319,6 +2506,230 @@ mod tests { assert_eq!(deleted.object_name, "missing-key"); } + #[test] + fn delete_objects_result_reducer_preserves_conditional_not_found_errors() { + let object = ObjectToDelete { + object_name: "missing-key".to_string(), + delete_condition: Some(DeleteObjectCondition::Exists), + ..Default::default() + }; + let deleted = StorageDeletedObject { + object_name: object.object_name.clone(), + ..Default::default() + }; + let error = reduce_delete_objects_result(&object, &deleted, Some(&StorageError::FileNotFound), false) + .expect_err("a failed existence condition is not an idempotent successful delete"); + assert_eq!(error.key.as_deref(), Some("missing-key")); + assert_eq!(error.code.as_deref(), Some("NoSuchKey")); + } + + #[test] + #[serial_test::serial] + fn execute_delete_objects_conditions_authorize_per_entry_and_keep_quiet_errors() { + crate::app::gating_test_env::run_large_stack_test("conditional-bulk-delete-api", || async { + use crate::app::storage_api::test::contract::bucket::{ + BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, + }; + + let store = crate::app::gating_test_env::shared_gating_ecstore().await; + if current_app_context().is_none() { + crate::app::runtime_sources::install_test_app_context(Arc::clone(&store)).await; + } + let bucket = format!("conditional-bulk-api-{}", Uuid::new_v4().simple()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create conditional bulk-delete bucket"); + let mut readable_etag = String::new(); + let mut denied_etag = String::new(); + let mut total_size = 0; + for key in [ + "readable", + "denied", + "wildcard", + "unconditional", + "invalid", + "version-combination", + ] { + let object = store + .put_object( + &bucket, + key, + &mut PutObjReader::from_vec(format!("payload-{key}").into_bytes()), + &ObjectOptions::default(), + ) + .await + .expect("seed object before request authorization"); + if key == "readable" { + readable_etag = object.etag.expect("readable fixture has an ETag"); + } else if key == "denied" { + denied_etag = object.etag.expect("denied fixture has an ETag"); + } + total_size += u64::try_from(object.size).expect("fixture size is nonnegative"); + } + crate::app::storage_api::test::data_usage::seed_bucket_usage_memory_for_test(&bucket, total_size).await; + let policy_json = format!( + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:DeleteObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}},{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::{bucket}/readable"]}},{{"Effect":"Deny","Principal":{{"AWS":"*"}},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::{bucket}/denied","arn:aws:s3:::{bucket}/wildcard"]}}]}}"# + ); + let mut metadata = (*crate::storage::get_bucket_metadata(&bucket) + .await + .expect("test bucket metadata")) + .clone(); + metadata.policy_config = Some(serde_json::from_str(&policy_json).expect("parse conditional delete policy")); + metadata.policy_config_json = policy_json.into_bytes(); + crate::storage::storage_api::set_bucket_metadata(bucket.clone(), metadata) + .await + .expect("publish conditional delete authorization policy"); + let requests = vec![ + ObjectIdentifier { + key: "readable".to_string(), + e_tag: Some(ETag::Strong(readable_etag.clone())), + ..Default::default() + }, + ObjectIdentifier { + key: "readable".to_string(), + e_tag: Some(ETag::Strong("stale-etag".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "readable".to_string(), + e_tag: Some(ETag::Strong(readable_etag)), + ..Default::default() + }, + ObjectIdentifier { + key: "denied".to_string(), + e_tag: Some(ETag::Strong(denied_etag)), + ..Default::default() + }, + ObjectIdentifier { + key: "wildcard".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "missing".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "invalid".to_string(), + e_tag: Some(ETag::Strong("".to_string())), + ..Default::default() + }, + ObjectIdentifier { + key: "version-combination".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + version_id: Some("null".to_string()), + ..Default::default() + }, + ObjectIdentifier { + key: "unconditional".to_string(), + ..Default::default() + }, + ]; + let input = DeleteObjectsInput::builder() + .bucket(bucket.clone()) + .delete(Delete { + objects: requests, + quiet: Some(false), + }) + .build() + .expect("build mixed conditional batch"); + let mut req = build_request(input, Method::POST); + req.extensions.insert(crate::storage::access::ReqInfo::default()); + let usecase = DefaultObjectUsecase::from_global(); + let response = usecase + .execute_delete_objects(req) + .await + .expect("mixed batch returns per-entry outcomes"); + let deleted = response + .output + .deleted + .expect("non-quiet batch reports every successful input entry"); + assert_eq!( + deleted.iter().map(|object| object.key.as_deref()).collect::>(), + vec![Some("readable"), Some("readable"), Some("wildcard"), Some("unconditional")], + "duplicate actions preserve both original input results" + ); + let errors = response.output.errors.expect("failed entries remain visible in Quiet mode"); + let actual = errors + .iter() + .map(|error| (error.key.as_deref(), error.code.as_deref())) + .collect::>(); + assert_eq!( + actual, + vec![ + (Some("readable"), Some("PreconditionFailed")), + (Some("denied"), Some("AccessDenied")), + (Some("missing"), Some("NoSuchKey")), + (Some("invalid"), Some("InvalidArgument")), + (Some("version-combination"), Some("InvalidRequest")), + ] + ); + for key in ["denied", "invalid", "version-combination"] { + store + .get_object_info(&bucket, key, &ObjectOptions::default()) + .await + .expect("rejected entry preserves data"); + } + for key in ["readable", "wildcard", "unconditional"] { + let error = store + .get_object_info(&bucket, key, &ObjectOptions::default()) + .await + .expect_err("successful entry deletes data"); + assert!(is_err_object_not_found(&error)); + } + let mut usage = rustfs_data_usage::DataUsageInfo::default(); + crate::app::storage_api::test::data_usage::apply_bucket_usage_memory_overlay(&mut usage).await; + let retained_size = ["denied", "invalid", "version-combination"] + .into_iter() + .map(|key| u64::try_from(format!("payload-{key}").len()).expect("fixture sizes fit u64")) + .sum::(); + assert_eq!( + usage.buckets_usage.get(&bucket).map(|value| value.size), + Some(retained_size), + "duplicate success and failed conditions must not subtract additional object bytes" + ); + + let quiet_input = DeleteObjectsInput::builder() + .bucket(bucket.clone()) + .delete(Delete { + objects: vec![ + ObjectIdentifier { + key: "unconditional".to_string(), + ..Default::default() + }, + ObjectIdentifier { + key: "missing".to_string(), + e_tag: Some(ETag::Strong("*".to_string())), + ..Default::default() + }, + ], + quiet: Some(true), + }) + .build() + .expect("build quiet mixed batch"); + let mut quiet_req = build_request(quiet_input, Method::POST); + quiet_req.extensions.insert(crate::storage::access::ReqInfo::default()); + let quiet_response = usecase.execute_delete_objects(quiet_req).await.expect("quiet batch succeeds"); + assert!(quiet_response.output.deleted.is_none(), "Quiet hides only successful entries"); + let quiet_errors = quiet_response.output.errors.expect("Quiet preserves errors"); + assert_eq!(quiet_errors.len(), 1); + assert_eq!(quiet_errors[0].key.as_deref(), Some("missing")); + assert_eq!(quiet_errors[0].code.as_deref(), Some("NoSuchKey")); + store + .delete_bucket( + &bucket, + &DeleteBucketOptions { + force: true, + ..Default::default() + }, + ) + .await + .expect("remove fixture bucket"); + }); + } + #[test] #[serial_test::serial] fn recursive_delete_holds_writer_exclusion_after_authorization() { diff --git a/rustfs/src/app/object/get.rs b/rustfs/src/app/object/get.rs index 712f0803d..f146c122c 100644 --- a/rustfs/src/app/object/get.rs +++ b/rustfs/src/app/object/get.rs @@ -277,6 +277,16 @@ const GET_OBJECT_STAGE_LIFECYCLE_EXPIRATION: &str = "lifecycle_expiration"; const GET_OBJECT_STAGE_METADATA_FILTER: &str = "metadata_filter"; +const GET_OBJECT_STAGE_BODY_CACHE_HOOK_HIT: &str = "body_cache_hook_hit"; +const GET_OBJECT_STAGE_BODY_CACHE_LOOKUP: &str = "body_cache_lookup"; +const GET_OBJECT_STAGE_BODY_CACHE_FILL_HANDOFF: &str = "body_cache_fill_handoff"; +const GET_OBJECT_STAGE_BODY_BUFFERED_HANDOFF: &str = "body_buffered_handoff"; +const GET_OBJECT_STAGE_OUTPUT_STRUCT_BUILD: &str = "output_struct_build"; +const GET_OBJECT_STAGE_PREPARE_READ_EXECUTION: &str = "prepare_read_execution"; +const GET_OBJECT_STAGE_RESPONSE_CORS_WRAP: &str = "response_cors_wrap"; +const GET_OBJECT_STAGE_RESPONSE_HEADER_INJECT: &str = "response_header_inject"; +const GET_OBJECT_STAGE_RESPONSE_EVENT_COMPLETE: &str = "response_event_complete"; + const GET_OBJECT_STREAM_WARN_THRESHOLD: Duration = Duration::from_secs(5); static GET_OBJECT_BUFFER_THRESHOLD_WARNED: AtomicBool = AtomicBool::new(false); @@ -555,10 +565,10 @@ fn get_object_stream_failure_reason(error_class: &'static str) -> &'static str { } } -/// A mid-stream reopen can recover only when another representation of the -/// committed object may exist. A single-disk, no-parity object has no alternate -/// shard after a bitrot/read-quorum failure, so retrying the same read only -/// repeats the failure and adds latency. +/// A mid-stream reopen can only recover when another representation of the +/// committed object may exist. Single-disk, no-parity objects have no alternate +/// shard to discover after a bitrot/read-quorum failure, so arming the normal +/// three-attempt resume loop only repeats the same failing read and adds delay. fn should_attach_get_object_resume(info: &ObjectInfo) -> bool { info.parity_blocks > 0 || info.data_blocks > 1 || !info.transitioned_object.tier.is_empty() } @@ -3470,12 +3480,15 @@ impl DefaultObjectUsecase { // already supplied this body, the request-level plan was built before // the authoritative lookup. Serve it without planning a second time. if cache_hook_served && let Some(bytes) = buffered_body.take() { - return Ok(Self::build_memory_bytes_blob( + let hook_hit_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let body = Self::build_memory_bytes_blob( bytes, response_content_length, GET_MEMORY_BODY_SOURCE_OBJECT_DATA_CACHE, lifecycle, - )); + ); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_HOOK_HIT, hook_hit_start); + return Ok(body); } if !cache_fill_allowed { @@ -3516,20 +3529,27 @@ impl DefaultObjectUsecase { // authoritative because it ran after fresh metadata resolution, so the // app layer skips its own lookup and only uses the plan to fill. if !cache_hook_probed { - match lookup_get_object_body_cache_hit(cache_adapter, &cache_plan).await { + let cache_lookup_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let cache_lookup = lookup_get_object_body_cache_hit(cache_adapter, &cache_plan).await; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_LOOKUP, cache_lookup_start); + match cache_lookup { GetObjectBodyCacheLookup::Hit(bytes) => { - return Ok(Self::build_memory_bytes_blob( + let cache_hit_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let body = Self::build_memory_bytes_blob( bytes, response_content_length, GET_MEMORY_BODY_SOURCE_OBJECT_DATA_CACHE, lifecycle, - )); + ); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_HOOK_HIT, cache_hit_start); + return Ok(body); } GetObjectBodyCacheLookup::Disabled | GetObjectBodyCacheLookup::Skip | GetObjectBodyCacheLookup::Miss => {} } } if let Some(buffered_body) = buffered_body { + let cache_fill_handoff_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); // ODC-15: the body is already fully in hand, so keep the fill off the // response's critical path. For a cacheable plan, run the fill in a // detached task (Bytes is a cheap clone) and return immediately. For @@ -3553,13 +3573,17 @@ impl DefaultObjectUsecase { } let _ = fill_get_object_body_cache_from_buffered_body(cache_adapter, &cache_plan, &buffered_body).await; } + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_CACHE_FILL_HANDOFF, cache_fill_handoff_start); - return Ok(Self::build_memory_bytes_blob( + let buffered_handoff_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); + let body = Self::build_memory_bytes_blob( buffered_body, response_content_length, GET_MEMORY_BODY_SOURCE_BUFFERED_BODY, lifecycle, - )); + ); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_BODY_BUFFERED_HANDOFF, buffered_handoff_start); + return Ok(body); } let should_materialize_for_cache = cache_adapter.materialize_fill_enabled() @@ -3613,6 +3637,7 @@ impl DefaultObjectUsecase { { Ok(buf) => { let bytes = Bytes::from(buf); + let cache_fill_handoff_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); // ODC-15: fill off the response's critical path (see the // buffered-body branch above). let cache_adapter = cache_adapter.clone(); @@ -3625,6 +3650,10 @@ impl DefaultObjectUsecase { } let _ = fill_get_object_body_cache_from_materialized_body(&cache_adapter, &cache_plan, &fill_bytes).await; }); + record_get_object_s3_handler_stage_duration( + GET_OBJECT_STAGE_BODY_CACHE_FILL_HANDOFF, + cache_fill_handoff_start, + ); return Ok(Self::build_memory_bytes_blob( bytes, @@ -3763,13 +3792,19 @@ impl DefaultObjectUsecase { None => helper, }; let helper = helper.version_id(version_id_for_event); + let cors_wrap_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let mut response = wrap_response_with_cors(bucket, method, headers, output).await; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_CORS_WRAP, cors_wrap_start); + let header_inject_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); inject_accept_ranges_header(&mut response.headers); // Emit XXHash3/64/128 and SHA-512 checksums that s3s GetObjectOutput cannot // carry (#1257). This is the download-side integrity path AWS SDKs verify. inject_additional_checksum_headers(&mut response.headers, &extra_checksum_headers); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_HEADER_INJECT, header_inject_start); let result = Ok(response); + let event_complete_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let _ = helper.complete(&result); + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_EVENT_COMPLETE, event_complete_start); result } @@ -3879,9 +3914,8 @@ impl DefaultObjectUsecase { let metadata_filter_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let metadata = filter_object_metadata(&info.user_defined); record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_METADATA_FILTER, metadata_filter_start); - + let output_struct_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now); let parts_count = Self::get_object_parts_count(part_number, &info); - let output = GetObjectOutput { body: Some(body), content_length: Some(response_content_length), @@ -3915,6 +3949,7 @@ impl DefaultObjectUsecase { storage_class, ..Default::default() }; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_OUTPUT_STRUCT_BUILD, output_struct_start); Ok(GetObjectOutputContext { output, @@ -4217,6 +4252,7 @@ impl DefaultObjectUsecase { let manager = get_concurrency_manager(); + let prepare_read_start = stage_metrics_enabled.then(std::time::Instant::now); let mut prepared_read = self .prepare_get_object_read_execution( &req, @@ -4232,6 +4268,7 @@ impl DefaultObjectUsecase { object_traffic_health.clone(), ) .await; + record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_PREPARE_READ_EXECUTION, prepare_read_start); // An object missing locally (and only missing — other errors keep // their semantics) may still be served by a remote copy. if let Err(err) = &prepared_read @@ -4275,6 +4312,7 @@ impl DefaultObjectUsecase { return Self::complete_get_object_error(helper.version_id(version_id_for_event), err); } Some(OdmGetOutcome::RetryLocal) => { + let prepare_read_retry_start = stage_metrics_enabled.then(std::time::Instant::now); prepared_read = self .prepare_get_object_read_execution( &req, @@ -4290,6 +4328,10 @@ impl DefaultObjectUsecase { object_traffic_health, ) .await; + record_get_object_s3_handler_stage_duration( + GET_OBJECT_STAGE_PREPARE_READ_EXECUTION, + prepare_read_retry_start, + ); } } } @@ -5893,8 +5935,18 @@ mod tests { *metadata.last_mut().expect("inline data bytes") ^= 1; tokio::fs::write(&meta_path, metadata).await.expect("damage inline payload"); } else { - let data_path = object_path.join(info.data_dir.expect("sharded object data directory").to_string()); - tokio::fs::remove_dir_all(data_path).await.expect("remove object data shards"); + // Discover version data dirs by listing the object directory + // (matching delete_object_part_shards) instead of constructing + // the path from info.data_dir, whose UUID string form may not + // match the on-disk directory name. + let mut entries = tokio::fs::read_dir(&object_path).await.expect("read object dir"); + while let Some(entry) = entries.next_entry().await.expect("read object dir entry") { + if entry.file_type().await.expect("entry file type").is_dir() { + tokio::fs::remove_dir_all(entry.path()) + .await + .expect("remove object data shards"); + } + } } } let resolved = store diff --git a/rustfs/src/app/object/internal_put.rs b/rustfs/src/app/object/internal_put.rs index 45dca295a..c27d6e3bf 100644 --- a/rustfs/src/app/object/internal_put.rs +++ b/rustfs/src/app/object/internal_put.rs @@ -37,7 +37,7 @@ use crate::app::storage_api::object_usecase::sse::{ use crate::capacity::record_capacity_write; use crate::runtime_sources::NotifyInterface; use crate::table_catalog::{ - EcStoreTableCatalogObjectBackend, TableCatalogLockGuard, TableCatalogObjectBackend, TableCatalogStoreError, + EcStoreTableCatalogObjectBackend, TableCatalogLockGuard, TableCatalogObjectBackend, default_table_bucket_publication_lock_path, }; use http::HeaderName; @@ -250,13 +250,7 @@ impl DefaultObjectUsecase { let guard = backend .acquire_read_lock(bucket, &default_table_bucket_publication_lock_path()) .await - .map_err(|err: TableCatalogStoreError| { - let code = match err { - TableCatalogStoreError::Unavailable(_) => S3ErrorCode::ServiceUnavailable, - _ => S3ErrorCode::InternalError, - }; - api_error_from_s3(S3Error::with_message(code, "failed to acquire table publication guard")) - })?; + .map_err(ApiError::from)?; Ok(Some(guard)) } diff --git a/rustfs/src/app/object/put.rs b/rustfs/src/app/object/put.rs index bb652fc65..9cce8a1cd 100644 --- a/rustfs/src/app/object/put.rs +++ b/rustfs/src/app/object/put.rs @@ -23,7 +23,7 @@ static PUT_FAILURE_LOGS: rustfs_utils::LogThrottle = rustfs_utils::LogThrottle:: #[cfg(test)] #[derive(Clone, Copy, PartialEq)] -enum PutReadQuorumTestStage { +enum PutStorageTestStage { DefaultSse, Prelookup, Commit, @@ -31,15 +31,15 @@ enum PutReadQuorumTestStage { #[cfg(test)] tokio::task_local! { - static PUT_READ_QUORUM_TEST_STAGE: PutReadQuorumTestStage; + static PUT_STORAGE_TEST_FAILURE: (PutStorageTestStage, StorageError); } #[cfg(test)] -fn injected_put_read_quorum_failure(stage: PutReadQuorumTestStage) -> Option { - PUT_READ_QUORUM_TEST_STAGE - .try_with(|configured| *configured == stage) - .unwrap_or(false) - .then(|| StorageError::InsufficientReadQuorum("bucket".into(), "object".into())) +fn injected_put_storage_failure(stage: PutStorageTestStage) -> Option { + PUT_STORAGE_TEST_FAILURE + .try_with(|(configured_stage, error)| (*configured_stage == stage).then(|| error.clone())) + .ok() + .flatten() } const DEFAULT_PUT_LARGE_CONCURRENCY_TUNING_MIN_SIZE_BYTES: i64 = 32 * 1024 * 1024; @@ -1261,6 +1261,7 @@ impl DefaultObjectUsecase { { return Err(s3_error!(InvalidStorageClass)); } + reject_non_wildcard_write_if_none_match(req.input.if_none_match.as_ref(), &req.headers)?; // An authorized inbound replication PUT must store the replica verbatim. // Legacy snowball-extracted members may still carry the auto-extract // metadata, which replication replays as a header. Do not interpret that @@ -1581,7 +1582,7 @@ impl DefaultObjectUsecase { let sse_config_stage_start = put_stage_metrics_enabled.then(Instant::now); let bucket_sse_config = load_bucket_default_sse_config(&bucket).await; #[cfg(test)] - let bucket_sse_config = injected_put_read_quorum_failure(PutReadQuorumTestStage::DefaultSse) + let bucket_sse_config = injected_put_storage_failure(PutStorageTestStage::DefaultSse) .map_or(bucket_sse_config, |err| Err(ApiError::from(err).into())); rustfs_io_metrics::record_put_object_stage_duration_from("app_sse_config_lookup", sse_config_stage_start); let bucket_sse_config = bucket_sse_config?; @@ -1734,8 +1735,7 @@ impl DefaultObjectUsecase { // metadata fanout per PUT. let prelookup_required = version_id.is_some() || object_lock_checks_required_for_state(&object_lock_config_state); #[cfg(test)] - let prelookup_required = - prelookup_required || injected_put_read_quorum_failure(PutReadQuorumTestStage::Prelookup).is_some(); + let prelookup_required = prelookup_required || injected_put_storage_failure(PutStorageTestStage::Prelookup).is_some(); // Outer None = prelookup skipped (accounting comes from the commit // backfill); Some(inner) = the previous current size as observed by the // lookup, with the pre-#1009 semantics kept bit-for-bit. @@ -1752,7 +1752,7 @@ impl DefaultObjectUsecase { }; #[cfg(test)] let previous_current_info = - injected_put_read_quorum_failure(PutReadQuorumTestStage::Prelookup).map_or(previous_current_info, Err); + injected_put_storage_failure(PutStorageTestStage::Prelookup).map_or(previous_current_info, Err); Some(match previous_current_info { Ok(existing_obj_info) => { validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &opts)?; @@ -1791,6 +1791,7 @@ impl DefaultObjectUsecase { )?; } + let checksum_input_stage_start = put_stage_metrics_enabled.then(Instant::now); let mut md5hex = match content_md5 { Some(PutObjectContentMd5::Base64(base64_md5)) => { let md5 = base64_simd::STANDARD @@ -1803,12 +1804,15 @@ impl DefaultObjectUsecase { }; let mut sha256hex = get_content_sha256_with_query(headers, query); + rustfs_io_metrics::record_put_object_stage_duration_from("app_checksum_input_prepare", checksum_input_stage_start); let mut write_plan = WritePlan::new(); // Additional-checksum (XXHash3/64/128, SHA-512) values to echo on the PutObject // response (#1256); captured at want_checksum set points before opts is moved. let mut put_extra_checksum_headers: Vec<(&'static str, String)> = Vec::new(); + let reader_prepare_stage_start = put_stage_metrics_enabled.then(Instant::now); let mut reader = if should_compress { + let compressed_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); let body = tokio::io::BufReader::with_capacity( buffer_size, StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), @@ -1831,51 +1835,98 @@ impl DefaultObjectUsecase { size = HashReader::SIZE_PRESERVE_LAYER; write_plan = write_plan.with_compression(algorithm); + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_reader_prepare_compressed", + compressed_reader_stage_start, + ); hrd } else { if use_zero_copy_eager_put_path { let zero_copy_start = std::time::Instant::now(); + let eager_read_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_body = read_zero_copy_put_body_exact(body, body_size as usize).await?; + rustfs_io_metrics::record_put_object_stage_duration_from("app_body_eager_read_zero_copy", eager_read_stage_start); rustfs_io_metrics::record_zero_copy_write(body_size as usize, zero_copy_start.elapsed().as_secs_f64() * 1000.0); - HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + let hash_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); + let reader = + HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_hash_reader_setup_zero_copy", + hash_reader_stage_start, + ); + reader } else if use_empty_or_small_eager_put_path { if (body_size as usize) <= POOL_BYPASS_MAX_SIZE { // Bypass BytesPool for very small objects to avoid Small-tier // Mutex contention under high concurrency. Direct allocation // for ≤4KiB is negligible cost. + let eager_read_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_body = read_small_put_body_exact_direct( StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), body_size as usize, ) .await?; - HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_body_eager_read_small_direct", + eager_read_stage_start, + ); + let hash_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); + let reader = HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false) + .map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_hash_reader_setup_small_direct", + hash_reader_stage_start, + ); + reader } else { let pool = get_concurrency_manager().bytes_pool(); + let eager_read_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_body = read_small_put_body_exact_pooled( StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), body_size as usize, pool.as_ref(), ) .await?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_body_eager_read_small_pooled", + eager_read_stage_start, + ); + let hash_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); let eager_reader = PooledBufferReader::new(eager_body, body_size as usize); - HashReader::from_stream(eager_reader, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + let reader = HashReader::from_stream(eager_reader, size, actual_size, md5hex, sha256hex, false) + .map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_hash_reader_setup_small_pooled", + hash_reader_stage_start, + ); + reader } } else { + let streaming_reader_stage_start = put_stage_metrics_enabled.then(Instant::now); let body = tokio::io::BufReader::with_capacity( buffer_size, StreamReader::new(body.map(|f| f.map_err(s3s_body_error_to_io))), ); - HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? + let reader = + HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_reader_prepare_streaming", + streaming_reader_stage_start, + ); + reader } }; + rustfs_io_metrics::record_put_object_stage_duration_from("app_reader_prepare", reader_prepare_stage_start); if size >= 0 { + let checksum_setup_stage_start = put_stage_metrics_enabled.then(Instant::now); if let Err(err) = reader.add_checksum(headers, trailer_source(trailing_headers.clone()), false) { return Err(ApiError::from(err).into()); } opts.want_checksum = reader.checksum(); put_extra_checksum_headers = additional_checksum_echo_pairs(&opts.want_checksum); + rustfs_io_metrics::record_put_object_stage_duration_from("app_reader_checksum_setup", checksum_setup_stage_start); } rustfs_io_metrics::record_put_object_path(put_path); rustfs_io_metrics::record_put_object_stage_duration_from("ingress_prepare", ingress_stage_start); @@ -2028,8 +2079,8 @@ impl DefaultObjectUsecase { None }; #[cfg(test)] - let injected_commit_error = injected_put_read_quorum_failure(PutReadQuorumTestStage::Commit); - let put_commit = spawn_traced_join({ + let injected_commit_error = injected_put_storage_failure(PutStorageTestStage::Commit); + let put_commit_future = { let store = Arc::clone(&store); let bucket = bucket.clone(); let key = key.clone(); @@ -2116,17 +2167,36 @@ impl DefaultObjectUsecase { wait_for_put_post_store_test_hook(&bucket).await; let post_store_stage_start = put_stage_metrics_enabled.then(Instant::now); + let transition_enqueue_stage_start = put_stage_metrics_enabled.then(Instant::now); maybe_enqueue_transition_immediate(&obj_info, LcEventSrc::S3PutObject).await; - let _ = invalidate_object_data_cache_after_put_success(&cache_adapter, &bucket, &key).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_transition_enqueue", + transition_enqueue_stage_start, + ); + let cache_invalidate_stage_start = put_stage_metrics_enabled.then(Instant::now); + let _ = invalidate_object_data_cache_after_put_success(&cache_adapter, &bucket, &key).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_data_cache_invalidate", + cache_invalidate_stage_start, + ); + + let versioning_lookup_stage_start = put_stage_metrics_enabled.then(Instant::now); let put_versioned = BucketVersioningSys::prefix_enabled(&bucket, &key).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_versioning_lookup", + versioning_lookup_stage_start, + ); // Fast in-memory update for immediate quota and admin usage consistency. // The previous current size comes from the prelookup when it ran, // otherwise from the rename_data backfill (rustfs/backlog#1009); the // backfill reproduces the lookup's observation bit for bit (latest // version's ObjectInfo.size — 0 for a delete-marker latest — or // not-found → None). + let quota_size_stage_start = put_stage_metrics_enabled.then(Instant::now); let committed_size = quota_accounting_object_size(&obj_info, quota_enabled)?; + rustfs_io_metrics::record_put_object_stage_duration_from("app_post_store_quota_size", quota_size_stage_start); + let usage_memory_stage_start = put_stage_metrics_enabled.then(Instant::now); match prelookup_previous_current_size.or_else(|| previous_current_size_from_backfill(backfilled_old_current_size)) { Some(previous_current_size) => { @@ -2152,16 +2222,27 @@ impl DefaultObjectUsecase { record_bucket_object_write_unknown_previous_memory(&bucket, committed_size, put_versioned).await; } } + rustfs_io_metrics::record_put_object_stage_duration_from("app_post_store_usage_memory", usage_memory_stage_start); if dsc.replicate_any() { + let replication_schedule_stage_start = put_stage_metrics_enabled.then(Instant::now); schedule_object_replication(obj_info.clone(), store, dsc).await; + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_replication_schedule", + replication_schedule_stage_start, + ); } + let dirty_usage_stage_start = put_stage_metrics_enabled.then(Instant::now); rustfs_scanner::record_dirty_usage_object_from_producer( &bucket, &key, rustfs_scanner::SegmentInvalidationProducerIdentity::PutObject, ); + rustfs_io_metrics::record_put_object_stage_duration_from( + "app_post_store_dirty_usage_record", + dirty_usage_stage_start, + ); rustfs_io_metrics::record_put_object_stage_duration_from("app_post_store_bookkeeping", post_store_stage_start); let capacity_update_stage_start = put_stage_metrics_enabled.then(Instant::now); @@ -2171,7 +2252,14 @@ impl DefaultObjectUsecase { Ok::<_, S3Error>(PutObjectCommitResult { obj_info, put_versioned }) } - }); + }; + #[cfg(feature = "diagnostic-read-version-caller")] + let put_commit_future = crate::app::storage_api::diagnostic::with_read_version_spawn_context( + crate::app::storage_api::diagnostic::ReadVersionEntrySource::AppPutOwner, + crate::app::storage_api::diagnostic::ReadVersionContextBoundary::AppPutOwner, + put_commit_future, + ); + let put_commit = spawn_traced_join(put_commit_future); let commit_stage_start = put_stage_metrics_enabled.then(Instant::now); let put_commit_result = if let Some(cancellation) = eager_put_commit_cancellation { EagerPutCommitOwner::new(put_commit, cancellation, EAGER_PUT_COMMIT_CANCELLATION_GRACE) @@ -2265,9 +2353,9 @@ mod tests { let usecase = DefaultObjectUsecase::with_context(Some(context)); for (index, stage) in [ - PutReadQuorumTestStage::DefaultSse, - PutReadQuorumTestStage::Prelookup, - PutReadQuorumTestStage::Commit, + PutStorageTestStage::DefaultSse, + PutStorageTestStage::Prelookup, + PutStorageTestStage::Commit, ] .into_iter() .enumerate() @@ -2285,8 +2373,11 @@ mod tests { if method == Method::POST { req.extensions.insert(PostObjectRequestMarker); } - let err = PUT_READ_QUORUM_TEST_STAGE - .scope(stage, usecase.execute_put_object(&FS::new(), req)) + let err = PUT_STORAGE_TEST_FAILURE + .scope( + (stage, StorageError::InsufficientReadQuorum("bucket".into(), "object".into())), + usecase.execute_put_object(&FS::new(), req), + ) .await .expect_err("read quorum failure must refuse the upload"); assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable, "stage {index}, method {method}"); @@ -2314,6 +2405,54 @@ mod tests { } } + #[tokio::test] + #[serial_test::serial] + async fn put_capacity_failure_returns_unavailable_without_publishing_object() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + + let (store, context) = real_store_test_context().await; + let bucket = format!("put-capacity-{}", Uuid::new_v4()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("capacity PUT bucket must be created"); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + for (index, failure) in [StorageError::DiskFull, StorageError::StorageFull].into_iter().enumerate() { + let key = format!("refused-{index}"); + let input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key(key.clone()) + .body(Some(StreamingBlob::from_bytes(Bytes::from_static(b"capacity probe")))) + .content_length(Some(14)) + .build() + .expect("capacity PUT input must build"); + let request = build_request(input, Method::PUT); + let error = PUT_STORAGE_TEST_FAILURE + .scope((PutStorageTestStage::Commit, failure), usecase.execute_put_object(&FS::new(), request)) + .await + .expect_err("capacity refusal must fail PUT"); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + assert!(matches!( + error.source().and_then(|source| source.downcast_ref::()), + Some(StorageError::DiskFull | StorageError::StorageFull) + )); + let response = error.to_http_response().expect("PUT error must serialize"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); + let body = http_body_util::BodyExt::collect(response.into_body()) + .await + .expect("PUT error XML must be readable") + .to_bytes(); + let xml = std::str::from_utf8(&body).expect("PUT error XML must be UTF-8"); + assert!(xml.contains("ServiceUnavailable")); + let lookup = store.get_object_info(&bucket, &key, &ObjectOptions::default()).await; + assert!( + lookup.as_ref().is_err_and(is_err_object_not_found), + "failed PUT must publish no object: {lookup:?}" + ); + } + } + #[test] fn failed_put_completion_normalizes_only_s3_errors() { let error = || S3Error::from(ApiError::from(StorageError::ErasureReadQuorum)); @@ -3437,6 +3576,25 @@ mod tests { assert!(should_use_zero_copy(2 * 1024 * 1024, &headers)); } + #[tokio::test] + async fn execute_put_object_rejects_post_object_etag_if_none_match_header() { + let input = PutObjectInput::builder() + .bucket("test-bucket".to_string()) + .key("test-key".to_string()) + .build() + .expect("POST-to-PUT input should build without conditional DTO fields"); + let mut req = build_request(input, Method::POST); + req.extensions.insert(PostObjectRequestMarker); + req.headers + .insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("\"different-etag\"")); + + let usecase = DefaultObjectUsecase::without_context(); + let err = Box::pin(usecase.execute_put_object(&FS::new(), req)) + .await + .expect_err("POST header-only ETag precondition must fail before storage access"); + assert_eq!(err.code(), &S3ErrorCode::NotImplemented); + } + #[tokio::test] async fn execute_put_object_rejects_post_object_sse_kms_from_input() { let input = PutObjectInput::builder() @@ -4019,6 +4177,62 @@ mod tests { assert!(!body_polled.load(Ordering::Acquire), "legacy quota rejection must not consume the body"); } + /// Regression (rustfs/backlog#2904, decision D3 in rustfs/backlog#2872): a + /// bucket admitted by the durable reservation ledger must accept a write + /// even when no scanner usage baseline exists anywhere, and the ledger + /// alone still refuses an over-limit write. Before the fix the checker's + /// `UsageUnavailable` was mapped to a retryable 503 for every PUT. + #[tokio::test] + #[serial_test::serial] + async fn durable_quota_put_succeeds_without_any_usage_baseline() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + + const LIMIT: u64 = 1 << 20; + let store = crate::app::gating_test_env::shared_gating_ecstore().await; + crate::app::runtime_sources::install_test_app_context(Arc::clone(&store)).await; + let bucket = format!("durable-no-usage-{}", Uuid::new_v4().simple()); + store + .make_bucket(&bucket, &MakeBucketOptions::default()) + .await + .expect("create durable quota test bucket"); + // Deliberately no usage seed: neither the memory cache nor a persisted + // snapshot knows this bucket, so the scanner-side lookup fails. + let metadata_sys = DefaultObjectUsecase::from_global() + .bucket_metadata_sys() + .expect("test app context should expose bucket metadata"); + QuotaChecker::new(metadata_sys) + .set_quota_config(&bucket, BucketQuota::new(Some(LIMIT))) + .await + .expect("configure durable quota"); + + let payload = Bytes::from(vec![0x42; 1024]); + let input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key("fits".to_string()) + .body(Some(StreamingBlob::from(s3s::Body::from(payload.clone())))) + .content_length(Some(i64::try_from(payload.len()).expect("test payload length must fit i64"))) + .build() + .expect("PUT input should build"); + DefaultObjectUsecase::from_global() + .execute_put_object(&FS::new(), build_request(input, Method::PUT)) + .await + .expect("a ledger-admitted bucket must accept a write without scanner usage"); + + let too_large = Bytes::from(vec![0x42; (LIMIT + 1) as usize]); + let input = PutObjectInput::builder() + .bucket(bucket) + .key("too-large".to_string()) + .body(Some(StreamingBlob::from(s3s::Body::from(too_large.clone())))) + .content_length(Some(i64::try_from(too_large.len()).expect("test payload length must fit i64"))) + .build() + .expect("PUT input should build"); + let err = DefaultObjectUsecase::from_global() + .execute_put_object(&FS::new(), build_request(input, Method::PUT)) + .await + .expect_err("the ledger must still reject a write above the quota limit"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + } + #[tokio::test] #[serial_test::serial] async fn concurrent_puts_share_durable_bucket_quota_reservations() { @@ -4094,7 +4308,7 @@ mod tests { #[tokio::test] #[serial_test::serial] - async fn put_rejects_rotated_quota_capability_before_rename() { + async fn put_commits_after_quota_fleet_proof_loss_before_rename() { use crate::app::storage_api::test::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause}; let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("rotated-proof-put-quota", 4096).await; @@ -4108,27 +4322,18 @@ mod tests { .await }); barrier.wait_until_paused().await; - assert!( - crate::storage::storage_api::ecstore_notification::rotate_cross_pool_fence_fleet_proof_for_test(), - "the gating environment must have a current fleet proof" - ); + let _without_proof = crate::storage::storage_api::ecstore_notification::without_cross_pool_fence_fleet_proof_for_test(); + assert!(crate::storage::storage_api::ecstore_notification::acquire_cross_pool_fence_fleet_proof().is_none()); barrier.release(); - let err = put - .await + put.await .expect("PUT task should not panic") - .expect_err("a replaced fleet proof must fence the authoritative rename"); - assert!(matches!( - err, - StorageError::NamespaceLockQuorumUnavailable { - mode: "quota_reservation", - .. - } - )); - store + .expect("fleet proof loss must not invalidate a live quota reservation and its disk fences"); + let stored = store .get_object_info(&bucket, "object", &ObjectOptions::default()) .await - .expect_err("proof rotation before rename must leave no committed object"); + .expect("the committed object must remain readable after fleet proof loss"); + assert_eq!(stored.size, 4096); } #[tokio::test] @@ -4411,6 +4616,83 @@ mod tests { assert!(is_err_object_not_found(&lookup_err), "{lookup_err}"); } + #[cfg(feature = "diagnostic-read-version-caller")] + #[test] + #[serial_test::serial] + fn diagnostic_s3_put_preserves_api_through_app_owner_and_tier_lookup() { + use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions}; + use metrics_util::debugging::{DebugValue, DebuggingRecorder}; + use rustfs_io_metrics::s3_http_metrics::{S3HttpRequestGuard, current_s3_http_operation_name}; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("test runtime"); + let (store, context) = runtime.block_on(Box::pin(real_store_test_context())); + let bucket = format!("put-context-{}", Uuid::new_v4()); + runtime + .block_on(Box::pin(store.make_bucket(&bucket, &MakeBucketOptions::default()))) + .expect("test bucket"); + let payload = Bytes::from_static(b"diagnostic PUT context"); + let input = PutObjectInput::builder() + .bucket(bucket.clone()) + .key("object.bin".to_string()) + .body(Some(StreamingBlob::from(s3s::Body::from(payload.clone())))) + .content_length(Some(i64::try_from(payload.len()).expect("payload length"))) + .build() + .expect("PUT input"); + let usecase = DefaultObjectUsecase::with_context(Some(context)); + let recorder = DebuggingRecorder::new(); + let snapshotter = recorder.snapshotter(); + rustfs_io_metrics::set_get_stage_metrics_enabled(true); + let mut request = S3HttpRequestGuard::new("PUT"); + metrics::with_local_recorder(&recorder, || { + request.in_scope(|| { + runtime.block_on(async { + Box::pin(usecase.execute_put_object(&FS::new(), build_request(input, Method::PUT))) + .await + .expect("S3 PUT must commit"); + assert_eq!(current_s3_http_operation_name(), Some("s3:PutObject")); + assert_eq!(crate::app::storage_api::diagnostic::current_read_version_entry_source(), None); + }) + }); + }); + let context_rows = snapshotter + .snapshot() + .into_vec() + .into_iter() + .filter(|(key, _, _, _)| key.key().name() == "rustfs_storage_diagnostic_context_transitions_total") + .map(|(key, _, _, value)| { + ( + key.key() + .labels() + .map(|l| (l.key().to_string(), l.value().to_string())) + .collect::>(), + value, + ) + }) + .collect::>(); + for boundary in ["app_put_owner", "put_commit_owner", "tier_free_version_source_lookup"] { + assert!( + context_rows.iter().any(|(labels, value)| { + labels.get("entry_source").map(String::as_str) == Some("app_put_owner") + && labels.get("boundary").map(String::as_str) == Some(boundary) + && labels.get("parent_api_operation").map(String::as_str) == Some("s3:PutObject") + && labels.get("child_api_operation").map(String::as_str) == Some("s3:PutObject") + && labels.get("api_context").map(String::as_str) == Some("preserved") + && matches!(value, DebugValue::Counter(count) if *count > 0) + }), + "actual S3 PUT boundary must preserve the parent API: {boundary}; {context_rows:?}" + ); + } + let info = runtime + .block_on(Box::pin(store.get_object_info(&bucket, "object.bin", &ObjectOptions::default()))) + .expect("committed object"); + assert_eq!(info.size, i64::try_from(payload.len()).expect("payload length")); + assert_eq!(current_s3_http_operation_name(), None); + assert_eq!(crate::app::storage_api::diagnostic::current_read_version_entry_source(), None); + } + #[tokio::test] #[serial_test::serial] async fn execute_put_object_still_writes_plaintext_without_bucket_encryption() { diff --git a/rustfs/src/app/object/shared.rs b/rustfs/src/app/object/shared.rs index 2ab083d1c..e4d7cc295 100644 --- a/rustfs/src/app/object/shared.rs +++ b/rustfs/src/app/object/shared.rs @@ -198,7 +198,7 @@ pub(super) fn request_body_is_aws_chunked_framed(headers: &HeaderMap) -> bool { /// Map a bucket-quota checker outcome onto the S3 admission result. /// -/// Hard is the only supported quota type, so a checker fault (bucket-config read, config parse, or usage lookup) must fail closed rather than admit the write: allowing it would silently bypass a configured hard quota. The no-quota happy path never reaches the error arm — `QuotaChecker::check_quota` returns `Ok(allowed)` via the zero-extra-I/O fast path when no quota is configured, so failing closed here cannot penalise buckets without a quota. A fault surfaces as a retryable `ServiceUnavailable` and is counted; the client-facing message stays generic so internal config/usage details are not leaked. +/// Hard is the only supported quota type, so a checker fault (bucket-config read, config parse, or usage lookup) must fail closed rather than admit the write: allowing it would silently bypass a configured hard quota. The no-quota happy path never reaches the error arm — `QuotaChecker::check_quota` returns `Ok(allowed)` via the zero-extra-I/O fast path when no quota is configured, so failing closed here cannot penalise buckets without a quota. A bucket admitted by the durable reservation ledger never reaches it for an unavailable usage lookup either: the checker reports that usage as absent (`current_usage = None`) and the ledger settles admission at commit (rustfs/backlog#2904); only the snapshot protocol still fails closed on it. A fault surfaces as a retryable `ServiceUnavailable` and is counted; the client-facing message stays generic so internal config/usage details are not leaked. pub(crate) fn map_quota_check_outcome(bucket: &str, outcome: Result) -> S3Result { match outcome { Ok(result) if !result.allowed => Err(S3Error::with_message( @@ -1107,6 +1107,28 @@ pub(crate) fn odm_check_source_preconditions(headers: &HeaderMap, head: &SourceH check_preconditions(headers, &info) } +/// S3 documents write-side `If-None-Match` only as `*` (create-only). +/// Reject unsupported ETag comparisons with 501 NotImplemented. +/// Treating it as an ETag comparison would let a write overwrite an existing key +/// while still satisfying a policy that requires `s3:if-none-match`. +pub(crate) fn reject_non_wildcard_write_if_none_match( + if_none_match: Option<&s3s::dto::ETagCondition>, + headers: &HeaderMap, +) -> S3Result<()> { + // POST parses conditional DTO fields from the form, so HTTP headers also need validation. + if matches!(if_none_match, Some(s3s::dto::ETagCondition::ETag(_))) + || headers.get_all(http::header::IF_NONE_MATCH).iter().any(|value| { + value.to_str().map_or(true, |value| { + let value = value.trim(); + !value.is_empty() && value != "*" + }) + }) + { + return Err(S3Error::new(S3ErrorCode::NotImplemented)); + } + Ok(()) +} + #[cfg(test)] mod tests { use super::*; @@ -1117,6 +1139,54 @@ mod tests { }; use std::sync::Arc; + #[test] + fn reject_non_wildcard_write_if_none_match_allows_only_wildcard() { + let headers = HeaderMap::new(); + assert!(reject_non_wildcard_write_if_none_match(None, &headers).is_ok()); + assert!(reject_non_wildcard_write_if_none_match(Some(&s3s::dto::ETagCondition::Any), &headers).is_ok()); + + let etag = s3s::dto::ETagCondition::ETag(ETag::Strong("9b2cf535f27731c974343645a3985328".to_owned())); + let err = + reject_non_wildcard_write_if_none_match(Some(&etag), &headers).expect_err("an ETag If-None-Match must be rejected"); + assert_eq!(*err.code(), S3ErrorCode::NotImplemented); + assert_eq!(err.status_code(), Some(http::StatusCode::NOT_IMPLEMENTED)); + assert_eq!( + err.message(), + Some("A header you provided implies functionality that is not implemented.") + ); + } + + #[test] + fn reject_non_wildcard_write_if_none_match_checks_header_without_dto() { + for value in ["*", " * ", "", " "] { + let mut headers = HeaderMap::new(); + headers.insert(http::header::IF_NONE_MATCH, http::HeaderValue::from_str(value).expect("valid header")); + assert!(reject_non_wildcard_write_if_none_match(None, &headers).is_ok(), "{value:?}"); + } + for value in ["\"different-etag\"", "W/\"weak-etag\"", "unquoted-etag"] { + let mut headers = HeaderMap::new(); + headers.insert(http::header::IF_NONE_MATCH, http::HeaderValue::from_str(value).expect("valid header")); + let err = reject_non_wildcard_write_if_none_match(None, &headers) + .expect_err("header-only ETag precondition must be rejected"); + assert_eq!(err.code(), &S3ErrorCode::NotImplemented); + } + } + + #[test] + fn reject_non_wildcard_write_if_none_match_rejects_conflicting_sources() { + for value in [ + http::HeaderValue::from_static("\"different-etag\""), + http::HeaderValue::from_bytes(b"\xff").expect("obs-text is a valid HTTP header"), + ] { + let mut headers = HeaderMap::new(); + headers.append(http::header::IF_NONE_MATCH, http::HeaderValue::from_static("*")); + headers.append(http::header::IF_NONE_MATCH, value); + let err = reject_non_wildcard_write_if_none_match(Some(&s3s::dto::ETagCondition::Any), &headers) + .expect_err("a form wildcard must not hide an unsupported raw HTTP precondition"); + assert_eq!(err.code(), &S3ErrorCode::NotImplemented); + } + } + #[test] fn s3_response_version_id_distinguishes_absent_null_and_uuid_versions() { let version = Uuid::parse_str("9341ae04-d4ce-468c-a4e1-6501d58cd6b7").unwrap(); @@ -1858,6 +1928,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; assert!(build_put_object_expiration_header(&event).is_none()); @@ -1873,6 +1944,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; // HTTP-date, not RFC 3339: the AWS SDKs parse this field with an @@ -1890,6 +1962,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; assert!(build_put_object_expiration_header(&event).is_none()); @@ -1901,6 +1974,7 @@ mod tests { noncurrent_days: 0, newer_noncurrent_versions: 0, storage_class: String::new(), + expiry_policy: None, }; assert!(build_put_object_expiration_header(&event).is_none()); @@ -2110,15 +2184,39 @@ mod tests { #[test] fn quota_admission_fails_closed_on_unknown_authoritative_usage() { + // Only the snapshot protocol still surfaces this: the checker never + // returns it for a bucket the durable reservation ledger admits. let err = map_quota_check_outcome( "bucket", Err(QuotaError::UsageUnavailable { bucket: "bucket".to_string(), }), ) - .expect_err("unknown authoritative usage must not admit a quota-controlled write"); + .expect_err("unknown authoritative usage must not admit a snapshot-protocol write"); assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable); } + + /// rustfs/backlog#2904: a durable-quota check whose scanner usage is + /// unknown admits the write at the handler and leaves the decision to the + /// ledger; nothing on the handler path treats the absent usage as zero. + #[test] + fn durable_quota_without_scanner_usage_admits_the_write() { + let unknown_usage = QuotaCheckResult { + allowed: true, + current_usage: None, + quota_limit: Some(2048), + operation_size: 512, + remaining: None, + uses_durable_reservations: true, + }; + let result = map_quota_check_outcome("bucket", Ok(unknown_usage)).expect("the ledger settles admission at commit"); + assert_eq!(result.current_usage, None); + + let mut opts = ObjectOptions::default(); + apply_quota_admission(&mut opts, &result).expect("durable admission carries no snapshot into the write"); + assert!(opts.quota_admission.is_none()); + ensure_object_size_within_quota(&result, 4096).expect("unknown usage cannot pre-reject an object"); + } } #[cfg(test)] diff --git a/rustfs/src/app/object_data_cache/planner.rs b/rustfs/src/app/object_data_cache/planner.rs index bead6a723..e0f5c00df 100644 --- a/rustfs/src/app/object_data_cache/planner.rs +++ b/rustfs/src/app/object_data_cache/planner.rs @@ -99,7 +99,7 @@ fn build_get_object_body_cache_plan_inner( return GetObjectBodyCachePlan::Skip; }; - // Nil version ids mean "no value" (see CLAUDE.md); map them to None so the + // Nil version ids mean "no value" (see AGENTS.md); map them to None so the // engine canonicalizes to the same "null" key as unversioned reads. let version_id = request .info diff --git a/rustfs/src/app/storage_api.rs b/rustfs/src/app/storage_api.rs index 8279118e6..4666a403f 100644 --- a/rustfs/src/app/storage_api.rs +++ b/rustfs/src/app/storage_api.rs @@ -24,6 +24,15 @@ pub(crate) fn EndpointServerPools( crate::storage::storage_api::EndpointServerPools::from(pools) } +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) mod diagnostic { + #[cfg(test)] + pub(crate) use crate::storage::storage_api::ecstore_diagnostic::current_read_version_entry_source; + pub(crate) use crate::storage::storage_api::ecstore_diagnostic::{ + ReadVersionContextBoundary, ReadVersionEntrySource, with_read_version_spawn_context, + }; +} + /// S3 wire types for app-layer modules, funneled here so new files stay off /// the direct s3s surface (s3s footprint ratchet, `scripts/check_s3s_footprint.sh`). pub(crate) mod s3 { @@ -1182,6 +1191,7 @@ pub(crate) mod bucket_usecase { pub(crate) mod object_usecase { pub(crate) use super::storage_contracts::BUCKET_LIFECYCLE_LOCK_OBJECT; + pub(crate) use super::storage_contracts::object::DeleteObjectCondition; pub(crate) use crate::storage::storage_api::ecstore_object::{WriteCommitGuard, WriteCompletion}; pub(crate) mod object_cache { diff --git a/rustfs/src/auth.rs b/rustfs/src/auth.rs index 68e17da96..36a163bb3 100644 --- a/rustfs/src/auth.rs +++ b/rustfs/src/auth.rs @@ -15,6 +15,7 @@ use crate::runtime_sources::{AppContext, ServerContextSlot, current_action_credentials, current_ready_iam_handle}; use http::HeaderMap; use http::Uri; +use http::header::{IF_MATCH, IF_NONE_MATCH}; use rustfs_credentials::Credentials; use rustfs_iam::error::Error as IamError; use rustfs_iam::sys::{ @@ -239,7 +240,6 @@ impl S3Auth for IAMAuth { event = EVENT_SECRET_KEY_LOOKUP_FAILED, component = LOG_COMPONENT_AUTH, subsystem = LOG_SUBSYSTEM_CREDENTIALS, - access_key = %MaskedAccessKey(access_key), reason = "no_such_user", "Secret key lookup rejected" ); @@ -249,7 +249,6 @@ impl S3Auth for IAMAuth { event = EVENT_SECRET_KEY_LOOKUP_FAILED, component = LOG_COMPONENT_AUTH, subsystem = LOG_SUBSYSTEM_CREDENTIALS, - access_key = %MaskedAccessKey(access_key), error = ?e, reason = "check_key_error", "Secret key lookup errored" @@ -262,7 +261,6 @@ impl S3Auth for IAMAuth { event = EVENT_SECRET_KEY_LOOKUP_FAILED, component = LOG_COMPONENT_AUTH, subsystem = LOG_SUBSYSTEM_CREDENTIALS, - access_key = %MaskedAccessKey(access_key), reason = "iam_not_initialized", "Secret key lookup skipped" ); @@ -730,10 +728,8 @@ pub fn get_condition_values_with_query_and_client_info( args.insert("LocationConstraint".to_owned(), vec![lc.to_string()]); } - let mut clone_header = header.clone(); - if let Some(v) = clone_header.get("x-amz-signature-age") { + if let Some(v) = header.get("x-amz-signature-age") { args.insert("signatureAge".to_string(), vec![v.to_str().unwrap_or("").to_string()]); - clone_header.remove("x-amz-signature-age"); } for obj_lock in &[ @@ -741,7 +737,7 @@ pub fn get_condition_values_with_query_and_client_info( AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, ] { - let values = clone_header + let values = header .get_all(*obj_lock) .iter() .map(|v| v.to_str().unwrap_or("").to_string()) @@ -749,7 +745,6 @@ pub fn get_condition_values_with_query_and_client_info( if !values.is_empty() { args.insert(obj_lock.trim_start_matches("x-amz-").to_string(), values); } - clone_header.remove(*obj_lock); } // S3 policy condition keys use "x-amz-grant-*" (policy key s3:x-amz-grant-* -> name() returns x-amz-grant-*) @@ -760,7 +755,7 @@ pub fn get_condition_values_with_query_and_client_info( "x-amz-grant-read-acp", "x-amz-grant-write-acp", ] { - let values = clone_header + let values = header .get_all(*grant_header) .iter() .map(|v| v.to_str().unwrap_or("").to_string()) @@ -768,7 +763,6 @@ pub fn get_condition_values_with_query_and_client_info( if !values.is_empty() { args.insert((*grant_header).to_string(), values); } - clone_header.remove(*grant_header); } // Claims and group membership are part of the verified identity, so they are @@ -776,7 +770,15 @@ pub fn get_condition_values_with_query_and_client_info( if let Some(claims) = &cred.claims { for (k, v) in claims { if let Some(v_str) = v.as_str() { - args.insert(k.trim_start_matches("ldap").to_lowercase(), vec![v_str.to_string()]); + let key = k.trim_start_matches("ldap"); + // Request conditions cannot originate from identity claims, even without a header. + if key.eq_ignore_ascii_case(OBJECT_CREATION_OPERATION_CONDITION) + || key.eq_ignore_ascii_case(IF_MATCH.as_str()) + || key.eq_ignore_ascii_case(IF_NONE_MATCH.as_str()) + { + continue; + } + args.insert(key.to_lowercase(), vec![v_str.to_string()]); } } @@ -804,9 +806,27 @@ pub fn get_condition_values_with_query_and_client_info( // both the ones already populated above and the well-known identity keys that are // absent for this credential, since an absent key is exactly what a spoofed header // would fill in. - for key in clone_header.keys() { - if key.as_str().eq_ignore_ascii_case("x-amz-tagging") { - continue; + for key in header.keys() { + match key.as_str() { + // These headers were consumed under their policy names above, or by tagging authorization. + "x-amz-signature-age" + | AMZ_OBJECT_LOCK_MODE_LOWER + | AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER + | AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER + | "x-amz-tagging" => continue, + "if-match" | "if-none-match" => { + // A blank header must not satisfy `Null: false` while storage ignores it. + if let Some(value) = header + .get(key) + .and_then(|value| value.to_str().ok()) + .map(str::trim) + .filter(|value| !value.is_empty()) + { + args.insert(key.as_str().to_string(), vec![value.to_string()]); + } + continue; + } + _ => {} } if is_reserved_condition_key(key.as_str(), &args) { continue; @@ -824,6 +844,9 @@ pub fn get_condition_values_with_query_and_client_info( args } +/// The S3 operation, rather than identity claims or request input, determines this value. +pub(crate) const OBJECT_CREATION_OPERATION_CONDITION: &str = "ObjectCreationOperation"; + /// Condition name for `s3:object-lock-remaining-retention-days`. Storage access /// derives it from the requested retain-until date, so no header may set it. pub(crate) const OBJECT_LOCK_REMAINING_RETENTION_DAYS_CONDITION: &str = "object-lock-remaining-retention-days"; @@ -1561,6 +1584,58 @@ mod tests { assert!(error.message().unwrap_or("").contains("Your account is not signed up")); } + #[tokio::test] + async fn credential_boundary_rejected_access_key_logs_omit_key_fragments() { + use rustfs_iam::{cache::Cache, manager::IamCache}; + use tracing::instrument::WithSubscriber; + + let (_temp_dir, _disk_paths, store) = crate::app::gating_test_env::isolated_multi_pool_ecstore().await; + let (send_chan, _receiver) = tokio::sync::mpsc::channel(1); + let cache = Arc::new(IamCache { + cache: Cache::default(), + api: ObjectStore::new(store.clone()), + state: Default::default(), + loading: Default::default(), + roles: HashMap::new(), + send_chan, + last_timestamp: Default::default(), + sync_failures: Default::default(), + sync_successes: Default::default(), + last_sync_duration_millis: Default::default(), + }); + let slot = ServerContextSlot::new(); + let context = AppContext::new( + store, + Arc::new(ContextIam { + handle: Arc::new(IamSys::new(cache)), + }), + Arc::new(TestKms), + ); + assert!(slot.install(Arc::new(context))); + for (slot, reason) in [(slot, "no_such_user"), (ServerContextSlot::new(), "iam_not_initialized")] { + let auth = IAMAuth::with_server_context("bootstrap-key", SecretKey::from("bootstrap-secret"), slot); + let log = tempfile::NamedTempFile::new().expect("capture file"); + let writer = log.reopen().expect("capture writer"); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::WARN) + .with_ansi(false) + .without_time() + .with_writer(move || writer.try_clone().expect("clone capture writer")) + .finish(); + let error = auth + .get_secret_key("LEAK-private-access-key-TAIL") + .with_subscriber(subscriber) + .await + .expect_err("unknown credentials must be rejected"); + assert_eq!(error.code(), &S3ErrorCode::InvalidAccessKeyId); + let captured = std::fs::read_to_string(log.path()).expect("read captured logs"); + assert!(captured.contains(reason), "capture the expected rejection: {captured}"); + for forbidden in ["LEAK", "TAIL", "private-access-key", "access_key="] { + assert!(!captured.contains(forbidden), "rejected access-key material must not be logged"); + } + } + } + #[test] fn test_iam_lookup_error_maps_to_internal_error() { let result = iam_lookup_error_to_s3_error(&IamError::Io(std::io::Error::other("load user failed"))); @@ -2039,6 +2114,131 @@ mod tests { ); } + #[test] + fn test_get_condition_values_with_conditional_write_headers() { + let cred = create_test_credentials(); + let mut headers = HeaderMap::new(); + headers.insert("If-None-Match", HeaderValue::from_static(" * ")); + headers.insert("If-Match", HeaderValue::from_static("\"9b2cf535f27731c974343645a3985328\"")); + + let conditions = get_condition_values(&headers, &cred, None, None, None); + + assert_eq!(conditions.get("if-none-match"), Some(&vec!["*".to_string()])); + assert_eq!( + conditions.get("if-match"), + Some(&vec!["\"9b2cf535f27731c974343645a3985328\"".to_string()]) + ); + } + + /// Storage ignores a blank precondition, and s3s rejects a non-UTF-8 one, so + /// neither may satisfy a `Null: false` check on `s3:if-none-match` / `s3:if-match`. + #[test] + fn test_get_condition_values_without_conditional_write_headers() { + let cred = create_test_credentials(); + let mut blank = HeaderMap::new(); + blank.insert("If-None-Match", HeaderValue::from_static("")); + blank.insert("If-Match", HeaderValue::from_static("")); + let mut non_utf8 = HeaderMap::new(); + non_utf8.insert( + "If-None-Match", + HeaderValue::from_bytes(b"\xff").expect("obs-text is a valid header value"), + ); + non_utf8.insert("If-Match", HeaderValue::from_bytes(b"\xff").expect("obs-text is a valid header value")); + + for headers in [HeaderMap::new(), blank, non_utf8] { + let conditions = get_condition_values(&headers, &cred, None, None, None); + + assert!(!conditions.contains_key("if-none-match"), "headers {headers:?}"); + assert!(!conditions.contains_key("if-match"), "headers {headers:?}"); + } + } + + #[test] + fn test_get_condition_values_conditional_write_headers_cannot_be_overridden_by_claims() { + let mut cred = create_service_account_credentials(); + cred.claims = Some(HashMap::from([ + ("if-none-match".to_string(), json!("*")), + ("if-match".to_string(), json!("\"claim-etag\"")), + ("IF-NONE-MATCH".to_string(), json!("*")), + ("ldapIF-MATCH".to_string(), json!("\"claim-etag\"")), + ])); + + for value in [None, Some(""), Some(" ")] { + let mut headers = HeaderMap::new(); + if let Some(value) = value { + headers.insert(IF_NONE_MATCH, HeaderValue::from_str(value).expect("valid header")); + headers.insert(IF_MATCH, HeaderValue::from_str(value).expect("valid header")); + } + let conditions = get_condition_values(&headers, &cred, None, None, None); + assert!( + !conditions.contains_key("if-none-match"), + "claim must not fabricate an absent precondition" + ); + assert!(!conditions.contains_key("if-match"), "claim must not fabricate an absent precondition"); + } + + let mut headers = HeaderMap::new(); + headers.insert(IF_NONE_MATCH, HeaderValue::from_static("*")); + headers.insert(IF_MATCH, HeaderValue::from_static("\"request-etag\"")); + let conditions = get_condition_values(&headers, &cred, None, None, None); + assert_eq!(conditions.get("if-none-match"), Some(&vec!["*".to_string()])); + assert_eq!(conditions.get("if-match"), Some(&vec!["\"request-etag\"".to_string()])); + } + + #[test] + fn test_get_condition_values_preserves_consumed_headers_and_multivalues() { + let cred = create_test_credentials(); + let mut headers = HeaderMap::new(); + headers.insert("x-amz-signature-age", HeaderValue::from_static("300")); + headers.insert("x-amz-object-lock-mode", HeaderValue::from_static("GOVERNANCE")); + headers.insert("x-amz-object-lock-legal-hold", HeaderValue::from_static("ON")); + headers.insert("x-amz-object-lock-retain-until-date", HeaderValue::from_static("2030-01-01T00:00:00Z")); + headers.insert("x-amz-tagging", HeaderValue::from_static("project=blue")); + for name in [ + "x-amz-grant-full-control", + "x-amz-grant-read", + "x-amz-grant-write", + "x-amz-grant-read-acp", + "x-amz-grant-write-acp", + "x-custom-header", + ] { + headers.append(name, HeaderValue::from_static("first")); + headers.append(name, HeaderValue::from_static("second")); + } + headers.append(IF_MATCH, HeaderValue::from_static(" \"first-etag\" ")); + headers.append(IF_MATCH, HeaderValue::from_static("\"second-etag\"")); + + let conditions = get_condition_values(&headers, &cred, None, None, None); + for (name, expected) in [ + ("signatureAge", "300"), + ("object-lock-mode", "GOVERNANCE"), + ("object-lock-legal-hold", "ON"), + ("object-lock-retain-until-date", "2030-01-01T00:00:00Z"), + ("if-match", "\"first-etag\""), + ] { + assert_eq!(conditions.get(name), Some(&vec![expected.to_string()]), "{name}"); + } + for name in [ + "x-amz-signature-age", + "x-amz-object-lock-mode", + "x-amz-object-lock-legal-hold", + "x-amz-object-lock-retain-until-date", + "x-amz-tagging", + ] { + assert!(!conditions.contains_key(name), "consumed header {name} must not be merged again"); + } + for name in [ + "x-amz-grant-full-control", + "x-amz-grant-read", + "x-amz-grant-write", + "x-amz-grant-read-acp", + "x-amz-grant-write-acp", + "x-custom-header", + ] { + assert_eq!(conditions.get(name), Some(&vec!["first".to_string(), "second".to_string()]), "{name}"); + } + } + #[test] fn test_get_condition_values_with_signature_age() { let cred = create_test_credentials(); diff --git a/rustfs/src/config/cli.rs b/rustfs/src/config/cli.rs index 867006bdf..106bca3cb 100644 --- a/rustfs/src/config/cli.rs +++ b/rustfs/src/config/cli.rs @@ -38,6 +38,7 @@ pub(super) const LONG_VERSION: &str = concat!( concat!("build time : ", build::BUILD_TIME, "\n"), concat!("build profile: ", build::BUILD_RUST_CHANNEL, "\n"), concat!("build os : ", build::BUILD_OS, "\n"), + concat!("build target : ", build::BUILD_TARGET, "\n"), concat!("rust version : ", build::RUST_VERSION, "\n"), concat!("rust channel : ", build::RUST_CHANNEL, "\n"), concat!("git branch : ", build::BRANCH, "\n"), diff --git a/rustfs/src/connect/diagnostics/perf_object.rs b/rustfs/src/connect/diagnostics/perf_object.rs index 6487c4d21..09d93a9ed 100644 --- a/rustfs/src/connect/diagnostics/perf_object.rs +++ b/rustfs/src/connect/diagnostics/perf_object.rs @@ -66,6 +66,9 @@ const MAX_OBJECT_RESPONSE_BYTES: usize = 16_384; const CLEANUP_RESERVE_MAX: Duration = Duration::from_millis(250); const CLEANUP_TIMEOUT: Duration = Duration::from_secs(5); const OUTPUT_MODE: u32 = 0o600; +const LOG_COMPONENT_CONNECT: &str = "connect"; +const LOG_SUBSYSTEM_PERFORMANCE_OBJECT: &str = "performance_object"; +const EVENT_OBJECT_CLEANUP_FAILED: &str = "connect_object_cleanup_failed"; static OBJECT_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); @@ -483,17 +486,61 @@ impl S3ObjectProbe { } async fn cleanup(&self, object_url: Url, bucket_url: Url, deadline: Instant) -> Result<(), ObjectProbeError> { - let object = self.send(Method::DELETE, object_url, Bytes::new(), deadline, None).await?; + let started = Instant::now(); + let object = self + .send(Method::DELETE, object_url, Bytes::new(), deadline, None) + .await + .inspect_err(|reason| Self::log_cleanup_failure("object_delete", None, *reason, started, deadline))?; if !object.status().is_success() && object.status() != StatusCode::NOT_FOUND { + Self::log_cleanup_failure( + "object_delete", + Some(object.status().as_u16()), + ObjectProbeError::CleanupFailed, + started, + deadline, + ); return Err(ObjectProbeError::CleanupFailed); } - let bucket = self.send(Method::DELETE, bucket_url, Bytes::new(), deadline, None).await?; + let bucket = self + .send(Method::DELETE, bucket_url, Bytes::new(), deadline, None) + .await + .inspect_err(|reason| Self::log_cleanup_failure("bucket_delete", None, *reason, started, deadline))?; if !bucket.status().is_success() { + Self::log_cleanup_failure( + "bucket_delete", + Some(bucket.status().as_u16()), + ObjectProbeError::CleanupFailed, + started, + deadline, + ); return Err(ObjectProbeError::CleanupFailed); } Ok(()) } + fn log_cleanup_failure( + stage: &'static str, + http_status: Option, + reason: ObjectProbeError, + started: Instant, + deadline: Instant, + ) { + // Only bounded local classifications are logged, never the signed URL, + // credentials, response body, or a transport error containing them. + tracing::error!( + event = EVENT_OBJECT_CLEANUP_FAILED, + component = LOG_COMPONENT_CONNECT, + subsystem = LOG_SUBSYSTEM_PERFORMANCE_OBJECT, + result = "failed", + stage, + reason = ?reason, + http_status, + elapsed_ms = started.elapsed().as_millis(), + remaining_ms = deadline.saturating_duration_since(Instant::now()).as_millis(), + "Object diagnostic cleanup failed" + ); + } + async fn send( &self, method: Method, diff --git a/rustfs/src/connect/diagnostics/trace_runtime.rs b/rustfs/src/connect/diagnostics/trace_runtime.rs index 04aa165ff..4c857071e 100644 --- a/rustfs/src/connect/diagnostics/trace_runtime.rs +++ b/rustfs/src/connect/diagnostics/trace_runtime.rs @@ -1911,6 +1911,14 @@ mod tests { .expect("telemetry subscriber"); } + async fn prepare_profile_test_provenance() { + // Cold hashing of the test executable is fixture setup, not IPC response time. + let provenance = super::super::job_delivery::executable_provenance() + .await + .expect("test executable provenance should be available"); + assert!(provenance.is_valid(), "profile capture requires valid executable provenance"); + } + #[tokio::test] #[serial] async fn local_runtime_captures_server_process_events() { @@ -2869,12 +2877,78 @@ mod tests { assert!(!state.path().join("identity").exists()); } + #[cfg(target_os = "linux")] + const NATIVE_THREADS_PREPARED_FILE: &str = "native-threads-provenance-ready"; + + #[cfg(target_os = "linux")] + const NATIVE_THREADS_FIXTURE_STARTED_FILE: &str = "native-threads-fixture-started"; + + #[cfg(target_os = "linux")] + fn spawn_native_threads_test_child(state: &std::path::Path, mode: &str) -> std::io::Result { + tokio::process::Command::new(std::env::current_exe()?) + .args([ + "connect::diagnostics::trace_runtime::tests::local_native_threads_service_child", + "--exact", + "--nocapture", + ]) + .env("RUSTFS_TEST_NATIVE_THREADS_STATE", state) + .env("RUSTFS_TEST_NATIVE_THREADS_FIXTURE_MODE", mode) + .stdin(std::process::Stdio::piped()) + .kill_on_drop(true) + .spawn() + } + + #[cfg(target_os = "linux")] + async fn wait_for_profile_child_marker( + child: &mut tokio::process::Child, + path: &std::path::Path, + budget: Duration, + ) -> std::io::Result<()> { + tokio::time::timeout(budget, async { + loop { + if let Some(status) = child.try_wait()? { + return Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + format!("profile fixture exited before readiness: {status}"), + )); + } + if path.try_exists()? { + return Ok(()); + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "profile fixture readiness timed out"))? + } + + #[cfg(target_os = "linux")] + async fn stop_profile_test_child( + child: &mut tokio::process::Child, + budget: Duration, + ) -> std::io::Result { + drop(child.stdin.take()); + match tokio::time::timeout(budget, child.wait()).await { + Ok(status) => status, + Err(_) => { + child.kill().await?; + child.wait().await + } + } + } + #[cfg(target_os = "linux")] #[test] fn local_native_threads_service_child() { let Some(state) = std::env::var_os("RUSTFS_TEST_NATIVE_THREADS_STATE") else { return; }; + let mode = std::env::var("RUSTFS_TEST_NATIVE_THREADS_FIXTURE_MODE").unwrap_or_else(|_| "capture".to_owned()); + match mode.as_str() { + "exit-before-ready" => return, + "capture" | "wait-before-ready" | "ignore-stop" => {} + _ => panic!("unknown native threads fixture mode"), + } let stop = CancellationToken::new(); let input_stop = stop.clone(); std::thread::spawn(move || { @@ -2882,6 +2956,14 @@ mod tests { let _ = std::io::stdin().read(&mut [0u8]); input_stop.cancel(); }); + if mode != "capture" { + std::fs::write(std::path::Path::new(&state).join(NATIVE_THREADS_FIXTURE_STARTED_FILE), b"started") + .expect("fixture started marker"); + while mode == "ignore-stop" || !stop.is_cancelled() { + std::thread::park_timeout(Duration::from_millis(10)); + } + return; + } let sleepers: Vec<_> = (0..16) .map(|_| { let stop = stop.clone(); @@ -2893,6 +2975,9 @@ mod tests { }) .collect(); tokio::runtime::Runtime::new().unwrap().block_on(async { + prepare_profile_test_provenance().await; + std::fs::write(std::path::Path::new(&state).join(NATIVE_THREADS_PREPARED_FILE), b"ready") + .expect("profile provenance ready marker"); let runtime = spawn_local_trace_capture_runtime(std::path::Path::new(&state), &stop).unwrap(); stop.cancelled().await; runtime.shutdown().await; @@ -2909,31 +2994,26 @@ mod tests { use std::io::Read as _; let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); - let (request, key) = runtime_request(state.path()); - let mut child = std::process::Command::new(std::env::current_exe().unwrap()) - .args([ - "connect::diagnostics::trace_runtime::tests::local_native_threads_service_child", - "--exact", - "--nocapture", - ]) - .env("RUSTFS_TEST_NATIVE_THREADS_STATE", state.path()) - .stdin(std::process::Stdio::piped()) - .spawn() - .unwrap(); - let ready = tokio::time::timeout(Duration::from_secs(5), async { - while !state.path().join(super::SOCKET_FILE).exists() { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) + let mut child = spawn_native_threads_test_child(state.path(), "capture").unwrap(); + let capture = async { + // This setup budget does not extend the socket or production request deadlines. + wait_for_profile_child_marker(&mut child, &state.path().join(NATIVE_THREADS_PREPARED_FILE), Duration::from_secs(120)) + .await + .map_err(LocalTraceCaptureError::Io)?; + wait_for_profile_child_marker(&mut child, &state.path().join(super::SOCKET_FILE), Duration::from_secs(5)) + .await + .map_err(LocalTraceCaptureError::Io)?; + let (request, key) = runtime_request(state.path()); + let export = + super::request_local_native_threads_profile(state.path(), request.clone(), &CancellationToken::new()).await?; + Ok::<_, LocalTraceCaptureError>((export, request, key)) + } .await; - let export = if ready.is_ok() { - super::request_local_native_threads_profile(state.path(), request.clone(), &CancellationToken::new()).await - } else { - Err(LocalTraceCaptureError::Protocol) - }; - drop(child.stdin.take()); - assert!(child.wait().unwrap().success()); - let export = export.unwrap(); + let status = stop_profile_test_child(&mut child, Duration::from_secs(10)) + .await + .expect("native threads fixture must be reaped"); + assert!(status.success(), "fixture exit: {status}; capture error: {:?}", capture.as_ref().err()); + let (export, request, key) = capture.unwrap(); let mut zip = zip::ZipArchive::new(std::io::Cursor::new(&export.archive_bytes)).unwrap(); let result: serde_json::Value = serde_json::from_reader(zip.by_name("result.json").unwrap()).unwrap(); assert_eq!(result["toolId"], "profile.threads"); @@ -2955,6 +3035,63 @@ mod tests { assert!(!state.path().join("identity").exists()); } + #[cfg(target_os = "linux")] + #[tokio::test] + async fn local_native_threads_preparation_detects_early_child_exit() { + let state = tempfile::tempdir().unwrap(); + let mut child = spawn_native_threads_test_child(state.path(), "exit-before-ready").unwrap(); + let ready = + wait_for_profile_child_marker(&mut child, &state.path().join(NATIVE_THREADS_PREPARED_FILE), Duration::from_secs(5)) + .await; + let status = stop_profile_test_child(&mut child, Duration::from_secs(1)).await.unwrap(); + assert_eq!(ready.unwrap_err().kind(), std::io::ErrorKind::UnexpectedEof); + assert!(status.success()); + assert!(child.try_wait().unwrap().is_some()); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn local_native_threads_preparation_timeout_reaps_child() { + let state = tempfile::tempdir().unwrap(); + let mut child = spawn_native_threads_test_child(state.path(), "wait-before-ready").unwrap(); + let started = wait_for_profile_child_marker( + &mut child, + &state.path().join(NATIVE_THREADS_FIXTURE_STARTED_FILE), + Duration::from_secs(5), + ) + .await; + let ready = wait_for_profile_child_marker( + &mut child, + &state.path().join(NATIVE_THREADS_PREPARED_FILE), + Duration::from_millis(20), + ) + .await; + let status = stop_profile_test_child(&mut child, Duration::from_secs(1)).await.unwrap(); + started.unwrap(); + assert_eq!(ready.unwrap_err().kind(), std::io::ErrorKind::TimedOut); + assert!(status.success()); + assert!(child.try_wait().unwrap().is_some()); + } + + #[cfg(target_os = "linux")] + #[tokio::test] + async fn local_native_threads_cleanup_kills_and_reaps_unresponsive_child() { + use std::os::unix::process::ExitStatusExt as _; + + let state = tempfile::tempdir().unwrap(); + let mut child = spawn_native_threads_test_child(state.path(), "ignore-stop").unwrap(); + let started = wait_for_profile_child_marker( + &mut child, + &state.path().join(NATIVE_THREADS_FIXTURE_STARTED_FILE), + Duration::from_secs(5), + ) + .await; + let status = stop_profile_test_child(&mut child, Duration::from_millis(20)).await.unwrap(); + started.unwrap(); + assert_eq!(status.signal(), Some(9)); + assert!(child.try_wait().unwrap().is_some()); + } + #[tokio::test] async fn local_native_threads_rejects_consent_scope_and_missing_identity() { let state = tempfile::tempdir().unwrap(); @@ -2996,14 +3133,14 @@ mod tests { use std::io::Read as _; let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + prepare_profile_test_provenance().await; let (mut request, key) = runtime_request(state.path()); request.capability = super::super::profile_cpu::MEMORY_PROFILE_CAPABILITY.to_owned(); let runtime = spawn_local_trace_capture_runtime(state.path(), &CancellationToken::new()).unwrap(); - let export = super::request_local_runtime_profile(state.path(), request.clone(), &CancellationToken::new()) - .await - .unwrap(); - assert_eq!(export.tool, super::ProfileTool::Memory); + let export = super::request_local_runtime_profile(state.path(), request.clone(), &CancellationToken::new()).await; runtime.shutdown().await; + let export = export.unwrap(); + assert_eq!(export.tool, super::ProfileTool::Memory); let mut zip = zip::ZipArchive::new(std::io::Cursor::new(&export.archive_bytes)).unwrap(); let result: serde_json::Value = serde_json::from_reader(zip.by_name("result.json").unwrap()).unwrap(); assert_eq!(result["toolId"], "profile.memory"); @@ -3029,6 +3166,7 @@ mod tests { use std::io::Read as _; let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); + prepare_profile_test_provenance().await; let (mut request, _key) = runtime_request(state.path()); request.capability = super::super::profile_cpu::CPU_PROFILE_CAPABILITY.to_owned(); request.duration_millis = 1_000; @@ -3090,6 +3228,7 @@ mod tests { async fn local_runtime_profile_listener_checks_real_peer_credentials() { use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + prepare_profile_test_provenance().await; for same_owner in [false, true] { let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); @@ -3212,10 +3351,7 @@ mod tests { async fn local_runtime_profile_cancellation_waits_for_lease_release() { let state = tempfile::tempdir().unwrap(); std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap(); - let provenance = super::super::job_delivery::executable_provenance() - .await - .expect("test executable provenance should be available"); - assert!(provenance.is_valid(), "profile capture requires valid executable provenance"); + prepare_profile_test_provenance().await; let (mut request, _) = runtime_request(state.path()); request.duration_millis = 5_000; let server_state_root = state.path().to_path_buf(); diff --git a/rustfs/src/connect/report_bundle.rs b/rustfs/src/connect/report_bundle.rs index 890629ac2..454626261 100644 --- a/rustfs/src/connect/report_bundle.rs +++ b/rustfs/src/connect/report_bundle.rs @@ -76,7 +76,11 @@ pub(crate) fn upload_source( Err(_) => return original_source(path, generated_bundle_uid), }; let names = (0..archive.len()) - .map(|index| archive.by_index(index).map(|entry| entry.name().to_owned())) + .map(|index| { + archive + .by_index(index) + .and_then(|entry| entry.name().map(|name| name.into_owned())) + }) .collect::, _>>()?; if names diff --git a/rustfs/src/error.rs b/rustfs/src/error.rs index 68fe9a365..56f488ff1 100644 --- a/rustfs/src/error.rs +++ b/rustfs/src/error.rs @@ -23,6 +23,13 @@ const MAX_VERSIONS_EXCEEDED_CODE: &str = "MaxVersionsExceeded"; const MAX_VERSIONS_EXCEEDED_MESSAGE: &str = "You've exceeded the limit on the number of versions you can create on this object"; const SLOW_DOWN_READ_CODE: &str = "SlowDownRead"; const SLOW_DOWN_READ_MESSAGE: &str = "Resource requested is unreadable, please reduce your request rate"; +/// Public message for `StorageError::UnsupportedDisk` on a request path. The +/// only request-time producer is the delete-transaction protocol when a +/// participant answers Unimplemented (a peer still on a release without the +/// recovery RPC, or a protocol-version mismatch); the variant's Display text +/// ("drive does not support O_DIRECT") describes the startup drive probe and +/// would send an operator mid-upgrade after the wrong cause. +const UNSUPPORTED_PARTICIPANT_MESSAGE: &str = "A storage participant does not support this operation's protocol version. If the cluster is being upgraded, finish upgrading every node and retry."; /// S3 error code for a request that names a KMS key the KMS does not hold. pub const KMS_KEY_NOT_FOUND_ERROR_CODE: &str = "KMS.NotFoundException"; @@ -625,7 +632,21 @@ fn error_chain_has_hyper_body_eof(err: &(dyn std::error::Error + 'static)) -> bo impl From for S3Error { fn from(err: ApiError) -> Self { let status = custom_error_status(&err.code); + // A typed lock acquisition timeout is transient contention. Keep its + // existing S3 code while advising a minimum delay before another attempt. + let lock_timeout = err.code == S3ErrorCode::ServiceUnavailable + && err.source.as_deref().is_some_and(|source| { + error_chain_find(source, |error| { + matches!(error.downcast_ref::(), Some(LockError::Timeout { .. })).then_some(()) + }) + .is_some() + }); let mut s3e = S3Error::with_message(err.code, err.message); + if lock_timeout { + let mut headers = http::HeaderMap::new(); + headers.insert(http::header::RETRY_AFTER, http::HeaderValue::from_static("1")); + s3e.set_headers(headers); + } if let Some(status) = status { s3e.set_status_code(status); } @@ -636,6 +657,26 @@ impl From for S3Error { } } +impl From for ApiError { + fn from(err: crate::table_catalog::TableCatalogStoreError) -> Self { + if let crate::table_catalog::TableCatalogStoreError::LockUnavailable { source, .. } = err { + // Use the storage boundary once so typed lock failures share its + // bounded metrics, public message, and retry advice. + return Self::from(StorageError::Lock(*source)); + } + let code = if err.is_unavailable() { + S3ErrorCode::ServiceUnavailable + } else { + S3ErrorCode::InternalError + }; + Self { + message: Self::error_code_to_message(&code), + code, + source: Some(Box::new(err)), + } + } +} + impl From for ApiError { fn from(err: StorageError) -> Self { record_s3_lock_failure(&err); @@ -751,7 +792,10 @@ impl From for ApiError { StorageError::ObjectNameInvalid(_, _) => S3ErrorCode::InvalidArgument, StorageError::BucketExists(_) => S3ErrorCode::BucketAlreadyOwnedByYou, StorageError::Io(io_error) if error_chain_has_unavailable_rpc(io_error) => S3ErrorCode::ServiceUnavailable, - StorageError::StorageFull => S3ErrorCode::ServiceUnavailable, + StorageError::DiskFull | StorageError::StorageFull => S3ErrorCode::ServiceUnavailable, + // Retryable on an upgraded node (or once the cluster upgrade + // completes); see UNSUPPORTED_PARTICIPANT_MESSAGE. + StorageError::UnsupportedDisk => S3ErrorCode::ServiceUnavailable, StorageError::SlowDown => S3ErrorCode::SlowDown, StorageError::FaultyDisk | StorageError::FaultyRemoteDisk @@ -788,6 +832,8 @@ impl From for ApiError { StorageError::EntityTooSmall(_, _, _) => S3ErrorCode::EntityTooSmall, StorageError::EntityTooLarge(_, _) => S3ErrorCode::EntityTooLarge, StorageError::PreconditionFailed => S3ErrorCode::PreconditionFailed, + StorageError::ConditionalDeleteConflict => S3ErrorCode::OperationAborted, + StorageError::GuardedDeleteUnsupported => S3ErrorCode::NotImplemented, StorageError::NotModified => S3ErrorCode::NotModified, StorageError::InvalidRangeSpec(_) => S3ErrorCode::InvalidRange, _ => S3ErrorCode::InternalError, @@ -795,6 +841,8 @@ impl From for ApiError { let message = if matches!(&err, StorageError::QuotaExceeded { .. }) { err.to_string() + } else if matches!(&err, StorageError::UnsupportedDisk) { + UNSUPPORTED_PARTICIPANT_MESSAGE.to_string() } else if matches!(&err, StorageError::MaxVersionsExceeded) || (code == S3ErrorCode::InternalError && matches!(&err, StorageError::Io(_))) { @@ -954,6 +1002,137 @@ mod tests { use s3s::{S3Error, S3ErrorCode}; use std::io::{Error as IoError, ErrorKind}; + #[tokio::test] + async fn lock_timeout_response_advises_retry_without_exposing_resource() { + use http_body_util::BodyExt as _; + + let storage_error = + StorageError::Lock(LockError::timeout("private-bucket/private-key@latest", std::time::Duration::from_secs(5))); + let error = S3Error::from(ApiError::from(storage_error)); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + let source = error.source().expect("retain the typed storage cause"); + assert!(source.downcast_ref::().is_some()); + assert!(error_chain_has_type::(source)); + let response = error.to_http_response().expect("serialize lock timeout response"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[http::header::RETRY_AFTER], "1"); + let body = response + .into_body() + .collect() + .await + .expect("read timeout error XML") + .to_bytes(); + let body = String::from_utf8(body.to_vec()).expect("S3 XML is UTF-8"); + assert!(body.contains("ServiceUnavailable")); + assert!(!body.contains("private-bucket")); + assert!(!body.contains("private-key")); + } + + #[tokio::test] + async fn unsupported_participant_is_retryable_with_an_actionable_message() { + use http_body_util::BodyExt as _; + + let error = S3Error::from(ApiError::from(StorageError::UnsupportedDisk)); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(error.message(), Some(UNSUPPORTED_PARTICIPANT_MESSAGE)); + assert!(matches!( + error.source().and_then(|source| source.downcast_ref::()), + Some(StorageError::UnsupportedDisk) + )); + + let response = error + .to_http_response() + .expect("unsupported participant error must serialize"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert!( + !response.headers().contains_key(http::header::RETRY_AFTER), + "retrying the same node does not help until it or its peers are upgraded" + ); + let body = response + .into_body() + .collect() + .await + .expect("error XML must be readable") + .to_bytes(); + let xml = std::str::from_utf8(&body).expect("error XML must be UTF-8"); + assert!(xml.contains("ServiceUnavailable")); + assert!( + !xml.contains("O_DIRECT"), + "the drive-probe Display text must not leak as the public message" + ); + } + + #[tokio::test] + async fn capacity_failures_share_service_unavailable_without_lock_retry_advice() { + use crate::storage_api::error::DiskError; + use http_body_util::BodyExt as _; + + for failure in [ + StorageError::DiskFull, + StorageError::StorageFull, + StorageError::from(DiskError::DiskFull), + ] { + let expected_code = failure.code(); + let error = S3Error::from(ApiError::from(failure)); + assert_eq!(error.code(), &S3ErrorCode::ServiceUnavailable); + assert_eq!(error.message(), Some("The service is unavailable. Please retry.")); + let source = error + .source() + .and_then(|source| source.downcast_ref::()) + .expect("retain the original capacity failure"); + assert_eq!(source.code(), expected_code); + let response = error.to_http_response().expect("serialize capacity refusal"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); + let body = response.into_body().collect().await.expect("read S3 error XML").to_bytes(); + let xml = std::str::from_utf8(&body).expect("S3 XML is UTF-8"); + assert!(xml.contains("ServiceUnavailable")); + assert!(!xml.contains("InternalError")); + } + } + + #[test] + fn lock_timeout_response_preserves_wrapped_source_and_requires_503() { + for code in [S3ErrorCode::ServiceUnavailable, S3ErrorCode::RequestTimeout] { + let should_advise_retry = code == S3ErrorCode::ServiceUnavailable; + let error = S3Error::from(ApiError { + message: ApiError::error_code_to_message(&code), + code: code.clone(), + source: Some(Box::new(IoError::other(LockError::timeout( + "bucket/object@latest", + std::time::Duration::from_secs(5), + )))), + }); + assert_eq!(error.code(), &code); + let source = error.source().expect("preserve wrapped lock timeout"); + assert!(error_chain_has_type::(source)); + let response = error.to_http_response().expect("serialize wrapped timeout"); + assert_eq!(response.headers().contains_key(http::header::RETRY_AFTER), should_advise_retry); + } + } + + #[test] + fn unrelated_unavailable_responses_do_not_inherit_lock_retry_advice() { + for failure in [ + StorageError::Timeout, + StorageError::DiskNotFound, + StorageError::FileCorrupt, + StorageError::NamespaceLockQuorumUnavailable { + mode: "read", + bucket: "bucket".to_owned(), + object: "object".to_owned(), + required: 2, + achieved: 1, + }, + StorageError::Io(IoError::other("Lock acquisition timeout for bucket/object after 5s")), + StorageError::SlowDown, + ] { + let error = S3Error::from(ApiError::from(failure)); + let response = error.to_http_response().expect("serialize unrelated storage failure"); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); + } + } + #[test] fn lock_503_mapping_records_operation_and_resource_class_without_resource_name() { let recorder = DebuggingRecorder::new(); @@ -987,6 +1166,42 @@ mod tests { assert!(!labels.values().any(|value| value.contains("durable-dirty-producer-replay"))); } + #[test] + fn catalog_lock_failure_uses_canonical_metrics_and_retry_advice_once() { + let recorder = DebuggingRecorder::new(); + let error = with_local_recorder(&recorder, || { + let mut request = S3HttpRequestGuard::new("PUT"); + request.in_scope(|| { + rustfs_io_metrics::record_s3_op(S3Operation::PutObject); + S3Error::from(ApiError::from(crate::table_catalog::TableCatalogStoreError::LockUnavailable { + message: "private catalog operation".to_owned(), + source: Box::new(LockError::timeout( + "private-bucket/.rustfs-table/warehouses/default/publication.lock@latest", + std::time::Duration::from_secs(5), + )), + })) + }) + }); + let response = error.to_http_response().expect("serialize catalog lock timeout"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[http::header::RETRY_AFTER], "1"); + let metrics = recorder.snapshotter().snapshot().into_vec(); + let (key, _, _, value) = metrics + .iter() + .find(|(key, _, _, _)| key.key().name() == "rustfs_s3_lock_failures_total") + .expect("catalog lock failures should use the canonical counter"); + assert!(matches!(value, DebugValue::Counter(1))); + let labels = key + .key() + .labels() + .map(|label| (label.key(), label.value())) + .collect::>(); + assert_eq!(labels["op"], S3Operation::PutObject.as_str()); + assert_eq!(labels["reason"], "timeout"); + assert_eq!(labels["resource_class"], "object_namespace"); + assert!(!labels.values().any(|value| value.contains("private"))); + } + #[test] fn multipart_lock_resources_have_a_distinct_bounded_class() { assert_eq!(lock_resource_class(Some(".rustfs.sys/multipart/upload-1")), "multipart_metadata"); @@ -1846,6 +2061,23 @@ mod tests { } } + #[test] + fn conditional_delete_storage_errors_preserve_api_contract() { + for (source, expected) in [ + (StorageError::ConditionalDeleteConflict, S3ErrorCode::OperationAborted), + (StorageError::GuardedDeleteUnsupported, S3ErrorCode::NotImplemented), + ] { + let error = ApiError::from(source.clone()); + assert_eq!(error.code, expected); + assert!( + error + .source + .as_ref() + .is_some_and(|value| value.downcast_ref::() == Some(&source)) + ); + } + } + #[test] fn test_unknown_authoritative_quota_usage_maps_to_retryable_error() { let api_error = ApiError::from(QuotaError::UsageUnavailable { @@ -1913,7 +2145,9 @@ mod tests { S3ErrorCode::InvalidArgument, ), (StorageError::BucketExists("test".into()), S3ErrorCode::BucketAlreadyOwnedByYou), + (StorageError::DiskFull, S3ErrorCode::ServiceUnavailable), (StorageError::StorageFull, S3ErrorCode::ServiceUnavailable), + (StorageError::UnsupportedDisk, S3ErrorCode::ServiceUnavailable), (StorageError::SlowDown, S3ErrorCode::SlowDown), (StorageError::FaultyDisk, S3ErrorCode::ServiceUnavailable), (StorageError::FaultyRemoteDisk, S3ErrorCode::ServiceUnavailable), diff --git a/rustfs/src/protocols/client.rs b/rustfs/src/protocols/client.rs index dfea6d36a..9b982356d 100644 --- a/rustfs/src/protocols/client.rs +++ b/rustfs/src/protocols/client.rs @@ -18,7 +18,7 @@ use crate::runtime_sources::current_object_store_handle; use crate::shared_types::RemoteAddr; #[cfg(feature = "webdav")] use crate::storage_api::protocols::client::capacity; -use crate::storage_api::protocols::client::{FS, ReqInfo, RequestContext, authorize_request}; +use crate::storage_api::protocols::client::{FS, MultipartStagingOperation, ReqInfo, RequestContext, authorize_request}; use http::{HeaderMap, Method}; use percent_encoding::{AsciiSet, CONTROLS, utf8_percent_encode}; use rustfs_credentials; @@ -286,6 +286,10 @@ impl ProtocolStorageClient { client_info.forwarded_proto = Some(if session.secure_transport { "https" } else { "http" }.to_owned()); request.extensions.insert(client_info); request.headers = session.request_headers.clone(); + // Preserve staging semantics before distinct operations collapse to PutObject. + if matches!(action, S3Action::CreateMultipartUpload | S3Action::UploadPart) { + request.extensions.insert(MultipartStagingOperation); + } let policy_action = if matches!(action, S3Action::HeadBucket) { S3Action::ListBucket.into() } else { diff --git a/rustfs/src/protocols/policy_test.rs b/rustfs/src/protocols/policy_test.rs index 7d4bdcb79..318e25436 100644 --- a/rustfs/src/protocols/policy_test.rs +++ b/rustfs/src/protocols/policy_test.rs @@ -561,3 +561,121 @@ fn protocol_policy_denials_and_request_conditions() { } }); } + +#[test] +#[serial_test::serial] +fn protocol_object_creation_operation_preserves_multipart_staging() { + crate::app::gating_test_env::run_large_stack_test("protocol-object-creation", || async { + let root = Credentials { + access_key: "creation-policy-root".into(), + secret_key: "creation-policy-root-secret".into(), + status: "on".into(), + ..Default::default() + }; + let (_temp, context, iam) = protocol_test_context(root).await; + let store = context.object_store(); + let slot = ServerContextSlot::new(); + assert!(slot.install(context)); + let backend = crate::init::protocol_storage_client_for_server(slot); + let bucket = "protocol-object-creation"; + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("create bucket"); + iam.create_user( + "creation-writer", + &rustfs_madmin::AddOrUpdateUserReq { + secret_key: "creation-writer-secret".into(), + policy: None, + status: rustfs_madmin::AccountStatus::Enabled, + }, + ) + .await + .expect("create protocol writer"); + let identity = iam + .check_key("creation-writer") + .await + .expect("load writer") + .0 + .expect("writer exists"); + let sessions: Vec<_> = [Protocol::Ftps, Protocol::Sftp] + .into_iter() + .map(|protocol| { + SessionContext::new( + ProtocolPrincipal::new(Arc::new(identity.clone())), + protocol, + "127.0.0.1".parse().expect("loopback"), + ) + }) + .collect(); + let resource = format!("arn:aws:s3:::{bucket}/*"); + let allow = json!({"Effect":"Allow", "Action":"s3:PutObject", "Resource": resource}); + iam.set_policy( + "creation-policy", + decode_policy(json!({"Version":"2012-10-17","Statement":[allow.clone()]})).expect("parse initial allow"), + ) + .await + .expect("save initial allow"); + iam.policy_db_set("creation-writer", UserType::Reg, false, "creation-policy") + .await + .expect("attach writer policy"); + for source in ["IAM", "bucket"] { + for denied_creation in [false, true] { + let deny = json!({"Effect":"Deny", "Action":"s3:PutObject", "Resource": resource, + "Condition":{"Bool":{"s3:ObjectCreationOperation":denied_creation.to_string()}}}); + let mut statements = vec![allow.clone()]; + let mut bucket_statements = Vec::new(); + if source == "IAM" { + statements.push(deny); + } else { + let mut deny = deny; + deny["Principal"] = json!("*"); + bucket_statements.push(deny); + } + iam.set_policy( + "creation-policy", + decode_policy(json!({"Version":"2012-10-17","Statement":statements})).expect("parse IAM policy"), + ) + .await + .expect("save IAM policy"); + store + .update_bucket_metadata_config( + bucket, + "policy.json", + serde_json::to_vec(&json!({"Version":"2012-10-17","Statement":bucket_statements})) + .expect("encode bucket policy"), + ) + .await + .expect("save bucket policy"); + for session in &sessions { + for (action, creates_object) in [ + (S3Action::CreateMultipartUpload, false), + (S3Action::UploadPart, false), + (S3Action::CompleteMultipartUpload, true), + (S3Action::PutObject, true), + (S3Action::CopyObject, true), + ] { + if session.protocol == Protocol::Ftps && action == S3Action::CopyObject { + continue; + } + let result = backend.authorize_operation(session, &action, bucket, Some("key")).await; + if creates_object == denied_creation { + assert!( + matches!(result, Err(AuthorizationError::AccessDenied)), + "{source} deny={denied_creation}, {:?}, {action:?}: {result:?}", + session.protocol + ); + } else { + result.unwrap_or_else(|error| { + panic!( + "{source} deny={denied_creation}, {:?}, {action:?} should allow: {error}", + session.protocol + ) + }); + } + } + } + } + } + }); +} diff --git a/rustfs/src/server/http.rs b/rustfs/src/server/http.rs index d5a90e6bc..3389fe736 100644 --- a/rustfs/src/server/http.rs +++ b/rustfs/src/server/http.rs @@ -2656,7 +2656,10 @@ fn check_auth(req: Request<()>) -> std::result::Result, Status> { .unwrap_or("unknown"); let peer_addr = req .extensions() - .get::() + .get::>() + .copied() + .flatten() + .or_else(|| req.extensions().get::().copied()) .map(|addr| addr.0.to_string()) .unwrap_or_else(|| "unknown".to_string()); let failure_reason = storage::tonic_rpc_auth_failure_reason(&e); @@ -2672,25 +2675,41 @@ fn check_auth(req: Request<()>) -> std::result::Result, Status> { INTERNODE_TRANSPORT_BACKEND_GRPC, failure_reason, ); - error!( - event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED, - component = LOG_COMPONENT_SERVER, - subsystem = LOG_SUBSYSTEM_HTTP, - failure_reason, - rpc_path, - rpc_service, - rpc_method, - expected_audience = %audience, - peer_addr = %peer_addr, - replay_scope_bootstrap_allowed = allow_replay_scope_bootstrap, - error = %e, - "RPC signature verification failed" - ); if failure_reason == "stale_boot_epoch" { // The signature is valid, but the peer restarted before this request. Reject it // before execution and let the client retry after its authenticated epoch refresh. + warn!( + event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_HTTP, + result = "retryable_rejection", + failure_reason, + rpc_path, + rpc_service, + rpc_method, + expected_audience = %audience, + peer_addr = %peer_addr, + replay_scope_bootstrap_allowed = allow_replay_scope_bootstrap, + error = %e, + "RPC boot epoch changed; request rejected for retry" + ); Status::unavailable("RPC boot epoch changed") } else { + error!( + event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_HTTP, + result = "rejected", + failure_reason, + rpc_path, + rpc_service, + rpc_method, + expected_audience = %audience, + peer_addr = %peer_addr, + replay_scope_bootstrap_allowed = allow_replay_scope_bootstrap, + error = %e, + "RPC signature verification failed" + ); Status::unauthenticated("No valid auth token") } })?; @@ -2818,6 +2837,163 @@ mod tests { use tokio::sync::{Notify, mpsc}; use tower::{Layer, Service, ServiceBuilder}; + type RateLimitTestCore = tower::util::BoxCloneService< + HttpRequest, + Response>>, + Box, + >; + type RateLimitTestStack = tower::util::BoxCloneService< + HttpRequest, + Response>>, + Box, + >; + + fn rate_limited_external_stack( + peer: SocketAddr, + proxy_layer: Option, + inner: RateLimitTestCore, + ) -> RateLimitTestStack { + use crate::server::rate_limit::RateLimitQuota; + + let readiness = Arc::new(GlobalReadiness::new()); + readiness.mark_stage(rustfs_common::SystemStage::FullReady); + RateLimitTestStack::new(external_service_stack!( + inner, + Some(RemoteAddr(peer)), + proxy_layer, + false, + Some(RateLimitLayer::new( + Some(RateLimitQuota { + requests_per_minute: 1, + burst: 1 + }), + None, + vec![] + )), + false, + readiness, + None::>, + HttpCompressionConfig::default(), + ServerContextSlot::new(), + false + )) + } + + #[tokio::test] + async fn external_stack_limits_socket_peers_without_trusted_proxies() { + use crate::server::hybrid::HybridBody; + use tower::ServiceExt; + + for peer in ["192.0.2.10:51000", "[2001:db8::10]:51000"] { + let peer: SocketAddr = peer.parse().expect("socket peer"); + for version in [Version::HTTP_11, Version::HTTP_2, Version::HTTP_3] { + let calls = Arc::new(AtomicUsize::new(0)); + let inner_calls = calls.clone(); + let inner = tower::service_fn(move |request: HttpRequest| { + assert_eq!( + request + .extensions() + .get::>() + .copied() + .flatten() + .map(|addr| addr.0), + Some(peer) + ); + inner_calls.fetch_add(1, Ordering::SeqCst); + async { + Ok::<_, Box>(Response::new( + HybridBody::>::Rest { + rest_body: s3s::Body::empty(), + }, + )) + } + }); + let mut service = rate_limited_external_stack(peer, None, RateLimitTestCore::new(inner)); + + for (forwarded, expected) in [ + ("203.0.113.1", StatusCode::OK), + ("203.0.113.2", StatusCode::TOO_MANY_REQUESTS), + ] { + let request = HttpRequest::builder() + .uri("/bucket/object") + .version(version) + .header("x-forwarded-for", forwarded) + .header("x-real-ip", forwarded) + .body(s3s::Body::empty()) + .expect("request"); + let response = service + .ready() + .await + .expect("service ready") + .call(request) + .await + .expect("response"); + assert_eq!(response.status(), expected, "peer={peer}, version={version:?}"); + if expected == StatusCode::TOO_MANY_REQUESTS { + assert!(response.headers().contains_key(http::header::RETRY_AFTER)); + let body = response.into_body().collect().await.expect("body").to_bytes(); + assert!(String::from_utf8_lossy(&body).contains("TooManyRequests")); + } + } + assert_eq!(calls.load(Ordering::SeqCst), 1, "throttled requests must not reach the inner service"); + } + } + } + + #[tokio::test] + async fn external_stack_limits_validated_proxy_clients_independently() { + use crate::server::hybrid::HybridBody; + use rustfs_trusted_proxies::{ + LegacyTrustedProxyLayer, TrustedProxy, TrustedProxyConfig, TrustedProxyLayer, ValidationMode, + }; + use tower::ServiceExt; + + let peer: SocketAddr = "192.0.2.100:51000".parse().expect("proxy peer"); + for version in [Version::HTTP_11, Version::HTTP_2, Version::HTTP_3] { + let proxy_config = + TrustedProxyConfig::new(vec![TrustedProxy::Single(peer.ip())], ValidationMode::HopByHop, false, 3, false, vec![]); + let proxy_layer = TrustedProxyLayer::legacy(LegacyTrustedProxyLayer::new(proxy_config, None, true)); + let calls = Arc::new(AtomicUsize::new(0)); + let inner_calls = calls.clone(); + let inner = tower::service_fn(move |request: HttpRequest| { + let info = request.extensions().get::().expect("validated client context"); + assert!(info.is_from_trusted_proxy); + assert_ne!(info.real_ip, peer.ip()); + inner_calls.fetch_add(1, Ordering::SeqCst); + async { + Ok::<_, Box>(Response::new( + HybridBody::>::Rest { + rest_body: s3s::Body::empty(), + }, + )) + } + }); + let mut service = rate_limited_external_stack(peer, Some(proxy_layer), RateLimitTestCore::new(inner)); + + for (client, expected) in [ + ("203.0.113.1", StatusCode::OK), + ("203.0.113.2", StatusCode::OK), + ("203.0.113.1", StatusCode::TOO_MANY_REQUESTS), + ] { + let request = HttpRequest::builder() + .uri("/bucket/object") + .version(version) + .header("x-forwarded-for", client) + .body(s3s::Body::empty()) + .expect("request"); + let response = service + .ready() + .await + .expect("service ready") + .call(request) + .await + .expect("response"); + assert_eq!(response.status(), expected, "client={client}, version={version:?}"); + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + } + } + #[test] fn peer_transport_close_kind_uses_io_error_sources() { #[derive(Debug)] @@ -4273,14 +4449,14 @@ mod tests { let _ = rustfs_credentials::set_global_rpc_secret("rpc-http-test-secret".to_string()); let previous_node_name = rustfs_common::get_global_local_node_name().await; let audience = "127.0.0.1:9000"; - let path = "/node_service.NodeService/ReadVersion"; + let path = "/node_service.NodeService/ReadAll"; rustfs_common::set_global_local_node_name(audience).await; let challenge = uuid::Uuid::new_v4(); let proof = storage::tonic_boot_epoch_response_headers(audience, challenge).expect("signed epoch proof"); let epoch = storage::verify_tonic_boot_epoch_response(audience, challenge, &proof).expect("authenticated epoch"); let stale_epoch = uuid::Uuid::from_u128(epoch.as_u128() ^ 1); let signed_headers = |audience: &str, epoch| { - let mut headers = storage::gen_tonic_signature_headers(audience, "node_service.NodeService", "ReadVersion", None) + let mut headers = storage::gen_tonic_signature_headers(audience, "node_service.NodeService", "ReadAll", None) .expect("method-bound signature"); let replay = storage::gen_tonic_replay_scope_headers( audience, @@ -4293,34 +4469,198 @@ mod tests { headers.extend(replay); headers }; - let request = |headers: HeaderMap| { + let request = |headers: HeaderMap, peer: Option| { let mut request = Request::new(()); request.metadata_mut().as_mut().extend(headers); request.extensions_mut().insert(RpcRequestTarget { uri: format!("http://{audience}{path}").parse().expect("RPC URI"), method: Method::POST, }); + // The connection stack stores this optional wrapper, not a bare RemoteAddr. + request.extensions_mut().insert(peer.map(RemoteAddr)); request }; - + let log_file = tempfile::NamedTempFile::new().expect("create RPC auth log capture"); + let subscriber = tracing_subscriber::fmt() + .json() + .with_ansi(false) + .with_max_level(Level::TRACE) + .with_writer(Arc::new(log_file.reopen().expect("open RPC auth log writer"))) + .finish(); + let peer: SocketAddr = "192.0.2.10:51000".parse().expect("test peer address"); let stale = signed_headers(audience, stale_epoch); - let error = check_auth(request(stale.clone())).expect_err("a stale epoch must still reject the request"); - assert_eq!(error.code(), tonic::Code::Unavailable); - assert_eq!(error.message(), "RPC boot epoch changed"); + tracing::subscriber::with_default(subscriber, || { + let error = check_auth(request(stale.clone(), Some(peer))).expect_err("a stale epoch must still reject the request"); + assert_eq!(error.code(), tonic::Code::Unavailable); + assert_eq!(error.message(), "RPC boot epoch changed"); + let mut forged = stale.clone(); + forged.insert("x-rustfs-rpc-signature-v3", HeaderValue::from_static("00")); + let error = check_auth(request(forged, Some(peer))).expect_err("a forged stale-epoch signature must remain terminal"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); + let error = check_auth(request(signed_headers("127.0.0.1:9001", stale_epoch), Some(peer))) + .expect_err("a stale epoch must not hide an audience mismatch"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); - let mut forged = stale; - forged.insert("x-rustfs-rpc-signature-v3", HeaderValue::from_static("00")); - let error = check_auth(request(forged)).expect_err("a forged stale-epoch signature must remain terminal"); - assert_eq!(error.code(), tonic::Code::Unauthenticated); - let error = check_auth(request(signed_headers("127.0.0.1:9001", stale_epoch))) - .expect_err("a stale epoch must not hide an audience mismatch"); - assert_eq!(error.code(), tonic::Code::Unauthenticated); - - let current = signed_headers(audience, epoch); - assert!(check_auth(request(current.clone())).is_ok(), "a fresh authenticated scope must succeed"); - let error = check_auth(request(current)).expect_err("a same-epoch nonce replay must remain terminal"); - assert_eq!(error.code(), tonic::Code::Unauthenticated); + let current = signed_headers(audience, epoch); + assert!( + check_auth(request(current.clone(), Some(peer))).is_ok(), + "a fresh authenticated scope must succeed" + ); + let error = check_auth(request(current, Some(peer))).expect_err("a same-epoch nonce replay must remain terminal"); + assert_eq!(error.code(), tonic::Code::Unauthenticated); + let error = + check_auth(request(stale.clone(), None)).expect_err("a missing peer address must not affect authentication"); + assert_eq!(error.code(), tonic::Code::Unavailable); + }); rustfs_common::set_global_local_node_name(&previous_node_name).await; + + let logs = std::fs::read_to_string(log_file.path()).expect("read RPC auth logs"); + let events: Vec = logs + .lines() + .map(|line| serde_json::from_str::(line).expect("valid JSON log event")) + .filter(|event| event["fields"]["event"] == EVENT_RPC_SIGNATURE_VERIFICATION_FAILED) + .collect(); + assert_eq!(events.len(), 5, "each authentication rejection must retain its diagnostic event"); + let stale_events: Vec<_> = events + .iter() + .filter(|event| event["fields"]["failure_reason"] == "stale_boot_epoch") + .collect(); + assert_eq!(stale_events.len(), 2); + for event in &stale_events { + assert_eq!(event["level"], "WARN"); + assert_eq!(event["fields"]["result"], "retryable_rejection"); + assert_eq!(event["fields"]["rpc_method"], "ReadAll"); + } + assert_eq!(stale_events[0]["fields"]["peer_addr"], peer.to_string()); + assert_eq!(stale_events[1]["fields"]["peer_addr"], "unknown"); + for event in events + .iter() + .filter(|event| event["fields"]["failure_reason"] != "stale_boot_epoch") + { + assert_eq!(event["level"], "ERROR"); + assert_eq!(event["fields"]["result"], "rejected"); + assert_eq!(event["fields"]["peer_addr"], peer.to_string()); + } + assert!(!logs.contains("rpc-http-test-secret"), "logs must not expose the RPC secret"); + for header in ["x-rustfs-signature", "x-rustfs-rpc-signature-v2", "x-rustfs-rpc-signature-v3"] { + if let Some(signature) = stale.get(header) { + assert!( + !logs.contains(signature.to_str().expect("ASCII signature")), + "logs must not expose {header}" + ); + } + } + } + + #[tokio::test] + #[serial_test::serial] + async fn rpc_auth_read_all_recovers_after_stale_boot_epoch() { + use storage::DiskAPI; + + temp_env::async_with_vars( + [ + ("RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES", Some("1")), + ("RUSTFS_INTERNODE_PREWARM", Some("false")), + ], + async { + let _ = rustfs_credentials::set_global_rpc_secret("rpc-http-test-secret".to_string()); + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind ReadAll epoch recovery peer"); + let addr = listener.local_addr().expect("ReadAll recovery address"); + let audience = addr.to_string(); + let previous_node_name = rustfs_common::get_global_local_node_name().await; + rustfs_common::set_global_local_node_name(&audience).await; + let challenge = uuid::Uuid::new_v4(); + let proof = storage::tonic_boot_epoch_response_headers(&audience, challenge).expect("current epoch proof"); + let epoch = storage::verify_tonic_boot_epoch_response(&audience, challenge, &proof).expect("current epoch"); + let stale_epoch = uuid::Uuid::from_u128(epoch.as_u128() ^ 1); + let admitted = Arc::new(AtomicUsize::new(0)); + let admitted_by_server = Arc::clone(&admitted); + let node = InterceptedService::new(NodeServiceServer::new(make_server()), move |request: Request<()>| { + let request = check_auth(request)?; + assert!( + request.extensions().get::>().copied().flatten().is_some(), + "production auth must receive the connection's optional peer address" + ); + admitted_by_server.fetch_add(1, Ordering::SeqCst); + Ok(request) + }); + let scopes = Arc::new(Mutex::new(Vec::new())); + let observed_scopes = Arc::clone(&scopes); + let service = ServiceBuilder::new() + .map_request(move |mut request: HttpRequest| { + assert_eq!(request.uri().path(), "/node_service.NodeService/ReadAll"); + let mut scopes = observed_scopes.lock().expect("captured replay scopes"); + if scopes.is_empty() { + assert!( + !request.headers().contains_key("x-rustfs-rpc-boot-epoch"), + "the fresh peer must start without a cached epoch" + ); + // Inject a correctly signed stale scope at the transport boundary. + // The retry must learn its scope from the real server response proof. + let replay = storage::gen_tonic_replay_scope_headers( + &audience, + request.uri().path(), + request.headers()["x-rustfs-timestamp"].to_str().expect("timestamp"), + request.headers()["x-rustfs-content-sha256"].to_str().expect("body digest"), + stale_epoch, + ) + .expect("signed stale scope"); + request.headers_mut().extend(replay); + } + let parse_uuid = |name: &str| { + uuid::Uuid::parse_str(request.headers()[name].to_str().expect("UUID header")) + .expect("valid replay scope UUID") + }; + scopes.push((parse_uuid("x-rustfs-rpc-boot-epoch"), parse_uuid("x-rustfs-rpc-replay-nonce"))); + request + }) + .service(RpcRequestPathService::new(Routes::new(node).prepare())); + let server = tokio::spawn(async move { + let (socket, peer) = listener.accept().await.expect("accept ReadAll recovery connection"); + let service = ServiceBuilder::new() + .layer(AddExtensionLayer::new(Some(RemoteAddr(peer)))) + .service(service); + ConnBuilder::new(TokioExecutor::new()) + .serve_connection(TokioIo::new(socket), TowerToHyperService::new(service)) + .await + .expect("serve ReadAll recovery connection"); + }); + let mut endpoint = + Endpoint::try_from(format!("http://{addr}/missing-read-all-disk").as_str()).expect("ReadAll remote endpoint"); + endpoint.is_local = false; + let disk = storage::new_disk( + &endpoint, + &storage::DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("production remote disk client"); + let result = tokio::time::timeout(Duration::from_secs(10), disk.read_all("bucket", "object")).await; + server.abort(); + let _ = server.await; + rustfs_common::set_global_local_node_name(&previous_node_name).await; + + let error = result + .expect("bounded ReadAll retry must complete") + .expect_err("missing disk must remain an error"); + assert_eq!( + error.to_string(), + "io error io error cannot find disk", + "the retry must reach the real ReadAll missing-disk handler" + ); + let scopes = scopes.lock().expect("captured replay scopes"); + assert_eq!(scopes.len(), 2, "stale epoch recovery must use one bounded retry"); + assert_eq!(scopes[0].0, stale_epoch); + assert_eq!(scopes[1].0, epoch, "the retry must use the authenticated response epoch"); + assert_ne!(scopes[0].1, scopes[1].1, "the retry must carry a fresh replay nonce"); + assert_eq!(admitted.load(Ordering::SeqCst), 1, "the stale request must not reach the handler"); + }, + ) + .await; } #[tokio::test] diff --git a/rustfs/src/server/hybrid.rs b/rustfs/src/server/hybrid.rs index c20d47215..170f7e6cf 100644 --- a/rustfs/src/server/hybrid.rs +++ b/rustfs/src/server/hybrid.rs @@ -19,6 +19,7 @@ use hyper::{Request, Response}; use pin_project_lite::pin_project; use std::pin::Pin; use std::task::{Context, Poll}; +use std::time::Instant; use tower::Service; type BoxError = Box; @@ -35,6 +36,15 @@ pub(crate) fn is_grpc_request(req: &Request) -> bool { ) } +fn rename_data_grpc_stage(path: &str) -> bool { + matches!( + path, + "/node_service.NodeService/RenameData" + | "/node_service.NodeService/RenameDataAtIncarnation" + | "/node_service.NodeService/RenameDataWithQuotaFence" + ) +} + /// The service that can serve both gRPC and REST HTTP Requests #[derive(Clone)] pub struct HybridService { @@ -71,8 +81,11 @@ where /// as a REST request fn call(&mut self, req: Request) -> Self::Future { if is_grpc_request(&req) { + let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(req.uri().path()); HybridFuture::Grpc { grpc_future: self.grpc.call(req), + observe_rename_data, + service_future_started: observe_rename_data.then(Instant::now), } } else { HybridFuture::Rest { @@ -93,7 +106,10 @@ pin_project! { }, Grpc { #[pin] - grpc_body: GrpcBody + grpc_body: GrpcBody, + first_frame_started: Option, + body_started: Option, + first_frame_recorded: bool, }, } } @@ -123,21 +139,44 @@ where fn is_end_stream(&self) -> bool { match self { Self::Rest { rest_body } => rest_body.is_end_stream(), - Self::Grpc { grpc_body } => grpc_body.is_end_stream(), + Self::Grpc { grpc_body, .. } => grpc_body.is_end_stream(), } } fn poll_frame(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll, Self::Error>>> { match self.project() { HybridBodyProj::Rest { rest_body } => rest_body.poll_frame(cx).map_err(Into::into), - HybridBodyProj::Grpc { grpc_body } => grpc_body.poll_frame(cx).map_err(Into::into), + HybridBodyProj::Grpc { + mut grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => { + let poll = grpc_body.as_mut().poll_frame(cx).map_err(Into::into); + if !*first_frame_recorded && let Poll::Ready(_) = &poll { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_FIRST_FRAME, + first_frame_started.take(), + ); + *first_frame_recorded = true; + } + let body_complete = matches!(&poll, Poll::Ready(None)) + || (matches!(&poll, Poll::Ready(Some(Ok(_)))) && grpc_body.as_ref().get_ref().is_end_stream()); + if body_complete { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_BODY_COMPLETE, + body_started.take(), + ); + } + poll + } } } fn size_hint(&self) -> http_body::SizeHint { match self { Self::Rest { rest_body } => rest_body.size_hint(), - Self::Grpc { grpc_body } => grpc_body.size_hint(), + Self::Grpc { grpc_body, .. } => grpc_body.size_hint(), } } } @@ -154,6 +193,8 @@ pin_project! { Grpc { #[pin] grpc_future: GrpcFuture, + observe_rename_data: bool, + service_future_started: Option, }, } } @@ -174,8 +215,27 @@ where Poll::Ready(Err(err)) => Poll::Ready(Err(err.into())), Poll::Pending => Poll::Pending, }, - HybridFutureProj::Grpc { grpc_future } => match grpc_future.poll(cx) { - Poll::Ready(Ok(res)) => Poll::Ready(Ok(res.map(|grpc_body| HybridBody::Grpc { grpc_body }))), + HybridFutureProj::Grpc { + grpc_future, + observe_rename_data, + service_future_started, + } => match grpc_future.poll(cx) { + Poll::Ready(Ok(res)) => { + if *observe_rename_data { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_TRANSPORT_SERVICE_FUTURE, + service_future_started.take(), + ); + } + let first_frame_started = (*observe_rename_data).then(Instant::now); + let body_started = (*observe_rename_data).then(Instant::now); + Poll::Ready(Ok(res.map(|grpc_body| HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded: !*observe_rename_data, + }))) + } Poll::Ready(Err(err)) => Poll::Ready(Err(err.into())), Poll::Pending => Poll::Pending, }, diff --git a/rustfs/src/server/layer.rs b/rustfs/src/server/layer.rs index cff0e72a8..bcc21fb37 100644 --- a/rustfs/src/server/layer.rs +++ b/rustfs/src/server/layer.rs @@ -587,6 +587,7 @@ where let watchdog_context = context.clone(); async move { tokio::select! { + biased; _ = watchdog.cancelled() => {} _ = tokio::time::sleep(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD) => { watchdog_context.log_slow_inflight(); @@ -596,11 +597,11 @@ where }); } + // An unpolled response future must also cancel its watchdog on drop. + let watchdog_guard = watchdog.map(CancellationToken::drop_guard); Box::pin(async move { let result = inner.call(req).await; - if let Some(watchdog) = watchdog { - watchdog.cancel(); - } + drop(watchdog_guard); match &result { Ok(response) => context.log_response(response), Err(error) => context.log_failure(error), @@ -863,7 +864,20 @@ where } Response::from_parts(parts, HybridBody::Rest { rest_body }) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1012,7 +1026,20 @@ where } Response::from_parts(parts, HybridBody::Rest { rest_body }) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1163,7 +1190,20 @@ where Response::from_parts(parts, HybridBody::Rest { rest_body }) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1234,7 +1274,20 @@ where }, ) } - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -1302,7 +1355,20 @@ where rest_body: RestBody::from(Bytes::new()), }, ), - HybridBody::Grpc { grpc_body } => Response::from_parts(parts, HybridBody::Grpc { grpc_body }), + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + } => Response::from_parts( + parts, + HybridBody::Grpc { + grpc_body, + first_frame_started, + body_started, + first_frame_recorded, + }, + ), }; Ok(response) @@ -5682,6 +5748,154 @@ mod tests { assert_eq!(redact_sensitive_uri_query(&uri), "/rustfs/admin/v3/users?token=not-a-download-token"); } + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_cancels_dropped_futures() { + for poll_request in [false, true] { + for deadline_ready in [false, true] { + let attempts = if deadline_ready { 32 } else { 1 }; + for attempt in 0..attempts { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = + tower::service_fn(|_: HttpRequest<()>| std::future::pending::, Infallible>>()); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + if poll_request { + assert!(futures::poll!(&mut response).is_pending()); + } + tokio::task::yield_now().await; + if deadline_ready { + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD).await; + } + drop(response); + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert!( + !output.contains(HTTP_REQUEST_INFLIGHT_SLOW_EVENT), + "polled={poll_request}, deadline_ready={deadline_ready}, attempt={attempt}: {output}" + ); + } + } + } + } + + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_cancels_panicking_future() { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = tower::service_fn(|_: HttpRequest<()>| { + std::future::poll_fn(|_| -> Poll, Infallible>> { panic!("test service panic") }) + }); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + tokio::task::yield_now().await; + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + response.as_mut().poll(&mut Context::from_waker(std::task::Waker::noop())) + })); + assert!(panic.is_err()); + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert!(!output.contains(HTTP_REQUEST_INFLIGHT_SLOW_EVENT), "{output}"); + drop(response); + } + + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_cancels_completed_futures() { + for fail_request in [false, true] { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = tower::service_fn(move |_: HttpRequest<()>| { + std::future::ready(if fail_request { + Err(io::Error::other("test service failure")) + } else { + Ok(Response::new(())) + }) + }); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + tokio::task::yield_now().await; + let Poll::Ready(result) = futures::poll!(&mut response) else { + panic!("service should complete immediately"); + }; + assert_eq!(result.is_err(), fail_request); + // Retaining a completed future must not retain an active watchdog. + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert!(!output.contains(HTTP_REQUEST_INFLIGHT_SLOW_EVENT), "{output}"); + let expected_event = if fail_request { + HTTP_REQUEST_FAILED_EVENT + } else { + HTTP_REQUEST_COMPLETED_EVENT + }; + assert_eq!(output.matches(expected_event).count(), 1, "{output}"); + drop(response); + } + } + + #[tokio::test(start_paused = true)] + async fn request_logging_watchdog_warns_once_for_pending_request() { + let writer = SharedWriter::default(); + let captured = writer.buffer.clone(); + let _subscriber = tracing::subscriber::set_default( + tracing_subscriber::fmt() + .without_time() + .with_ansi(false) + .with_writer(writer) + .finish(), + ); + assert!(tracing::enabled!(target: HTTP_SERVER_LOG_TARGET, Level::WARN)); + let inner = tower::service_fn(|_: HttpRequest<()>| std::future::pending::, Infallible>>()); + let mut service = RequestLoggingLayer.layer(inner); + let mut response = service.call(HttpRequest::new(())); + assert!(futures::poll!(&mut response).is_pending()); + tokio::task::yield_now().await; + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD - Duration::from_millis(1)).await; + tokio::task::yield_now().await; + assert!(captured.lock().expect("logs").is_empty()); + + tokio::time::advance(Duration::from_millis(1)).await; + tokio::task::yield_now().await; + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert_eq!(output.matches(HTTP_REQUEST_INFLIGHT_SLOW_EVENT).count(), 1, "{output}"); + assert!(output.contains("response_pending"), "{output}"); + + tokio::time::advance(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD).await; + tokio::task::yield_now().await; + let output = String::from_utf8(captured.lock().expect("logs").clone()).expect("UTF-8 logs"); + assert_eq!(output.matches(HTTP_REQUEST_INFLIGHT_SLOW_EVENT).count(), 1, "{output}"); + drop(response); + } + #[tokio::test] async fn request_logging_bounds_s3_failure_bursts_without_losing_counts_or_leaking_queries() { use rustfs_io_metrics::s3_http_metrics::s3_http_metrics_snapshot; diff --git a/rustfs/src/server/mod.rs b/rustfs/src/server/mod.rs index 4841bf4fe..dea1ef980 100644 --- a/rustfs/src/server/mod.rs +++ b/rustfs/src/server/mod.rs @@ -83,6 +83,7 @@ pub(crate) use readiness::ReadinessDegradedReason; pub(crate) use readiness::ReadinessGateLayer; pub(crate) use readiness::collect_dependency_readiness_report; pub(crate) use readiness::collect_node_readiness_report; +pub(crate) use readiness::observe_peer_drive_admission; pub use readiness::publish_ready_when_runtime_ready; pub(crate) use readiness::snapshot_dependency_readiness_report; pub(crate) use readiness::{collect_cluster_read_health_report, collect_cluster_write_health_report}; diff --git a/rustfs/src/server/rate_limit.rs b/rustfs/src/server/rate_limit.rs index 483b805a0..ca39b8d28 100644 --- a/rustfs/src/server/rate_limit.rs +++ b/rustfs/src/server/rate_limit.rs @@ -301,14 +301,16 @@ impl RateLimiter { /// Resolve the client identity for rate limiting. /// /// Only trusted sources are consulted: `ClientInfo` is inserted by the -/// trusted-proxy layer after validating the proxy chain, and `RemoteAddr` is -/// the raw socket peer. Spoofable headers (`X-Forwarded-For`, `X-Real-IP`) -/// are deliberately never read here. +/// trusted-proxy layer after validating the proxy chain, and `Option` +/// carries the socket peer injected by the external HTTP stack. Spoofable +/// headers (`X-Forwarded-For`, `X-Real-IP`) are deliberately never read here. pub(crate) fn client_ip(req: &Request) -> Option { if let Some(info) = req.extensions().get::() { return Some(info.real_ip); } - req.extensions().get::().map(|remote| remote.0.ip()) + req.extensions() + .get::>() + .and_then(|remote| remote.map(|remote| remote.0.ip())) } /// Resolve the host a request addresses, for virtual-hosted-style bucket @@ -929,7 +931,7 @@ mod tests { fn request_from(peer: IpAddr, path: &str) -> Request<()> { let mut req = Request::builder().uri(path).body(()).expect("request"); - req.extensions_mut().insert(RemoteAddr(SocketAddr::new(peer, 51000))); + req.extensions_mut().insert(Some(RemoteAddr(SocketAddr::new(peer, 51000)))); req } @@ -1047,7 +1049,11 @@ mod tests { async fn missing_client_identity_fails_open() { let mut service = service_with_quota(60, 1); for _ in 0..3 { - let req = Request::builder().uri("/bucket/object").body(()).expect("request"); + let mut req = Request::builder().uri("/bucket/object").body(()).expect("request"); + req.extensions_mut().insert(None::); + req.headers_mut() + .insert("x-forwarded-for", HeaderValue::from_static("203.0.113.1")); + req.headers_mut().insert("x-real-ip", HeaderValue::from_static("203.0.113.2")); assert_eq!(service.call(req).await.expect("ok").status(), StatusCode::OK); } } diff --git a/rustfs/src/server/readiness.rs b/rustfs/src/server/readiness.rs index 5de55bb7e..dc513f219 100644 --- a/rustfs/src/server/readiness.rs +++ b/rustfs/src/server/readiness.rs @@ -44,6 +44,9 @@ use tokio::sync::Mutex; use tower::{Layer, Service}; use tracing::{debug, info}; +mod peer_readmission; +pub(crate) use peer_readmission::observe_peer_drive_admission; + /// Default upper bound for the startup runtime-readiness wait. /// /// This is the compile-time fallback used when @@ -234,30 +237,41 @@ pub async fn publish_ready_when_runtime_ready( readiness: &GlobalReadiness, state_manager: Option<&ServiceStateManager>, ) -> Result<(), std::io::Error> { + let max_wait = startup_runtime_readiness_max_wait(); + let startup_deadline = tokio::time::Instant::now() + max_wait; + let mut local_readiness = None; wait_for_runtime_readiness_with( - startup_runtime_readiness_max_wait(), + max_wait, STARTUP_RUNTIME_READINESS_POLL_INTERVAL, // Publish service readiness at the same per-set read and shared-lock // boundary reported by the node probe. Mutation paths independently // require write quorum and a writable pool-metadata state. || async { collect_node_readiness_report().await.readiness }, - |dependency_readiness| { - readiness.mark_stage(rustfs_common::SystemStage::FullReady); - mark_get_metadata_read_version_coalescing_service_ready(); - if let Some(state_manager) = state_manager { - state_manager.update(ServiceState::Ready); - } - info!( - target: "rustfs::server::readiness", - storage_ready = dependency_readiness.storage_ready, - iam_ready = dependency_readiness.iam_ready, - lock_quorum_ready = dependency_readiness.lock_quorum_ready, - peer_health_ready = dependency_readiness.peer_health_ready, - "Runtime node readiness reached; publishing ready state" - ); - }, + |dependency_readiness| local_readiness = Some(dependency_readiness), ) - .await + .await?; + + // This node's own view is ready. Hold the startup FullReady transition until + // the peers count this node's drives online too (rustfs#8669). The gate + // shares the startup budget and always ends in publication. + let readmission = peer_readmission::wait_for_startup_peer_readmission(startup_deadline).await; + peer_readmission::log_readmission_outcome(&readmission); + + let dependency_readiness = local_readiness.unwrap_or_default(); + readiness.mark_stage(rustfs_common::SystemStage::FullReady); + mark_get_metadata_read_version_coalescing_service_ready(); + if let Some(state_manager) = state_manager { + state_manager.update(ServiceState::Ready); + } + info!( + target: "rustfs::server::readiness", + storage_ready = dependency_readiness.storage_ready, + iam_ready = dependency_readiness.iam_ready, + lock_quorum_ready = dependency_readiness.lock_quorum_ready, + peer_health_ready = dependency_readiness.peer_health_ready, + "Runtime node readiness reached; publishing ready state" + ); + Ok(()) } #[derive(Debug, Clone, Copy)] @@ -1506,6 +1520,91 @@ mod tests { )); } + /// Two-node model of rustfs#8669: `node-2` restarts while the survivor's + /// lock client still cannot dial it. The restarted node must keep the + /// startup readiness gate closed until the survivor's own readiness + /// predicate counts node-2's drive online, then publish. + #[tokio::test(start_paused = true)] + async fn restarted_node_stays_unready_until_survivor_counts_its_drive_online() { + use peer_readmission::{ + PeerDriveView, PeerProbe, ReadmissionOutcome, UNREACHABLE_PEER_GRACE, host_drive_observations, readmission_plan, + wait_for_peer_readmission_with, + }; + use std::sync::atomic::AtomicBool; + + // Survivor side: the shared 3-drive set, one drive per node. + let survivor = runtime_inventory(&[(1, 3, 1)]).await; + // Restarted side: the same topology seen from node-2. + let restarted_topology = EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 3, + endpoints: Endpoints::from( + (0..3) + .map(|disk_idx| Endpoint { + url: url::Url::parse(&format!("http://node-{disk_idx}:9000/pool-0-set-0")).expect("valid test endpoint"), + is_local: disk_idx == 2, + pool_idx: 0, + set_idx: 0, + disk_idx, + }) + .collect::>(), + ), + cmd_line: String::new(), + platform: String::new(), + }]); + let plan = readmission_plan(&restarted_topology).expect("restarted node plan"); + assert_eq!(plan.local_host, "node-2:9000"); + + let survivor_reaches_restarted = AtomicBool::new(false); + let readiness = GlobalReadiness::new(); + readiness.mark_stage(rustfs_common::SystemStage::IamReady); + let deadline = tokio::time::Instant::now() + Duration::from_secs(120); + + let gate = async { + let outcome = + wait_for_peer_readmission_with(&plan, deadline, Duration::from_secs(1), UNREACHABLE_PEER_GRACE, || async { + let mut online_hosts = HashSet::from(["node-0:9000".to_string(), "node-1:9000".to_string()]); + if survivor_reaches_restarted.load(Ordering::SeqCst) { + online_hosts.insert("node-2:9000".to_string()); + } + let drives = host_drive_observations(&survivor, &plan.local_host, &online_hosts) + .await + .expect("survivor inventory"); + assert_eq!(drives.len(), 1, "the survivor reports only the restarted host's drives"); + let view = PeerProbe::Answered( + drives + .into_iter() + .map(|drive| PeerDriveView { + pool_index: drive.pool_index, + set_index: drive.set_index, + disk_index: drive.disk_index, + online: drive.online, + }) + .collect(), + ); + HashMap::from([("node-0:9000".to_string(), view.clone()), ("node-1:9000".to_string(), view)]) + }) + .await; + readiness.mark_stage(rustfs_common::SystemStage::FullReady); + outcome + }; + let survivor_recovers = async { + tokio::time::sleep(Duration::from_secs(10)).await; + assert!(!readiness.is_ready(), "the survivor still counts node-2 unavailable"); + assert!(readiness_gate_blocks_path("/bucket/object", &readiness)); + survivor_reaches_restarted.store(true, Ordering::SeqCst); + }; + + let (outcome, ()) = tokio::join!(gate, survivor_recovers); + let ReadmissionOutcome::Admitted { waited } = outcome else { + panic!("expected re-admission, got {outcome:?}"); + }; + assert!(waited >= Duration::from_secs(10), "published after {waited:?}"); + assert!(readiness.is_ready()); + assert!(!readiness_gate_blocks_path("/bucket/object", &readiness)); + } + #[tokio::test] async fn node_storage_snapshot_bounds_waits_and_propagates_inventory_failure() { let mut store = runtime_inventory(&[(1, 4, 2)]).await; diff --git a/rustfs/src/server/readiness/peer_readmission.rs b/rustfs/src/server/readiness/peer_readmission.rs new file mode 100644 index 000000000..ad40772b8 --- /dev/null +++ b/rustfs/src/server/readiness/peer_readmission.rs @@ -0,0 +1,795 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Startup peer re-admission gate (rustfs#8669). +//! +//! Local startup readiness only proves that *this* node sees storage, IAM and +//! lock quorum. During a one-pod-at-a-time rollout the peers can still count +//! the restarted node's drives as unavailable: their dials keep hitting the +//! previous pod IP until the platform DNS TTL expires, and a recovered remote +//! drive then walks Offline -> Returning -> Online. An orchestrator that +//! advances on this node's Ready can stop the next pod while the survivors are +//! still short of read quorum. +//! +//! The gate asks the peers in every erasure set this node serves how they +//! currently see its drives, over the authenticated internode RPC channel, and +//! holds the startup `FullReady` transition until a strict majority of each +//! set's peers count all of this node's drives in that set as online. +//! +//! The gate is bounded and fails open. It publishes readiness when the startup +//! budget elapses, immediately when older peers cannot answer and the majority +//! is out of reach, and after [`UNREACHABLE_PEER_GRACE`] when unreachable peers +//! keep the majority out of reach. It runs only on the startup transition; +//! steady-state `/health/ready` and the S3 request path never consult it. + +use super::{ + NODE_STORAGE_READINESS_TIMEOUT, STARTUP_RUNTIME_READINESS_POLL_INTERVAL, collect_lock_quorum_observation, + exclusive_lock_quorum, health_cluster_timeout, node_disk_snapshot, runtime_sources, +}; +use crate::storage_api::error::StorageError; +use crate::storage_api::server::readiness::contract::admin::{DiskSetSelector, StorageAdminApi}; +use crate::storage_api::server::readiness::{ + DiskStore, EndpointServerPools, PeerRestClient, disk_endpoint_snapshot, get_global_notification_sys, is_dist_erasure, +}; +use rustfs_madmin::BackendInfo; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; +use std::fmt; +use std::future::Future; +use std::time::Duration; +use tracing::{debug, info, warn}; + +const EVENT_STARTUP_PEER_READMISSION: &str = "startup_peer_readmission"; +const LOG_COMPONENT_SERVER: &str = "server"; +const LOG_SUBSYSTEM_READINESS: &str = "readiness"; + +/// How long unreachable peers may keep a set's majority out of reach before the +/// gate stops waiting for them. A dead peer must not block a rollout for the +/// whole startup budget. +pub(super) const UNREACHABLE_PEER_GRACE: Duration = Duration::from_secs(15); + +fn startup_peer_readmission_enabled() -> bool { + rustfs_utils::get_env_bool( + rustfs_config::ENV_STARTUP_PEER_READMISSION_ENABLE, + rustfs_config::DEFAULT_STARTUP_PEER_READMISSION_ENABLE, + ) +} + +// --------------------------------------------------------------------------- +// Peer side: report how this node sees another host's drives. +// --------------------------------------------------------------------------- + +/// This node's readiness view of one drive served by another host. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct PeerDriveObservation { + pub(crate) pool_index: i32, + pub(crate) set_index: i32, + pub(crate) disk_index: i32, + /// Same predicate as `/health/ready`: the drive counts towards read quorum. + pub(crate) online: bool, + pub(crate) runtime_state: String, + pub(crate) host_online: bool, +} + +#[derive(Debug)] +pub(crate) enum PeerDriveAdmissionError { + NotInitialized, + UnknownHost, + Storage(StorageError), +} + +impl PeerDriveAdmissionError { + pub(crate) fn not_initialized(&self) -> bool { + matches!(self, Self::NotInitialized) + } +} + +impl fmt::Display for PeerDriveAdmissionError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::NotInitialized => f.write_str("storage layer not initialized"), + Self::UnknownHost => f.write_str("host is not a remote member of this node's topology"), + Self::Storage(err) => write!(f, "drive inventory unavailable: {err}"), + } + } +} + +/// Answer a peer's `PeerDriveAdmission` RPC from local runtime state only. +/// +/// Reuses the node readiness inputs (cached lock-client reachability and the +/// in-memory drive health tracker); no disk-info RPC or filesystem probe runs. +pub(crate) async fn observe_peer_drive_admission(host: &str) -> Result, PeerDriveAdmissionError> { + let (Some(store), Some(endpoint_pools)) = ( + runtime_sources::current_object_store_handle(), + runtime_sources::current_endpoints_handle(), + ) else { + return Err(PeerDriveAdmissionError::NotInitialized); + }; + if !is_remote_topology_host(&endpoint_pools, host) { + return Err(PeerDriveAdmissionError::UnknownHost); + } + let lock_observation = collect_lock_quorum_observation().await; + host_drive_observations(store.as_ref(), host, &lock_observation.online_hosts) + .await + .map_err(PeerDriveAdmissionError::Storage) +} + +fn is_remote_topology_host(endpoint_pools: &EndpointServerPools, host: &str) -> bool { + !host.is_empty() + && endpoint_pools + .as_ref() + .iter() + .flat_map(|pool| pool.endpoints.as_ref().iter()) + .any(|endpoint| !endpoint.is_local && endpoint.host_port() == host) +} + +pub(super) async fn host_drive_observations( + store: &S, + host: &str, + online_hosts: &HashSet, +) -> Result, StorageError> +where + S: StorageAdminApi, +{ + tokio::time::timeout(NODE_STORAGE_READINESS_TIMEOUT, async { + let backend = store.backend_info().await; + let mut drives = Vec::new(); + for (pool_idx, &set_count) in backend.total_sets.iter().enumerate() { + for set_idx in 0..set_count { + for disk in store + .disk_set_inventory(DiskSetSelector::new(pool_idx, set_idx)) + .await? + .into_iter() + .flatten() + { + let endpoint = disk_endpoint_snapshot(&disk); + if endpoint.is_local || endpoint.host_port() != host { + continue; + } + let (pool_index, set_index, disk_index) = (endpoint.pool_idx, endpoint.set_idx, endpoint.disk_idx); + let runtime_state = disk.runtime_state().as_str().to_string(); + let (_, unavailable) = node_disk_snapshot(endpoint, &runtime_state, online_hosts); + drives.push(PeerDriveObservation { + pool_index, + set_index, + disk_index, + online: unavailable.is_none(), + host_online: online_hosts.contains(host), + runtime_state, + }); + } + } + } + Ok(drives) + }) + .await + .map_err(|_| StorageError::Timeout)? +} + +// --------------------------------------------------------------------------- +// Restarting node: wait for a majority of each set's peers to re-admit us. +// --------------------------------------------------------------------------- + +/// One erasure set that mixes this node's drives with peer drives. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct ReadmissionSet { + pub(super) pool_index: i32, + pub(super) set_index: i32, + pub(super) local_disks: BTreeSet, + pub(super) peers: BTreeSet, + pub(super) required_peers: usize, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct ReadmissionPlan { + pub(super) local_host: String, + pub(super) sets: Vec, +} + +impl ReadmissionPlan { + fn peers(&self) -> BTreeSet { + self.sets.iter().flat_map(|set| set.peers.iter().cloned()).collect() + } +} + +/// Build the per-set peer quorum this node must reach before startup readiness. +/// +/// Each set requires a strict majority of its distinct peer hosts. A simple +/// half would let a 3-node set publish on one peer's view, and that peer may be +/// the next one the orchestrator stops. +pub(super) fn readmission_plan(endpoint_pools: &EndpointServerPools) -> Result { + let mut local_hosts = BTreeSet::new(); + let mut sets: BTreeMap<(i32, i32), (BTreeSet, BTreeSet)> = BTreeMap::new(); + for endpoint in endpoint_pools.as_ref().iter().flat_map(|pool| pool.endpoints.as_ref().iter()) { + let host = endpoint.host_port(); + let (local_disks, peers) = sets.entry((endpoint.pool_idx, endpoint.set_idx)).or_default(); + if endpoint.is_local { + local_disks.insert(endpoint.disk_idx); + if !host.is_empty() { + local_hosts.insert(host); + } + } else if !host.is_empty() { + peers.insert(host); + } + } + + let mut local_hosts = local_hosts.into_iter(); + let (Some(local_host), None) = (local_hosts.next(), local_hosts.next()) else { + return Err("local_topology_host_ambiguous"); + }; + + let sets = sets + .into_iter() + .filter(|(_, (local_disks, peers))| !local_disks.is_empty() && !peers.is_empty()) + .map(|((pool_index, set_index), (local_disks, peers))| ReadmissionSet { + pool_index, + set_index, + required_peers: exclusive_lock_quorum(peers.len()), + local_disks, + peers, + }) + .collect::>(); + if sets.is_empty() { + return Err("no_set_shared_with_peers"); + } + Ok(ReadmissionPlan { local_host, sets }) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) struct PeerDriveView { + pub(super) pool_index: i32, + pub(super) set_index: i32, + pub(super) disk_index: i32, + pub(super) online: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum PeerProbe { + /// The peer answered with its view of this node's drives. + Answered(Vec), + /// The peer predates the RPC and will never answer. + Unsupported, + /// The RPC failed or timed out; the peer may answer later. + Unreachable, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum SetState { + Admitted, + Waiting, + /// The majority needs a peer that is currently unreachable. + NeedsUnreachablePeer, + /// Even if every unreachable peer answered, the majority is out of reach. + Unattainable, +} + +fn peer_admits_set(drives: &[PeerDriveView], set: &ReadmissionSet) -> bool { + set.local_disks.iter().all(|disk_index| { + drives.iter().any(|drive| { + drive.pool_index == set.pool_index + && drive.set_index == set.set_index + && drive.disk_index == *disk_index + && drive.online + }) + }) +} + +fn set_state(set: &ReadmissionSet, probes: &HashMap) -> SetState { + let (mut admitted, mut pending, mut unreachable) = (0usize, 0usize, 0usize); + for peer in &set.peers { + match probes.get(peer) { + Some(PeerProbe::Answered(drives)) if peer_admits_set(drives, set) => admitted += 1, + Some(PeerProbe::Answered(_)) => pending += 1, + Some(PeerProbe::Unsupported) => {} + Some(PeerProbe::Unreachable) | None => unreachable += 1, + } + } + if admitted >= set.required_peers { + SetState::Admitted + } else if admitted + pending + unreachable < set.required_peers { + SetState::Unattainable + } else if admitted + pending < set.required_peers { + SetState::NeedsUnreachablePeer + } else { + SetState::Waiting + } +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(super) struct ReadmissionProgress { + pub(super) total_sets: usize, + pub(super) admitted_sets: usize, + pub(super) waiting_sets: usize, + pub(super) unreachable_sets: usize, + pub(super) unattainable_sets: usize, +} + +fn evaluate(plan: &ReadmissionPlan, probes: &HashMap) -> ReadmissionProgress { + let mut progress = ReadmissionProgress { + total_sets: plan.sets.len(), + ..Default::default() + }; + for set in &plan.sets { + match set_state(set, probes) { + SetState::Admitted => progress.admitted_sets += 1, + SetState::Waiting => progress.waiting_sets += 1, + SetState::NeedsUnreachablePeer => progress.unreachable_sets += 1, + SetState::Unattainable => progress.unattainable_sets += 1, + } + } + progress +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum GiveUpReason { + DeadlineElapsed, + PeersUnreachable, + PeersCannotReport, +} + +impl GiveUpReason { + fn as_str(self) -> &'static str { + match self { + Self::DeadlineElapsed => "deadline_elapsed", + Self::PeersUnreachable => "peers_unreachable", + Self::PeersCannotReport => "peers_cannot_report", + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum ReadmissionOutcome { + Admitted { + waited: Duration, + }, + Skipped(&'static str), + GaveUp { + reason: GiveUpReason, + waited: Duration, + progress: ReadmissionProgress, + }, +} + +/// Poll `probe` until every planned set is re-admitted or the gate gives up. +/// +/// The loop always terminates: by `deadline`, by `unreachable_grace` of +/// continuous unreachable shortfall, or immediately when peers that cannot +/// report make the majority unattainable. +pub(super) async fn wait_for_peer_readmission_with( + plan: &ReadmissionPlan, + deadline: tokio::time::Instant, + poll_interval: Duration, + unreachable_grace: Duration, + mut probe: P, +) -> ReadmissionOutcome +where + P: FnMut() -> Fut, + Fut: Future>, +{ + let started = tokio::time::Instant::now(); + let mut unreachable_since: Option = None; + let mut last_logged: Option = None; + + loop { + let probes = probe().await; + let progress = evaluate(plan, &probes); + let now = tokio::time::Instant::now(); + let waited = now.saturating_duration_since(started); + let give_up = |reason| ReadmissionOutcome::GaveUp { + reason, + waited, + progress, + }; + + if progress.admitted_sets == progress.total_sets { + return ReadmissionOutcome::Admitted { waited }; + } + if progress.unattainable_sets > 0 { + return give_up(GiveUpReason::PeersCannotReport); + } + if progress.unreachable_sets > 0 { + let since = *unreachable_since.get_or_insert(now); + if now.saturating_duration_since(since) >= unreachable_grace { + return give_up(GiveUpReason::PeersUnreachable); + } + } else { + unreachable_since = None; + } + if now >= deadline { + return give_up(GiveUpReason::DeadlineElapsed); + } + + if last_logged != Some(progress) { + info!( + event = EVENT_STARTUP_PEER_READMISSION, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_READINESS, + state = "waiting", + total_sets = progress.total_sets, + admitted_sets = progress.admitted_sets, + waiting_sets = progress.waiting_sets, + unreachable_sets = progress.unreachable_sets, + "waiting for peers to re-admit this node" + ); + last_logged = Some(progress); + } + tokio::time::sleep(poll_interval.min(deadline.saturating_duration_since(now))).await; + } +} + +async fn probe_peers(clients: &[(String, PeerRestClient)], local_host: &str, timeout: Duration) -> HashMap { + futures::future::join_all(clients.iter().map(|(host, client)| async move { + let probe = match tokio::time::timeout(timeout, client.peer_drive_admission(local_host)).await { + Ok(Ok(Some(drives))) => PeerProbe::Answered( + drives + .into_iter() + .map(|drive| PeerDriveView { + pool_index: drive.pool_index, + set_index: drive.set_index, + disk_index: drive.disk_index, + online: drive.online, + }) + .collect(), + ), + Ok(Ok(None)) => PeerProbe::Unsupported, + Ok(Err(err)) => { + debug!( + event = EVENT_STARTUP_PEER_READMISSION, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_READINESS, + result = "peer_error", + peer = %host, + error = %err, + "peer re-admission probe failed" + ); + PeerProbe::Unreachable + } + Err(_) => PeerProbe::Unreachable, + }; + (host.clone(), probe) + })) + .await + .into_iter() + .collect() +} + +/// Run the startup re-admission gate against the live topology and peers. +pub(super) async fn wait_for_startup_peer_readmission(deadline: tokio::time::Instant) -> ReadmissionOutcome { + if !startup_peer_readmission_enabled() { + return ReadmissionOutcome::Skipped("disabled"); + } + if !is_dist_erasure().await { + return ReadmissionOutcome::Skipped("not_distributed"); + } + let Some(endpoint_pools) = runtime_sources::current_endpoints_handle() else { + return ReadmissionOutcome::Skipped("topology_unavailable"); + }; + let plan = match readmission_plan(&endpoint_pools) { + Ok(plan) => plan, + Err(reason) => return ReadmissionOutcome::Skipped(reason), + }; + let Some(notification_sys) = get_global_notification_sys() else { + return ReadmissionOutcome::Skipped("peer_clients_unavailable"); + }; + let peers = plan.peers(); + // A planned peer without a client stays unreachable, so the grace bound applies. + let clients = notification_sys + .peer_clients + .iter() + .flatten() + .filter(|client| peers.contains(client.topology_member())) + .map(|client| (client.topology_member().to_string(), client.clone())) + .collect::>(); + let timeout = health_cluster_timeout(); + let local_host = plan.local_host.as_str(); + wait_for_peer_readmission_with(&plan, deadline, STARTUP_RUNTIME_READINESS_POLL_INTERVAL, UNREACHABLE_PEER_GRACE, || { + probe_peers(&clients, local_host, timeout) + }) + .await +} + +pub(super) fn log_readmission_outcome(outcome: &ReadmissionOutcome) { + match outcome { + ReadmissionOutcome::Admitted { waited } => info!( + event = EVENT_STARTUP_PEER_READMISSION, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_READINESS, + result = "admitted", + waited_ms = u64::try_from(waited.as_millis()).unwrap_or(u64::MAX), + "peers re-admitted this node" + ), + // Single-node deployments have no peers; keep their startup log quiet. + ReadmissionOutcome::Skipped(reason @ "not_distributed") => debug!( + event = EVENT_STARTUP_PEER_READMISSION, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_READINESS, + result = "skipped", + reason = *reason, + "peer re-admission gate skipped" + ), + ReadmissionOutcome::Skipped(reason) => info!( + event = EVENT_STARTUP_PEER_READMISSION, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_READINESS, + result = "skipped", + reason = *reason, + "peer re-admission gate skipped" + ), + ReadmissionOutcome::GaveUp { + reason, + waited, + progress, + } => warn!( + event = EVENT_STARTUP_PEER_READMISSION, + component = LOG_COMPONENT_SERVER, + subsystem = LOG_SUBSYSTEM_READINESS, + result = "gave_up", + reason = reason.as_str(), + waited_ms = u64::try_from(waited.as_millis()).unwrap_or(u64::MAX), + total_sets = progress.total_sets, + admitted_sets = progress.admitted_sets, + waiting_sets = progress.waiting_sets, + unreachable_sets = progress.unreachable_sets, + unattainable_sets = progress.unattainable_sets, + "publishing startup readiness without peer re-admission" + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage_api::server::readiness::{Endpoint, Endpoints, PoolEndpoints}; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + + fn endpoint(host: &str, is_local: bool, set_idx: i32, disk_idx: i32) -> Endpoint { + Endpoint { + url: url::Url::parse(&format!("http://{host}:9000/data{disk_idx}")).expect("test endpoint url"), + is_local, + pool_idx: 0, + set_idx, + disk_idx, + } + } + + fn pools(set_count: usize, drives_per_set: usize, endpoints: Vec) -> EndpointServerPools { + EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count, + drives_per_set, + endpoints: Endpoints::from(endpoints), + cmd_line: String::new(), + platform: String::new(), + }]) + } + + /// Three pods, one drive each, one set: the #8669 topology seen from node-2. + fn three_node_plan() -> ReadmissionPlan { + readmission_plan(&pools( + 1, + 3, + vec![ + endpoint("node-0", false, 0, 0), + endpoint("node-1", false, 0, 1), + endpoint("node-2", true, 0, 2), + ], + )) + .expect("three-node plan") + } + + fn answered(online: bool) -> PeerProbe { + PeerProbe::Answered(vec![PeerDriveView { + pool_index: 0, + set_index: 0, + disk_index: 2, + online, + }]) + } + + fn probes(entries: &[(&str, PeerProbe)]) -> HashMap { + entries + .iter() + .map(|(host, probe)| (format!("{host}:9000"), probe.clone())) + .collect() + } + + #[test] + fn plan_requires_a_strict_majority_of_each_shared_sets_peers() { + let plan = three_node_plan(); + assert_eq!(plan.local_host, "node-2:9000"); + assert_eq!(plan.sets.len(), 1); + assert_eq!(plan.sets[0].local_disks, BTreeSet::from([2])); + assert_eq!(plan.sets[0].required_peers, 2, "a 3-node set must not publish on one peer's view"); + + // Two sets of four drives on four nodes; node-0 serves only set 0. + let plan = readmission_plan(&pools( + 2, + 4, + vec![ + endpoint("node-0", true, 0, 0), + endpoint("node-1", false, 0, 1), + endpoint("node-2", false, 0, 2), + endpoint("node-3", false, 0, 3), + endpoint("node-1", false, 1, 0), + endpoint("node-2", false, 1, 1), + endpoint("node-3", false, 1, 2), + endpoint("node-3", false, 1, 3), + ], + )) + .expect("multi-set plan"); + assert_eq!(plan.sets.len(), 1, "sets without a local drive are not gated"); + assert_eq!(plan.sets[0].peers.len(), 3); + assert_eq!(plan.sets[0].required_peers, 2); + } + + #[test] + fn plan_skips_topologies_without_peers_or_a_single_local_host() { + assert_eq!( + readmission_plan(&pools(1, 2, vec![endpoint("node-0", true, 0, 0), endpoint("node-0", true, 0, 1)])), + Err("no_set_shared_with_peers") + ); + assert_eq!( + readmission_plan(&pools(1, 2, vec![endpoint("node-0", false, 0, 0), endpoint("node-1", false, 0, 1)])), + Err("local_topology_host_ambiguous") + ); + } + + #[test] + fn set_state_separates_waiting_unreachable_and_unattainable_shortfalls() { + let plan = three_node_plan(); + let set = &plan.sets[0]; + let state = |entries: &[(&str, PeerProbe)]| set_state(set, &probes(entries)); + + assert_eq!(state(&[("node-0", answered(true)), ("node-1", answered(true))]), SetState::Admitted); + assert_eq!(state(&[("node-0", answered(true)), ("node-1", answered(false))]), SetState::Waiting); + assert_eq!( + state(&[("node-0", answered(true)), ("node-1", PeerProbe::Unreachable)]), + SetState::NeedsUnreachablePeer + ); + assert_eq!( + state(&[("node-0", answered(true))]), + SetState::NeedsUnreachablePeer, + "missing probe is unreachable" + ); + assert_eq!( + state(&[("node-0", answered(true)), ("node-1", PeerProbe::Unsupported)]), + SetState::Unattainable + ); + // A peer that answers without listing our drive has not re-admitted it. + assert_eq!( + state(&[("node-0", answered(true)), ("node-1", PeerProbe::Answered(Vec::new()))]), + SetState::Waiting + ); + } + + #[tokio::test(start_paused = true)] + async fn gate_holds_until_peers_count_the_drive_online() { + let plan = three_node_plan(); + let polls = Arc::new(AtomicUsize::new(0)); + let deadline = tokio::time::Instant::now() + Duration::from_secs(120); + let outcome = wait_for_peer_readmission_with(&plan, deadline, Duration::from_secs(1), UNREACHABLE_PEER_GRACE, || { + let poll = polls.fetch_add(1, Ordering::SeqCst); + async move { + // node-1 keeps dialing the previous pod IP for five polls. + probes(&[("node-0", answered(true)), ("node-1", answered(poll >= 5))]) + } + }) + .await; + + assert_eq!( + outcome, + ReadmissionOutcome::Admitted { + waited: Duration::from_secs(5) + } + ); + assert_eq!(polls.load(Ordering::SeqCst), 6); + } + + #[tokio::test(start_paused = true)] + async fn gate_gives_up_at_the_startup_deadline() { + let plan = three_node_plan(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + let outcome = wait_for_peer_readmission_with(&plan, deadline, Duration::from_secs(1), UNREACHABLE_PEER_GRACE, || async { + probes(&[("node-0", answered(true)), ("node-1", answered(false))]) + }) + .await; + + let ReadmissionOutcome::GaveUp { + reason, + waited, + progress, + } = outcome + else { + panic!("expected the bounded wait to give up, got {outcome:?}"); + }; + assert_eq!(reason, GiveUpReason::DeadlineElapsed); + assert_eq!(waited, Duration::from_secs(30)); + assert_eq!(progress.waiting_sets, 1); + } + + #[tokio::test(start_paused = true)] + async fn dead_peer_releases_the_gate_after_the_unreachable_grace() { + let plan = three_node_plan(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(120); + let outcome = wait_for_peer_readmission_with(&plan, deadline, Duration::from_secs(1), UNREACHABLE_PEER_GRACE, || async { + probes(&[("node-0", answered(true)), ("node-1", PeerProbe::Unreachable)]) + }) + .await; + + let ReadmissionOutcome::GaveUp { reason, waited, .. } = outcome else { + panic!("expected a dead peer to release the gate, got {outcome:?}"); + }; + assert_eq!(reason, GiveUpReason::PeersUnreachable); + assert_eq!(waited, UNREACHABLE_PEER_GRACE); + } + + #[tokio::test(start_paused = true)] + async fn unreachable_grace_restarts_when_the_peer_comes_back() { + let plan = three_node_plan(); + let polls = Arc::new(AtomicUsize::new(0)); + let deadline = tokio::time::Instant::now() + Duration::from_secs(120); + let outcome = wait_for_peer_readmission_with(&plan, deadline, Duration::from_secs(1), UNREACHABLE_PEER_GRACE, || { + let poll = polls.fetch_add(1, Ordering::SeqCst); + async move { + let node_1 = match poll { + 0..10 => PeerProbe::Unreachable, + 10..30 => answered(false), + _ => answered(true), + }; + probes(&[("node-0", answered(true)), ("node-1", node_1)]) + } + }) + .await; + + assert_eq!( + outcome, + ReadmissionOutcome::Admitted { + waited: Duration::from_secs(30) + } + ); + } + + #[tokio::test(start_paused = true)] + async fn older_peers_that_cannot_report_release_the_gate_immediately() { + let plan = three_node_plan(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(120); + let outcome = wait_for_peer_readmission_with(&plan, deadline, Duration::from_secs(1), UNREACHABLE_PEER_GRACE, || async { + probes(&[("node-0", PeerProbe::Unsupported), ("node-1", PeerProbe::Unsupported)]) + }) + .await; + + let ReadmissionOutcome::GaveUp { reason, waited, .. } = outcome else { + panic!("expected a rolling upgrade from older peers to skip the wait, got {outcome:?}"); + }; + assert_eq!(reason, GiveUpReason::PeersCannotReport); + assert_eq!(waited, Duration::ZERO); + } + + #[test] + fn peer_side_rejects_hosts_outside_its_remote_topology() { + let topology = pools( + 1, + 3, + vec![ + endpoint("node-0", true, 0, 0), + endpoint("node-1", false, 0, 1), + endpoint("node-2", false, 0, 2), + ], + ); + assert!(is_remote_topology_host(&topology, "node-2:9000")); + assert!(!is_remote_topology_host(&topology, "node-0:9000"), "a node does not answer for itself"); + assert!(!is_remote_topology_host(&topology, "node-9:9000")); + assert!(!is_remote_topology_host(&topology, "")); + } +} diff --git a/rustfs/src/startup_entrypoint.rs b/rustfs/src/startup_entrypoint.rs index e2402b2a4..adf5f056d 100644 --- a/rustfs/src/startup_entrypoint.rs +++ b/rustfs/src/startup_entrypoint.rs @@ -39,7 +39,41 @@ const LOG_SUBSYSTEM_STARTUP: &str = "startup"; const EVENT_SERVER_RUNTIME_FAILED: &str = "server_runtime_failed"; const OBSERVABILITY_INIT_FATAL_ALREADY_REPORTED: &str = "observability initialization failure already reported"; +/// Thread name for the thread that drives the root startup future. +const STARTUP_THREAD_NAME: &str = "rustfs-main"; + +/// Stack reserved for the thread that drives the root startup future. +/// +/// `block_on` polls `async_main` on the calling thread, so the whole startup +/// chain runs on that thread's stack rather than on a Tokio worker. The native +/// main thread only gets what the platform hands out: 8 MiB from `ulimit -s` +/// on Linux and macOS, but the 1 MiB PE default on Windows, where a debug +/// build overflows before it starts listening. `RUST_MIN_STACK` does not apply +/// to the native main thread, so the budget is set here explicitly instead. +/// Startup currently peaks near 2 MiB in a debug build and under 1 MiB in a +/// release build; 8 MiB keeps the budget Linux and macOS already had. +const STARTUP_THREAD_STACK_SIZE: usize = 8 * 1024 * 1024; + pub fn run_process() { + run_on_startup_thread(run_process_on_current_thread); +} + +/// Run `f` on a thread whose stack is [`STARTUP_THREAD_STACK_SIZE`] on every +/// platform, propagating its panic so the process exit status is unchanged. +fn run_on_startup_thread(f: impl FnOnce() -> T + Send + 'static) -> T { + // Failing to spawn the startup thread is a startup fatal boundary. + let handle = std::thread::Builder::new() + .name(STARTUP_THREAD_NAME.to_owned()) + .stack_size(STARTUP_THREAD_STACK_SIZE) + .spawn(f) + .expect("Failed to spawn startup thread"); + match handle.join() { + Ok(value) => value, + Err(panic) => std::panic::resume_unwind(panic), + } +} + +fn run_process_on_current_thread() { // Building the process runtime is a startup fatal boundary. let (runtime, dial9_guard) = crate::server::build_tokio_runtime().expect("Failed to build Tokio runtime"); let result = runtime.block_on(async_main()); @@ -2034,4 +2068,36 @@ mod tests { "[FATAL] Observability initialization failed: collector unavailable" ); } + + /// Consume roughly `bytes` of stack through non-elidable recursive frames. + #[inline(never)] + fn consume_stack(bytes: usize) -> u8 { + const FRAME: usize = 64 * 1024; + let mut frame = [0_u8; FRAME]; + std::hint::black_box(&mut frame); + if bytes <= FRAME { + return frame[0]; + } + frame[0].wrapping_add(consume_stack(bytes - FRAME)) + } + + #[test] + fn startup_thread_provides_documented_stack_budget() { + // Half the budget exceeds both the 2 MiB Rust thread default and the + // 1 MiB Windows main thread, so this overflows unless the startup + // thread really gets its explicit budget. + let used = STARTUP_THREAD_STACK_SIZE / 2; + let name = run_on_startup_thread(move || { + std::hint::black_box(consume_stack(used)); + std::thread::current().name().map(str::to_owned) + }); + assert_eq!(name.as_deref(), Some(STARTUP_THREAD_NAME)); + } + + #[test] + fn startup_thread_propagates_panics() { + let result = std::panic::catch_unwind(|| run_on_startup_thread(|| panic!("startup thread panic"))); + let payload = result.expect_err("startup thread panic must propagate"); + assert_eq!(payload.downcast_ref::<&str>(), Some(&"startup thread panic")); + } } diff --git a/rustfs/src/startup_fs_guard.rs b/rustfs/src/startup_fs_guard.rs index 53964f2e2..918baebcf 100644 --- a/rustfs/src/startup_fs_guard.rs +++ b/rustfs/src/startup_fs_guard.rs @@ -17,6 +17,7 @@ use rustfs_config::{ DEFAULT_RUSTFS_UNSUPPORTED_FS_POLICY, ENV_RUSTFS_UNSUPPORTED_FS_POLICY, RUSTFS_UNSUPPORTED_FS_POLICY_FAIL, RUSTFS_UNSUPPORTED_FS_POLICY_WARN, }; +use rustfs_utils::os::DiskInfo; use std::collections::BTreeSet; use std::io::{Error, Result}; use tracing::warn; @@ -60,6 +61,17 @@ fn is_unsupported_fs_type(fs_type: &str) -> bool { ) || normalized.starts_with("fuse.") } +/// Describes why the filesystem behind `info` is outside the supported +/// boundary, or returns `None` when it is supported. +fn unsupported_fs_label(info: &DiskInfo) -> Option { + if info.remote { + // Windows names the server's filesystem for SMB shares, which would + // otherwise pass as a local NTFS volume. + return Some(format!("{} on network share", info.fstype)); + } + is_unsupported_fs_type(&info.fstype).then(|| info.fstype.clone()) +} + fn collect_local_paths(endpoint_pools: &EndpointServerPools) -> Vec { let mut local_paths = BTreeSet::new(); for pool in endpoint_pools.as_ref() { @@ -92,8 +104,8 @@ pub(crate) fn enforce_unsupported_fs_policy(endpoint_pools: &EndpointServerPools } }; - if is_unsupported_fs_type(&info.fstype) { - unsupported.push((path, info.fstype)); + if let Some(label) = unsupported_fs_label(&info) { + unsupported.push((path, label)); } } @@ -103,7 +115,7 @@ pub(crate) fn enforce_unsupported_fs_policy(endpoint_pools: &EndpointServerPools let detail = unsupported .iter() - .map(|(path, fs_type)| format!("{path} ({fs_type})")) + .map(|(path, label)| format!("{path} ({label})")) .collect::>() .join(", "); @@ -150,6 +162,29 @@ mod tests { assert!(!is_unsupported_fs_type("BTRFS")); } + #[test] + fn remote_volume_is_unsupported_whatever_its_fstype() { + // An SMB share on Windows reports the server's filesystem name. + let share = DiskInfo { + fstype: "NTFS".to_string(), + remote: true, + ..Default::default() + }; + assert_eq!(unsupported_fs_label(&share).as_deref(), Some("NTFS on network share")); + + let local = DiskInfo { + fstype: "NTFS".to_string(), + ..Default::default() + }; + assert_eq!(unsupported_fs_label(&local), None); + + let cifs = DiskInfo { + fstype: "CIFS".to_string(), + ..Default::default() + }; + assert_eq!(unsupported_fs_label(&cifs).as_deref(), Some("CIFS")); + } + #[test] fn policy_parser_handles_valid_and_invalid_values() { assert_eq!(UnsupportedFsPolicy::parse("warn"), Some(UnsupportedFsPolicy::Warn)); diff --git a/rustfs/src/startup_lifecycle.rs b/rustfs/src/startup_lifecycle.rs index d3360935e..1118abd5b 100644 --- a/rustfs/src/startup_lifecycle.rs +++ b/rustfs/src/startup_lifecycle.rs @@ -21,7 +21,7 @@ use crate::{ startup_iam::{IamBootstrapDisposition, publish_ready_for_iam_bootstrap}, startup_runtime_sources, startup_services::StartupServiceRuntime, - startup_shutdown::run_startup_shutdown_sequence, + startup_shutdown::{StartupBackgroundTasks, run_startup_shutdown_sequence}, }; use rustfs_common::GlobalReadiness; use rustfs_object_capacity::capacity_manager::CapacityBackgroundTasks; @@ -130,6 +130,7 @@ pub(crate) async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifec readiness, } = lifecycle; let StartupServiceRuntime { + table_maintenance, oidc_recovery, optional_runtimes, heartbeat, @@ -163,7 +164,10 @@ pub(crate) async fn run_startup_runtime_lifecycle(lifecycle: StartupRuntimeLifec s3_shutdown_tx, console_shutdown_tx, optional_runtimes, - capacity_tasks, + StartupBackgroundTasks { + capacity: capacity_tasks, + table_maintenance, + }, shutdown_token, ) .await; diff --git a/rustfs/src/startup_observability.rs b/rustfs/src/startup_observability.rs index 01ec050b8..114b1a808 100644 --- a/rustfs/src/startup_observability.rs +++ b/rustfs/src/startup_observability.rs @@ -13,12 +13,16 @@ // limitations under the License. use std::sync::Arc; +use std::time::Duration; use crate::init::{init_auto_tuner, init_update_check, print_server_info}; use crate::startup_runtime_sources; use crate::storage_api::startup::storage::{ECStore, init_compression_total_memory_from_backend}; +use tokio::io::AsyncWriteExt; use tokio_util::sync::CancellationToken; +const GET_STAGE_LOCAL_SUMMARY_FILE: &str = "/var/log/rustfs/get-stage-summary.log"; + pub(crate) async fn init_observability_runtime(store: Arc, ctx: CancellationToken) { print_server_info(); init_update_check(); @@ -27,6 +31,10 @@ pub(crate) async fn init_observability_runtime(store: Arc, ctx: Cancell let metrics_enabled = startup_runtime_sources::observability_metric_enabled(); configure_metric_gates(metrics_enabled); + if rustfs_io_metrics::get_stage_local_summary_enabled() { + init_get_stage_local_summary_runtime(ctx.clone()); + } + if metrics_enabled { // Load persisted compression stats into memory early, before any PUTs can occur. init_compression_total_memory_from_backend(store).await; @@ -37,21 +45,115 @@ pub(crate) async fn init_observability_runtime(store: Arc, ctx: Cancell } fn configure_metric_gates(metrics_enabled: bool) { + let local_summary_enabled = !metrics_enabled + && rustfs_utils::get_env_bool(rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, false); + let sample_rate = + rustfs_utils::get_env_u64(rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE, 64).max(1); + startup_runtime_sources::set_get_stage_local_summary_enabled(local_summary_enabled); + startup_runtime_sources::set_get_stage_local_summary_sample_rate(sample_rate); let put_stage_metrics_enabled = metrics_enabled && rustfs_utils::get_env_bool( rustfs_config::observability::ENV_OBS_PUT_STAGE_METRICS_ENABLED, rustfs_config::DEFAULT_OBS_PUT_STAGE_METRICS_ENABLED, ); startup_runtime_sources::set_put_stage_metrics_enabled(put_stage_metrics_enabled); - startup_runtime_sources::set_get_stage_metrics_enabled(metrics_enabled); + startup_runtime_sources::set_get_stage_metrics_enabled(metrics_enabled || local_summary_enabled); startup_runtime_sources::set_metrics_enabled(metrics_enabled); } +fn init_get_stage_local_summary_runtime(ctx: CancellationToken) { + let interval_secs = + rustfs_utils::get_env_u64(rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS, 15).max(1); + tokio::spawn(async move { + let mut interval = tokio::time::interval(Duration::from_secs(interval_secs)); + interval.tick().await; + loop { + tokio::select! { + _ = ctx.cancelled() => break, + _ = interval.tick() => { + let rows = rustfs_io_metrics::take_get_stage_local_summary(); + if rows.is_empty() { + continue; + } + let summary = rows.iter().map(|row| { + format!("{}|{}|{}|{}|{}|{}", row.path, row.stage, row.object_class, row.size_bucket, row.sampled_count, row.duration_nanoseconds) + }).collect::>().join(";"); + let timestamp_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_millis(); + let line = format!( + "timestamp_ms={timestamp_ms}\tsample_rate={}\tstage_count={}\tstages={summary}\n", + rustfs_io_metrics::get_stage_local_summary_sample_rate(), + rows.len(), + ); + let Ok(mut file) = tokio::fs::OpenOptions::new() + .create(true) + .append(true) + .open(GET_STAGE_LOCAL_SUMMARY_FILE) + .await + else { + break; + }; + if file.write_all(line.as_bytes()).await.is_err() { + break; + } + } + } + } + }); +} + #[cfg(test)] mod tests { use super::*; const PUT_STAGE_ENV: &str = rustfs_config::observability::ENV_OBS_PUT_STAGE_METRICS_ENABLED; + const LOCAL_SUMMARY_ENV: &str = rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED; + const LOCAL_SUMMARY_RATE_ENV: &str = rustfs_config::observability::ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE; + + #[test] + #[serial_test::serial] + fn get_stage_local_summary_can_run_without_metrics_export() { + let previous_metrics = rustfs_io_metrics::metrics_enabled(); + let previous_get_stages = rustfs_io_metrics::get_stage_metrics_enabled(); + let previous_local_summary = rustfs_io_metrics::get_stage_local_summary_enabled(); + let previous_sample_rate = rustfs_io_metrics::get_stage_local_summary_sample_rate(); + + temp_env::with_var(LOCAL_SUMMARY_ENV, Some("true"), || { + temp_env::with_var(LOCAL_SUMMARY_RATE_ENV, Some("1"), || { + configure_metric_gates(false); + assert!(!rustfs_io_metrics::metrics_enabled()); + assert!(rustfs_io_metrics::get_stage_metrics_enabled()); + assert!(rustfs_io_metrics::get_stage_local_summary_enabled()); + rustfs_io_metrics::take_get_stage_local_summary(); + + rustfs_io_metrics::record_get_object_stage_duration("legacy_duplex", "metadata", 0.001); + rustfs_io_metrics::record_get_object_stage_duration("legacy_duplex", "metadata", 0.003); + rustfs_io_metrics::record_get_object_stage_duration_by_size( + "legacy_duplex", + "inline_direct.decode", + "plain_single_part", + "le_4kib", + 0.002, + ); + + let rows = rustfs_io_metrics::take_get_stage_local_summary(); + let metadata = rows + .iter() + .find(|row| row.path == "legacy_duplex" && row.stage == "metadata") + .expect("local summary should retain sampled metadata observations"); + assert_eq!(metadata.sampled_count, 2); + assert_eq!(metadata.duration_nanoseconds, 4_000_000); + assert!(rows.iter().any(|row| row.size_bucket == "le_4kib")); + }); + }); + + startup_runtime_sources::set_metrics_enabled(previous_metrics); + startup_runtime_sources::set_get_stage_metrics_enabled(previous_get_stages); + startup_runtime_sources::set_get_stage_local_summary_enabled(previous_local_summary); + startup_runtime_sources::set_get_stage_local_summary_sample_rate(previous_sample_rate); + } #[test] #[serial_test::serial] diff --git a/rustfs/src/startup_runtime_sources.rs b/rustfs/src/startup_runtime_sources.rs index 73501a592..f69546688 100644 --- a/rustfs/src/startup_runtime_sources.rs +++ b/rustfs/src/startup_runtime_sources.rs @@ -99,6 +99,14 @@ pub(crate) fn set_get_stage_metrics_enabled(enabled: bool) { rustfs_io_metrics::set_get_stage_metrics_enabled(enabled); } +pub(crate) fn set_get_stage_local_summary_enabled(enabled: bool) { + rustfs_io_metrics::set_get_stage_local_summary_enabled(enabled); +} + +pub(crate) fn set_get_stage_local_summary_sample_rate(sample_rate: u64) { + rustfs_io_metrics::set_get_stage_local_summary_sample_rate(sample_rate); +} + pub(crate) fn set_metrics_enabled(enabled: bool) { rustfs_io_metrics::set_metrics_enabled(enabled); } diff --git a/rustfs/src/startup_services.rs b/rustfs/src/startup_services.rs index 4a3b66a5e..acf2b7a51 100644 --- a/rustfs/src/startup_services.rs +++ b/rustfs/src/startup_services.rs @@ -43,6 +43,7 @@ const LOG_COMPONENT_CONNECT: &str = "connect"; const LOG_SUBSYSTEM_INVENTORY: &str = "inventory"; pub(crate) struct StartupServiceRuntime { + pub(crate) table_maintenance: Option, pub(crate) oidc_recovery: Option>, pub(crate) optional_runtimes: OptionalRuntimeServices, pub(crate) heartbeat: Option, @@ -83,6 +84,7 @@ pub(crate) async fn init_startup_runtime_services( state_manager: Arc, server_ctx: Arc, ) -> Result { + let maintenance_config = crate::table_catalog::PeriodicMaintenanceConfig::from_env().map_err(std::io::Error::other)?; init_kms_system(config, store.clone()).await?; let optional_runtimes = init_optional_runtime_services(server_ctx.clone()).await?; @@ -102,7 +104,7 @@ pub(crate) async fn init_startup_runtime_services( init_deadlock_detector_runtime(); let buckets = init_bucket_metadata_runtime(store.clone(), ctx.clone()).await?; - let iam_bootstrap = init_iam_runtime(store.clone(), ctx.clone(), readiness, state_manager, server_ctx).await?; + let iam_bootstrap = init_iam_runtime(store.clone(), ctx.clone(), readiness, state_manager, server_ctx.clone()).await?; // Audit initialization requires the AppContext (server config + object store) // which is published by ensure_startup_after_iam inside init_iam_runtime. @@ -119,7 +121,11 @@ pub(crate) async fn init_startup_runtime_services( let heartbeat = start_heartbeat_runtime(heartbeat_config.clone(), heartbeat_nodes, &ctx)?; let inventory = start_inventory_runtime(heartbeat_config, heartbeat_nodes, inventory_drives, store, &ctx)?; + let table_maintenance = + maintenance_config.map(|config| crate::table_catalog::PeriodicMaintenanceRuntime::spawn(config, server_ctx, &ctx)); + Ok(StartupServiceRuntime { + table_maintenance, oidc_recovery, optional_runtimes, heartbeat, diff --git a/rustfs/src/startup_shutdown.rs b/rustfs/src/startup_shutdown.rs index 3234eca22..7a863e24b 100644 --- a/rustfs/src/startup_shutdown.rs +++ b/rustfs/src/startup_shutdown.rs @@ -206,10 +206,10 @@ fn background_shutdown_steps(enable_scanner: bool, enable_heal: bool) -> Vec( - heal_handoff_complete: bool, + background_handoff_complete: bool, detached_mutation_timeout: std::time::Duration, wait_for_detached_mutations: W, clear_unclean_shutdown_markers: ClearFut, @@ -233,22 +233,31 @@ where "Detached mutations did not finish before shutdown; retaining unclean-shutdown markers" ); } - if !(heal_handoff_complete && detached_mutations_drained) { + if !(background_handoff_complete && detached_mutations_drained) { return false; } clear_unclean_shutdown_markers.await; true } +pub(crate) struct StartupBackgroundTasks { + pub(crate) capacity: Option, + pub(crate) table_maintenance: Option, +} + pub(crate) async fn run_startup_shutdown_sequence( state_manager: &ServiceStateManager, shutdown_signal: ShutdownSignal, s3_shutdown_handle: Option, console_shutdown_handle: Option, optional_runtimes: OptionalRuntimeServices, - capacity_tasks: Option, + background_tasks: StartupBackgroundTasks, ctx: CancellationToken, ) { + let StartupBackgroundTasks { + capacity: capacity_tasks, + table_maintenance, + } = background_tasks; ctx.cancel(); // Stop long-lived peer/disk background monitors while the runtime and tracing @@ -463,10 +472,14 @@ pub(crate) async fn run_startup_shutdown_sequence( console_shutdown_handle.shutdown().await; } shutdown_optional_runtime_services(optional_runtime_shutdowns).await; + let maintenance_drained = match table_maintenance { + Some(runtime) => runtime.shutdown(SHUTDOWN_TIMEOUT).await, + None => true, + }; // FTP, SFTP and WebDAV can start deletes too: wait for detached mutations // only after every front end has stopped. finish_detached_mutations_and_clear_unclean_shutdown_markers( - heal_handoff_complete, + heal_handoff_complete && maintenance_drained, DETACHED_MUTATION_SHUTDOWN_TIMEOUT, wait_for_detached_mutations, rustfs_heal::heal::clear_unclean_shutdown_markers(), diff --git a/rustfs/src/storage/access.rs b/rustfs/src/storage/access.rs index b601c9333..35cf01aee 100644 --- a/rustfs/src/storage/access.rs +++ b/rustfs/src/storage/access.rs @@ -16,11 +16,11 @@ use super::ObjectOptions; use super::ecfs::FS; use super::{ECStore, PolicySys, ReplicationStatusType, StorageError, get_lock_acquire_timeout, is_err_bucket_not_found}; use crate::auth::{ - AuthType, OBJECT_LOCK_REMAINING_RETENTION_DAYS_CONDITION, RUSTFS_MAX_CONTENT_LENGTH_QUERY, - RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest, VerifiedSigV4Request, check_key_valid_with_context, - get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, get_request_auth_type_with_query, - get_session_token, parse_presigned_multipart_max_total_object_size, parse_presigned_put_max_content_length, - reject_unsigned_amz_headers_on_sigv4_request, + AuthType, OBJECT_CREATION_OPERATION_CONDITION, OBJECT_LOCK_REMAINING_RETENTION_DAYS_CONDITION, + RUSTFS_MAX_CONTENT_LENGTH_QUERY, RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest, VerifiedSigV4Request, + check_key_valid_with_context, get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, + get_request_auth_type_with_query, get_session_token, parse_presigned_multipart_max_total_object_size, + parse_presigned_put_max_content_length, reject_unsigned_amz_headers_on_sigv4_request, }; use crate::error::{ApiError, normalize_put_read_quorum_error}; use crate::license::license_check; @@ -143,6 +143,17 @@ struct RequestedObjectLockRetainUntil(OffsetDateTime); #[derive(Clone, Debug)] struct RequestedObjectLockRetention(ObjectLockRetention); +/// These operations stage upload state without publishing an object. Keep this +/// separate from the marker controlling multipart publication fences. +#[derive(Clone, Copy, Debug)] +pub(crate) struct MultipartStagingOperation; + +impl MultipartStagingOperation { + fn for_operation(operation: &str) -> Option { + matches!(operation, "CreateMultipartUpload" | "UploadPart" | "UploadPartCopy").then_some(Self) + } +} + #[derive(Clone, Debug)] struct StagedMultipartPartAuthorization; @@ -887,8 +898,15 @@ fn authorization_conditions( req.uri.query(), client_info, ); - if req.extensions.get::().is_some() { + let internal_object_authorization = req.extensions.get::().is_some(); + if internal_object_authorization { retain_internal_object_authorization_conditions(&mut conditions, cred); + } + if action == Action::S3Action(S3Action::PutObjectAction) { + let creates_object = internal_object_authorization || req.extensions.get::().is_none(); + conditions.insert(OBJECT_CREATION_OPERATION_CONDITION.to_string(), vec![creates_object.to_string()]); + } + if internal_object_authorization { return Ok(conditions); } merge_list_bucket_query_conditions(action, req.uri.query(), &mut conditions); @@ -1749,11 +1767,7 @@ fn table_catalog_store_for_data_plane( } fn table_publication_guard_error(err: crate::table_catalog::TableCatalogStoreError) -> S3Error { - let code = match &err { - crate::table_catalog::TableCatalogStoreError::Unavailable(_) => S3ErrorCode::ServiceUnavailable, - _ => S3ErrorCode::InternalError, - }; - S3Error::with_message(code, format!("failed to acquire table publication guard: {err}")) + S3Error::from(ApiError::from(err)) } async fn retain_table_data_plane_publication_guard( @@ -1889,8 +1903,8 @@ fn table_list_cursor_key(req: &S3Request) -> S3Result<[u8; 32]> { } fn table_list_catalog_error(err: crate::table_catalog::TableCatalogStoreError) -> S3Error { - if matches!(err, crate::table_catalog::TableCatalogStoreError::Unavailable(_)) { - S3Error::from(ApiError::service_unavailable()) + if err.is_unavailable() { + S3Error::from(ApiError::from(err)) } else { S3Error::from(ApiError::access_denied()) } @@ -2032,11 +2046,7 @@ async fn table_data_plane_resource_for_request( error = %err, "failed to resolve table data-plane resource" ); - if matches!(err, crate::table_catalog::TableCatalogStoreError::Unavailable(_)) { - S3Error::from(ApiError::service_unavailable()) - } else { - s3_error!(AccessDenied, "Access Denied") - } + table_list_catalog_error(err) })?; let resource = require_owned_reserved_table_object(object, resource)?; let bucket_fence_key = (bucket.to_string(), crate::table_catalog::default_table_bucket_publication_lock_path()); @@ -2224,12 +2234,16 @@ impl S3Access for FS { let auth_type = get_request_auth_type_with_query(cx.headers(), cx.uri().query()); let verified_presigned = matches!(auth_type, AuthType::Presigned); let verified_sigv4 = matches!(auth_type, AuthType::Presigned | AuthType::Signed); + let multipart_staging_operation = MultipartStagingOperation::for_operation(cx.s3_op().name()); // Publish this server's context slot so downstream data-plane handlers // resolve the same store (backlog#1052 S6). { let ext = cx.extensions_mut(); ext.insert(self.server_ctx().clone()); ext.insert(req_info); + if let Some(operation) = multipart_staging_operation { + ext.insert(operation); + } if verified_presigned { ext.insert(VerifiedPresignedRequest); } @@ -3827,7 +3841,7 @@ mod tests { "", ] { let error = TableCatalogStoreError::Unavailable(message.to_string()); - let expected_message = format!("failed to acquire table publication guard: {error}"); + let expected_message = ApiError::error_code_to_message(&S3ErrorCode::ServiceUnavailable); let err = table_publication_guard_error(error); assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable, "{message}"); assert_eq!( @@ -3835,6 +3849,8 @@ mod tests { Some(http::StatusCode::SERVICE_UNAVAILABLE) ); assert_eq!(err.message(), Some(expected_message.as_str())); + let response = err.to_http_response().expect("serialize catalog failure"); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); } } @@ -3853,7 +3869,7 @@ mod tests { TableCatalogStoreError::AlreadyExists(message.clone()), TableCatalogStoreError::Conflict(message), ] { - let expected_message = format!("failed to acquire table publication guard: {error}"); + let expected_message = ApiError::error_code_to_message(&S3ErrorCode::InternalError); let err = table_publication_guard_error(error); assert_eq!(err.code(), &S3ErrorCode::InternalError); assert_eq!( @@ -3861,13 +3877,16 @@ mod tests { Some(http::StatusCode::INTERNAL_SERVER_ERROR) ); assert_eq!(err.message(), Some(expected_message.as_str())); + let response = err.to_http_response().expect("serialize catalog failure"); + assert!(!response.headers().contains_key(http::header::RETRY_AFTER)); } } #[tokio::test] #[serial] - async fn table_publication_guard_timeout_blocks_put_and_multipart_authorization() { + async fn table_publication_guard_timeout_blocks_put_delete_and_multipart_authorization() { use crate::storage::storage_api::contract::namespace::NamespaceLocking as _; + use http_body_util::BodyExt as _; use std::time::Duration; let store = crate::app::gating_test_env::shared_gating_ecstore().await; @@ -3880,7 +3899,7 @@ mod tests { .await .expect("test bucket should be created"); let policy_json = format!( - r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:PutObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}}]}}"# + r#"{{"Version":"2012-10-17","Statement":[{{"Effect":"Allow","Principal":{{"AWS":"*"}},"Action":["s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::{bucket}/*"]}}]}}"# ); let mut metadata = (*crate::storage::get_bucket_metadata(&bucket) .await @@ -3916,6 +3935,16 @@ mod tests { .expect("multipart input should build"), Method::POST, ); + let mut delete_req = build_request( + DeleteObjectInput::builder() + .bucket(bucket.clone()) + .key("object".to_string()) + .build() + .expect("DELETE input should build"), + Method::DELETE, + ); + ensure_req_info(&mut delete_req); + delete_req.extensions.insert(fs.server_ctx().clone()); ensure_req_info(&mut put_req); ensure_req_info(&mut multipart_req); put_req.extensions.insert(fs.server_ctx().clone()); @@ -3928,6 +3957,12 @@ mod tests { .await .expect_err("PUT must wait for the publication writer"), ), + ( + "DeleteObject", + fs.delete_object(&mut delete_req) + .await + .expect_err("DELETE must wait for the publication writer"), + ), ( "CreateMultipartUpload", fs.create_multipart_upload(&mut multipart_req) @@ -3936,8 +3971,24 @@ mod tests { ), ] { assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable, "{operation}: {err}"); + let source = err.source().expect("retain the typed publication lock failure"); + let storage = source + .downcast_ref::() + .expect("use the canonical storage error boundary"); + assert!(matches!( + storage, + crate::storage_api::error::StorageError::Lock(rustfs_lock::LockError::Timeout { .. }) + )); + let response = err.to_http_response().expect("serialize publication failure"); + assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[http::header::RETRY_AFTER], "1"); + let body = response.into_body().collect().await.expect("read error XML").to_bytes(); + let body = String::from_utf8(body.to_vec()).expect("S3 XML is UTF-8"); + assert!(body.contains("ServiceUnavailable")); + assert!(!body.contains(&bucket)); + assert!(!body.contains(&lock_object)); } - for extensions in [&put_req.extensions, &multipart_req.extensions] { + for extensions in [&put_req.extensions, &delete_req.extensions, &multipart_req.extensions] { assert!(extensions.get::().is_none()); } @@ -3949,7 +4000,10 @@ mod tests { fs.create_multipart_upload(&mut multipart_req) .await .expect("multipart initialization may retry after publication finishes"); - for extensions in [&put_req.extensions, &multipart_req.extensions] { + fs.delete_object(&mut delete_req) + .await + .expect("DELETE may retry after publication finishes"); + for extensions in [&put_req.extensions, &delete_req.extensions, &multipart_req.extensions] { let retained = extensions .get::() .expect("successful admission retains the guard"); @@ -3960,6 +4014,7 @@ mod tests { } drop(put_req); drop(multipart_req); + drop(delete_req); lock.get_write_lock(Duration::from_secs(1)) .await .expect("dropping the request must release its publication guard"); @@ -4376,6 +4431,170 @@ mod tests { } } + #[test] + fn object_creation_operation_classifies_only_multipart_staging() { + for (operation, staging) in [ + ("CreateMultipartUpload", true), + ("UploadPart", true), + ("UploadPartCopy", true), + ("PutObject", false), + ("CopyObject", false), + ("CompleteMultipartUpload", false), + ("PostObject", false), + ("GetObject", false), + ("InternalObjectWrite", false), + ] { + assert_eq!( + super::MultipartStagingOperation::for_operation(operation).is_some(), + staging, + "{operation}" + ); + } + } + + #[test] + fn object_creation_operation_ignores_request_and_claim_spoofing() { + use crate::auth::{OBJECT_CREATION_OPERATION_CONDITION, get_condition_values}; + for forged in ["false", "true"] { + for claim in [ + "ObjectCreationOperation", + "objectcreationoperation", + "OBJECTCREATIONOPERATION", + "ldapObjectCreationOperation", + "ldapldapObjectCreationOperation", + ] { + let credentials = rustfs_credentials::Credentials { + claims: Some(HashMap::from([(claim.to_string(), serde_json::json!(forged))])), + ..Default::default() + }; + let mut req = build_request((), Method::GET); + req.uri = format!("/bucket/key?ObjectCreationOperation={forged}&uploadId=fake&partNumber=1") + .parse() + .expect("spoofed query should parse"); + req.headers + .insert("objectcreationoperation", HeaderValue::from_str(forged).expect("header should parse")); + assert!( + !get_condition_values(&req.headers, &credentials, None, None, None) + .keys() + .any(|key| key.eq_ignore_ascii_case(OBJECT_CREATION_OPERATION_CONDITION)) + ); + for staging in [false, true] { + if staging { + req.extensions.insert(super::MultipartStagingOperation); + } + for internal in [false, true] { + if internal { + req.extensions.insert(InternalObjectAuthorization); + } else { + req.extensions.remove::(); + } + for action in [ + S3Action::PutObjectAction, + S3Action::GetObjectAction, + S3Action::GetObjectVersionAction, + ] { + let conditions = + authorization_conditions(&req, &credentials, None, None, None, None, Action::S3Action(action)) + .expect("conditions should build"); + let values = conditions + .iter() + .filter(|(key, _)| key.eq_ignore_ascii_case(OBJECT_CREATION_OPERATION_CONDITION)) + .collect::>(); + if action == S3Action::PutObjectAction { + assert_eq!(values.len(), 1, "one canonical server-derived value"); + assert_eq!(values[0].0, OBJECT_CREATION_OPERATION_CONDITION); + assert_eq!( + values[0].1, + &vec![(internal || !staging).to_string()], + "{claim}, internal={internal}, staging={staging}" + ); + } else { + assert!(values.is_empty(), "non-PutObject action {action:?} must have no creation condition"); + } + } + } + } + } + } + } + + #[tokio::test] + async fn object_creation_operation_controls_identity_and_bucket_policies() { + use rustfs_policy::policy::{Args, Policy}; + for effect in ["Allow", "Deny"] { + for expected in [false, true] { + let mut statements = vec![serde_json::json!({ + "Effect": effect, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::bucket/*", + "Condition": { "Bool": { "s3:ObjectCreationOperation": expected.to_string() } } + })]; + if effect == "Deny" { + statements.push(serde_json::json!({ + "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::bucket/*" + })); + } + let identity: Policy = serde_json::from_str( + &serde_json::json!({ + "Version": "2012-10-17", "Statement": statements + }) + .to_string(), + ) + .expect("identity policy should accept the AWS Bool key"); + for statement in &mut statements { + statement["Principal"] = serde_json::json!("*"); + } + let bucket: BucketPolicy = serde_json::from_str( + &serde_json::json!({ + "Version": "2012-10-17", "Statement": statements + }) + .to_string(), + ) + .expect("bucket policy should accept the AWS Bool key"); + for actual in [None, Some(false), Some(true)] { + let conditions = actual + .map(|value| HashMap::from([("ObjectCreationOperation".to_string(), vec![value.to_string()])])) + .unwrap_or_default(); + let allowed = if effect == "Allow" { + actual == Some(expected) + } else { + actual != Some(expected) + }; + assert_eq!( + identity + .is_allowed(&Args { + account: "writer", + groups: &None, + action: Action::S3Action(S3Action::PutObjectAction), + bucket: "bucket", + object: "object", + conditions: &conditions, + is_owner: false, + claims: &HashMap::new(), + deny_only: false, + }) + .await, + allowed, + "identity {effect}, expected={expected}, actual={actual:?}" + ); + assert_eq!( + bucket + .is_allowed(&BucketPolicyArgs { + account: "writer", + groups: &None, + action: Action::S3Action(S3Action::PutObjectAction), + bucket: "bucket", + object: "object", + conditions: &conditions, + is_owner: false, + }) + .await, + allowed, + "bucket {effect}, expected={expected}, actual={actual:?}" + ); + } + } + } + } + fn remaining_retention_days_conditions(req: &S3Request, action: Action) -> HashMap> { let credentials = rustfs_credentials::Credentials::default(); authorization_conditions(req, &credentials, None, None, None, None, action).expect("conditions should build") diff --git a/rustfs/src/storage/helper.rs b/rustfs/src/storage/helper.rs index b8eb3c15d..805b5a870 100644 --- a/rustfs/src/storage/helper.rs +++ b/rustfs/src/storage/helper.rs @@ -13,7 +13,7 @@ // limitations under the License. use crate::module_switches::{is_audit_module_enabled, is_notify_module_enabled}; -use crate::shared_types::convert_ecstore_object_info; +use crate::shared_types::{RemoteAddr, convert_ecstore_object_info}; use crate::storage::access::{ReqInfo, request_context_from_req}; use crate::storage::request_context::RequestContext; use crate::storage::sse::KmsRequestAuditScope; @@ -33,6 +33,7 @@ use rustfs_s3_types::EventName; use rustfs_targets::{ extract_params_header, extract_req_params, extract_resp_elements, get_request_host, get_request_port, get_request_user_agent, }; +use rustfs_trusted_proxies::ClientInfo; use rustfs_utils::http::headers::AMZ_REQUEST_ID; use s3s::{S3Request, S3Response, S3Result}; use serde_json::Value; @@ -150,15 +151,6 @@ impl OperationHelper { .filter(|value| !value.is_empty()) .unwrap_or(path_object_key); - // Infer remote address - let remote_host = req - .headers - .get(rustfs_utils::http::X_FORWARDED_FOR) - .and_then(|v| v.to_str().ok()) - .or_else(|| req.headers.get(rustfs_utils::http::X_REAL_IP).and_then(|v| v.to_str().ok())) - .unwrap_or("") - .to_string(); - let trigger = op.as_str(); // Initialize audit builder @@ -179,15 +171,30 @@ impl OperationHelper { let request_id = request_context.request_id.clone(); let audit_builder = if audit_enabled { - Some( - AuditEntryBuilder::new("1.0", event, trigger, api_builder.clone().build()) - .remote_host(remote_host) - .user_agent(get_request_user_agent(&req.headers)) - .req_host(get_request_host(&req.headers)) - .req_path(req.uri.path().to_string()) - .req_query(extract_req_params(req)) - .request_id(&request_id), - ) + let remote_host = req + .extensions + .get::() + .map(|info| info.real_ip.to_string()) + .or_else(|| { + req.extensions + .get::>() + .and_then(|addr| addr.map(|addr| addr.0.ip().to_string())) + }) + .unwrap_or_default(); + let mut builder = AuditEntryBuilder::new("1.0", event, trigger, api_builder.clone().build()) + .remote_host(remote_host) + .user_agent(get_request_user_agent(&req.headers)) + .req_host(get_request_host(&req.headers)) + .req_path(req.uri.path().to_string()) + .req_query(extract_req_params(req)) + .request_id(&request_id); + if let Some(cred) = req_info + .and_then(|info| info.cred.as_ref()) + .filter(|cred| !cred.access_key.is_empty()) + { + builder = builder.access_key(&cred.access_key); + } + Some(builder) } else { None }; @@ -346,10 +353,6 @@ impl OperationHelper { final_builder = final_builder.error(err); } - if let Some(cred) = runtime_sources::current_action_credentials() { - final_builder = final_builder.access_key(&cred.access_key); - } - // Inject OpenTelemetry trace context into audit tags for distributed tracing correlation let mut tags = HashMap::new(); if let Some(ref tid) = state.request_context.trace_id { @@ -572,6 +575,133 @@ mod tests { ); } + #[test] + fn operation_helper_audits_the_authenticated_request_identity() { + with_vars( + [ + (rustfs_config::ENV_NOTIFY_ENABLE, Some("false")), + (rustfs_config::ENV_AUDIT_ENABLE, Some("true")), + ], + || { + refresh_notify_module_enabled(); + refresh_audit_module_enabled(); + + for (access_key, parent_user) in [ + (Some("testuser"), ""), + (Some("rustfsadmin"), ""), + (Some("service-account"), "testuser"), + (Some("sts-session"), "testuser"), + (None, ""), + (Some(""), ""), + ] { + let expected_access_key = access_key.filter(|key| !key.is_empty()); + for outcome in [None, Some(true), Some(false)] { + let input = GetObjectInput::builder() + .bucket("audit-bucket".to_string()) + .key("object.txt".to_string()) + .build() + .expect("get object input should build"); + let mut req = build_request(input, Method::GET, Uri::from_static("/audit-bucket/object.txt")); + req.extensions.insert(ReqInfo { + cred: access_key.map(|key| Credentials { + access_key: key.to_string(), + secret_key: "audit-secret-must-not-leak".to_string(), + session_token: "audit-token-must-not-leak".to_string(), + parent_user: parent_user.to_string(), + ..Default::default() + }), + ..Default::default() + }); + let mut helper = OperationHelper::new(&req, EventName::ObjectAccessedGet, S3Operation::GetObject); + if let Some(success) = outcome { + let result = if success { + Ok(S3Response::new(GetObjectOutput::default())) + } else { + Err(S3Error::new(S3ErrorCode::NoSuchKey)) + }; + helper = helper.complete(&result); + } + let OperationHelper::Enabled(state) = &mut helper else { + panic!("helper should be enabled when the audit switch is on"); + }; + let entry = state.audit_builder.take().expect("audit builder should exist").build(); + assert_eq!(entry.access_key.as_deref(), expected_access_key); + let json = serde_json::to_value(&entry).expect("audit entry should serialize"); + assert_eq!(json.get("accessKey").and_then(|key| key.as_str()), expected_access_key); + let rendered = serde_json::to_string(&entry).expect("audit entry should serialize"); + assert!(!rendered.contains("audit-secret-must-not-leak")); + assert!(!rendered.contains("audit-token-must-not-leak")); + } + } + }, + ); + } + + #[test] + fn operation_helper_audits_verified_client_addresses() { + use crate::shared_types::RemoteAddr; + use rustfs_trusted_proxies::{ClientInfo, ValidationMode}; + use std::net::SocketAddr; + + with_vars( + [ + (rustfs_config::ENV_NOTIFY_ENABLE, Some("false")), + (rustfs_config::ENV_AUDIT_ENABLE, Some("true")), + ], + || { + refresh_notify_module_enabled(); + refresh_audit_module_enabled(); + + let peer: SocketAddr = "192.0.2.10:54321".parse().unwrap(); + let ipv6_peer: SocketAddr = "[2001:db8::10]:54321".parse().unwrap(); + let proxied = ClientInfo::from_trusted_proxy( + "198.51.100.20".parse().unwrap(), + None, + None, + peer.ip(), + 1, + ValidationMode::Strict, + Vec::new(), + ); + for (remote_addr, client_info, expected) in [ + (Some(RemoteAddr(peer)), None, "192.0.2.10"), + (Some(RemoteAddr(ipv6_peer)), None, "2001:db8::10"), + (Some(RemoteAddr(peer)), Some(proxied), "198.51.100.20"), + (Some(RemoteAddr(peer)), Some(ClientInfo::direct(peer)), "192.0.2.10"), + (None, None, ""), + ] { + for spoofed in [false, true] { + let input = GetObjectInput::builder() + .bucket("audit-bucket".to_string()) + .key("object.txt".to_string()) + .build() + .expect("get object input should build"); + let mut req = build_request(input, Method::GET, Uri::from_static("/audit-bucket/object.txt")); + req.extensions.insert(remote_addr); + if let Some(client_info) = client_info.as_ref() { + req.extensions.insert(client_info.clone()); + } + if spoofed { + req.headers + .insert("x-forwarded-for", HeaderValue::from_static("203.0.113.99")); + req.headers.insert("x-real-ip", HeaderValue::from_static("203.0.113.98")); + } + let result = Ok(S3Response::new(GetObjectOutput::default())); + let mut helper = + OperationHelper::new(&req, EventName::ObjectAccessedGet, S3Operation::GetObject).complete(&result); + let OperationHelper::Enabled(state) = &mut helper else { + panic!("helper should be enabled when the audit switch is on"); + }; + let entry = state.audit_builder.take().expect("audit builder should exist").build(); + assert_eq!(entry.remote_host.as_deref(), Some(expected)); + let rendered = serde_json::to_value(&entry).expect("audit entry should serialize"); + assert_eq!(rendered["remotehost"], expected); + } + } + }, + ); + } + #[test] fn operation_helper_adds_objects_to_audit_details() { with_vars( diff --git a/rustfs/src/storage/rpc/node_service.rs b/rustfs/src/storage/rpc/node_service.rs index 255bf1044..318b38158 100644 --- a/rustfs/src/storage/rpc/node_service.rs +++ b/rustfs/src/storage/rpc/node_service.rs @@ -591,6 +591,14 @@ pub struct NodeService { local_peer: LocalPeerS3Client, context: Option>, server_ctx: Option>, + quota_topology_cache: std::sync::Mutex>, + #[cfg(test)] + quota_topology_computations: std::sync::atomic::AtomicUsize, +} + +struct QuotaTopologyCacheEntry { + store: std::sync::Weak, + fingerprint: String, } enum LocalMutationTarget { @@ -710,6 +718,9 @@ pub fn make_server_for_context(context: Option> local_peer, context, server_ctx: None, + quota_topology_cache: std::sync::Mutex::new(None), + #[cfg(test)] + quota_topology_computations: std::sync::atomic::AtomicUsize::new(0), } } @@ -1711,7 +1722,7 @@ impl Node for NodeService { if !request.get_ref().bucket_incarnation_id.is_empty() { return Err(Status::invalid_argument("incarnation-bound rename requires RenameDataAtIncarnation")); } - self.handle_rename_data(request).await + self.handle_rename_data(request, false).await } async fn rename_data_at_incarnation( @@ -1726,7 +1737,14 @@ impl Node for NodeService { { return Err(Status::invalid_argument("bucket incarnation must be a non-nil UUID")); } - self.handle_rename_data(request).await + self.handle_rename_data(request, false).await + } + + async fn rename_data_with_quota_fence( + &self, + request: Request, + ) -> Result, Status> { + self.handle_rename_data(request, true).await } async fn make_volumes(&self, request: Request) -> Result, Status> { @@ -1752,7 +1770,13 @@ impl Node for NodeService { &self, request: Request, ) -> Result, Status> { - self.handle_acquire_snapshot_lease(request).await + self.handle_acquire_snapshot_lease(request, false).await + } + async fn acquire_quota_mutation_fence( + &self, + request: Request, + ) -> Result, Status> { + self.handle_acquire_snapshot_lease(request, true).await } async fn renew_snapshot_lease( &self, @@ -1764,7 +1788,13 @@ impl Node for NodeService { &self, request: Request, ) -> Result, Status> { - self.handle_release_snapshot_lease(request).await + self.handle_release_snapshot_lease(request, false).await + } + async fn release_quota_mutation_fence( + &self, + request: Request, + ) -> Result, Status> { + self.handle_release_snapshot_lease(request, true).await } async fn read_metadata(&self, request: Request) -> Result, Status> { self.handle_read_metadata(request).await @@ -1848,6 +1878,13 @@ impl Node for NodeService { self.handle_delete_version(request, true).await } + async fn guarded_delete_versions( + &self, + request: Request, + ) -> Result, Status> { + self.handle_guarded_delete_versions(request).await + } + async fn delete_versions(&self, request: Request) -> Result, Status> { self.handle_delete_versions(request).await } @@ -2762,6 +2799,39 @@ impl Node for NodeService { } } + async fn peer_drive_admission( + &self, + request: Request, + ) -> Result, Status> { + let host = request.into_inner().host; + match crate::server::observe_peer_drive_admission(&host).await { + Ok(drives) => Ok(Response::new(PeerDriveAdmissionResponse { + success: true, + drives: drives + .into_iter() + .map(|drive| PeerDriveAdmission { + pool_index: drive.pool_index, + set_index: drive.set_index, + disk_index: drive.disk_index, + online: drive.online, + runtime_state: drive.runtime_state, + host_online: drive.host_online, + }) + .collect(), + error_info: None, + error_code: None, + })), + Err(err) => Ok(Response::new(PeerDriveAdmissionResponse { + success: false, + drives: Vec::new(), + error_info: Some(err.to_string()), + error_code: err + .not_initialized() + .then_some(ControlPlaneErrorCode::ControlPlaneErrorNotInitialized as i32), + })), + } + } + async fn get_metacache_listing( &self, _request: Request, @@ -3075,12 +3145,12 @@ mod tests { DownloadProfileDataRequest, GenerallyLockRequest, GetAllBucketStatsRequest, GetBucketInfoRequest, GetBucketStatsDataRequest, GetCpusRequest, GetMemInfoRequest, GetMetacacheListingRequest, GetMetricsRequest, GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, - GetSrMetricsDataRequest, GetSysConfigRequest, GetSysErrorsRequest, HealBucketRequest, HealControlRequest, - HealControlResponse, ListBucketRequest, ListDirRequest, ListVolumesRequest, LoadBucketMetadataRequest, LoadGroupRequest, - LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest, - LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, MakeBucketRequest, MakeVolumeRequest, - MakeVolumesRequest, Mss, PingRequest, PreparePartTransactionRequest, ReadAllRequest, ReadAtRequest, ReadMultipleRequest, - ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest, ReloadPoolMetaRequest, + GetSrMetricsDataRequest, GetSysConfigRequest, GetSysErrorsRequest, GuardedDeleteVersionsRequest, HealBucketRequest, + HealControlRequest, HealControlResponse, ListBucketRequest, ListDirRequest, ListVolumesRequest, + LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, + LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, MakeBucketRequest, + MakeVolumeRequest, MakeVolumesRequest, Mss, PingRequest, PreparePartTransactionRequest, ReadAllRequest, ReadAtRequest, + ReadMultipleRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, RenameDataRequest, RenameFileRequest, RenamePartRequest, ScannerActivityRequest, ScannerDirtyUsageSnapshotRequest, ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseRequest, ServerInfoRequest, SettlePartTransactionRequest, SignalServiceRequest, SnapshotLeaseReleaseRequest, @@ -3105,15 +3175,17 @@ mod tests { use tonic::{Request, Response, Status}; use uuid::Uuid; - const DISK_MUTATION_RPC_METHODS: [&str; 25] = [ + const DISK_MUTATION_RPC_METHODS: [&str; 29] = [ "renamedata", "renamedataatincarnation", + "renamedatawithquotafence", "writemetadataatincarnation", "deleteatincarnation", "deleteversionatincarnation", "deleteversion", "deleteretiredmarker", "deleteversions", + "guardeddeleteversions", "writemetadata", "updatemetadata", "writeall", @@ -3129,8 +3201,10 @@ mod tests { "makevolume", "makevolumes", "acquiresnapshotlease", + "acquirequotamutationfence", "renewsnapshotlease", "releasesnapshotlease", + "releasequotamutationfence", ]; fn normalized_rpc_method(method: &str) -> String { @@ -3209,6 +3283,13 @@ mod tests { Ok(None) } + async fn list_buckets_for_admin_heal( + &self, + _selector: Option, + ) -> rustfs_heal::Result> { + self.list_buckets().await + } + async fn list_buckets(&self) -> rustfs_heal::Result> { Ok(Vec::new()) } @@ -4289,12 +4370,56 @@ mod tests { }]) } + fn quota_two_pool_topologies() -> (EndpointServerPools, EndpointServerPools) { + let mut original = heal_control_test_endpoints("node-d").as_ref().to_vec(); + let mut second = heal_control_test_endpoints("node-e").as_ref()[0].clone(); + for (index, endpoint) in second.endpoints.as_mut().iter_mut().enumerate() { + *endpoint = Endpoint::try_from(format!("http://other-node-{index}:9000/disk{}", index + 1).as_str()) + .expect("disjoint second pool endpoint"); + endpoint.set_set_index(index / 2); + endpoint.set_disk_index(index % 2); + } + original.push(second); + let reindex = |pools: &mut Vec| { + for (index, pool) in pools.iter_mut().enumerate() { + for endpoint in pool.endpoints.as_mut() { + endpoint.set_pool_index(index); + } + } + }; + reindex(&mut original); + let mut swapped = original.clone(); + swapped.swap(0, 1); + reindex(&mut swapped); + (EndpointServerPools::from(original), EndpointServerPools::from(swapped)) + } + + #[test] + fn quota_topology_fingerprint_binds_pool_order_after_reindexing() { + let (original, swapped) = quota_two_pool_topologies(); + assert_ne!( + heal_topology_fingerprint(&original).expect("valid original pools"), + heal_topology_fingerprint(&swapped).expect("valid reordered pools"), + "different pool zero lock owners must not share a quota coordinator identity" + ); + } + fn mark_v2_authenticated(request: &mut Request) { request .metadata_mut() .insert("x-rustfs-rpc-auth-version", "2".parse().expect("valid metadata value")); } + fn sign_quota_test_request(request: &mut Request, body: &[u8], topology: &str) { + request.metadata_mut().insert( + rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER, + topology.parse().expect("quota topology metadata"), + ); + let body = rustfs_protos::canonical_quota_mutation_request_body(topology, body).expect("quota topology envelope"); + set_tonic_canonical_body_digest(request, &body).expect("quota request digest"); + mark_v2_authenticated(request); + } + fn signed_tier_prepare_request(mutation_id: uuid::Uuid, canonical_payload: Bytes) -> Request { let mut request = Request::new(TierMutationPrepareRequest { version: rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION, @@ -4497,6 +4622,98 @@ mod tests { } } + #[tokio::test] + async fn orphan_data_dir_probe_requires_body_bound_authentication() { + let service = make_server(); + let operation = crate::storage::storage_api::DeleteRecoveryRpcOperation::InspectOrphanDataDir { + path: format!("object/{}", uuid::Uuid::new_v4()), + }; + let message = RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode orphan probe").into(), + }; + let error = service + .recover_delete_transaction(Request::new(message.clone())) + .await + .expect_err("unsigned probe must fail"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + let mut tampered = Request::new(message.clone()); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(tampered.get_ref()).expect("canonical probe"); + set_tonic_canonical_body_digest(&mut tampered, &body).expect("probe digest"); + mark_v2_authenticated(&mut tampered); + tampered.get_mut().volume = "other-bucket".into(); + let error = service + .recover_delete_transaction(tampered) + .await + .expect_err("bucket scope must be body-bound"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + let mut signed = Request::new(message); + set_tonic_canonical_body_digest(&mut signed, &body).expect("probe digest"); + mark_v2_authenticated(&mut signed); + let result = service + .recover_delete_transaction(signed) + .await + .expect("signed probe reaches disk lookup") + .into_inner(); + assert!(!result.success); + assert!(result.error.is_some()); + } + + #[tokio::test] + async fn delete_recovery_exists_requires_body_bound_authentication() { + use crate::storage::storage_api::DeleteRecoveryRpcOperation; + let service = make_server(); + let operation = DeleteRecoveryRpcOperation::Exists { + object: "object".to_owned(), + }; + let message = RecoverDeleteTransactionRequest { + disk: "unknown".into(), + volume: "bucket".into(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&operation).expect("encode existence probe").into(), + }; + let error = service + .recover_delete_transaction(Request::new(message.clone())) + .await + .expect_err("unsigned existence probe must fail"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + for change_object in [false, true] { + let mut request = Request::new(message.clone()); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical existence probe"); + set_tonic_canonical_body_digest(&mut request, &body).expect("existence digest"); + mark_v2_authenticated(&mut request); + if change_object { + request.get_mut().operation = rmp_serde::to_vec_named(&DeleteRecoveryRpcOperation::Exists { + object: "other-object".to_owned(), + }) + .expect("encode changed key") + .into(); + } else { + request.get_mut().volume = "other-bucket".into(); + } + let error = service + .recover_delete_transaction(request) + .await + .expect_err("bucket and exact-key scopes must be body-bound"); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + let mut request = Request::new(message); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical existence probe"); + set_tonic_canonical_body_digest(&mut request, &body).expect("existence digest"); + mark_v2_authenticated(&mut request); + let result = service + .recover_delete_transaction(request) + .await + .expect("signed probe reaches disk lookup") + .into_inner(); + assert!(!result.success, "an unavailable disk cannot establish absence"); + assert!(result.error.is_some()); + } + #[tokio::test] async fn delete_recovery_requires_digest_and_valid_protocol_payload() { let service = make_server(); @@ -4590,8 +4807,16 @@ mod tests { // lookup, so it must NOT fail with PermissionDenied. let mut ok = Request::new(msg.clone()); let body = $canonical(ok.get_ref()).expect("canonical body should encode"); - set_tonic_canonical_body_digest(&mut ok, &body).expect("digest metadata should encode"); - mark_v2_authenticated(&mut ok); + let quota_method = matches!( + stringify!($method), + "acquire_quota_mutation_fence" | "release_quota_mutation_fence" | "rename_data_with_quota_fence" + ); + if quota_method { + sign_quota_test_request(&mut ok, &body, &"a".repeat(64)); + } else { + set_tonic_canonical_body_digest(&mut ok, &body).expect("digest metadata should encode"); + mark_v2_authenticated(&mut ok); + } if let Err(status) = service.$method(ok).await { assert_ne!( status.code(), @@ -4602,8 +4827,13 @@ mod tests { // Mismatched digest: the gate must reject before any disk work. let mut bad = Request::new(msg); - set_tonic_canonical_body_digest(&mut bad, b"unrelated-canonical-body").expect("digest metadata should encode"); - mark_v2_authenticated(&mut bad); + if quota_method { + sign_quota_test_request(&mut bad, b"unrelated-canonical-body", &"a".repeat(64)); + } else { + set_tonic_canonical_body_digest(&mut bad, b"unrelated-canonical-body") + .expect("digest metadata should encode"); + mark_v2_authenticated(&mut bad); + } let err = service .$method(bad) .await @@ -4631,6 +4861,15 @@ mod tests { }, rustfs_protos::canonical_rename_data_request_body ); + assert_gated!( + rename_data_with_quota_fence, + RenameDataRequest { + disk: disk.clone(), + file_info: "{}".into(), + ..Default::default() + }, + rustfs_protos::canonical_rename_data_request_body + ); assert_gated!( delete_version, DeleteVersionRequest { @@ -4693,6 +4932,16 @@ mod tests { }, rustfs_protos::canonical_delete_request_body ); + assert_gated!( + guarded_delete_versions, + GuardedDeleteVersionsRequest { + disk: disk.clone(), + volume: "v".into(), + payload: vec![0x80].into(), + probe_only: false, + }, + rustfs_protos::canonical_guarded_delete_versions_request_body + ); assert_gated!( delete_versions, DeleteVersionsRequest { @@ -4773,6 +5022,16 @@ mod tests { }, rustfs_protos::canonical_snapshot_lease_request_body ); + assert_gated!( + acquire_quota_mutation_fence, + SnapshotLeaseRequest { + disk: disk.clone(), + volume: ".rustfs.sys".into(), + path: format!("tmp/quota-mutation-fences/{}", "a".repeat(64)), + ttl_ms: 60_000, + }, + rustfs_protos::canonical_snapshot_lease_request_body + ); assert_gated!( renew_snapshot_lease, SnapshotLeaseRenewRequest { @@ -4794,6 +5053,16 @@ mod tests { }, rustfs_protos::canonical_snapshot_lease_release_request_body ); + assert_gated!( + release_quota_mutation_fence, + SnapshotLeaseReleaseRequest { + disk: disk.clone(), + volume: ".rustfs.sys".into(), + path: format!("tmp/quota-mutation-fences/{}", "a".repeat(64)), + token: vec![0; 16].into(), + }, + rustfs_protos::canonical_snapshot_lease_release_request_body + ); assert_gated!( delete_paths, DeletePathsRequest { @@ -4892,6 +5161,254 @@ mod tests { ); } + #[tokio::test] + async fn quota_fence_handlers_reject_unsigned_requests_wrong_scopes_and_invalid_tokens() { + use crate::storage::storage_api::ecstore_disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX; + use rustfs_filemeta::FileInfo; + + let service = super::make_server_for_slot(super::ServerContextSlot::new()); + macro_rules! signed { + ($message:expr, $canonical:path) => {{ + let mut request = Request::new($message); + let body = $canonical(request.get_ref()).expect("quota request canonical body"); + sign_quota_test_request(&mut request, &body, &"a".repeat(64)); + request + }}; + } + assert_eq!( + service + .acquire_quota_mutation_fence(Request::new(SnapshotLeaseRequest::default())) + .await + .expect_err("unsigned acquire") + .code(), + tonic::Code::PermissionDenied + ); + assert_eq!( + service + .release_quota_mutation_fence(Request::new(SnapshotLeaseReleaseRequest::default())) + .await + .expect_err("unsigned release") + .code(), + tonic::Code::PermissionDenied + ); + assert_eq!( + service + .rename_data_with_quota_fence(Request::new(RenameDataRequest::default())) + .await + .expect_err("unsigned rename") + .code(), + tonic::Code::PermissionDenied + ); + + for (volume, path) in [ + ("bucket", format!("tmp/quota-mutation-fences/{}", "a".repeat(64))), + (".rustfs.sys", "tmp/quota-mutation-fences/not-a-digest".to_string()), + ] { + let acquire = SnapshotLeaseRequest { + volume: volume.into(), + path: path.clone(), + ..Default::default() + }; + let release = SnapshotLeaseReleaseRequest { + volume: volume.into(), + path, + token: vec![0; 16].into(), + ..Default::default() + }; + assert_eq!( + service + .acquire_quota_mutation_fence(signed!(acquire, rustfs_protos::canonical_snapshot_lease_request_body)) + .await + .expect_err("invalid acquire scope") + .code(), + tonic::Code::InvalidArgument + ); + assert_eq!( + service + .release_quota_mutation_fence(signed!(release, rustfs_protos::canonical_snapshot_lease_release_request_body)) + .await + .expect_err("invalid release scope") + .code(), + tonic::Code::InvalidArgument + ); + } + + for token in [None, Some("not-a-uuid".to_string()), Some(Uuid::nil().to_string())] { + let mut file_info = FileInfo::default(); + if let Some(token) = token { + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + token, + ); + } + let message = RenameDataRequest { + file_info: serde_json::to_string(&file_info).expect("file info"), + ..Default::default() + }; + assert_eq!( + service + .rename_data_with_quota_fence(signed!(message, rustfs_protos::canonical_rename_data_request_body)) + .await + .expect_err("invalid quota token") + .code(), + tonic::Code::InvalidArgument + ); + } + let mut file_info = FileInfo::default(); + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::new_v4().to_string(), + ); + file_info.metadata.insert( + format!("x-minio-internal-{QUOTA_MUTATION_FENCE_METADATA_SUFFIX}"), + Uuid::new_v4().to_string(), + ); + let conflicting = RenameDataRequest { + file_info: serde_json::to_string(&file_info).expect("file info"), + ..Default::default() + }; + assert_eq!( + service + .rename_data_with_quota_fence(signed!(conflicting, rustfs_protos::canonical_rename_data_request_body)) + .await + .expect_err("conflicting quota tokens") + .code(), + tonic::Code::InvalidArgument + ); + + let path = format!("tmp/quota-mutation-fences/{}", "a".repeat(64)); + let acquire = SnapshotLeaseRequest { + volume: ".rustfs.sys".into(), + path: path.clone(), + ..Default::default() + }; + assert_eq!( + service + .acquire_quota_mutation_fence(signed!(acquire, rustfs_protos::canonical_snapshot_lease_request_body)) + .await + .expect_err("quota acquire requires a captured ready target") + .code(), + tonic::Code::FailedPrecondition + ); + let release = SnapshotLeaseReleaseRequest { + volume: ".rustfs.sys".into(), + path, + token: vec![0; 16].into(), + ..Default::default() + }; + assert_eq!( + service + .release_quota_mutation_fence(signed!(release, rustfs_protos::canonical_snapshot_lease_release_request_body)) + .await + .expect_err("quota release requires a captured ready target") + .code(), + tonic::Code::FailedPrecondition + ); + rustfs_utils::http::metadata_compat::insert_str( + &mut file_info.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + Uuid::new_v4().to_string(), + ); + let valid = RenameDataRequest { + file_info: serde_json::to_string(&file_info).expect("file info"), + ..Default::default() + }; + assert_eq!( + service + .rename_data_with_quota_fence(signed!(valid, rustfs_protos::canonical_rename_data_request_body)) + .await + .expect_err("quota rename requires a captured ready target") + .code(), + tonic::Code::FailedPrecondition + ); + } + + #[tokio::test] + async fn quota_fence_handlers_reject_missing_duplicate_malformed_and_tampered_topology() { + let service = super::make_server_for_slot(super::ServerContextSlot::new()); + macro_rules! rejects_invalid_topology { + ($method:ident, $message:expr, $canonical:path) => {{ + for case in [ + "missing", + "duplicate", + "empty", + "short", + "uppercase", + "nonhex", + "tampered", + "v1", + ] { + let mut request = Request::new($message); + let body = $canonical(request.get_ref()).expect("quota canonical body"); + sign_quota_test_request(&mut request, &body, &"a".repeat(64)); + let header = rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER; + match case { + "missing" => { + request.metadata_mut().remove(header); + } + "duplicate" => { + request + .metadata_mut() + .append(header, "a".repeat(64).parse().expect("header")); + } + "empty" => { + request.metadata_mut().insert(header, "".parse().expect("header")); + } + "short" => { + request + .metadata_mut() + .insert(header, "a".repeat(63).parse().expect("header")); + } + "uppercase" => { + request + .metadata_mut() + .insert(header, "A".repeat(64).parse().expect("header")); + } + "nonhex" => { + request + .metadata_mut() + .insert(header, "g".repeat(64).parse().expect("header")); + } + "tampered" => { + request + .metadata_mut() + .insert(header, "b".repeat(64).parse().expect("header")); + } + "v1" => { + request + .metadata_mut() + .insert("x-rustfs-rpc-auth-version", "1".parse().expect("auth version")); + } + _ => unreachable!(), + } + assert_eq!( + service.$method(request).await.expect_err(case).code(), + tonic::Code::PermissionDenied, + "{} must reject {case} topology authentication before resolving a disk", + stringify!($method) + ); + } + }}; + } + rejects_invalid_topology!( + acquire_quota_mutation_fence, + SnapshotLeaseRequest::default(), + rustfs_protos::canonical_snapshot_lease_request_body + ); + rejects_invalid_topology!( + release_quota_mutation_fence, + SnapshotLeaseReleaseRequest::default(), + rustfs_protos::canonical_snapshot_lease_release_request_body + ); + rejects_invalid_topology!( + rename_data_with_quota_fence, + RenameDataRequest::default(), + rustfs_protos::canonical_rename_data_request_body + ); + } + #[tokio::test] async fn snapshot_lease_acquire_and_renew_handlers_fail_closed_for_missing_disk() { let service = make_server(); @@ -6205,7 +6722,7 @@ mod tests { "the standard builder must use this exact instance context" ); super::timeout(Duration::from_secs(10), async { - while env.ecstore.scanner_data_usage_publication_blocked().await { + while env.ecstore.namespace_commits_pending_for_test() { tokio::task::yield_now().await; } }) @@ -6244,6 +6761,124 @@ mod tests { request } + #[tokio::test] + async fn delete_recovery_exists_rpc_reports_exact_key_evidence_without_mutation() { + use crate::storage::storage_api::{DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply}; + let fixture = target_rpc_fixture().await; + let set = fixture + .env + .ecstore + .all_set_disks() + .into_iter() + .next() + .expect("target erasure set"); + let configured = set.disks.read().await.iter().find_map(Clone::clone).expect("configured disk"); + let service = make_server_for_context(Some(fixture.context.clone())); + let disk = service + .find_disk(&configured.endpoint().to_string()) + .await + .expect("RPC target disk"); + let volume = "evidence-rpc"; + disk.make_volume(volume).await.expect("create evidence bucket"); + let record = format!("object/{}/delete-prepared", Uuid::new_v4()); + disk.write_all(volume, &record, Bytes::from_static(b"unresolved recovery record")) + .await + .expect("write evidence"); + for (object, expected) in [("object", true), ("other", false)] { + let mut request = Request::new(RecoverDeleteTransactionRequest { + disk: configured.endpoint().to_string(), + volume: volume.to_owned(), + protocol_version: 1, + operation: rmp_serde::to_vec_named(&DeleteRecoveryRpcOperation::Exists { + object: object.to_owned(), + }) + .expect("encode existence probe") + .into(), + }); + let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()) + .expect("canonical existence probe"); + set_tonic_canonical_body_digest(&mut request, &body).expect("existence digest"); + mark_v2_authenticated(&mut request); + let response = service + .recover_delete_transaction(request) + .await + .expect("existence RPC response") + .into_inner(); + assert!(response.success, "existence RPC failed: {:?}", response.error); + assert!(matches!( + rustfs_protos::decode_delete_recovery_payload::(&response.result).expect("decode evidence reply"), + DeleteRecoveryRpcReply::Evidence { present } if present == expected + )); + } + assert_eq!( + disk.read_all(volume, &record).await.expect("read retained evidence"), + b"unresolved recovery record"[..] + ); + } + + #[tokio::test] + async fn orphan_data_dir_delete_rpc_reports_snapshot_deferral_as_pending() { + let fixture = target_rpc_fixture().await; + let set = fixture + .env + .ecstore + .all_set_disks() + .into_iter() + .next() + .expect("target erasure set"); + let configured = set.disks.read().await.iter().find_map(Clone::clone).expect("configured disk"); + let service = make_server_for_context(Some(fixture.context.clone())); + let disk = service + .find_disk(&configured.endpoint().to_string()) + .await + .expect("RPC target disk"); + let volume = "orphan-rpc"; + disk.make_volume(volume).await.expect("create orphan bucket"); + let mut info = rustfs_filemeta::FileInfo::new("object", 1, 1); + info.data_dir = Some(Uuid::new_v4()); + info.mod_time = Some(OffsetDateTime::now_utc()); + info.size = 1; + let mut metadata = rustfs_filemeta::FileMeta::default(); + metadata.add_version(info).expect("add parent version"); + disk.write_all(volume, "object/xl.meta", metadata.marshal_msg().expect("encode parent metadata").into()) + .await + .expect("write parent metadata"); + let path = format!("object/{}", Uuid::new_v4()); + disk.write_all(volume, &format!("{path}/part.1"), Bytes::from_static(b"snapshot payload")) + .await + .expect("write orphan payload"); + let token = disk.acquire_snapshot_lease(volume, &path).await.expect("acquire snapshot"); + let mut request = Request::new(DeleteRequest { + disk: configured.endpoint().to_string(), + volume: volume.into(), + path: path.clone(), + options: serde_json::to_string(&crate::storage::storage_api::DeleteOptions { + orphan_data_dir: true, + immediate: true, + ..Default::default() + }) + .expect("encode orphan options"), + ..Default::default() + }); + let body = rustfs_protos::canonical_delete_request_body(request.get_ref()).expect("canonical orphan delete"); + set_tonic_canonical_body_digest(&mut request, &body).expect("delete digest"); + mark_v2_authenticated(&mut request); + let reply = service.delete(request).await.expect("run delete handler").into_inner(); + assert!(!reply.success, "the legacy RPC must not report deferred reclamation as completed"); + let error: DiskError = reply.error.expect("pending error code").into(); + assert_eq!(error, DiskError::DiskOngoingReq); + assert_eq!( + disk.read_all(volume, &format!("{path}/part.1")) + .await + .expect("read retained snapshot"), + b"snapshot payload"[..] + ); + disk.release_snapshot_lease(volume, &path, token) + .await + .expect("release completes safe cleanup"); + assert!(matches!(disk.list_dir("", volume, &path, 1).await, Err(DiskError::FileNotFound))); + } + #[tokio::test] async fn compare_and_update_file_handler_preserves_atomic_outcomes() { use rustfs_protos::proto_gen::node_service::CompareAndUpdateFileOutcome; @@ -6355,6 +6990,251 @@ mod tests { assert!(matches!(disk.read_all(volume, &path).await, Err(DiskError::FileNotFound))); } + #[cfg(not(windows))] + #[tokio::test] + async fn guarded_delete_remote_production_handler_preserves_probe_and_typed_results() { + use crate::storage::storage_api::{ + ecstore_disk::{ + CurrentDeleteGeneration, DeleteOptions, DiskOption, FileInfoVersions, GuardedFileInfoVersions, new_disk, + }, + init_local_disks_with_instance_ctx, + }; + use std::{ + collections::BTreeMap, + path::{Path, PathBuf}, + sync::Mutex, + }; + use tower_http::trace::TraceLayer; + + fn snapshot_tree(root: &Path) -> BTreeMap>> { + let mut entries = BTreeMap::new(); + let mut pending = vec![root.to_path_buf()]; + while let Some(directory) = pending.pop() { + for entry in std::fs::read_dir(&directory).expect("read fixture directory") { + let entry = entry.expect("read fixture entry"); + let path = entry.path(); + let relative = path + .strip_prefix(root) + .expect("fixture path remains beneath root") + .to_path_buf(); + if entry.file_type().expect("fixture entry type").is_dir() { + entries.insert(relative, None); + pending.push(path); + } else { + entries.insert(relative, Some(std::fs::read(path).expect("read fixture file"))); + } + } + } + entries + } + + fn inline_info(object: &str, etag: &str) -> rustfs_filemeta::FileInfo { + let mut info = rustfs_filemeta::FileInfo::new(object, 1, 0); + info.name = object.to_owned(); + info.erasure.index = 1; + info.version_id = Some(Uuid::nil()); + info.data_dir = Some(Uuid::new_v4()); + info.mod_time = Some(OffsetDateTime::now_utc()); + info.size = 7; + info.parts = vec![rustfs_filemeta::ObjectPartInfo { + number: 1, + size: 7, + actual_size: 7, + ..Default::default() + }]; + info.data = Some(Bytes::from_static(b"payload")); + info.metadata.insert("etag".into(), etag.into()); + info.set_inline_data(); + info + } + + fn guarded_group(info: rustfs_filemeta::FileInfo) -> GuardedFileInfoVersions { + let expected_current = CurrentDeleteGeneration::from_file_info(&info).expect("capture fixture generation"); + GuardedFileInfoVersions { + group: FileInfoVersions { + name: info.name.clone(), + versions: vec![info], + ..Default::default() + }, + expected_current, + } + } + + let fixture = target_rpc_fixture().await; + let _ = rustfs_credentials::set_global_rpc_secret(Uuid::new_v4().to_string()); + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind guarded production RPC target"); + let address = listener.local_addr().expect("guarded RPC target address"); + let mut endpoints = fixture.env.endpoint_pools.as_ref()[0].endpoints.as_ref().clone(); + let mut remote_endpoint = Endpoint::try_from(format!("http://{address}{}", fixture.env.disk_paths[0].display()).as_str()) + .expect("guarded RPC endpoint"); + remote_endpoint.set_pool_index(0); + remote_endpoint.set_set_index(0); + remote_endpoint.set_disk_index(0); + remote_endpoint.is_local = true; + endpoints[0] = remote_endpoint.clone(); + let mut pool = fixture.env.endpoint_pools.as_ref()[0].clone(); + pool.endpoints = Endpoints::from(endpoints); + init_local_disks_with_instance_ctx(&fixture.instance, EndpointServerPools::from(vec![pool])) + .await + .expect("register production local disk at its RPC endpoint"); + let service = make_server_for_context(Some(fixture.context.clone())); + let local = service + .find_disk(&remote_endpoint.to_string()) + .await + .expect("registered production local disk"); + let bucket = "guarded-delete-rpc"; + local.make_volume(bucket).await.expect("create guarded fixture volume"); + let first = inline_info("matching-first", "etag-first"); + let stale = inline_info("stale-middle", "etag-old"); + let last = inline_info("matching-last", "etag-last"); + for info in [&first, &stale, &last] { + local + .write_metadata(bucket, bucket, &info.name, info.clone()) + .await + .expect("write fixture object metadata"); + } + for info in [&first, &stale, &last] { + let published = local + .read_version( + bucket, + bucket, + &info.name, + "", + &crate::storage::storage_api::ecstore_disk::ReadOptions::default(), + ) + .await + .expect("read published fixture generation"); + assert_eq!( + CurrentDeleteGeneration::from_file_info(&published).expect("capture published fixture generation"), + CurrentDeleteGeneration::from_file_info(info).expect("capture original fixture generation"), + "the fixture must capture the exact generation published to xl.meta" + ); + } + let mut replacement = stale.clone(); + replacement.data_dir = Some(Uuid::new_v4()); + replacement.mod_time = stale.mod_time.map(|time| time + time::Duration::seconds(1)); + replacement.metadata.insert("etag".into(), "etag-replacement".into()); + local + .write_metadata(bucket, bucket, &replacement.name, replacement.clone()) + .await + .expect("commit a replacement generation"); + let disk_path = local.path(); + let volume_path = disk_path.join(bucket); + let before_probe = snapshot_tree(&disk_path); + let preserved_replacement = snapshot_tree(&volume_path.join(&stale.name)); + let calls = Arc::new(Mutex::new(Vec::::new())); + let recorded_calls = Arc::clone(&calls); + let trace = TraceLayer::new_for_grpc().on_request(move |request: &http::Request<_>, _: &tracing::Span| { + recorded_calls + .lock() + .expect("RPC method recording lock") + .push(request.uri().path().to_owned()); + }); + let (shutdown, stopped) = tokio::sync::oneshot::channel(); + let server = tokio::spawn(async move { + // This uses the production service and protobuf transport. Signature + // middleware has separate coverage; RemoteDisk still signs each body. + tonic::transport::Server::builder() + .layer(trace) + .add_service(NodeServiceServer::new(service)) + .serve_with_incoming_shutdown(TcpListenerStream::new(listener), async { + let _ = stopped.await; + }) + .await + .expect("serve guarded production RPC target"); + }); + remote_endpoint.is_local = false; + let remote = new_disk( + &remote_endpoint, + &DiskOption { + cleanup: false, + health_check: false, + }, + ) + .await + .expect("construct RemoteDisk client"); + assert!(!remote.is_local(), "the test must exercise the RemoteDisk implementation"); + remote + .supports_guarded_delete_versions() + .await + .expect("production peer supports guarded deletion"); + assert_eq!( + snapshot_tree(&disk_path), + before_probe, + "capability probing creates no metadata or recovery artifacts anywhere on the target disk" + ); + let transaction = Uuid::new_v4(); + let groups = vec![ + guarded_group(first.clone()), + guarded_group(stale.clone()), + guarded_group(last.clone()), + ]; + let payload = crate::storage::storage_api::ecstore_disk::GuardedDeleteVersionsPayload { + versions: groups.clone(), + opts: DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + }; + let encoded = rmp_serde::to_vec_named(&payload).expect("encode guarded fixture payload"); + let decoded: crate::storage::storage_api::ecstore_disk::GuardedDeleteVersionsPayload = + rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode guarded fixture payload"); + for (original, decoded) in groups.iter().zip(&decoded.versions) { + assert_eq!( + decoded.expected_current, original.expected_current, + "guarded generation must survive its RPC codec exactly" + ); + } + let errors = remote + .delete_versions_guarded_recoverable( + bucket, + groups, + DeleteOptions { + old_data_dir: Some(transaction), + ..Default::default() + }, + ) + .await; + assert_eq!( + errors, + vec![None, Some(DiskError::ConditionalDeleteConflict), None], + "typed per-item results retain the original batch indices" + ); + for info in [&first, &last] { + let object = volume_path.join(&info.name); + assert!(!object.join("xl.meta").exists(), "matching guarded objects are deleted"); + assert!( + object.join(transaction.to_string()).join("xl.meta.bkp").is_file(), + "successful guarded deletion retains its recoverable receipt" + ); + } + assert_eq!( + snapshot_tree(&volume_path.join(&stale.name)), + preserved_replacement, + "the stale guard preserves replacement metadata and creates no recovery artifacts" + ); + let deletion_methods: Vec<_> = calls + .lock() + .expect("RPC method recording lock") + .iter() + .filter(|path| path.contains("Delete")) + .cloned() + .collect(); + assert_eq!( + deletion_methods, + vec!["/node_service.NodeService/GuardedDeleteVersions"; 2], + "probe and application use the additive protocol without legacy fallback" + ); + remote.close().await.expect("close guarded RemoteDisk client"); + let _ = shutdown.send(()); + super::timeout(Duration::from_secs(10), server) + .await + .expect("guarded RPC server shuts down") + .expect("guarded RPC server task"); + } + #[cfg(not(windows))] #[tokio::test] async fn replacement_resume_selection_falls_back_to_a_remote_survivor() { @@ -6546,6 +7426,218 @@ mod tests { request } + fn target_quota_rename_request( + disk: &super::DiskStore, + fi: &rustfs_filemeta::FileInfo, + topology: &str, + ) -> Request { + let mut request = target_rename_request(disk, fi); + let body = rustfs_protos::canonical_rename_data_request_body(request.get_ref()).expect("quota rename body"); + sign_quota_test_request(&mut request, &body, topology); + request + } + + #[cfg(not(windows))] + #[tokio::test] + async fn quota_fence_handlers_commit_real_data_and_drain_before_revocation() { + use crate::storage::storage_api::{ + LocalPublicationPause, LocalPublicationStage, + ecstore_disk::{DiskAPI, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, ReadOptions}, + }; + use sha2::{Digest, Sha256}; + + let fixture = target_rpc_fixture().await; + // Mirror server startup: the disk fixture builds the store but leaves + // its write-once instance topology for the listener to initialize. + fixture.instance.set_endpoints(fixture.env.endpoint_pools.clone()); + let (disk, mut fi, _) = stage_target_rpc(&fixture).await; + let service = make_server_for_context(Some(fixture.context.clone())); + let topology = + super::heal::heal_topology_fingerprint(&fixture.env.ecstore.instance_endpoints().expect("target topology")) + .expect("target fingerprint"); + let _without_proof = crate::storage::storage_api::ecstore_notification::without_cross_pool_fence_fleet_proof_for_test(); + let path = format!( + "tmp/quota-mutation-fences/{}", + rustfs_utils::crypto::hex(Sha256::digest(b"target-rpc\0destination")) + ); + assert!(crate::storage::storage_api::ecstore_notification::acquire_cross_pool_fence_fleet_proof().is_none()); + let (_, swapped) = quota_two_pool_topologies(); + let wrong_topology = heal_topology_fingerprint(&swapped).expect("reordered pool fingerprint"); + assert_ne!(wrong_topology, topology); + let set = fixture + .env + .ecstore + .all_set_disks() + .into_iter() + .next() + .expect("quota erasure set"); + let disks = set.disks.read().await.iter().filter_map(Clone::clone).collect::>(); + assert_eq!(disks.len(), 4); + let mut successful = 0; + for participant in &disks { + let message = SnapshotLeaseRequest { + disk: participant.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: path.clone(), + ttl_ms: 60_000, + }; + let mut wrong = Request::new(message.clone()); + let body = rustfs_protos::canonical_snapshot_lease_request_body(wrong.get_ref()).expect("acquire body"); + sign_quota_test_request(&mut wrong, &body, &wrong_topology); + assert_eq!( + service + .acquire_quota_mutation_fence(wrong) + .await + .expect_err("wrong layout must not issue a token") + .code(), + tonic::Code::FailedPrecondition + ); + let mut correct = Request::new(message); + sign_quota_test_request(&mut correct, &body, &topology); + let response = service + .acquire_quota_mutation_fence(correct) + .await + .expect("same layout acquire") + .into_inner(); + assert!(response.success, "same layout participant must issue a token: {:?}", response.error); + successful += 1; + } + assert_eq!( + successful, 4, + "correct topology admits every available quorum participant without a fleet proof" + ); + let acquire_message = SnapshotLeaseRequest { + disk: disk.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: path.clone(), + ttl_ms: 60_000, + }; + let acquire_request = || { + let mut request = Request::new(acquire_message.clone()); + let body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()).expect("acquire body"); + sign_quota_test_request(&mut request, &body, &topology); + request + }; + let acquired = service + .acquire_quota_mutation_fence(acquire_request()) + .await + .expect("production quota acquire handler") + .into_inner(); + assert!(acquired.success, "real quota fence must be acquired: {:?}", acquired.error); + let token = Uuid::from_slice(&acquired.token).expect("quota token UUID"); + assert!(!token.is_nil()); + rustfs_utils::http::metadata_compat::insert_str( + &mut fi.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + token.to_string(), + ); + let mut wrong_release = Request::new(SnapshotLeaseReleaseRequest { + disk: disk.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: path.clone(), + token: vec![0; 16].into(), + }); + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(wrong_release.get_ref()).expect("release body"); + sign_quota_test_request(&mut wrong_release, &body, &wrong_topology); + assert_eq!( + service + .release_quota_mutation_fence(wrong_release) + .await + .expect_err("wrong topology must not revoke") + .code(), + tonic::Code::FailedPrecondition + ); + assert_eq!( + service + .rename_data_with_quota_fence(target_quota_rename_request(&disk, &fi, &wrong_topology)) + .await + .expect_err("wrong topology must not publish") + .code(), + tonic::Code::FailedPrecondition + ); + assert!(disk.path().join("target-rpc/staged/xl.meta").exists()); + assert!(!disk.path().join("target-rpc/destination/xl.meta").exists()); + // The following publication uses the same token: rejecting the wrong + // topology must neither revoke it nor consume the staged object. + let mut pause = + LocalPublicationPause::install(&disk, "target-rpc", "destination/xl.meta", LocalPublicationStage::PreparedRename) + .expect("pause the real quota publication"); + let mut rename = Box::pin(service.rename_data_with_quota_fence(target_quota_rename_request(&disk, &fi, &topology))); + super::timeout(Duration::from_secs(10), async { + tokio::select! { + result = &mut rename => panic!("quota rename finished before the physical pause: {result:?}"), + entered = pause.entered() => entered.expect("quota rename reached physical publication"), + } + }) + .await + .expect("bounded quota publication entry"); + + let mut request = Request::new(SnapshotLeaseReleaseRequest { + disk: disk.endpoint().to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path, + token: vec![0; 16].into(), + }); + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()).expect("release body"); + sign_quota_test_request(&mut request, &body, &topology); + let mut revoke = Box::pin(service.release_quota_mutation_fence(request)); + super::timeout(Duration::from_secs(10), async { + loop { + tokio::select! { + biased; + result = &mut revoke => panic!("revocation returned before the physical commit drained: {result:?}"), + result = service.acquire_quota_mutation_fence(acquire_request()) => { + if !result.expect("probe revoked quota fence").into_inner().success { + break; + } + tokio::task::yield_now().await; + } + } + } + }) + .await + .expect("revocation must reject new tokens while waiting for the active commit"); + drop(pause); + let (renamed, revoked) = super::timeout(Duration::from_secs(10), async { tokio::join!(rename, revoke) }) + .await + .expect("quota publication and revocation drain"); + let renamed = renamed.expect("quota rename response").into_inner(); + assert!(renamed.success, "real quota rename must commit: {:?}", renamed.error); + let revoked = revoked.expect("quota revoke response").into_inner(); + assert!(revoked.success, "real quota revoke must finish: {:?}", revoked.error); + let read = disk + .read_version( + "target-rpc", + "target-rpc", + "destination", + &fi.version_id.expect("version").to_string(), + &ReadOptions { + read_data: true, + ..Default::default() + }, + ) + .await + .expect("read quota RPC committed object"); + assert_eq!(read.data, fi.data); + assert!( + !rustfs_utils::http::metadata_compat::contains_key_str(&read.metadata, QUOTA_MUTATION_FENCE_METADATA_SUFFIX), + "the transient quota token must not persist in object metadata" + ); + disk.write_metadata("target-rpc", "target-rpc", "staged", fi.clone()) + .await + .expect("restage valid data for a revoked-token replay"); + let replay = service + .rename_data_with_quota_fence(target_quota_rename_request(&disk, &fi, &topology)) + .await + .expect("revoked token replay response") + .into_inner(); + assert!(!replay.success, "revoked quota token must not authorize another real rename"); + assert!( + disk.path().join("target-rpc/staged/xl.meta").exists(), + "rejected replay must retain its staged input" + ); + } + #[tokio::test] async fn target_slot_rejects_mismatched_and_repeated_install_before_global_publication() { let fixture = target_rpc_fixture().await; @@ -6656,7 +7748,7 @@ mod tests { before ); assert!(!disk.path().join("target-rpc/destination").exists()); - assert!(!fixture.env.ecstore.scanner_data_usage_publication_blocked().await); + assert!(!fixture.env.ecstore.namespace_commits_pending_for_test()); } #[tokio::test] @@ -6689,7 +7781,7 @@ mod tests { .expect("source remains"), before ); - assert!(!fixture.env.ecstore.scanner_data_usage_publication_blocked().await); + assert!(!fixture.env.ecstore.namespace_commits_pending_for_test()); } #[cfg(not(windows))] @@ -6916,12 +8008,12 @@ mod tests { .expect("bounded physical entry"); drop(handler); assert!( - fixture.env.ecstore.scanner_data_usage_publication_blocked().await, + fixture.env.ecstore.namespace_commits_pending_for_test(), "dropping the actual target handler must not release its physical owner" ); drop(pause); super::timeout(Duration::from_secs(10), async { - while fixture.env.ecstore.scanner_data_usage_publication_blocked().await { + while fixture.env.ecstore.namespace_commits_pending_for_test() { tokio::task::yield_now().await; } }) @@ -7002,10 +8094,10 @@ mod tests { .await .expect("bounded physical restore entry"); drop(handler); - assert!(fixture.env.ecstore.scanner_data_usage_publication_blocked().await); + assert!(fixture.env.ecstore.namespace_commits_pending_for_test()); drop(pause); super::timeout(Duration::from_secs(10), async { - while fixture.env.ecstore.scanner_data_usage_publication_blocked().await { + while fixture.env.ecstore.namespace_commits_pending_for_test() { tokio::task::yield_now().await; } }) @@ -7063,6 +8155,7 @@ mod tests { platform: "test".to_string(), }]); let instance = new_instance_ctx(); + instance.set_endpoints(pools.clone()); init_local_disks_with_instance_ctx(&instance, pools.clone()) .await .expect("register this instance's real disks"); @@ -7148,7 +8241,7 @@ mod tests { // Copy the complete internal trees, including erasure part data, // without editing disk IDs, cluster identity, epochs or pool topology. super::timeout(Duration::from_secs(10), async { - while store_b.scanner_data_usage_publication_blocked().await { + while store_b.namespace_commits_pending_for_test() { tokio::task::yield_now().await; } }) @@ -7171,7 +8264,7 @@ mod tests { } assert_eq!(internal_snapshot(root_a.path()).await, snapshot, "A must receive the complete physical snapshot"); assert_eq!(internal_snapshot(root_b.path()).await, snapshot, "B's source snapshot must remain unchanged"); - assert!(!store_b.scanner_data_usage_publication_blocked().await); + assert!(!store_b.namespace_commits_pending_for_test()); assert_eq!(store_b.scanner_namespace_mutation_generation(), snapshot_generation); let store_a = build_store(root_a.path()).await; assert_eq!( @@ -7186,8 +8279,13 @@ mod tests { pool_identity, "A must preserve the initialized cluster identity and epoch" ); - let service = make_server_for_context(Some(context(&store_a).await)); - assert!(Arc::ptr_eq(&service.resolve_object_store().expect("captured store"), &store_a)); + let slot_a = super::ServerContextSlot::new(); + slot_a.try_install(context(&store_a).await).expect("install captured instance A"); + let service = super::make_server_for_slot(slot_a); + let super::LocalMutationTarget::Ready(captured_store) = service.local_mutation_target() else { + panic!("slot A must capture a ready mutation target"); + }; + assert!(Arc::ptr_eq(&captured_store, &store_a)); assert!(Arc::ptr_eq( &crate::runtime_sources::current_object_store_handle().expect("global store"), &store_b @@ -7204,6 +8302,64 @@ mod tests { .expect("global UUID lookup must resolve B before the request"); assert!(Arc::ptr_eq(&global_disk, &disk_b)); + let topology_a = heal_topology_fingerprint(&store_a.instance_endpoints().expect("A layout")).expect("A identity"); + let topology_b = heal_topology_fingerprint(&store_b.instance_endpoints().expect("B layout")).expect("B identity"); + assert_ne!(topology_a, topology_b, "the captured and ambient layouts use different physical paths"); + let target_a = super::LocalMutationTarget::Ready(store_a.clone()); + let target_b = super::LocalMutationTarget::Ready(store_b.clone()); + for _ in 0..4 { + assert!(Arc::ptr_eq(service.quota_store_for_topology(&target_a, &topology_a).expect("cached A"), &store_a)); + } + assert_eq!(service.quota_topology_computations.load(std::sync::atomic::Ordering::Relaxed), 1, + "repeated fanout requests for one captured instance must hash its topology only once"); + assert!(Arc::ptr_eq(service.quota_store_for_topology(&target_b, &topology_b).expect("replacement B"), &store_b)); + assert_eq!(service.quota_topology_computations.load(std::sync::atomic::Ordering::Relaxed), 2, + "a different Arc with the same deployment identity must replace the cache entry"); + assert!(Arc::ptr_eq(service.quota_store_for_topology(&target_a, &topology_a).expect("return to A"), &store_a)); + assert_eq!(service.quota_topology_computations.load(std::sync::atomic::Ordering::Relaxed), 3, + "the bounded single-entry cache must recompute after an instance switch"); + let fence_path = format!("tmp/quota-mutation-fences/{}", "d".repeat(64)); + let acquire = |topology: &str| { + let mut request = Request::new(SnapshotLeaseRequest { + disk: disk_id.to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: fence_path.clone(), + ttl_ms: 60_000, + }); + let body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()).expect("acquire body"); + sign_quota_test_request(&mut request, &body, topology); + request + }; + assert_eq!( + service.acquire_quota_mutation_fence(acquire(&topology_b)).await.expect_err("ambient layout must not authorize A").code(), + tonic::Code::FailedPrecondition + ); + let response = service.acquire_quota_mutation_fence(acquire(&topology_a)).await.expect("captured acquire").into_inner(); + assert!(response.success, "A must acquire its own token: {:?}", response.error); + let token_a = crate::storage::storage_api::SnapshotLeaseToken::from_slice(&response.token).expect("A token"); + assert!(disk_b.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.is_err()); + let token_a = disk_a.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.expect("token belongs to A"); + let token_b = disk_b.acquire_snapshot_lease(RUSTFS_META_BUCKET, &fence_path).await.expect("independent B token"); + let release = |topology: &str| { + let mut request = Request::new(SnapshotLeaseReleaseRequest { + disk: disk_id.to_string(), + volume: RUSTFS_META_BUCKET.to_string(), + path: fence_path.clone(), + token: vec![0; 16].into(), + }); + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()).expect("release body"); + sign_quota_test_request(&mut request, &body, topology); + request + }; + assert_eq!( + service.release_quota_mutation_fence(release(&topology_b)).await.expect_err("ambient layout must not revoke A").code(), + tonic::Code::FailedPrecondition + ); + let token_a = disk_a.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.expect("wrong layout did not revoke A"); + assert!(service.release_quota_mutation_fence(release(&topology_a)).await.expect("captured release").into_inner().success); + assert!(disk_a.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_a).await.is_err()); + assert!(disk_b.renew_snapshot_lease(RUSTFS_META_BUCKET, &fence_path, token_b).await.is_ok(), "A release must leave ambient B untouched"); + let volume = "namespace-target-context"; let object = "destination"; let staging = "staged"; @@ -7238,8 +8394,8 @@ mod tests { assert!(tokio::fs::try_exists(&a_staging).await.expect("A staging exists")); assert_ne!(a_before, b_before, "the old on-disk bodies must distinguish A from B"); super::timeout(Duration::from_secs(10), async { - while store_a.scanner_data_usage_publication_blocked().await - || store_b.scanner_data_usage_publication_blocked().await + while store_a.namespace_commits_pending_for_test() + || store_b.namespace_commits_pending_for_test() { tokio::task::yield_now().await; } @@ -7288,8 +8444,8 @@ mod tests { store_b.scanner_namespace_mutation_generation(), ); let pending_after = ( - store_a.scanner_data_usage_publication_blocked().await, - store_b.scanner_data_usage_publication_blocked().await, + store_a.namespace_commits_pending_for_test(), + store_b.namespace_commits_pending_for_test(), ); for disk in store_a.disk_map.values().chain(store_b.disk_map.values()).flatten().flatten() { disk.close().await.expect("close real fixture disk before assertions and directory cleanup"); diff --git a/rustfs/src/storage/rpc/node_service/disk.rs b/rustfs/src/storage/rpc/node_service/disk.rs index 22c1a88c1..601d17bd1 100644 --- a/rustfs/src/storage/rpc/node_service/disk.rs +++ b/rustfs/src/storage/rpc/node_service/disk.rs @@ -13,10 +13,13 @@ // limitations under the License. use super::{LocalMutationTarget, NodeService}; +use crate::storage::storage_api::ecstore_disk::{ + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, is_quota_mutation_fence_path, +}; use crate::storage::storage_api::rpc_consumer::node_service::{ BatchReadVersionReq, BatchReadVersionResp, ConditionalFileUpdate, DeleteOptions, DiskError, DiskInfoOptions, - FileInfoVersions, ReadMultipleReq, ReadMultipleResp, ReadOptions, StorageDiskRpcExt as _, UpdateMetadataOpts, - validate_batch_read_version_item_count, + FileInfoVersions, GuardedDeleteVersionsPayload, ReadMultipleReq, ReadMultipleResp, ReadOptions, StorageDiskRpcExt as _, + UpdateMetadataOpts, validate_batch_read_version_item_count, }; use crate::storage::storage_api::runtime_sources_consumer::runtime_sources; use crate::storage::storage_api::{ @@ -286,6 +289,33 @@ fn verify_disk_mutation_digest( .map_err(|err| Status::permission_denied(format!("{op} authentication failed: {err}"))) } +fn verify_quota_mutation_digest( + request: &Request, + canonical_body: std::result::Result, std::num::TryFromIntError>, +) -> Result { + let values = request.metadata().get_all(rustfs_protos::QUOTA_MUTATION_TOPOLOGY_HEADER); + let mut values = values.iter(); + let topology = values + .next() + .and_then(|value| value.to_str().ok()) + .filter(|value| { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + }) + .ok_or_else(|| Status::permission_denied("quota mutation requires a valid topology header"))?; + if values.next().is_some() { + return Err(Status::permission_denied("quota mutation requires exactly one topology header")); + } + let body = canonical_body.map_err(|_| Status::invalid_argument("quota mutation request length cannot be represented"))?; + let body = rustfs_protos::canonical_quota_mutation_request_body(topology, &body) + .map_err(|_| Status::invalid_argument("quota mutation envelope length cannot be represented"))?; + verify_tonic_canonical_body_digest(request, &body) + .map_err(|_| Status::permission_denied("quota mutation requires topology-bound v2 authentication"))?; + Ok(topology.to_owned()) +} + /// JSON compatibility string for a dual-encoded response field. Returns an empty string when /// msgpack-only mode and its explicit fleet confirmation guard are both enabled, or when /// request-local `_bin` payloads prove the caller can consume the paired response `_bin` field, @@ -391,19 +421,71 @@ fn encode_rename_data_response_payloads( } impl NodeService { + pub(super) fn quota_store_for_topology<'a>( + &self, + target: &'a LocalMutationTarget, + coordinator_topology: &str, + ) -> Result<&'a Arc, Status> { + let LocalMutationTarget::Ready(store) = target else { + return Err(Status::failed_precondition("quota mutation requires a ready storage instance")); + }; + let mut cache = self + .quota_topology_cache + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let identity = Arc::downgrade(store); + if !cache.as_ref().is_some_and(|entry| entry.store.ptr_eq(&identity)) { + // Instance endpoints are immutable once initialized. Keep one weak + // identity so a replacement instance cannot inherit this layout, + // even when its deployment or disk UUIDs are unchanged. + let endpoints = store + .instance_endpoints() + .ok_or_else(|| Status::failed_precondition("quota mutation target topology is unavailable"))?; + let fingerprint = super::heal::heal_topology_fingerprint(&endpoints) + .map_err(|_| Status::failed_precondition("quota mutation target topology is invalid"))?; + #[cfg(test)] + self.quota_topology_computations + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + *cache = Some(super::QuotaTopologyCacheEntry { + store: identity, + fingerprint, + }); + } + if cache.as_ref().is_none_or(|entry| entry.fingerprint != coordinator_topology) { + return Err(Status::failed_precondition( + "quota mutation coordinator topology does not match the target", + )); + } + Ok(store) + } + pub(super) async fn handle_acquire_snapshot_lease( &self, request: Request, + quota_fence: bool, ) -> Result, Status> { - verify_disk_mutation_digest( - &request, - rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()), - "acquire_snapshot_lease", - )?; + let body = rustfs_protos::canonical_snapshot_lease_request_body(request.get_ref()); + let quota_store = if quota_fence { + let topology = verify_quota_mutation_digest(&request, body)?; + if request.get_ref().volume != RUSTFS_META_BUCKET || !is_quota_mutation_fence_path(&request.get_ref().path) { + return Err(Status::invalid_argument("quota fence requires its internal mutation fence path")); + } + Some(Arc::clone(self.quota_store_for_topology(&self.local_mutation_target(), &topology)?)) + } else { + verify_disk_mutation_digest(&request, body, "acquire_snapshot_lease")?; + None + }; let request = request.into_inner(); - let result = match self.find_disk(&request.disk).await { - Some(disk) => disk.acquire_snapshot_lease(&request.volume, &request.path).await, - None => Err(DiskError::other("cannot find disk")), + let disk = match quota_store { + Some(store) => store.quota_mutation_fence_disk(&request.disk).await, + None => self + .find_disk(&request.disk) + .await + .ok_or_else(|| DiskError::other("cannot find disk")), + }; + let result = match disk { + Ok(disk) => disk.acquire_snapshot_lease(&request.volume, &request.path).await, + Err(err) => Err(err), }; Ok(snapshot_lease_response(result)) } @@ -430,25 +512,37 @@ impl NodeService { pub(super) async fn handle_release_snapshot_lease( &self, request: Request, + quota_fence: bool, ) -> Result, Status> { - verify_disk_mutation_digest( - &request, - rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()), - "release_snapshot_lease", - )?; + let body = rustfs_protos::canonical_snapshot_lease_release_request_body(request.get_ref()); + let quota_store = if quota_fence { + let topology = verify_quota_mutation_digest(&request, body)?; + if request.get_ref().volume != RUSTFS_META_BUCKET || !is_quota_mutation_fence_path(&request.get_ref().path) { + return Err(Status::invalid_argument("quota fence requires its internal mutation fence path")); + } + Some(Arc::clone(self.quota_store_for_topology(&self.local_mutation_target(), &topology)?)) + } else { + verify_disk_mutation_digest(&request, body, "release_snapshot_lease")?; + None + }; let request = request.into_inner(); let token = if request.token.as_ref() == SnapshotLeaseToken::revoke_all().as_bytes() { SnapshotLeaseToken::revoke_all() } else { SnapshotLeaseToken::from_slice(&request.token).map_err(|_| Status::invalid_argument("invalid lease token"))? }; - let Some(disk) = self.find_disk(&request.disk).await else { - return Ok(Response::new(SnapshotLeaseMutationResponse { - success: false, - error: Some(DiskError::other("cannot find disk").into()), - })); + let disk = match quota_store { + Some(store) => store.quota_mutation_fence_disk(&request.disk).await, + None => self + .find_disk(&request.disk) + .await + .ok_or_else(|| DiskError::other("cannot find disk")), }; - match disk.release_snapshot_lease(&request.volume, &request.path, token).await { + let result = match disk { + Ok(disk) => disk.release_snapshot_lease(&request.volume, &request.path, token).await, + Err(err) => Err(err), + }; + match result { Ok(()) => Ok(Response::new(SnapshotLeaseMutationResponse { success: true, error: None, @@ -690,6 +784,58 @@ impl NodeService { } } + pub(super) async fn handle_guarded_delete_versions( + &self, + request: Request, + ) -> Result, Status> { + if request.get_ref().payload.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES { + return Err(Status::resource_exhausted("guarded delete payload exceeds limit")); + } + let body = rustfs_protos::canonical_guarded_delete_versions_request_body(request.get_ref()) + .map_err(|_| Status::invalid_argument("invalid guarded delete body length"))?; + verify_tonic_canonical_body_digest(&request, &body) + .map_err(|_| Status::permission_denied("guarded delete requires body-bound v2 authentication"))?; + let request = request.into_inner(); + let disk = self + .find_disk(&request.disk) + .await + .ok_or_else(|| Status::not_found("cannot find disk"))?; + if request.probe_only { + if !request.payload.is_empty() || !request.volume.is_empty() { + return Err(Status::invalid_argument("guarded capability probe must have no mutation payload")); + } + if let Err(error) = disk.supports_guarded_delete_versions().await { + return Ok(Response::new(DeleteVersionsResponse { + success: false, + errors: Vec::new(), + error: Some(error.into()), + item_errors: Vec::new(), + })); + } + return Ok(Response::new(DeleteVersionsResponse { + success: true, + errors: Vec::new(), + error: None, + item_errors: Vec::new(), + })); + } + let payload: GuardedDeleteVersionsPayload = rustfs_protos::decode_delete_recovery_payload(&request.payload) + .map_err(|_| Status::invalid_argument("invalid guarded delete payload"))?; + payload + .validate() + .map_err(|_| Status::invalid_argument("invalid guarded delete options or batch"))?; + let (errors, item_errors) = encode_delete_versions_errors( + disk.delete_versions_guarded_recoverable(&request.volume, payload.versions, payload.opts) + .await, + ); + Ok(Response::new(DeleteVersionsResponse { + success: true, + errors, + error: None, + item_errors, + })) + } + pub(super) async fn handle_delete_versions( &self, request: Request, @@ -1364,7 +1510,10 @@ impl NodeService { pub(super) async fn handle_rename_data( &self, request: Request, + quota_fence: bool, ) -> Result, Status> { + let service_return_started = rustfs_io_metrics::put_stage_timer(); + let service_entry_started = rustfs_io_metrics::put_stage_timer(); if !request.get_ref().scanner_publication_lease_token.is_empty() || !request.get_ref().bucket_incarnation_id.is_empty() { let has_body_digest = request .metadata() @@ -1372,14 +1521,40 @@ impl NodeService { .and_then(|value| value.to_str().ok()) .is_some_and(|value| value != "UNSIGNED-PAYLOAD"); if !has_body_digest { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + service_entry_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); return Err(Status::permission_denied("scanner publication lease rename requires a body-bound digest")); } } - verify_disk_mutation_digest( - &request, - rustfs_protos::canonical_rename_data_request_body(request.get_ref()), - "rename_data", - )?; + let body = rustfs_protos::canonical_rename_data_request_body(request.get_ref()); + let mut quota_topology = None; + let authenticated = if quota_fence { + verify_quota_mutation_digest(&request, body).map(|topology| quota_topology = Some(topology)) + } else { + verify_disk_mutation_digest(&request, body, "rename_data") + }; + if let Err(err) = authenticated { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + service_entry_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); + return Err(err); + } + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_ENTRY, + service_entry_started, + ); + let before_handler_started = rustfs_io_metrics::put_stage_timer(); let request = request.into_inner(); let target = self.local_mutation_target(); let bucket_incarnation = if request.bucket_incarnation_id.is_empty() { @@ -1390,6 +1565,14 @@ impl NodeService { .filter(|id| !id.is_nil()) .ok_or_else(|| Status::invalid_argument("bucket incarnation must be a non-nil UUID"))?; if !request.scanner_publication_lease_token.is_empty() { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); return Err(Status::invalid_argument( "heal incarnation and scanner publication lease cannot be combined", )); @@ -1401,12 +1584,26 @@ impl NodeService { let decoded_file_info = match decode_rename_data_request_file_info(&request.file_info_bin, &request.file_info) { Ok(file_info) => file_info, Err(err) => { - return Ok(Response::new(RenameDataResponse { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + let response_build_started = rustfs_io_metrics::put_stage_timer(); + let response = Response::new(RenameDataResponse { success: false, rename_data_resp: String::new(), rename_data_resp_bin: Vec::new().into(), error: Some(DiskError::other(format!("decode FileInfo failed: {err}")).into()), - })); + }); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD, + response_build_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); + return Ok(response); } }; let scanner_publication_lease_token = if request.scanner_publication_lease_token.is_empty() { @@ -1415,13 +1612,42 @@ impl NodeService { let token = Uuid::from_slice(&request.scanner_publication_lease_token) .map_err(|_| Status::invalid_argument("scanner publication lease token must be a UUID"))?; if token.is_nil() { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); return Err(Status::invalid_argument("scanner publication lease token must not be nil")); } Some(token) }; + if quota_fence { + let token = rustfs_utils::http::metadata_compat::get_consistent_str( + &decoded_file_info.value.metadata, + QUOTA_MUTATION_FENCE_METADATA_SUFFIX, + ) + .and_then(|value| Uuid::parse_str(value).ok()) + .filter(|token| !token.is_nil()); + if token.is_none() { + return Err(Status::invalid_argument( + "quota rename requires a consistent non-nil mutation fence token", + )); + } + } + if let Some(topology) = quota_topology { + self.quota_store_for_topology(&target, &topology)?; + } let request_decoded_from_msgpack = decoded_file_info.from_msgpack; #[cfg(feature = "e2e-test-hooks")] let observation = startup_cas_rename_observation(&target, &request, &decoded_file_info.value); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_BEFORE_HANDLER, + before_handler_started, + ); + let handler_await_started = rustfs_io_metrics::put_stage_timer(); let result = target .rename_local_data( &request.disk, @@ -1432,6 +1658,11 @@ impl NodeService { bucket_incarnation, ) .await; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_HANDLER_AWAIT, + handler_await_started, + ); + let after_handler_started = rustfs_io_metrics::put_stage_timer(); #[cfg(feature = "e2e-test-hooks")] if let Some(mut observation) = observation { observation["ok"] = serde_json::json!(result.is_ok()); @@ -1439,28 +1670,47 @@ impl NodeService { let line = format!("RUSTFS_E2E_STARTUP_CAS {observation}\n"); let _ = std::io::Write::write_all(&mut std::io::stderr().lock(), line.as_bytes()); } - match result { + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_AFTER_HANDLER, + after_handler_started, + ); + let response_build_started = rustfs_io_metrics::put_stage_timer(); + let response_encode_started = rustfs_io_metrics::put_stage_timer(); + let response = match result { Ok(rename_data_resp) => match encode_rename_data_response_payloads(&rename_data_resp, request_decoded_from_msgpack) { - Ok((rename_data_resp, rename_data_resp_bin)) => Ok(Response::new(RenameDataResponse { + Ok((rename_data_resp, rename_data_resp_bin)) => Response::new(RenameDataResponse { success: true, rename_data_resp, rename_data_resp_bin: rename_data_resp_bin.into(), error: None, - })), - Err(err) => Ok(Response::new(RenameDataResponse { + }), + Err(err) => Response::new(RenameDataResponse { success: false, rename_data_resp: String::new(), rename_data_resp_bin: Vec::new().into(), error: Some(err.into()), - })), + }), }, - Err(err) => Ok(Response::new(RenameDataResponse { + Err(err) => Response::new(RenameDataResponse { success: false, rename_data_resp: String::new(), rename_data_resp_bin: Vec::new().into(), error: Some(err.into()), - })), - } + }), + }; + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_ENCODE, + response_encode_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_BUILD, + response_build_started, + ); + rustfs_io_metrics::record_put_object_stage_duration_from( + rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_SERVICE_RESPONSE_RETURN, + service_return_started, + ); + Ok(response) } pub(super) async fn handle_list_dir(&self, request: Request) -> Result, Status> { @@ -1651,6 +1901,7 @@ impl NodeService { || opts.undo_write || opts.recursive || opts.immediate + || opts.orphan_data_dir || opts.expected_delete_marker.is_some() || opts .delete_recovery_bucket_incarnation @@ -1672,6 +1923,14 @@ impl NodeService { } let result = if let Some(disk) = self.find_disk(&request.disk).await { match operation { + DeleteRecoveryRpcOperation::Exists { object } => disk + .has_delete_transaction_evidence(&request.volume, &object) + .await + .map(|present| DeleteRecoveryRpcReply::Evidence { present }), + DeleteRecoveryRpcOperation::InspectOrphanDataDir { path } => disk + .is_plain_orphan_data_dir(&request.volume, &path) + .await + .map(|plain| DeleteRecoveryRpcReply::OrphanDataDir { plain }), DeleteRecoveryRpcOperation::ApplyVersion { object, file_info, diff --git a/rustfs/src/storage/storage_api.rs b/rustfs/src/storage/storage_api.rs index c0a9608d5..3e9114c80 100644 --- a/rustfs/src/storage/storage_api.rs +++ b/rustfs/src/storage/storage_api.rs @@ -27,6 +27,15 @@ pub(crate) mod ecstore_integrity { IntegrityError, JobRequest, control_job, create_job, get_job, inventory, readiness, resume_job, }; } + +#[cfg(feature = "diagnostic-read-version-caller")] +pub(crate) mod ecstore_diagnostic { + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::diagnostic::current_read_version_entry_source; + pub(crate) use rustfs_ecstore::api::diagnostic::{ + ReadVersionContextBoundary, ReadVersionEntrySource, with_read_version_spawn_context, + }; +} use tokio::sync::{Mutex, OwnedMutexGuard}; use tokio_util::sync::CancellationToken; @@ -59,6 +68,8 @@ fn bucket_targets_metadata_lock_shard(bucket: &str) -> usize { pub(crate) mod contract { pub(crate) mod admin { + #[cfg(test)] + pub(crate) use super::super::storage_contracts::DiskSetSelector; pub(crate) use super::super::storage_contracts::StorageAdminApi; } @@ -125,10 +136,10 @@ pub(crate) use super::sse::{ pub(crate) mod access_consumer { pub(crate) use super::super::access::{ - PostObjectRequestMarker, ReqInfo, TABLE_DATA_PLANE_LIST_CURSOR_PREFIX, TableDataPlaneListAccess, - TableDataPlaneListCursorPosition, TableDataPlanePublicationGuards, apply_bucket_generation_guard, - apply_copy_source_bucket_generation_guard, authorize_internal_object_request, authorize_request, - bucket_config_mutation_incarnation, delete_object_authorize_action, has_bypass_governance_header, + MultipartStagingOperation, PostObjectRequestMarker, ReqInfo, TABLE_DATA_PLANE_LIST_CURSOR_PREFIX, + TableDataPlaneListAccess, TableDataPlaneListCursorPosition, TableDataPlanePublicationGuards, + apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard, authorize_internal_object_request, + authorize_request, bucket_config_mutation_incarnation, delete_object_authorize_action, has_bypass_governance_header, load_bucket_generation_from_store, log_list_buckets_iam_implicit_deny, odm_read_generation, prepare_list_buckets_iam_authorization, prepare_odm_read_generation, recursive_force_delete_has_authenticated_caller, replication_request_authorized, req_info_mut, req_info_ref, retained_table_data_plane_publication_guards, @@ -270,12 +281,13 @@ pub(crate) mod rpc_consumer { }; pub(crate) use super::super::{ BatchReadVersionReq, BatchReadVersionResp, CollectMetricsOpts, ConditionalFileUpdate, DeleteOptions, DiskError, - DiskInfoOptions, DiskStore, ECStore, Error, FileInfoVersions, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MetricType, - PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, ReadMultipleReq, ReadMultipleResp, ReadOptions, - SCANNER_PUBLICATION_LEASE_TTL_MS, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, StorageDiskRpcExt, - StoragePeerS3ClientExt, TierDailyStatsWire, UpdateMetadataOpts, all_local_disk_path, collect_local_metrics, - find_local_disk_by_ref, get_global_transition_state, get_local_server_property, reload_bucket_metadata, - reload_transition_tier_config, remove_bucket_metadata_if_absent, validate_batch_read_version_item_count, + DiskInfoOptions, DiskStore, ECStore, Error, FileInfoVersions, GuardedDeleteVersionsPayload, KMS_SIGNAL_SUBSYSTEM, + LocalPeerS3Client, MetricType, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, ReadMultipleReq, + ReadMultipleResp, ReadOptions, SCANNER_PUBLICATION_LEASE_TTL_MS, SERVICE_SIGNAL_REFRESH_CONFIG, + SERVICE_SIGNAL_RELOAD_DYNAMIC, StorageDiskRpcExt, StoragePeerS3ClientExt, TierDailyStatsWire, UpdateMetadataOpts, + all_local_disk_path, collect_local_metrics, find_local_disk_by_ref, get_global_transition_state, + get_local_server_property, reload_bucket_metadata, reload_transition_tier_config, remove_bucket_metadata_if_absent, + validate_batch_read_version_item_count, }; pub(crate) type StorageResult = super::super::Result; @@ -482,12 +494,13 @@ pub(crate) mod ecstore_disk { BUCKET_META_PREFIX, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, - DiskInfoOptions, DiskStore, FileInfoVersions, FileReader, FileWriter, OldCurrentSize, PartTransactionAction, - RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, - UpdateMetadataOpts, VolumeInfo, WalkDirOptions, get_object_disk_read_timeout, validate_batch_read_version_item_count, + DiskInfoOptions, DiskStore, FileInfoVersions, FileReader, FileWriter, GuardedDeleteVersionsPayload, + GuardedFileInfoVersions, OldCurrentSize, PartTransactionAction, QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, + ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, + WalkDirOptions, get_object_disk_read_timeout, is_quota_mutation_fence_path, validate_batch_read_version_item_count, }; #[cfg(test)] - pub(crate) use rustfs_ecstore::api::disk::{DiskOption, new_disk}; + pub(crate) use rustfs_ecstore::api::disk::{CurrentDeleteGeneration, DiskOption, new_disk}; pub(crate) use rustfs_ecstore::api::disk::{endpoint, error, error_reduce}; } @@ -528,12 +541,14 @@ pub(crate) mod ecstore_metrics { #[allow(unused_imports)] pub(crate) mod ecstore_notification { - #[cfg(test)] - pub(crate) use rustfs_ecstore::api::notification::rotate_cross_pool_fence_fleet_proof_for_test; pub(crate) use rustfs_ecstore::api::notification::{ ClusterTierDailyStats, CrossPoolFenceFleetProofToken, NotificationSys, acquire_cross_pool_fence_fleet_proof, - cross_pool_fence_fleet_proof_matches, get_global_notification_sys, ilm_recovery_export_local_process_epoch, - new_global_notification_sys, start_remote_version_state_fleet_probe, + configured_fleet_topology_fingerprint, cross_pool_fence_fleet_proof_matches, get_global_notification_sys, + ilm_recovery_export_local_process_epoch, new_global_notification_sys, start_remote_version_state_fleet_probe, + }; + #[cfg(test)] + pub(crate) use rustfs_ecstore::api::notification::{ + rotate_cross_pool_fence_fleet_proof_for_test, without_cross_pool_fence_fleet_proof_for_test, }; } @@ -708,6 +723,8 @@ pub(crate) type EndpointServerPools = ecstore_layout::EndpointServerPools; pub(crate) type EventArgs = ecstore_event::EventArgs; pub(crate) type ExpiryState = ecstore_bucket::lifecycle::bucket_lifecycle_ops::ExpiryState; pub(crate) type FileInfoVersions = ecstore_disk::FileInfoVersions; +pub(crate) type GuardedFileInfoVersions = ecstore_disk::GuardedFileInfoVersions; +pub(crate) type GuardedDeleteVersionsPayload = ecstore_disk::GuardedDeleteVersionsPayload; pub(crate) type FileReader = ecstore_disk::FileReader; pub(crate) type FileWriter = ecstore_disk::FileWriter; pub(crate) type SnapshotLeaseToken = ecstore_disk::SnapshotLeaseToken; @@ -716,7 +733,6 @@ pub(crate) type HashReader = ecstore_rio::HashReader; pub(crate) type InstanceContext = ecstore_runtime::InstanceContext; pub(crate) type ServerContextSlot = crate::storage::runtime_sources::ServerContextSlot; pub(crate) type LocalPeerS3Client = ecstore_rpc::LocalPeerS3Client; -#[cfg(test)] pub(crate) type PeerRestClient = ecstore_rpc::PeerRestClient; #[cfg(test)] pub(crate) type ScannerScopedDirtyUsageAckEntry = ecstore_rpc::ScannerScopedDirtyUsageAckEntry; @@ -975,23 +991,45 @@ pub(crate) async fn get_local_server_property() -> rustfs_madmin::ServerProperti ecstore_admin::get_local_server_property().await } +/// Bridge an ECStore committed replication-path mutation into the scanner's +/// dirty-usage tracking. +/// +/// A replication status writeback changes only replication state on an +/// existing version. It is status class: it must not count as namespace +/// activity, or every scanner cycle is `Superseded` for as long as replication +/// runs (#8375). The scanner's own queued replication heals write back through +/// the same path and are covered by the same rule. Tier-expiration cleanup +/// removes data and stays on the namespace path. +pub(crate) fn record_committed_replication_mutation( + bucket: &str, + object: &str, + source: ecstore_bucket::replication::ScannerDirtyUsageMutationSource, +) { + match source { + ecstore_bucket::replication::ScannerDirtyUsageMutationSource::Replication => { + rustfs_scanner::record_status_class_mutation( + bucket, + object, + rustfs_scanner::SegmentInvalidationProducerIdentity::Replication, + ); + } + ecstore_bucket::replication::ScannerDirtyUsageMutationSource::TierExpiration => { + rustfs_scanner::record_dirty_usage_object_from_producer( + bucket, + object, + rustfs_scanner::SegmentInvalidationProducerIdentity::TierExpiration, + ); + } + } +} + pub(crate) async fn init_background_replication(store: Arc) { let durable_dirty_usage_journal = super::scanner_dirty_journal::start_durable_dirty_usage_journal(store.clone()).await; let mutation_journal = durable_dirty_usage_journal.clone(); rustfs_scanner::set_scanner_dirty_usage_mutation_observer(Some(Arc::new(move |bucket, object, producer| { mutation_journal.record_committed_mutation(bucket, object, producer); }))); - ecstore_bucket::replication::set_scanner_dirty_usage_mutation_observer(Some(Arc::new(move |bucket, object, source| { - let producer = match source { - ecstore_bucket::replication::ScannerDirtyUsageMutationSource::Replication => { - rustfs_scanner::SegmentInvalidationProducerIdentity::Replication - } - ecstore_bucket::replication::ScannerDirtyUsageMutationSource::TierExpiration => { - rustfs_scanner::SegmentInvalidationProducerIdentity::TierExpiration - } - }; - rustfs_scanner::record_dirty_usage_object_from_producer(bucket, object, producer); - }))); + ecstore_bucket::replication::set_scanner_dirty_usage_mutation_observer(Some(Arc::new(record_committed_replication_mutation))); rustfs_scanner::set_scanner_dirty_usage_clear_observer(Some(Arc::new(move |cleared| { durable_dirty_usage_journal.clear_confirmed_buckets(cleared); }))); @@ -1380,6 +1418,13 @@ pub(crate) async fn try_migrate_server_config(store: Arc) { } pub(crate) trait StorageDiskRpcExt { + async fn supports_guarded_delete_versions(&self) -> DiskResult<()>; + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec>; async fn disk_info(&self, opts: &DiskInfoOptions) -> DiskResult; async fn delete_volume(&self, volume: &str, force_delete: bool) -> DiskResult<()>; async fn read_multiple(&self, req: ReadMultipleReq) -> DiskResult>; @@ -1469,6 +1514,7 @@ pub(crate) trait StorageDiskRpcExt { versions: Vec, opts: DeleteOptions, ) -> Vec>; + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> DiskResult; async fn discover_delete_transactions( &self, volume: &str, @@ -1513,6 +1559,17 @@ impl StorageDiskRpcExt for T where T: ecstore_disk::DiskAPI, { + async fn supports_guarded_delete_versions(&self) -> DiskResult<()> { + ecstore_disk::DiskAPI::supports_guarded_delete_versions(self).await + } + async fn delete_versions_guarded_recoverable( + &self, + volume: &str, + versions: Vec, + opts: DeleteOptions, + ) -> Vec> { + ecstore_disk::DiskAPI::delete_versions_guarded_recoverable(self, volume, versions, opts).await + } async fn disk_info(&self, opts: &DiskInfoOptions) -> DiskResult { ecstore_disk::DiskAPI::disk_info(self, opts).await } @@ -1681,6 +1738,10 @@ where ecstore_disk::DiskAPI::delete_versions_recoverable(self, volume, versions, opts).await } + async fn has_delete_transaction_evidence(&self, volume: &str, object: &str) -> DiskResult { + ecstore_disk::DiskAPI::has_delete_transaction_evidence(self, volume, object).await + } + async fn discover_delete_transactions( &self, volume: &str, @@ -2379,6 +2440,34 @@ mod tests { }; use std::time::Duration; + #[test] + #[serial_test::serial] + fn replication_status_writebacks_are_status_class_and_tier_expiration_is_namespace() { + use ecstore_bucket::replication::ScannerDirtyUsageMutationSource; + let bucket = format!("status-class-{}", uuid::Uuid::new_v4().simple()); + let before = rustfs_scanner::scanner_dirty_usage_state(); + + super::record_committed_replication_mutation(&bucket, "2026/object", ScannerDirtyUsageMutationSource::Replication); + let after_replication = rustfs_scanner::scanner_dirty_usage_state(); + assert_eq!( + after_replication.generation, before.generation, + "a replication status writeback must not advance the scanner dirty-usage generation" + ); + assert!( + rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket).is_none(), + "a replication status writeback must not mark the bucket dirty" + ); + + super::record_committed_replication_mutation(&bucket, "2026/object", ScannerDirtyUsageMutationSource::TierExpiration); + let after_expiry = rustfs_scanner::scanner_dirty_usage_state(); + assert!( + after_expiry.generation > before.generation, + "tier expiration removes data and must stay on the namespace path" + ); + assert!(rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket).is_some()); + rustfs_scanner::clear_dirty_usage_bucket(&bucket); + } + #[tokio::test] async fn site_metrics_snapshot_includes_live_failure_windows() { let stats = StorageReplicationStatsHandle::new(); diff --git a/rustfs/src/storage/timeout_wrapper.rs b/rustfs/src/storage/timeout_wrapper.rs index f0ca0f882..94f0bed24 100644 --- a/rustfs/src/storage/timeout_wrapper.rs +++ b/rustfs/src/storage/timeout_wrapper.rs @@ -412,21 +412,21 @@ impl RequestTimeoutWrapper { debug!(timeout_secs = timeout_duration.as_secs(), "Starting timed operation"); - rustfs_io_metrics::record_get_object_request_started(); + rustfs_io_metrics::record_get_object_timeout_operation_start(); let cancel_token_for_op = self.cancel_token.clone(); match tokio::time::timeout(timeout_duration, operation(cancel_token_for_op)).await { Ok(Ok(result)) => { let elapsed = self.elapsed(); - rustfs_io_metrics::record_get_object_request_result("success", elapsed.as_secs_f64()); + rustfs_io_metrics::record_get_object_timeout_operation_result("success", elapsed.as_secs_f64()); debug!(elapsed_ms = elapsed.as_millis(), "Operation completed successfully"); TimedGetObjectResult::Success(result) } Ok(Err(e)) => { let elapsed = self.elapsed(); - rustfs_io_metrics::record_get_object_request_result("error", elapsed.as_secs_f64()); + rustfs_io_metrics::record_get_object_timeout_operation_result("error", elapsed.as_secs_f64()); debug!(elapsed_ms = elapsed.as_millis(), "Operation failed with error"); TimedGetObjectResult::Error(e) @@ -436,7 +436,7 @@ impl RequestTimeoutWrapper { self.cancel_token.cancel(); rustfs_io_metrics::record_get_object_timeout(None, Some(elapsed.as_secs_f64())); - rustfs_io_metrics::record_get_object_request_result("timeout", elapsed.as_secs_f64()); + rustfs_io_metrics::record_get_object_timeout_operation_result("timeout", elapsed.as_secs_f64()); warn!( timeout_secs = timeout_duration.as_secs(), diff --git a/rustfs/src/storage_api.rs b/rustfs/src/storage_api.rs index ecc668e0d..7cf05126f 100644 --- a/rustfs/src/storage_api.rs +++ b/rustfs/src/storage_api.rs @@ -134,7 +134,7 @@ pub(crate) mod kms { pub(crate) mod protocols { pub(crate) mod client { - pub(crate) use crate::storage::storage_api::access_consumer::{ReqInfo, authorize_request}; + pub(crate) use crate::storage::storage_api::access_consumer::{MultipartStagingOperation, ReqInfo, authorize_request}; #[cfg(test)] pub(crate) use crate::storage::storage_api::contract::bucket::{BucketOperations, MakeBucketOptions}; #[cfg(test)] @@ -265,7 +265,9 @@ pub(crate) mod server { } pub(crate) use crate::storage::storage_api::ecstore_disk::DiskStore; - pub(crate) use crate::storage::storage_api::{Endpoint, EndpointServerPools, is_dist_erasure}; + pub(crate) use crate::storage::storage_api::{ + Endpoint, EndpointServerPools, PeerRestClient, get_global_notification_sys, is_dist_erasure, + }; pub(crate) fn disk_endpoint_snapshot(disk: &DiskStore) -> Endpoint { crate::storage::storage_api::ecstore_disk::DiskAPI::endpoint(disk.as_ref()) @@ -460,6 +462,7 @@ pub(crate) mod workload { } pub(crate) mod table { + pub(crate) use crate::storage::storage_api::ecstore_bucket::utils::check_valid_bucket_name_strict; pub(crate) use crate::storage::storage_api::ecstore_object::{WriteCommitGuard, WriteCompletion}; pub(crate) mod contract { pub(crate) mod http { diff --git a/rustfs/src/table_catalog/error.rs b/rustfs/src/table_catalog/error.rs index f06bc7c95..db3c93160 100644 --- a/rustfs/src/table_catalog/error.rs +++ b/rustfs/src/table_catalog/error.rs @@ -55,7 +55,7 @@ impl fmt::Display for TableObjectMutationError { } impl std::error::Error for TableObjectMutationError {} -#[derive(Debug, Clone, PartialEq, Eq)] +#[derive(Debug)] pub(crate) enum TableCatalogStoreError { NotFound(String), NamespaceNotFound(String), @@ -65,6 +65,10 @@ pub(crate) enum TableCatalogStoreError { Invalid(String), Unsupported(String), Unavailable(String), + LockUnavailable { + message: String, + source: Box, + }, Internal(String), } @@ -78,12 +82,27 @@ impl fmt::Display for TableCatalogStoreError { Self::Conflict(message) => write!(f, "table catalog conflict: {message}"), Self::Invalid(message) => write!(f, "invalid table catalog entry: {message}"), Self::Unsupported(message) => write!(f, "unsupported table catalog operation: {message}"), - Self::Unavailable(message) => write!(f, "table catalog temporarily unavailable: {message}"), + Self::Unavailable(message) | Self::LockUnavailable { message, .. } => { + write!(f, "table catalog temporarily unavailable: {message}") + } Self::Internal(message) => write!(f, "table catalog store error: {message}"), } } } -impl std::error::Error for TableCatalogStoreError {} +impl TableCatalogStoreError { + pub(crate) fn is_unavailable(&self) -> bool { + matches!(self, Self::Unavailable(_) | Self::LockUnavailable { .. }) + } +} + +impl std::error::Error for TableCatalogStoreError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::LockUnavailable { source, .. } => Some(source.as_ref()), + _ => None, + } + } +} pub(crate) type TableCatalogStoreResult = Result; diff --git a/rustfs/src/table_catalog/iceberg/commit.rs b/rustfs/src/table_catalog/iceberg/commit.rs index 2b91593ad..19ee14ca3 100644 --- a/rustfs/src/table_catalog/iceberg/commit.rs +++ b/rustfs/src/table_catalog/iceberg/commit.rs @@ -277,7 +277,7 @@ fn table_catalog_store_result_label(result: &TableCatalogStoreResult) -> & | TableCatalogStoreError::TableNotFound(_), ) => "not_found", Err(TableCatalogStoreError::Unsupported(_)) => "unsupported", - Err(TableCatalogStoreError::Unavailable(_)) => "unavailable", + Err(TableCatalogStoreError::Unavailable(_) | TableCatalogStoreError::LockUnavailable { .. }) => "unavailable", Err(TableCatalogStoreError::Internal(_)) => "failure", } } @@ -286,7 +286,7 @@ fn duration_millis_u64(duration: StdDuration) -> u64 { u64::try_from(duration.as_millis()).unwrap_or(u64::MAX) } -pub(crate) fn record_table_commit_cas_result(operation: &str, started: Instant, result: &TableCatalogStoreResult<()>) { +pub(crate) fn record_table_commit_cas_result(operation: &str, started: Instant, result: &TableCatalogStoreResult) { let elapsed = started.elapsed(); let result_label = table_catalog_store_result_label(result); counter!( @@ -390,10 +390,15 @@ mod tests { #[test] fn table_catalog_unavailable_result_label_is_stable() { - let result: TableCatalogStoreResult<()> = Err(TableCatalogStoreError::Unavailable( - "catalog publication authority is temporarily unavailable".to_string(), - )); - - assert_eq!(table_catalog_store_result_label(&result), "unavailable"); + for error in [ + TableCatalogStoreError::Unavailable("catalog publication authority is temporarily unavailable".to_string()), + TableCatalogStoreError::LockUnavailable { + message: "catalog publication authority is temporarily unavailable".to_string(), + source: Box::new(rustfs_lock::LockError::timeout("catalog/lock", StdDuration::from_secs(5))), + }, + ] { + let result: TableCatalogStoreResult<()> = Err(error); + assert_eq!(table_catalog_store_result_label(&result), "unavailable"); + } } } diff --git a/rustfs/src/table_catalog/iceberg/manifest.rs b/rustfs/src/table_catalog/iceberg/manifest.rs index 437c156a2..b2c7a8967 100644 --- a/rustfs/src/table_catalog/iceberg/manifest.rs +++ b/rustfs/src/table_catalog/iceberg/manifest.rs @@ -581,6 +581,7 @@ fn avro_i64_value(value: &apache_avro::types::Value) -> Option { #[cfg(test)] mod tests { use super::*; + use std::assert_matches; #[test] fn rejects_v2_manifest_lists_without_required_count_fields() { @@ -609,9 +610,9 @@ mod tests { Ok(_) => panic!("v2 count fields must be declared in the writer schema"), Err(error) => error, }; - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("Iceberg v2 manifest list Avro schema is missing added_files_count".to_string()) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v2 manifest list Avro schema is missing added_files_count" ); } @@ -622,15 +623,15 @@ mod tests { ("added_rows_count".to_string(), apache_avro::types::Value::Long(-1)), ]); - assert_eq!( - avro_nullable_non_negative_i32(&value, "added_files_count", "manifest list") - .expect_err("negative file counts must be rejected"), - TableCatalogStoreError::Invalid("manifest list field added_files_count must be a non-negative int".to_string()) + assert_matches!( + avro_nullable_non_negative_i32(&value, "added_files_count", "manifest list") + .expect_err("negative file counts must be rejected"), + TableCatalogStoreError::Invalid(message) if message == "manifest list field added_files_count must be a non-negative int" ); - assert_eq!( - avro_nullable_non_negative_i64(&value, "added_rows_count", "manifest list") - .expect_err("negative row counts must be rejected"), - TableCatalogStoreError::Invalid("manifest list field added_rows_count must be a non-negative long".to_string()) + assert_matches!( + avro_nullable_non_negative_i64(&value, "added_rows_count", "manifest list") + .expect_err("negative row counts must be rejected"), + TableCatalogStoreError::Invalid(message) if message == "manifest list field added_rows_count must be a non-negative long" ); } @@ -685,9 +686,9 @@ mod tests { Ok(_) => panic!("manifest partitions must preserve their record shape"), Err(error) => error, }; - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest data file partition must be a record".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest data file partition must be a record" ); } diff --git a/rustfs/src/table_catalog/iceberg/metadata.rs b/rustfs/src/table_catalog/iceberg/metadata.rs index d8061f441..74955f284 100644 --- a/rustfs/src/table_catalog/iceberg/metadata.rs +++ b/rustfs/src/table_catalog/iceberg/metadata.rs @@ -35,17 +35,6 @@ where .map(Some) } -pub(crate) fn metadata_log_locations( - current_metadata: &serde_json::Value, - table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, -) -> BTreeSet { - metadata_log_locations_matching(current_metadata, table_bucket, |location| { - is_valid_table_metadata_location(namespace, table, location) - }) -} - pub(crate) fn metadata_log_locations_matching( current_metadata: &serde_json::Value, table_bucket: &str, diff --git a/rustfs/src/table_catalog/maintenance/mod.rs b/rustfs/src/table_catalog/maintenance/mod.rs index e38b5a4de..723db7d8d 100644 --- a/rustfs/src/table_catalog/maintenance/mod.rs +++ b/rustfs/src/table_catalog/maintenance/mod.rs @@ -13,11 +13,14 @@ // limitations under the License. mod config; +mod periodic; +mod periodic_runtime; mod planner; mod recovery; mod worker; pub(crate) use config::*; +pub(crate) use periodic_runtime::{PeriodicMaintenanceConfig, PeriodicMaintenanceRuntime}; pub(crate) use planner::*; pub(crate) use recovery::*; pub(crate) use worker::*; diff --git a/rustfs/src/table_catalog/maintenance/periodic.rs b/rustfs/src/table_catalog/maintenance/periodic.rs new file mode 100644 index 000000000..9cfbdd4e0 --- /dev/null +++ b/rustfs/src/table_catalog/maintenance/periodic.rs @@ -0,0 +1,812 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::super::*; +use std::collections::BTreeSet; +use std::num::NonZeroUsize; +use std::time::Duration as StdDuration; +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +const PERIODIC_MAINTENANCE_TABLES_PER_PASS: usize = 8; +const PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS: usize = 64; + +struct BucketCursor { + bucket: String, + namespace_cursor: Option, + namespace: Option, + table_cursor: Option, +} + +impl BucketCursor { + fn reset(&mut self) { + self.namespace_cursor = None; + self.namespace = None; + self.table_cursor = None; + } +} + +/// Queue-only admission. The caller owns this future until it returns: a +/// deadline stops new operations, never cancels a catalog mutation in flight. +pub(crate) struct PeriodicMaintenanceScheduler { + store: ConfiguredTableCatalogStore, + buckets: Vec, + next_bucket: usize, + interval: StdDuration, + scheduler_id: String, +} + +impl PeriodicMaintenanceScheduler { + /// Bucket names and the interval have already passed runtime configuration validation. + pub(crate) fn new(store: ConfiguredTableCatalogStore, buckets: Vec, interval: StdDuration) -> Self { + Self { + store, + buckets: buckets + .into_iter() + .map(|bucket| BucketCursor { + bucket, + namespace_cursor: None, + namespace: None, + table_cursor: None, + }) + .collect(), + next_bucket: 0, + interval, + scheduler_id: format!("periodic-{}", uuid::Uuid::new_v4()), + } + } + + pub(crate) async fn run_pass( + &mut self, + cancellation: &CancellationToken, + admission_deadline: Instant, + ) -> TableCatalogStoreResult<()> { + if self.buckets.is_empty() { + return Ok(()); + } + let mut visited = BTreeSet::new(); + let mut unavailable_buckets = BTreeSet::new(); + let mut first_error = None; + for _ in 0..PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS { + if !admission_open(cancellation, admission_deadline) || visited.len() == PERIODIC_MAINTENANCE_TABLES_PER_PASS { + break; + } + let bucket_index = self.next_bucket; + self.next_bucket = (bucket_index + 1) % self.buckets.len(); + if unavailable_buckets.contains(&bucket_index) { + continue; + } + let table = match self.discover_next_table(bucket_index, cancellation, admission_deadline).await { + Ok(Some(table)) => table, + Ok(None) => continue, + Err(error) => { + if matches!(&error, TableCatalogStoreError::NotFound(_)) { + // A namespace removed between pages must not pin this bucket's cursor forever. + self.buckets[bucket_index].reset(); + } + unavailable_buckets.insert(bucket_index); + first_error.get_or_insert(error); + continue; + } + }; + if !visited.insert((table.table_bucket.clone(), table.table_id.clone())) { + continue; + } + if let Err(error) = self.queue_if_eligible(&table, cancellation, admission_deadline).await { + // A removed/renamed table or one unhealthy bucket must not starve other buckets. + first_error.get_or_insert(error); + } + } + match first_error { + Some(error) => Err(error), + None => Ok(()), + } + } + + async fn discover_next_table( + &mut self, + bucket_index: usize, + cancellation: &CancellationToken, + admission_deadline: Instant, + ) -> TableCatalogStoreResult> { + let cursor = &mut self.buckets[bucket_index]; + if !admission_open(cancellation, admission_deadline) { + return Ok(None); + } + if self + .store + .get_table_bucket(&cursor.bucket) + .await? + .is_none_or(|bucket| bucket.state != TableCatalogEntryState::Active) + { + cursor.reset(); + return Ok(None); + } + if cursor.namespace.is_none() { + if !admission_open(cancellation, admission_deadline) { + return Ok(None); + } + let page = self + .store + .list_namespaces_page(&cursor.bucket, cursor.namespace_cursor.as_deref(), NonZeroUsize::MIN) + .await?; + cursor.namespace_cursor = page.next_cursor; + cursor.namespace = page.entries.into_iter().next().map(|namespace| namespace.namespace); + cursor.table_cursor = None; + } + let Some(namespace) = cursor.namespace.as_deref() else { + return Ok(None); + }; + if !admission_open(cancellation, admission_deadline) { + return Ok(None); + } + let page = self + .store + .list_tables_page(&cursor.bucket, namespace, cursor.table_cursor.as_deref(), NonZeroUsize::MIN) + .await?; + cursor.table_cursor = page.next_cursor; + if cursor.table_cursor.is_none() { + cursor.namespace = None; + } + // Object-backed discovery can return an empty page with a continuation. + Ok(page.entries.into_iter().next()) + } + + async fn queue_if_eligible( + &self, + table: &TableEntry, + cancellation: &CancellationToken, + admission_deadline: Instant, + ) -> TableCatalogStoreResult<()> { + if !admission_open(cancellation, admission_deadline) { + return Ok(()); + } + let scheduler = self + .store + .get_table_maintenance_scheduler_report(&table.table_bucket, &table.namespace, &table.table) + .await?; + if scheduler.table_id != table.table_id { + return Err(TableCatalogStoreError::Conflict( + "table identity changed during periodic maintenance discovery".to_string(), + )); + } + if !scheduler.background_enabled || scheduler.worker_paused || scheduler.quarantine.active { + return Ok(()); + } + if let Some(current) = scheduler.current_job.as_ref() { + if current.status != TableMetadataMaintenanceJobStatus::Successful { + // Periodic admission never renews an abandoned queue or resets a failed job's retry budget. + return Ok(()); + } + let finished_at = current + .finished_at + .as_deref() + .and_then(|timestamp| OffsetDateTime::parse(timestamp, &time::format_description::well_known::Rfc3339).ok()) + .ok_or_else(|| { + TableCatalogStoreError::Invalid( + "periodic maintenance requires a valid successful job completion timestamp".to_string(), + ) + })?; + let interval = time::Duration::try_from(self.interval) + .map_err(|_| TableCatalogStoreError::Invalid("periodic maintenance interval is out of range".to_string()))?; + let next_admission = finished_at.checked_add(interval).ok_or_else(|| { + TableCatalogStoreError::Invalid("periodic maintenance completion timestamp is out of range".to_string()) + })?; + if OffsetDateTime::now_utc() < next_admission { + return Ok(()); + } + } + if !admission_open(cancellation, admission_deadline) { + return Ok(()); + } + self.store + .queue_periodic_table_maintenance(table, self.scheduler_id.clone(), scheduler.current_job.as_ref()) + .await?; + Ok(()) + } +} + +fn admission_open(cancellation: &CancellationToken, deadline: Instant) -> bool { + !cancellation.is_cancelled() && Instant::now() < deadline +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::table_catalog::test_support::TestCatalogObjectBackend; + use crate::table_catalog::tests::{test_bucket_entry, test_namespace_entry, test_table_entry}; + use std::collections::BTreeMap; + + type TestStore = ConfiguredTableCatalogStore; + + fn backings() -> [TableCatalogBackingMode; 2] { + [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] + } + + fn enabled_config() -> TableMaintenanceConfig { + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + } + } + + async fn seed_table( + store: &TestStore, + backend: &TestCatalogObjectBackend, + bucket: &str, + namespace: &str, + table: &str, + config: Option, + ) -> TableEntry { + let namespace = Namespace::parse(namespace).expect("fixture namespace"); + let table = IdentifierSegment::parse(table).expect("fixture table"); + if store.get_table_bucket(bucket).await.expect("read fixture bucket").is_none() { + store + .put_table_bucket(test_bucket_entry(bucket)) + .await + .expect("create fixture bucket"); + } + if store + .get_namespace(bucket, &namespace.public_name()) + .await + .expect("read fixture namespace") + .is_none() + { + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .expect("create fixture namespace"); + } + let metadata = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let mut entry = test_table_entry(bucket, &namespace, &table, metadata.clone()); + entry.table_id = uuid::Uuid::new_v4().to_string(); + entry.table_uuid = entry.table_id.clone(); + entry.warehouse_location = format!("s3://{bucket}/tables/{}", entry.table_id); + store.create_table(entry.clone()).await.expect("create fixture table"); + backend + .seed_object(bucket, &metadata, br#"{"metadata-log":[]}"#.to_vec()) + .await; + if let Some(config) = config { + store + .put_table_maintenance_config(bucket, &namespace.public_name(), table.as_str(), config) + .await + .expect("configure fixture table"); + } + entry + } + + async fn current(store: &TestStore, table: &TableEntry) -> Option { + store + .get_table_metadata_maintenance_report( + &table.table_bucket, + &table.namespace, + &table.table, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await + .expect("read current fixture job") + } + + async fn seed_report(backend: &TestCatalogObjectBackend, table: &TableEntry, report: &TableMetadataMaintenanceReport) { + let namespace = Namespace::parse(&table.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&table.table).expect("fixture table"); + let paths = TableCatalogObjectPaths::default(); + for path in [ + paths.table_maintenance_job_path(&table.table_bucket, &namespace, &name, &table.table_id, &report.job.job_id), + paths.table_maintenance_latest_job_path(&table.table_bucket, &namespace, &name, &table.table_id), + paths.table_maintenance_current_job_path(&table.table_bucket, &namespace, &name, &table.table_id), + ] { + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).expect("serialize fixture report")) + .await; + } + } + + async fn object_versions(backend: &TestCatalogObjectBackend) -> BTreeMap<(String, String), (Vec, String)> { + backend + .state + .lock() + .await + .objects + .iter() + .map(|(key, value)| (key.clone(), (value.data.clone(), value.etag.clone()))) + .collect() + } + + async fn run_pass(scheduler: &mut PeriodicMaintenanceScheduler) -> TableCatalogStoreResult<()> { + scheduler + .run_pass(&CancellationToken::new(), Instant::now() + StdDuration::from_secs(60)) + .await + } + + #[tokio::test] + async fn periodic_queue_is_idempotent_and_never_runs_delete_worker() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let namespace = Namespace::parse(&table.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&table.table).expect("fixture table"); + let orphan = default_table_metadata_file_path(&namespace, &name, "00000.metadata.json"); + backend + .seed_object(&table.table_bucket, &orphan, br#"{"metadata-log":[]}"#.to_vec()) + .await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec![table.table_bucket.clone()], StdDuration::from_secs(300)); + + run_pass(&mut scheduler).await.expect("first periodic pass queues work"); + let queued = current(&store, &table).await.expect("queued job"); + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.job.operation, TableMetadataMaintenanceOperation::Delete); + assert!(queued.job.worker_id.is_none()); + assert_eq!(queued.job.attempt, 0); + assert!( + backend + .read_object(&table.table_bucket, &orphan) + .await + .expect("read orphan") + .is_some() + ); + assert!(backend.state.lock().await.delete_attempts.is_empty()); + assert_eq!( + store + .load_table(&table.table_bucket, &table.namespace, &table.table) + .await + .expect("read unchanged table"), + Some(table.clone()) + ); + + let versions = object_versions(&backend).await; + run_pass(&mut scheduler).await.expect("repeated pass skips queued work"); + assert_eq!( + object_versions(&backend).await, + versions, + "queue-only repetition must not rewrite history or aliases" + ); + } + } + + #[tokio::test] + async fn periodic_queue_respects_disabled_paused_and_inherited_configuration() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let disabled = seed_table(&store, &backend, "disabled", "sales", "orders", None).await; + let paused = seed_table( + &store, + &backend, + "paused", + "sales", + "orders", + Some(TableMaintenanceConfig { + worker_paused: true, + ..enabled_config() + }), + ) + .await; + let inherited = seed_table(&store, &backend, "inherited", "sales", "orders", None).await; + backend + .seed_object( + RUSTFS_META_BUCKET, + &TableCatalogObjectPaths::default().table_bucket_maintenance_config_path("inherited"), + serde_json::to_vec(&enabled_config()).expect("serialize bucket defaults"), + ) + .await; + let mut scheduler = PeriodicMaintenanceScheduler::new( + store.clone(), + vec!["disabled".to_string(), "paused".to_string(), "inherited".to_string()], + StdDuration::from_secs(300), + ); + run_pass(&mut scheduler).await.expect("configuration-aware pass"); + assert!( + current(&store, &disabled).await.is_none(), + "disabled tables must not gain control history" + ); + assert!(current(&store, &paused).await.is_none(), "paused tables must not gain control history"); + let queued = current(&store, &inherited) + .await + .expect("bucket defaults must enable admission"); + assert_eq!(queued.job.config_source, TableMaintenanceConfigSource::TableBucketDefault); + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + } + } + + #[tokio::test] + async fn periodic_queue_never_recovers_failed_expired_or_quarantined_jobs() { + for backing in backings() { + for status in [ + TableMetadataMaintenanceJobStatus::Failed, + TableMetadataMaintenanceJobStatus::Queued, + TableMetadataMaintenanceJobStatus::Running, + TableMetadataMaintenanceJobStatus::Successful, + ] { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut report = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "fixture".to_string()) + .await + .expect("seed real queued report") + .report; + report.job.status = status.clone(); + report.job.attempt = 4; + report.job.max_retry_attempts = 3; + report.job.scheduled_at = Some(maintenance_timestamp(OffsetDateTime::UNIX_EPOCH)); + report.job.heartbeat_at = report.job.scheduled_at.clone(); + report.job.finished_at = report.job.scheduled_at.clone(); + report.job.next_retry_after = None; + if status == TableMetadataMaintenanceJobStatus::Successful { + report.job.quarantine_enabled = true; + report.job.quarantined_object_count = 1; + report.job.quarantine_retention_seconds = 3600; + } + seed_report(&backend, &table, &report).await; + let versions = object_versions(&backend).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + run_pass(&mut scheduler) + .await + .expect("manual recovery boundary is a normal skip"); + assert_eq!( + object_versions(&backend).await, + versions, + "periodic passes must preserve failed budgets, expired leases and quarantine" + ); + } + } + } + + #[tokio::test] + async fn periodic_queue_success_cooldown_requires_valid_persisted_timestamp() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut report = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "fixture".to_string()) + .await + .expect("seed real report") + .report; + report.job.status = TableMetadataMaintenanceJobStatus::Successful; + report.job.finished_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + seed_report(&backend, &table, &report).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + let versions = object_versions(&backend).await; + run_pass(&mut scheduler).await.expect("recent success is cooling down"); + assert_eq!(object_versions(&backend).await, versions); + for timestamp in [None, Some("not-a-timestamp".to_string())] { + report.job.finished_at = timestamp; + seed_report(&backend, &table, &report).await; + let versions = object_versions(&backend).await; + assert!(matches!(run_pass(&mut scheduler).await, Err(TableCatalogStoreError::Invalid(_)))); + assert_eq!(object_versions(&backend).await, versions, "invalid cooldown must fail without a new job"); + } + report.job.finished_at = Some(maintenance_timestamp(OffsetDateTime::UNIX_EPOCH)); + seed_report(&backend, &table, &report).await; + run_pass(&mut scheduler).await.expect("expired cooldown permits a new job"); + let queued = current(&store, &table).await.expect("next queue"); + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_ne!(queued.job.job_id, report.job.job_id); + } + } + + #[tokio::test] + async fn periodic_queue_pages_and_round_robins_buckets_across_passes() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let buckets = ["first", "second", "third"]; + let mut tables = Vec::new(); + for bucket in buckets { + for index in 0..5 { + tables.push( + seed_table( + &store, + &backend, + bucket, + if index < 2 { "sales" } else { "support" }, + &format!("table{index}"), + Some(enabled_config()), + ) + .await, + ); + } + } + let mut scheduler = PeriodicMaintenanceScheduler::new( + store.clone(), + buckets.iter().map(|bucket| (*bucket).to_string()).collect(), + StdDuration::from_secs(300), + ); + run_pass(&mut scheduler).await.expect("first bounded fair pass"); + let mut per_bucket = BTreeMap::new(); + for table in &tables { + if current(&store, table).await.is_some() { + *per_bucket.entry(table.table_bucket.clone()).or_insert(0_usize) += 1; + } + } + assert_eq!(per_bucket.values().sum::(), PERIODIC_MAINTENANCE_TABLES_PER_PASS); + assert_eq!(per_bucket.len(), buckets.len(), "one bucket must not consume every admission"); + assert!(per_bucket.values().all(|count| *count <= 3)); + run_pass(&mut scheduler).await.expect("resume all retained cursors"); + for table in &tables { + assert_eq!( + current(&store, table) + .await + .expect("every table reached through retained cursors") + .job + .status, + TableMetadataMaintenanceJobStatus::Queued + ); + } + } + } + + #[tokio::test] + async fn periodic_queue_cancelled_or_expired_pass_does_not_start_discovery() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store, vec!["analytics".to_string()], StdDuration::from_secs(300)); + backend.reset_call_counts().await; + let cancelled = CancellationToken::new(); + cancelled.cancel(); + scheduler + .run_pass(&cancelled, Instant::now() + StdDuration::from_secs(60)) + .await + .expect("cancelled pass"); + scheduler + .run_pass(&CancellationToken::new(), Instant::now()) + .await + .expect("expired pass"); + assert_eq!(backend.read_call_count().await, 0); + assert_eq!(backend.list_call_count().await, 0); + } + } + + #[tokio::test(start_paused = true)] + async fn periodic_queue_retains_inflight_publication_after_cancellation_and_deadline() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let first = seed_table(&store, &backend, "analytics", "sales", "a", Some(enabled_config())).await; + let second = seed_table(&store, &backend, "analytics", "sales", "b", Some(enabled_config())).await; + let namespace = Namespace::parse(&first.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&first.table).expect("fixture table"); + let path = TableCatalogObjectPaths::default().table_maintenance_current_job_path( + "analytics", + &namespace, + &name, + &first.table_id, + ); + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &path).await; + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + let cancellation = CancellationToken::new(); + let runner_cancellation = cancellation.clone(); + let deadline = Instant::now() + StdDuration::from_secs(30); + let task = tokio::spawn(async move { scheduler.run_pass(&runner_cancellation, deadline).await }); + tokio::time::timeout(StdDuration::from_secs(5), pause.wait_started()) + .await + .expect("queue reaches publication pause"); + cancellation.cancel(); + tokio::time::advance(StdDuration::from_secs(31)).await; + assert!( + !task.is_finished(), + "cancellation must retain the mutating future until its write returns" + ); + pause.release(); + task.await.expect("runner task").expect("in-flight queue completes"); + assert_eq!( + current(&store, &first).await.expect("complete first publication").job.status, + TableMetadataMaintenanceJobStatus::Queued + ); + assert!( + current(&store, &second).await.is_none(), + "no operation starts after cancellation or admission deadline" + ); + } + } + + #[tokio::test] + async fn periodic_queue_atomic_guard_rejects_a_claim_changed_during_planning() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let pause = backend.pause_next_read(&table.table_bucket, &table.metadata_location).await; + let claimant = store.clone(); + let observed = table.clone(); + let task = tokio::spawn(async move { + claimant + .queue_periodic_table_maintenance(&observed, "first".to_string(), None) + .await + }); + tokio::time::timeout(StdDuration::from_secs(5), pause.wait_started()) + .await + .expect("first claimant starts unlocked planning"); + let winner = store + .queue_periodic_table_maintenance(&table, "second".to_string(), None) + .await + .expect("second claimant wins while first is planning"); + let versions = object_versions(&backend).await; + pause.release(); + assert!(matches!( + task.await.expect("first claimant task"), + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!( + current(&store, &table).await.expect("winning current job").job.job_id, + winner.report.job.job_id + ); + assert_eq!(object_versions(&backend).await, versions, "a stale claim must not publish any sidecar"); + assert_eq!( + store + .load_table(&table.table_bucket, &table.namespace, &table.table) + .await + .expect("read table"), + Some(table) + ); + } + } + + #[tokio::test] + async fn periodic_queue_atomic_guard_rejects_changed_identity_config_and_partial_aliases() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let mut old_identity = table.clone(); + old_identity.table_id = uuid::Uuid::new_v4().to_string(); + let versions = object_versions(&backend).await; + assert!(matches!( + store + .queue_periodic_table_maintenance(&old_identity, "periodic".to_string(), None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!(object_versions(&backend).await, versions); + + for config in [ + TableMaintenanceConfig::default(), + TableMaintenanceConfig { + worker_paused: true, + ..enabled_config() + }, + ] { + store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .expect("change config after selection"); + let versions = object_versions(&backend).await; + assert!(matches!( + store + .queue_periodic_table_maintenance(&table, "periodic".to_string(), None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!( + object_versions(&backend).await, + versions, + "changed config must not cause Disabled or Paused control writes" + ); + } + store + .put_table_maintenance_config("analytics", "sales", "orders", enabled_config()) + .await + .expect("restore fixture config"); + let mut report = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "fixture".to_string()) + .await + .expect("seed full report") + .report; + report.job.status = TableMetadataMaintenanceJobStatus::Successful; + report.job.finished_at = Some(maintenance_timestamp(OffsetDateTime::UNIX_EPOCH)); + seed_report(&backend, &table, &report).await; + let expected = table_maintenance_scheduler_job_summary(&report); + report.job.status = TableMetadataMaintenanceJobStatus::Failed; + let namespace = Namespace::parse(&table.namespace).expect("fixture namespace"); + let name = IdentifierSegment::parse(&table.table).expect("fixture table"); + let latest = TableCatalogObjectPaths::default().table_maintenance_latest_job_path( + "analytics", + &namespace, + &name, + &table.table_id, + ); + backend + .seed_object( + RUSTFS_META_BUCKET, + &latest, + serde_json::to_vec(&report).expect("serialize partial publication"), + ) + .await; + let versions = object_versions(&backend).await; + assert!(matches!( + store + .queue_periodic_table_maintenance(&table, "periodic".to_string(), Some(&expected)) + .await, + Err(TableCatalogStoreError::Conflict(_)) + )); + assert_eq!( + object_versions(&backend).await, + versions, + "partial report publication requires explicit recovery" + ); + } + } + + #[tokio::test] + async fn periodic_queue_empty_discovery_is_bounded_and_inactive_buckets_are_skipped() { + for backing in backings() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), backing); + store + .put_table_bucket(test_bucket_entry("empty")) + .await + .expect("empty active bucket"); + let mut inactive = test_bucket_entry("inactive"); + inactive.state = TableCatalogEntryState::Deleted; + store.put_table_bucket(inactive).await.expect("inactive fixture bucket"); + backend.reset_call_counts().await; + let versions = object_versions(&backend).await; + let mut scheduler = PeriodicMaintenanceScheduler::new( + store, + vec!["empty".to_string(), "inactive".to_string(), "missing".to_string()], + StdDuration::from_secs(300), + ); + run_pass(&mut scheduler) + .await + .expect("bounded discovery with no eligible tables"); + assert_eq!(object_versions(&backend).await, versions); + assert!( + backend.list_call_count().await <= PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS, + "empty discovery must stop at the pass budget" + ); + assert_eq!(scheduler.next_bucket, PERIODIC_MAINTENANCE_DISCOVERY_STEPS_PER_PASS % 3); + } + } + + #[tokio::test] + async fn periodic_queue_preserves_empty_object_page_continuations() { + let backend = TestCatalogObjectBackend::default(); + let store = TestStore::new_for_test(backend.clone(), TableCatalogBackingMode::ObjectBacked); + let table = seed_table(&store, &backend, "analytics", "sales", "orders", Some(enabled_config())).await; + let prefix = TableCatalogObjectPaths::default().namespace_entries_prefix("analytics"); + for index in 0..TABLE_CATALOG_LIST_MAX_KEYS { + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{prefix}!non-entry-{index:04}"), b"fixture".to_vec()) + .await; + } + let first_page = store + .list_namespaces_page("analytics", None, NonZeroUsize::MIN) + .await + .expect("bounded empty namespace page"); + assert!(first_page.entries.is_empty()); + assert!(first_page.next_cursor.is_some()); + let mut scheduler = + PeriodicMaintenanceScheduler::new(store.clone(), vec!["analytics".to_string()], StdDuration::from_secs(300)); + run_pass(&mut scheduler).await.expect("continue past an empty page"); + assert_eq!( + current(&store, &table) + .await + .expect("table after the empty page must be reached") + .job + .status, + TableMetadataMaintenanceJobStatus::Queued + ); + } +} diff --git a/rustfs/src/table_catalog/maintenance/periodic_runtime.rs b/rustfs/src/table_catalog/maintenance/periodic_runtime.rs new file mode 100644 index 000000000..088063efc --- /dev/null +++ b/rustfs/src/table_catalog/maintenance/periodic_runtime.rs @@ -0,0 +1,339 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::periodic::PeriodicMaintenanceScheduler; +use crate::storage_api::startup::services::ServerContextSlot; +use crate::storage_api::table::check_valid_bucket_name_strict; +use crate::table_catalog::{ + ConfiguredTableCatalogStore, EcStoreTableCatalogObjectBackend, ObjectTableCatalogStore, StrongTableCatalogStore, + TableCatalogBackingMode, TableCatalogStoreError, TableCatalogStoreResult, +}; +use std::{collections::BTreeSet, future::Future, sync::Arc, time::Duration}; +use tokio::{task::JoinHandle, time::Instant}; +use tokio_util::sync::CancellationToken; +use tracing::{info, warn}; + +const ENV_ENABLED: &str = "RUSTFS_TABLE_MAINTENANCE_SCHEDULER_ENABLED"; +const ENV_BUCKETS: &str = "RUSTFS_TABLE_MAINTENANCE_SCHEDULER_BUCKETS"; +const ENV_INTERVAL: &str = "RUSTFS_TABLE_MAINTENANCE_SCHEDULER_INTERVAL_SECONDS"; +const ADMISSION_BUDGET: Duration = Duration::from_secs(30); +const LOG_COMPONENT: &str = "table_catalog"; +const LOG_SUBSYSTEM: &str = "maintenance"; +const EVENT_PERIODIC_SCHEDULER: &str = "table_maintenance_periodic_scheduler"; + +pub(crate) struct PeriodicMaintenanceConfig { + buckets: Vec, + interval: Duration, + backing: TableCatalogBackingMode, +} + +impl PeriodicMaintenanceConfig { + pub(crate) fn from_env() -> TableCatalogStoreResult> { + Self::parse( + |key| match std::env::var(key) { + Ok(value) => Ok(Some(value)), + Err(std::env::VarError::NotPresent) => Ok(None), + Err(std::env::VarError::NotUnicode(_)) => { + Err(TableCatalogStoreError::Invalid(format!("{key} must be valid UTF-8"))) + } + }, + TableCatalogBackingMode::from_env, + ) + } + + fn parse( + read: impl Fn(&str) -> TableCatalogStoreResult>, + backing: impl FnOnce() -> TableCatalogStoreResult, + ) -> TableCatalogStoreResult> { + match read(ENV_ENABLED)?.as_deref().map(str::trim) { + None | Some("false") => return Ok(None), + Some("true") => {} + _ => return Err(TableCatalogStoreError::Invalid(format!("{ENV_ENABLED} must be true or false"))), + } + let raw = read(ENV_BUCKETS)? + .ok_or_else(|| TableCatalogStoreError::Invalid(format!("{ENV_BUCKETS} is required when scheduling is enabled")))?; + if raw.len() > 2048 { + return Err(TableCatalogStoreError::Invalid(format!("{ENV_BUCKETS} exceeds the allowlist size limit"))); + } + let mut buckets = BTreeSet::new(); + for bucket in raw.split(',').map(str::trim) { + check_valid_bucket_name_strict(bucket) + .map_err(|_| TableCatalogStoreError::Invalid(format!("{ENV_BUCKETS} contains an invalid bucket name")))?; + if !buckets.insert(bucket.to_string()) || buckets.len() > 32 { + return Err(TableCatalogStoreError::Invalid(format!( + "{ENV_BUCKETS} requires 1 to 32 distinct bucket names" + ))); + } + } + let interval = read(ENV_INTERVAL)?.unwrap_or_else(|| "300".to_string()); + let seconds = interval + .trim() + .parse::() + .ok() + .filter(|seconds| (30..=86_400).contains(seconds)) + .ok_or_else(|| TableCatalogStoreError::Invalid(format!("{ENV_INTERVAL} must be between 30 and 86400")))?; + Ok(Some(Self { + buckets: buckets.into_iter().collect(), + interval: Duration::from_secs(seconds), + backing: backing()?, + })) + } +} + +/// Owns queueing until completion; never runs a worker or deletes table objects. +pub(crate) struct PeriodicMaintenanceRuntime { + cancellation: CancellationToken, + task: Option>, +} + +impl Drop for PeriodicMaintenanceRuntime { + fn drop(&mut self) { + // Detach, never abort: in-flight sidecar writes retain their publication owner. + self.cancellation.cancel(); + } +} + +impl PeriodicMaintenanceRuntime { + pub(crate) fn spawn(config: PeriodicMaintenanceConfig, server: Arc, shutdown: &CancellationToken) -> Self { + let cancellation = shutdown.child_token(); + let task = tokio::spawn(run(config, server, cancellation.clone())); + Self { + cancellation, + task: Some(task), + } + } + + pub(crate) async fn shutdown(mut self, grace: Duration) -> bool { + self.cancellation.cancel(); + let Some(task) = self.task.take() else { + return true; + }; + let joined = matches!(tokio::time::timeout(grace, task).await, Ok(Ok(()))); + if !joined { + warn!( + event = EVENT_PERIODIC_SCHEDULER, + component = LOG_COMPONENT, + subsystem = LOG_SUBSYSTEM, + state = "drain_unconfirmed", + "Maintenance scheduler completion is unconfirmed; owned work was not aborted" + ); + } + joined + } +} + +fn tick_delay(interval: Duration) -> Duration { + // Delay after completion avoids catch-up bursts; jitter spreads node scans. + interval + Duration::from_secs(rand::random_range(0..=interval.as_secs() / 10)) +} + +async fn wait_for_tick(cancellation: &CancellationToken, delay: Duration) -> bool { + tokio::select! { + biased; + _ = cancellation.cancelled() => false, + _ = tokio::time::sleep(delay) => true, + } +} + +async fn finish_owned_pass(pass: F, cancellation: &CancellationToken, deadline: Instant) -> F::Output { + tokio::pin!(pass); + tokio::select! { + biased; + result = &mut pass => result, + _ = cancellation.cancelled() => pass.await, + _ = tokio::time::sleep_until(deadline) => { + warn!( + event = EVENT_PERIODIC_SCHEDULER, component = LOG_COMPONENT, subsystem = LOG_SUBSYSTEM, + state = "draining", reason = "admission_deadline", + "Maintenance scheduler stopped admitting work while its store call drains" + ); + pass.await + } + } +} + +async fn run(config: PeriodicMaintenanceConfig, server: Arc, cancellation: CancellationToken) { + let mut scheduler = None; + let mut degraded = false; + while wait_for_tick(&cancellation, tick_delay(config.interval)).await { + if scheduler.is_none() { + // Deferred IAM may not have installed this server yet. Do not use + // another embedded server's ambient context as a fallback. + let Some(context) = server.installed_app_context() else { + continue; + }; + let backend = EcStoreTableCatalogObjectBackend::new_with_strong_runtime( + context.object_store(), + context.table_catalog_strong_runtime(), + ); + let store = match config.backing { + TableCatalogBackingMode::ObjectBacked => { + ConfiguredTableCatalogStore::ObjectBacked(ObjectTableCatalogStore::new(backend)) + } + TableCatalogBackingMode::DurableStrong => { + ConfiguredTableCatalogStore::DurableStrong(StrongTableCatalogStore::new_requiring_snapshot(backend)) + } + }; + scheduler = Some(PeriodicMaintenanceScheduler::new(store, config.buckets.clone(), config.interval)); + } + let deadline = Instant::now() + ADMISSION_BUDGET; + let Some(scheduler) = scheduler.as_mut() else { + continue; + }; + let result = finish_owned_pass(scheduler.run_pass(&cancellation, deadline), &cancellation, deadline).await; + match result { + Ok(()) if degraded => { + info!( + event = EVENT_PERIODIC_SCHEDULER, + component = LOG_COMPONENT, + subsystem = LOG_SUBSYSTEM, + state = "recovered", + "Maintenance scheduler pass recovered" + ); + degraded = false; + } + Err(_) if !degraded => { + warn!( + event = EVENT_PERIODIC_SCHEDULER, + component = LOG_COMPONENT, + subsystem = LOG_SUBSYSTEM, + state = "degraded", + reason = "queue_pass_failed", + "Maintenance scheduler could not complete a queueing pass" + ); + degraded = true; + } + _ => {} + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::{AtomicBool, Ordering}; + + fn config(values: &[(&str, &str)]) -> TableCatalogStoreResult> { + PeriodicMaintenanceConfig::parse( + |key| { + Ok(values + .iter() + .find(|(name, _)| *name == key) + .map(|(_, value)| value.to_string())) + }, + || Ok(TableCatalogBackingMode::DurableStrong), + ) + } + + #[test] + fn periodic_config_is_opt_in_and_validates_limits() { + assert!(config(&[]).unwrap().is_none()); + assert!( + config(&[(ENV_ENABLED, "false"), (ENV_BUCKETS, "invalid/path")]) + .unwrap() + .is_none() + ); + for value in ["", "1", "yes", "TRUE"] { + assert!(config(&[(ENV_ENABLED, value)]).is_err()); + } + assert!(config(&[(ENV_ENABLED, "true")]).is_err()); + for buckets in ["", "foo,", "foo,foo", "../foo", "127.0.0.1", "Upper", ".minio.sys"] { + assert!(config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, buckets)]).is_err(), "{buckets}"); + } + let too_many = (0..33).map(|n| format!("bucket-{n}")).collect::>().join(","); + assert!(config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, &too_many)]).is_err()); + for interval in ["0", "29", "86401", "-1", "NaN", "18446744073709551616"] { + assert!(config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, "foo"), (ENV_INTERVAL, interval)]).is_err()); + } + let parsed = config(&[(ENV_ENABLED, "true"), (ENV_BUCKETS, "bbb, aaa")]).unwrap().unwrap(); + assert_eq!(parsed.buckets, ["aaa", "bbb"]); + assert_eq!(parsed.interval, Duration::from_secs(300)); + for _ in 0..100 { + assert!((Duration::from_secs(300)..=Duration::from_secs(330)).contains(&tick_delay(parsed.interval))); + } + } + + #[tokio::test(start_paused = true)] + async fn periodic_tick_waits_and_cancels_without_work() { + let cancellation = CancellationToken::new(); + let start = Instant::now(); + assert!(wait_for_tick(&cancellation, Duration::from_secs(300)).await); + assert_eq!(Instant::now() - start, Duration::from_secs(300)); + cancellation.cancel(); + assert!(!wait_for_tick(&cancellation, Duration::from_secs(300)).await); + assert_eq!(Instant::now() - start, Duration::from_secs(300)); + } + + #[tokio::test(start_paused = true)] + async fn periodic_deadline_and_shutdown_retain_owned_work() { + for cancel in [false, true] { + let cancellation = CancellationToken::new(); + let completed = Arc::new(AtomicBool::new(false)); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, resume) = tokio::sync::oneshot::channel(); + let done = completed.clone(); + let token = cancellation.clone(); + let task = tokio::spawn(async move { + finish_owned_pass( + async move { + entered.send(()).unwrap(); + resume.await.unwrap(); + done.store(true, Ordering::SeqCst); + }, + &token, + Instant::now() + Duration::from_secs(30), + ) + .await; + }); + started.await.unwrap(); + if cancel { + cancellation.cancel(); + } else { + tokio::time::advance(Duration::from_secs(31)).await; + } + tokio::task::yield_now().await; + assert!(!task.is_finished()); + assert!(!completed.load(Ordering::SeqCst)); + release.send(()).unwrap(); + task.await.unwrap(); + assert!(completed.load(Ordering::SeqCst)); + } + } + + #[tokio::test(start_paused = true)] + async fn periodic_shutdown_reports_incomplete_drain_without_aborting() { + let completed = Arc::new(AtomicBool::new(false)); + let done = completed.clone(); + let (release, resume) = tokio::sync::oneshot::channel(); + let (finished, observed) = tokio::sync::oneshot::channel(); + let runtime = PeriodicMaintenanceRuntime { + cancellation: CancellationToken::new(), + task: Some(tokio::spawn(async move { + resume.await.unwrap(); + done.store(true, Ordering::SeqCst); + finished.send(()).unwrap(); + })), + }; + assert!(!runtime.shutdown(Duration::from_secs(30)).await); + assert!(!completed.load(Ordering::SeqCst)); + release.send(()).unwrap(); + observed.await.unwrap(); + assert!(completed.load(Ordering::SeqCst)); + let runtime = PeriodicMaintenanceRuntime { + cancellation: CancellationToken::new(), + task: Some(tokio::spawn(async {})), + }; + assert!(runtime.shutdown(Duration::from_secs(30)).await); + } +} diff --git a/rustfs/src/table_catalog/maintenance/planner.rs b/rustfs/src/table_catalog/maintenance/planner.rs index e6b39354e..fdc6afaf0 100644 --- a/rustfs/src/table_catalog/maintenance/planner.rs +++ b/rustfs/src/table_catalog/maintenance/planner.rs @@ -213,9 +213,7 @@ where Some(manifest_list) => { let planning = match compaction_data_file_candidates( backend, - table_bucket, - namespace, - table, + entry, warehouse_object_prefix.as_deref(), manifest_list, &config, @@ -326,9 +324,7 @@ where async fn compaction_data_file_candidates( backend: &B, - table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + entry: &TableEntry, warehouse_object_prefix: Option<&str>, manifest_list: &str, config: &TableCompactionPlanningConfig, @@ -336,12 +332,13 @@ async fn compaction_data_file_candidates( where B: TableCatalogObjectBackend, { + let table_bucket = &entry.table_bucket; let Some(manifest_list_key) = table_catalog_object_key_from_location(table_bucket, manifest_list) else { return Err(TableCatalogStoreError::Invalid( "compaction manifest list must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_list_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &manifest_list_key) != Some(TableMetadataMaintenanceObjectKind::ManifestList) { return Err(TableCatalogStoreError::Invalid( @@ -367,7 +364,7 @@ where "compaction manifest must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &manifest_key) != Some(TableMetadataMaintenanceObjectKind::ManifestFile) { return Err(TableCatalogStoreError::Invalid( @@ -384,9 +381,7 @@ where if reference.object_kind != TableMetadataMaintenanceObjectKind::DataFile { record_compaction_row_level_delete_file( backend, - table_bucket, - namespace, - table, + entry, warehouse_object_prefix, &mut planning.row_level_planning, &reference, @@ -400,7 +395,7 @@ where "compaction data file must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &data_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &data_key) != Some(TableMetadataMaintenanceObjectKind::DataFile) { return Err(TableCatalogStoreError::Invalid( @@ -413,7 +408,7 @@ where let size_bytes = u64::try_from(data_object.data.len()).unwrap_or(u64::MAX); if size_bytes <= config.small_file_threshold_bytes { planning.candidates.push(TableCompactionDataFileCandidate { - rewrite_prefix: compaction_data_file_rewrite_prefix(namespace, table, warehouse_object_prefix, &data_key) + rewrite_prefix: compaction_data_file_rewrite_prefix_for_entry(entry, warehouse_object_prefix, &data_key) .unwrap_or_else(|| data_key.clone()), location: data_key, size_bytes, @@ -427,9 +422,7 @@ where async fn record_compaction_row_level_delete_file( backend: &B, - table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + entry: &TableEntry, warehouse_object_prefix: Option<&str>, planning: &mut TableRowLevelMaintenancePlanningReport, reference: &ManifestDataFileReference, @@ -437,6 +430,7 @@ async fn record_compaction_row_level_delete_file( where B: TableCatalogObjectBackend, { + let table_bucket = &entry.table_bucket; let Some(content) = row_level_delete_file_content(reference.content) else { return Ok(()); }; @@ -445,7 +439,7 @@ where "compaction delete file must be inside the table bucket".to_string(), )); }; - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &delete_key) + if table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, &delete_key) != Some(TableMetadataMaintenanceObjectKind::DeleteFile) { return Err(TableCatalogStoreError::Invalid( @@ -714,6 +708,29 @@ pub(crate) fn compaction_data_file_rewrite_prefix( None } +fn compaction_data_file_rewrite_prefix_for_entry( + entry: &TableEntry, + warehouse_object_prefix: Option<&str>, + location: &str, +) -> Option { + let metadata_dir = table_metadata_dir_path_for_entry(entry).ok()?; + let table_root = metadata_dir.strip_suffix(&format!("/{METADATA_DIR}"))?; + let entry_data_prefix = format!("{table_root}/{DATA_DIR}/"); + let warehouse_data_prefix = warehouse_object_prefix + .map(|prefix| format!("{prefix}{DATA_DIR}")) + .unwrap_or_else(|| format!("{table_root}/{DATA_DIR}")); + if let Some(relative_path) = location.strip_prefix(&entry_data_prefix) { + return Some(compaction_data_file_output_prefix(&warehouse_data_prefix, relative_path)); + } + if let Some(warehouse_object_prefix) = warehouse_object_prefix { + let warehouse_input_prefix = format!("{warehouse_object_prefix}{DATA_DIR}/"); + if let Some(relative_path) = location.strip_prefix(&warehouse_input_prefix) { + return Some(compaction_data_file_output_prefix(&warehouse_data_prefix, relative_path)); + } + } + None +} + fn compaction_data_file_output_prefix(output_data_prefix: &str, relative_path: &str) -> String { relative_path .rsplit_once('/') diff --git a/rustfs/src/table_catalog/maintenance/recovery.rs b/rustfs/src/table_catalog/maintenance/recovery.rs index fc39cebec..a60b9600c 100644 --- a/rustfs/src/table_catalog/maintenance/recovery.rs +++ b/rustfs/src/table_catalog/maintenance/recovery.rs @@ -72,6 +72,51 @@ fn insert_referenced_object_report( report.reasons.insert(reason); } +#[derive(Clone, Copy)] +enum TableMaintenanceObjectScope<'a> { + Logical { + namespace: &'a Namespace, + table: &'a IdentifierSegment, + }, + Entry(&'a TableEntry), +} + +impl TableMaintenanceObjectScope<'_> { + fn object_kind( + self, + warehouse_object_prefix: Option<&str>, + object_location: &str, + ) -> Option { + match self { + Self::Logical { namespace, table } => { + table_maintenance_object_kind(namespace, table, warehouse_object_prefix, object_location) + } + Self::Entry(entry) => table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix, object_location), + } + } + + fn object_directories(self) -> TableCatalogStoreResult<(String, String, String)> { + match self { + Self::Logical { namespace, table } => Ok(( + default_table_metadata_dir_path(namespace, table), + default_table_data_dir_path(namespace, table), + default_table_delete_dir_path(namespace, table), + )), + Self::Entry(entry) => { + let metadata_dir = table_metadata_dir_path_for_entry(entry)?; + let table_root = metadata_dir.strip_suffix(&format!("/{METADATA_DIR}")).ok_or_else(|| { + TableCatalogStoreError::Invalid( + "current metadata location must be inside a table metadata directory".to_string(), + ) + })?; + let data_dir = format!("{table_root}/{DATA_DIR}"); + let delete_dir = format!("{table_root}/{DELETE_DIR}"); + Ok((metadata_dir, data_dir, delete_dir)) + } + } + } +} + pub(crate) async fn metadata_maintenance_referenced_object_reports( backend: &B, table_bucket: &str, @@ -81,6 +126,50 @@ pub(crate) async fn metadata_maintenance_referenced_object_reports( current_metadata: &serde_json::Value, retained_metadata_locations: &[String], ) -> TableCatalogStoreResult> +where + B: TableCatalogObjectBackend, +{ + metadata_maintenance_referenced_object_reports_in_scope( + backend, + table_bucket, + TableMaintenanceObjectScope::Logical { namespace, table }, + warehouse_object_prefix, + current_metadata, + retained_metadata_locations, + ) + .await +} + +pub(crate) async fn metadata_maintenance_referenced_object_reports_for_entry( + backend: &B, + entry: &TableEntry, + warehouse_object_prefix: Option<&str>, + current_metadata: &serde_json::Value, + retained_metadata_locations: &[String], +) -> TableCatalogStoreResult> +where + B: TableCatalogObjectBackend, +{ + table_metadata_dir_path_for_entry(entry)?; + metadata_maintenance_referenced_object_reports_in_scope( + backend, + &entry.table_bucket, + TableMaintenanceObjectScope::Entry(entry), + warehouse_object_prefix, + current_metadata, + retained_metadata_locations, + ) + .await +} + +async fn metadata_maintenance_referenced_object_reports_in_scope( + backend: &B, + table_bucket: &str, + scope: TableMaintenanceObjectScope<'_>, + warehouse_object_prefix: Option<&str>, + current_metadata: &serde_json::Value, + retained_metadata_locations: &[String], +) -> TableCatalogStoreResult> where B: TableCatalogObjectBackend, { @@ -88,8 +177,7 @@ where metadata_maintenance_referenced_object_reports_for_metadata( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, current_metadata, &mut reports, @@ -114,8 +202,7 @@ where metadata_maintenance_referenced_object_reports_for_metadata( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, &metadata, &mut reports, @@ -137,8 +224,7 @@ where async fn metadata_maintenance_referenced_object_reports_for_metadata( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, metadata: &serde_json::Value, reports: &mut BTreeMap, @@ -155,8 +241,7 @@ where metadata_maintenance_referenced_manifest_list( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, manifest_list_location, reports, @@ -182,8 +267,7 @@ where metadata_maintenance_referenced_manifest_file( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, manifest_location, reports, @@ -198,8 +282,7 @@ where async fn metadata_maintenance_referenced_manifest_list( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, manifest_list_location: &str, reports: &mut BTreeMap, @@ -217,9 +300,12 @@ where ); return Ok(()); }; - if !table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_list_key).is_some_and(|kind| { - table_maintenance_object_kind_matches_reference(&kind, &TableMetadataMaintenanceObjectKind::ManifestList) - }) { + if !scope + .object_kind(warehouse_object_prefix, &manifest_list_key) + .is_some_and(|kind| { + table_maintenance_object_kind_matches_reference(&kind, &TableMetadataMaintenanceObjectKind::ManifestList) + }) + { insert_referenced_object_report( reports, manifest_list_key, @@ -272,8 +358,7 @@ where metadata_maintenance_referenced_manifest_file( backend, table_bucket, - namespace, - table, + scope, warehouse_object_prefix, &manifest_location, reports, @@ -287,8 +372,7 @@ where async fn metadata_maintenance_referenced_manifest_file( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, manifest_location: &str, reports: &mut BTreeMap, @@ -306,7 +390,7 @@ where ); return Ok(()); }; - if !table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &manifest_key).is_some_and(|kind| { + if !scope.object_kind(warehouse_object_prefix, &manifest_key).is_some_and(|kind| { table_maintenance_object_kind_matches_reference(&kind, &TableMetadataMaintenanceObjectKind::ManifestFile) }) { insert_referenced_object_report( @@ -360,7 +444,8 @@ where ); continue; }; - if !table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &file_key) + if !scope + .object_kind(warehouse_object_prefix, &file_key) .is_some_and(|kind| table_maintenance_object_kind_matches_reference(&kind, &object_kind)) { insert_referenced_object_report( @@ -643,8 +728,50 @@ pub(crate) async fn metadata_maintenance_object_cleanup_reports( where B: TableCatalogObjectBackend, { - let scanned_objects = - table_maintenance_cleanup_objects(backend, table_bucket, namespace, table, warehouse_object_prefix).await?; + metadata_maintenance_object_cleanup_reports_in_scope( + backend, + table_bucket, + TableMaintenanceObjectScope::Logical { namespace, table }, + warehouse_object_prefix, + referenced_object_reports, + now, + ) + .await +} + +pub(crate) async fn metadata_maintenance_object_cleanup_reports_for_entry( + backend: &B, + entry: &TableEntry, + warehouse_object_prefix: Option<&str>, + referenced_object_reports: &[TableMetadataMaintenanceReferencedObjectReport], + now: OffsetDateTime, +) -> TableCatalogStoreResult<(usize, Vec, Vec, Vec)> +where + B: TableCatalogObjectBackend, +{ + metadata_maintenance_object_cleanup_reports_in_scope( + backend, + &entry.table_bucket, + TableMaintenanceObjectScope::Entry(entry), + warehouse_object_prefix, + referenced_object_reports, + now, + ) + .await +} + +async fn metadata_maintenance_object_cleanup_reports_in_scope( + backend: &B, + table_bucket: &str, + scope: TableMaintenanceObjectScope<'_>, + warehouse_object_prefix: Option<&str>, + referenced_object_reports: &[TableMetadataMaintenanceReferencedObjectReport], + now: OffsetDateTime, +) -> TableCatalogStoreResult<(usize, Vec, Vec, Vec)> +where + B: TableCatalogObjectBackend, +{ + let scanned_objects = table_maintenance_cleanup_objects(backend, table_bucket, scope, warehouse_object_prefix).await?; if referenced_object_reports .iter() .any(|report| report.state == TableMetadataMaintenanceObjectState::ManualReviewRequired) @@ -700,17 +827,17 @@ where async fn table_maintenance_cleanup_objects( backend: &B, table_bucket: &str, - namespace: &Namespace, - table: &IdentifierSegment, + scope: TableMaintenanceObjectScope<'_>, warehouse_object_prefix: Option<&str>, ) -> TableCatalogStoreResult> where B: TableCatalogObjectBackend, { let mut objects = BTreeMap::new(); - let mut metadata_prefixes = vec![format!("{}/", default_table_metadata_dir_path(namespace, table))]; - let mut data_prefixes = vec![format!("{}/", default_table_data_dir_path(namespace, table))]; - let mut delete_prefixes = vec![format!("{}/", default_table_delete_dir_path(namespace, table))]; + let (metadata_dir, data_dir, delete_dir) = scope.object_directories()?; + let mut metadata_prefixes = vec![format!("{metadata_dir}/")]; + let mut data_prefixes = vec![format!("{data_dir}/")]; + let mut delete_prefixes = vec![format!("{delete_dir}/")]; if let Some(warehouse_object_prefix) = warehouse_object_prefix { metadata_prefixes.push(format!("{warehouse_object_prefix}{METADATA_DIR}/")); data_prefixes.push(format!("{warehouse_object_prefix}{DATA_DIR}/")); @@ -725,7 +852,7 @@ where for metadata_prefix in metadata_prefixes { for object in backend.list_objects(table_bucket, &metadata_prefix).await? { - if let Some(kind) = table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &object) + if let Some(kind) = scope.object_kind(warehouse_object_prefix, &object) && matches!( kind, TableMetadataMaintenanceObjectKind::ManifestList | TableMetadataMaintenanceObjectKind::ManifestFile @@ -738,9 +865,7 @@ where for data_prefix in data_prefixes { for object in backend.list_objects(table_bucket, &data_prefix).await? { - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &object) - == Some(TableMetadataMaintenanceObjectKind::DataFile) - { + if scope.object_kind(warehouse_object_prefix, &object) == Some(TableMetadataMaintenanceObjectKind::DataFile) { objects.insert(object, TableMetadataMaintenanceObjectKind::DataFile); } } @@ -748,9 +873,7 @@ where for delete_prefix in delete_prefixes { for object in backend.list_objects(table_bucket, &delete_prefix).await? { - if table_maintenance_object_kind(namespace, table, warehouse_object_prefix, &object) - == Some(TableMetadataMaintenanceObjectKind::DeleteFile) - { + if scope.object_kind(warehouse_object_prefix, &object) == Some(TableMetadataMaintenanceObjectKind::DeleteFile) { objects.insert(object, TableMetadataMaintenanceObjectKind::DeleteFile); } } diff --git a/rustfs/src/table_catalog/maintenance/worker.rs b/rustfs/src/table_catalog/maintenance/worker.rs index 7c75aa413..834278297 100644 --- a/rustfs/src/table_catalog/maintenance/worker.rs +++ b/rustfs/src/table_catalog/maintenance/worker.rs @@ -223,3 +223,205 @@ pub(crate) fn apply_maintenance_retry_after( let delay_seconds = i64::try_from(delay_seconds).unwrap_or(i64::MAX); job.next_retry_after = Some(maintenance_timestamp(now.saturating_add(Duration::seconds(delay_seconds)))); } + +pub(crate) fn validate_table_maintenance_worker_condition( + worker_id: &str, + condition: &TableMaintenanceWorkerCondition, +) -> TableCatalogStoreResult<()> { + IdentifierSegment::parse(condition.expected_job_id.clone()) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid expected maintenance job ID: {err}")))?; + if matches!( + condition.expected_job_id.as_str(), + MAINTENANCE_JOB_ALIAS_CURRENT | MAINTENANCE_JOB_ALIAS_LATEST + ) { + return Err(TableCatalogStoreError::Invalid( + "expected maintenance job ID cannot be an alias".to_string(), + )); + } + if worker_id.trim().is_empty() || condition.expected_attempt == u16::MAX { + return Err(TableCatalogStoreError::Invalid( + "conditional maintenance requires a nonempty worker ID and an incrementable expected attempt".to_string(), + )); + } + Ok(()) +} + +/// True means that this request already claimed the recorded attempt. It must +/// return that report without executing its candidates again. +pub(crate) fn conditional_table_maintenance_is_replay( + report: &TableMetadataMaintenanceReport, + worker_id: &str, + condition: &TableMaintenanceWorkerCondition, +) -> bool { + report.job.job_id == condition.expected_job_id + && condition.expected_attempt.checked_add(1) == Some(report.job.attempt) + && report.job.worker_id.as_deref() == Some(worker_id) + && !report.job.lease_id.is_empty() + && matches!( + report.job.status, + TableMetadataMaintenanceJobStatus::Running + | TableMetadataMaintenanceJobStatus::Successful + | TableMetadataMaintenanceJobStatus::Failed + ) +} + +pub(crate) fn validate_conditional_table_maintenance_claim( + report: &TableMetadataMaintenanceReport, + entry: &TableEntry, + effective: &TableMaintenanceEffectiveConfig, + worker_id: &str, + condition: &TableMaintenanceWorkerCondition, + now: OffsetDateTime, +) -> TableCatalogStoreResult { + let job = &report.job; + if job.job_id != condition.expected_job_id + || job.table_id != entry.table_id + || report.current_metadata_location != entry.metadata_location + || job.current_metadata_location != entry.metadata_location + || job.current_generation != entry.generation + { + return Err(TableCatalogStoreError::Conflict( + "maintenance job identity or catalog state changed before conditional worker claim".to_string(), + )); + } + if conditional_table_maintenance_is_replay(report, worker_id, condition) { + if matches!(job.status, TableMetadataMaintenanceJobStatus::Running) { + if !table_maintenance_job_lease_is_active(job, effective.config.worker_lease_timeout_seconds, now) { + return Err(TableCatalogStoreError::Conflict( + "conditional maintenance does not take over an expired worker; inspect the persisted job".to_string(), + )); + } + } else if job.finished_at.as_deref().and_then(parse_maintenance_timestamp).is_none() { + return Err(TableCatalogStoreError::Conflict( + "maintenance terminal report is incomplete; inspect the persisted job".to_string(), + )); + } + return Ok(true); + } + if job.attempt != condition.expected_attempt { + return Err(TableCatalogStoreError::Conflict( + "maintenance attempt changed before conditional worker claim".to_string(), + )); + } + let config = &effective.config; + if !config.background_enabled || config.worker_paused { + return Err(TableCatalogStoreError::Conflict( + "conditional maintenance is disabled or paused".to_string(), + )); + } + if job.retain_recent_metadata_files != config.retain_recent_metadata_files + || job.quarantine_enabled != config.quarantine_enabled + || job.quarantine_retention_seconds != config.quarantine_retention_seconds + || job.config_source != effective.source + || (matches!(job.operation, TableMetadataMaintenanceOperation::Delete) && !config.delete_enabled) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance policy changed before conditional worker claim".to_string(), + )); + } + if job.quarantined_object_count > 0 || job.max_retry_attempts > 10 { + return Err(TableCatalogStoreError::Conflict( + "maintenance quarantine or invalid persisted attempt budget requires operator recovery".to_string(), + )); + } + match job.status { + TableMetadataMaintenanceJobStatus::Queued + if job.attempt == 0 + && job.worker_id.is_none() + && job.lease_id.is_empty() + && job.started_at.is_none() + && job.finished_at.is_none() + && table_maintenance_scheduler_lease_is_active(job, config.worker_lease_timeout_seconds, now) => {} + TableMetadataMaintenanceJobStatus::Failed if job.attempt > 0 && job.attempt < job.max_retry_attempts => { + let finished_at = job + .finished_at + .as_deref() + .and_then(parse_maintenance_timestamp) + .ok_or_else(|| { + TableCatalogStoreError::Conflict( + "maintenance failed report is incomplete; inspect the persisted job".to_string(), + ) + })?; + let retry_after = job + .next_retry_after + .as_deref() + .and_then(parse_maintenance_timestamp) + .ok_or_else(|| { + TableCatalogStoreError::Conflict("maintenance retry is not scheduled; inspect the persisted job".to_string()) + })?; + if retry_after < finished_at || job.lease_id.is_empty() || job.worker_id.as_deref().is_none_or(str::is_empty) { + return Err(TableCatalogStoreError::Conflict( + "maintenance failed attempt has invalid retry or lease state; inspect the persisted job".to_string(), + )); + } + if retry_after > now { + return Err(TableCatalogStoreError::Conflict("maintenance retry backoff has not elapsed".to_string())); + } + } + _ => { + return Err(TableCatalogStoreError::Conflict( + "maintenance job is not eligible for a conditional claim; inspect its status and persisted attempt budget" + .to_string(), + )); + } + } + Ok(false) +} + +/// Only a completed transition from this exact persisted Running report can +/// repair its aliases. A partial claim is deliberately not resumable here. +pub(crate) fn maintenance_terminal_report_continues_running( + terminal: &TableMetadataMaintenanceReport, + running: &TableMetadataMaintenanceReport, +) -> bool { + let finished = &terminal.job; + let started = &running.job; + let terminal_action = match finished.status { + TableMetadataMaintenanceJobStatus::Successful => TableMaintenanceAuditAction::WorkerSucceeded, + TableMetadataMaintenanceJobStatus::Failed => TableMaintenanceAuditAction::WorkerFailed, + _ => return false, + }; + matches!(started.status, TableMetadataMaintenanceJobStatus::Running) + && finished + .finished_at + .as_deref() + .and_then(parse_maintenance_timestamp) + .is_some() + && finished.job_id == started.job_id + && finished.table_bucket == started.table_bucket + && finished.namespace == started.namespace + && finished.table == started.table + && finished.table_id == started.table_id + && finished.operation == started.operation + && finished.config_source == started.config_source + && finished.worker_id == started.worker_id + && !finished.lease_id.is_empty() + && finished.lease_id == started.lease_id + && finished.attempt == started.attempt + && finished.max_retry_attempts == started.max_retry_attempts + && finished.current_metadata_location == started.current_metadata_location + && finished.current_generation == started.current_generation + && finished.retain_recent_metadata_files == started.retain_recent_metadata_files + && finished.safety_window_seconds == started.safety_window_seconds + && finished.cleanup_watermark_unix_seconds == started.cleanup_watermark_unix_seconds + && finished.quarantine_enabled == started.quarantine_enabled + && finished.quarantine_retention_seconds == started.quarantine_retention_seconds + && finished.started_at == started.started_at + && finished.heartbeat_at == started.heartbeat_at + && finished.scheduler_id == started.scheduler_id + && finished.scheduler_lease_id == started.scheduler_lease_id + && finished.scheduled_at == started.scheduled_at + && terminal.current_metadata_location == running.current_metadata_location + && terminal.cleanup_candidate_locations == running.cleanup_candidate_locations + && terminal.deletable_metadata_locations == running.deletable_metadata_locations + && terminal.cleanup_object_candidate_locations == running.cleanup_object_candidate_locations + && terminal.deletable_object_locations == running.deletable_object_locations + && terminal.audit_events.len() == running.audit_events.len().saturating_add(1) + && terminal.audit_events.starts_with(&running.audit_events) + && terminal.audit_events.last().is_some_and(|event| { + event.actor == TableMaintenanceAuditActor::Worker + && event.action == terminal_action + && event.before_status == Some(TableMetadataMaintenanceJobStatus::Running) + && event.after_status.as_ref() == Some(&finished.status) + }) +} diff --git a/rustfs/src/table_catalog/model.rs b/rustfs/src/table_catalog/model.rs index 56456bed2..5d98d9d38 100644 --- a/rustfs/src/table_catalog/model.rs +++ b/rustfs/src/table_catalog/model.rs @@ -582,6 +582,12 @@ pub(crate) struct TableMaintenanceSchedulerJobSummary { pub audit_events: Vec, } +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct TableMaintenanceWorkerCondition { + pub expected_job_id: String, + pub expected_attempt: u16, +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub(crate) struct TableMetadataMaintenanceJob { pub job_id: String, diff --git a/rustfs/src/table_catalog/store/backup.rs b/rustfs/src/table_catalog/store/backup.rs index fcc1a6559..ff90385fe 100644 --- a/rustfs/src/table_catalog/store/backup.rs +++ b/rustfs/src/table_catalog/store/backup.rs @@ -817,8 +817,8 @@ where } TableCatalogEntryState::Active => {} } - let namespace = parse_namespace_for_store(&entry.namespace)?; - let table = parse_table_for_store(&entry.table)?; + parse_namespace_for_store(&entry.namespace)?; + parse_table_for_store(&entry.table)?; if !is_valid_table_metadata_location_for_entry(entry, &entry.metadata_location) { return Err(TableCatalogStoreError::Invalid(format!( "table backup metadata location is invalid for {}/{}/{}", @@ -892,11 +892,9 @@ where let kind = StrongTableCatalogBackupObjectKind::from_statistics_kind(reference.kind); insert_backup_object_kind(&mut keys, &entry.table_bucket, &object_key, kind)?; } - let referenced = metadata_maintenance_referenced_object_reports( + let referenced = metadata_maintenance_referenced_object_reports_for_entry( &self.object_backend, - &entry.table_bucket, - &namespace, - &table, + &metadata_entry, Some(warehouse_prefix.as_str()), &metadata, std::slice::from_ref(&metadata_location), @@ -1435,6 +1433,8 @@ where let mutation_fence = fence.mutation_fence(); let _write_guard = self.write_lock.lock().await; self.hydrate_state().await?; + self.recover_table_maintenance_rename_unlocked(table_bucket, Some(&mutation_fence)) + .await?; if self.read_restore_intent(table_bucket).await?.is_some() { return Err(TableCatalogStoreError::Conflict( "catalog backup requires pending restore recovery before sealing".to_string(), @@ -1584,6 +1584,8 @@ where Self::validate_maintenance_backup_objects(table_bucket, &backup.snapshot, &backup.maintenance_objects)?; self.hydrate_state().await?; + self.recover_table_maintenance_rename_unlocked(table_bucket, Some(&mutation_fence)) + .await?; let existing_intent = self.read_restore_intent(table_bucket).await?; if let Some((intent, _)) = existing_intent.as_ref() && intent.backup_id != backup_id diff --git a/rustfs/src/table_catalog/store/mod.rs b/rustfs/src/table_catalog/store/mod.rs index 378b01f66..91cc264cc 100644 --- a/rustfs/src/table_catalog/store/mod.rs +++ b/rustfs/src/table_catalog/store/mod.rs @@ -36,6 +36,11 @@ pub(super) use strong::{ }; pub(crate) use strong::{StrongTableCatalogRuntime, StrongTableCatalogStore}; +struct PeriodicTableMaintenanceQueueGuard<'a> { + table_id: &'a str, + current_job: Option<&'a TableMaintenanceSchedulerJobSummary>, +} + pub(in crate::table_catalog) fn catalog_lock_acquisition_error( operation: &str, err: rustfs_lock::LockError, @@ -51,7 +56,10 @@ pub(in crate::table_catalog) fn catalog_lock_acquisition_error( ); let message = format!("failed to {operation}: {err}"); if unavailable { - TableCatalogStoreError::Unavailable(message) + TableCatalogStoreError::LockUnavailable { + message, + source: Box::new(err), + } } else { TableCatalogStoreError::Internal(message) } @@ -468,6 +476,11 @@ pub(crate) trait TableCommitPublication: Send + Sync { None } + /// Storage-level lease signals, available only from publications that retain their guards. + fn object_mutation_fence(&self) -> Option { + None + } + fn complete(&self); } @@ -496,6 +509,7 @@ struct TableCommitLockPublication<'a, B> { struct TableCommitLockPublicationState { table_bucket: Option, table: Option<(String, String, String)>, + migration_read_guard: Option, guards: Vec, } @@ -506,6 +520,45 @@ impl<'a, B> TableCommitLockPublication<'a, B> { state: parking_lot::Mutex::new(TableCommitLockPublicationState::default()), } } + + fn attach_catalog_migration_read_guard(&self, guard: TableCatalogLockGuard) -> TableCatalogStoreResult<()> { + if guard.is_lock_lost() { + return Err(TableCatalogStoreError::Conflict( + "table-bucket catalog migration read permit was lost".to_string(), + )); + } + let mut state = self.state.lock(); + if state.migration_read_guard.is_some() || state.table_bucket.is_some() || state.table.is_some() { + return Err(TableCatalogStoreError::Internal( + "catalog migration read permit must be attached before publication".to_string(), + )); + } + state.migration_read_guard = Some(guard); + Ok(()) + } + + fn attach_catalog_write_guard(&self, guard: TableCatalogLockGuard) -> TableCatalogStoreResult<()> { + if guard.is_lock_lost() { + return Err(TableCatalogStoreError::Conflict("table catalog write lock was lost".to_string())); + } + let mut state = self.state.lock(); + if state.table_bucket.is_none() || state.table.is_some() { + return Err(TableCatalogStoreError::Internal( + "catalog write guard must be attached between bucket and table publication locks".to_string(), + )); + } + state.guards.push(guard); + Ok(()) + } + + fn release_publication_locks(&self) { + // Read-only planning may proceed without excluding unrelated tables' commits. + // The migration permit remains attached until the whole operation completes. + let mut state = self.state.lock(); + state.table = None; + state.table_bucket = None; + state.guards.clear(); + } } #[async_trait::async_trait] @@ -516,6 +569,15 @@ where async fn begin_table_bucket(&self, table_bucket: &str) -> TableCatalogStoreResult<()> { { let mut state = self.state.lock(); + if state + .migration_read_guard + .as_ref() + .is_some_and(TableCatalogLockGuard::is_lock_lost) + { + return Err(TableCatalogStoreError::Conflict( + "table-bucket catalog migration read permit was lost".to_string(), + )); + } if state.table_bucket.as_deref() == Some(table_bucket) { return Ok(()); } @@ -568,7 +630,9 @@ where fn holds_table_bucket(&self, table_bucket: &str) -> bool { let state = self.state.lock(); - state.table_bucket.as_deref() == Some(table_bucket) && state.guards.iter().all(|guard| !guard.is_lock_lost()) + state.table_bucket.as_deref() == Some(table_bucket) + && state.migration_read_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) + && state.guards.iter().all(|guard| !guard.is_lock_lost()) } fn holds_table(&self, table_bucket: &str, namespace: &str, table: &str) -> bool { @@ -577,9 +641,29 @@ where .table .as_ref() .is_some_and(|held| held.0 == table_bucket && held.1 == namespace && held.2 == table) + && state.migration_read_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) && state.guards.iter().all(|guard| !guard.is_lock_lost()) } + fn catalog_migration_read_permit_status(&self) -> Option { + self.state + .lock() + .migration_read_guard + .as_ref() + .map(|guard| !guard.is_lock_lost()) + } + + fn object_mutation_fence(&self) -> Option { + let state = self.state.lock(); + Some(TableCatalogObjectMutationFence::from_signals( + state + .migration_read_guard + .iter() + .chain(state.guards.iter()) + .map(TableCatalogLockGuard::lock_lost_signal), + )) + } + fn complete(&self) { *self.state.lock() = TableCommitLockPublicationState::default(); } @@ -627,6 +711,11 @@ impl TableCatalogLockGuard { } } + #[cfg(test)] + pub(crate) fn test_namespace(guard: rustfs_lock::NamespaceLockGuard) -> Self { + Self::namespace(guard) + } + fn write_namespace(guard: WriteCommitGuard) -> Self { let lock_lost = guard.lock_lost_signal(); Self { @@ -907,6 +996,10 @@ pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static { None } + fn table_commit_object_mutation_fence(&self) -> Option { + None + } + fn complete_table_commit_publication(&self) {} } @@ -1033,6 +1126,10 @@ where TableCatalogObjectBackend::catalog_migration_read_permit_status(self) } + fn object_mutation_fence(&self) -> Option { + self.table_commit_object_mutation_fence() + } + fn complete(&self) { self.complete_table_commit_publication(); } @@ -1762,7 +1859,7 @@ where ) -> TableCatalogStoreResult { match self { Self::ObjectBacked(store) => store.get_table_maintenance_config(table_bucket, namespace, table).await, - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance config")), + Self::DurableStrong(store) => store.get_table_maintenance_config(table_bucket, namespace, table).await, } } @@ -1779,7 +1876,11 @@ where .put_table_maintenance_config(table_bucket, namespace, table, config) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance config")), + Self::DurableStrong(store) => { + store + .put_table_maintenance_config(table_bucket, namespace, table, config) + .await + } } } @@ -1796,7 +1897,11 @@ where .get_table_metadata_maintenance_report(table_bucket, namespace, table, job_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance report")), + Self::DurableStrong(store) => { + store + .get_table_metadata_maintenance_report(table_bucket, namespace, table, job_id) + .await + } } } @@ -1812,7 +1917,11 @@ where .get_table_maintenance_scheduler_report(table_bucket, namespace, table) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance scheduler")), + Self::DurableStrong(store) => { + store + .get_table_maintenance_scheduler_report(table_bucket, namespace, table) + .await + } } } @@ -1829,7 +1938,48 @@ where .run_table_maintenance_scheduler_once(table_bucket, namespace, table, scheduler_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance scheduler")), + Self::DurableStrong(store) => { + store + .run_table_maintenance_scheduler_once(table_bucket, namespace, table, scheduler_id) + .await + } + } + } + + pub(crate) async fn queue_periodic_table_maintenance( + &self, + entry: &TableEntry, + scheduler_id: String, + expected_job: Option<&TableMaintenanceSchedulerJobSummary>, + ) -> TableCatalogStoreResult { + let guard = PeriodicTableMaintenanceQueueGuard { + table_id: &entry.table_id, + current_job: expected_job, + }; + match self { + Self::ObjectBacked(store) => { + store + .run_table_maintenance_scheduler_once_with_guard_at( + &entry.table_bucket, + &entry.namespace, + &entry.table, + scheduler_id, + OffsetDateTime::now_utc(), + Some(&guard), + ) + .await + } + Self::DurableStrong(store) => { + store + .run_table_maintenance_scheduler_once_with_guard( + &entry.table_bucket, + &entry.namespace, + &entry.table, + scheduler_id, + Some(&guard), + ) + .await + } } } @@ -1847,7 +1997,11 @@ where .apply_table_maintenance_quarantine_operation(table_bucket, namespace, table, job_id, request) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance quarantine")), + Self::DurableStrong(store) => { + store + .apply_table_maintenance_quarantine_operation(table_bucket, namespace, table, job_id, request) + .await + } } } @@ -1864,7 +2018,45 @@ where .run_table_metadata_maintenance_worker_once(table_bucket, namespace, table, worker_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance worker")), + Self::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once(table_bucket, namespace, table, worker_id) + .await + } + } + } + + pub(crate) async fn run_table_metadata_maintenance_worker_once_conditionally( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + ) -> TableCatalogStoreResult { + match self { + Self::ObjectBacked(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally( + table_bucket, + namespace, + table, + worker_id, + condition, + ) + .await + } + Self::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally( + table_bucket, + namespace, + table, + worker_id, + condition, + ) + .await + } } } @@ -1883,7 +2075,11 @@ where .heartbeat_table_metadata_maintenance_job(table_bucket, namespace, table, job_id, lease_id, worker_id) .await } - Self::DurableStrong(_) => Err(Self::unsupported_for_durable_strong("table maintenance heartbeat")), + Self::DurableStrong(store) => { + store + .heartbeat_table_metadata_maintenance_job(table_bucket, namespace, table, job_id, lease_id, worker_id) + .await + } } } @@ -2448,48 +2644,693 @@ where mod tests { use super::*; use crate::storage_api::protocols::client::{BucketOperations as _, MakeBucketOptions}; - use rustfs_lock::{LockClient, LockId, LockInfo, LockRequest, LockResponse, LockStats}; + use crate::table_catalog::test_support::{RecordingLeaseClient, TestCatalogObjectBackend}; + use crate::table_catalog::tests::{test_bucket_entry, test_namespace_entry, test_table_entry}; + use rustfs_lock::LockClient; use std::sync::{Arc, Mutex}; use std::time::Duration; - #[derive(Debug)] - struct RecordingLeaseClient { - inner: rustfs_lock::LocalClient, - acquired: Mutex>, + #[tokio::test] + async fn table_maintenance_publication_reuses_attached_migration_read_permit() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + let migration_path = TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(); + let guard = backend.acquire_read_lock(RUSTFS_META_BUCKET, &migration_path).await.unwrap(); + let publication = TableCommitLockPublication::new(&backend); + publication.attach_catalog_migration_read_guard(guard).unwrap(); + assert_eq!(publication.catalog_migration_read_permit_status(), Some(true)); + assert!( + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .is_err() + ); + let reads = backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_path).await; + let entry = test_table_entry( + bucket, + &namespace, + &table, + default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"), + ); + store.register_table_with_publication(entry, &publication).await.unwrap(); + assert_eq!(backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_path).await, reads); + assert_eq!(publication.catalog_migration_read_permit_status(), None); + tokio::time::timeout(Duration::from_secs(5), backend.acquire_write_lock(RUSTFS_META_BUCKET, &migration_path)) + .await + .expect("completion releases the attached migration read permit") + .unwrap(); } - #[async_trait::async_trait] - impl LockClient for RecordingLeaseClient { - async fn acquire_lock(&self, request: &LockRequest) -> rustfs_lock::Result { - let response = self.inner.acquire_lock(request).await?; - if let Some(info) = response.lock_info.as_ref() { - *self.acquired.lock().expect("record lease") = Some(info.id.clone()); + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_migration_lease_loss_blocks_sidecars_and_cleanup() { + let backend = TestCatalogObjectBackend::default(); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let orphan = default_table_metadata_file_path(&namespace, &table, "00000.metadata.json"); + let entry = test_table_entry(bucket, &namespace, &table, current.clone()); + backend + .seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object(bucket, &orphan, b"{}".to_vec()).await; + let sidecar = ObjectTableCatalogStore::new(backend.clone()); + let config = TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }; + sidecar + .put_table_maintenance_config_for_entry(&entry, config.clone()) + .await + .unwrap(); + let report = sidecar.plan_table_metadata_maintenance_for_entry(&entry, 0).await.unwrap(); + assert_eq!(report.deletable_metadata_locations, vec![orphan.clone()]); + let client = Arc::new(RecordingLeaseClient { + inner: rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new( + rustfs_lock::FastObjectLockManager::new(), + )))), + acquired: Mutex::new(None), + }); + let lock = rustfs_lock::NamespaceLockWrapper::new( + rustfs_lock::NamespaceLock::with_clients_and_quorum( + "maintenance-probe".to_string(), + vec![client.clone() as Arc], + 1, + ), + rustfs_lock::ObjectKey::new( + RUSTFS_META_BUCKET, + TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(), + ), + "maintenance-migration-probe".to_string(), + ); + let guard = TableCatalogLockGuard::namespace(lock.get_read_lock(Duration::from_secs(5)).await.unwrap()); + let publication = TableCommitLockPublication::new(&backend); + publication.attach_catalog_migration_read_guard(guard).unwrap(); + publication.begin_table_bucket(bucket).await.unwrap(); + publication.prepare(bucket, "sales", "orders").await.unwrap(); + assert!(publication.holds_table(bucket, "sales", "orders")); + let id = client + .acquired + .lock() + .unwrap() + .clone() + .expect("actual migration lease acquired"); + assert!(client.force_release(&id).await.unwrap()); + tokio::time::timeout(Duration::from_secs(15), async { + while publication.catalog_migration_read_permit_status() != Some(false) { + tokio::time::sleep(Duration::from_millis(10)).await; } - Ok(response) + }) + .await + .expect("heartbeat observes the released migration lease"); + assert!(!publication.holds_table_bucket(bucket)); + assert!(!publication.holds_table(bucket, "sales", "orders")); + assert!(matches!( + publication.begin_table_bucket(bucket).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("migration read permit was lost") + )); + let mut changed_config = config.clone(); + changed_config.worker_paused = true; + assert!( + sidecar + .put_table_maintenance_config_for_entry_with_publication(&entry, changed_config, Some(&publication)) + .await + .is_err() + ); + assert_eq!(sidecar.get_table_maintenance_config_for_entry(&entry).await.unwrap(), config); + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + assert!( + sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, true, &effective, report, &publication) + .await + .is_err() + ); + assert!(backend.object_exists(bucket, ¤t).await.unwrap()); + assert!(backend.object_exists(bucket, &orphan).await.unwrap()); + let paths = TableCatalogObjectPaths::default(); + assert!( + !backend + .object_exists( + RUSTFS_META_BUCKET, + &paths.table_maintenance_current_job_path(bucket, &namespace, &table, &entry.table_id) + ) + .await + .unwrap() + ); + let store = StrongTableCatalogStore::new(backend.clone()); + assert!(matches!( + store.register_table_with_publication(entry, &publication).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("migration read permit was lost") + )); + let lost = publication.state.lock().migration_read_guard.take().unwrap(); + let rejected = TableCommitLockPublication::new(&backend); + assert!(matches!( + rejected.attach_catalog_migration_read_guard(lost), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("migration read permit was lost") + )); + assert_eq!(rejected.catalog_migration_read_permit_status(), None); + } + + struct MaintenanceMutationFixture { + backend: TestCatalogObjectBackend, + sidecar: ObjectTableCatalogStore, + entry: TableEntry, + config: TableMaintenanceConfig, + report: TableMetadataMaintenanceReport, + orphan: String, + } + + async fn maintenance_mutation_fixture(data_orphan: bool) -> MaintenanceMutationFixture { + let backend = TestCatalogObjectBackend::default(); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let orphan = if data_orphan { + format!("{}/orphan.parquet", default_table_data_dir_path(&namespace, &table)) + } else { + default_table_metadata_file_path(&namespace, &table, "00000.metadata.json") + }; + let entry = test_table_entry("analytics", &namespace, &table, current.clone()); + backend + .seed_object("analytics", ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object("analytics", &orphan, b"{}".to_vec()).await; + let sidecar = ObjectTableCatalogStore::new(backend.clone()); + let config = TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }; + sidecar + .put_table_maintenance_config_for_entry(&entry, config.clone()) + .await + .unwrap(); + let report = sidecar.plan_table_metadata_maintenance_for_entry(&entry, 0).await.unwrap(); + assert!(if data_orphan { + report.deletable_object_locations.contains(&orphan) + } else { + report.deletable_metadata_locations.contains(&orphan) + }); + MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + report, + orphan, } - async fn release(&self, id: &LockId) -> rustfs_lock::Result { - self.inner.release(id).await + } + + async fn maintenance_loss_publication<'a>( + backend: &'a TestCatalogObjectBackend, + entry: &TableEntry, + ) -> (TableCommitLockPublication<'a, TestCatalogObjectBackend>, Arc) { + let client = Arc::new(RecordingLeaseClient { + inner: rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new( + rustfs_lock::FastObjectLockManager::new(), + )))), + acquired: Mutex::new(None), + }); + let lock = rustfs_lock::NamespaceLockWrapper::new( + rustfs_lock::NamespaceLock::with_clients_and_quorum( + "maintenance-await-probe".to_string(), + vec![client.clone() as Arc], + 1, + ), + rustfs_lock::ObjectKey::new( + RUSTFS_META_BUCKET, + TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(), + ), + "maintenance-await-migration".to_string(), + ); + let publication = TableCommitLockPublication::new(backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::namespace( + lock.get_read_lock(Duration::from_secs(5)).await.unwrap(), + )) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + (publication, client) + } + + async fn lose_maintenance_lease( + client: &RecordingLeaseClient, + publication: &TableCommitLockPublication<'_, TestCatalogObjectBackend>, + ) { + let id = client.acquired.lock().unwrap().clone().unwrap(); + assert!(client.force_release(&id).await.unwrap()); + tokio::time::timeout(Duration::from_secs(15), async { + while publication.catalog_migration_read_permit_status() != Some(false) { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("heartbeat observes loss while backend mutation is suspended"); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_await_loss_prevents_each_sidecar_storage_commit() { + for target in 0..4 { + let fixture = maintenance_mutation_fixture(false).await; + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + mut report, + .. + } = fixture; + sidecar + .put_table_metadata_maintenance_report_for_entry(&report, &entry) + .await + .unwrap(); + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + let paths = TableCatalogObjectPaths::default(); + let sidecars = [ + paths.table_maintenance_config_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + paths.table_maintenance_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id, &report.job.job_id), + paths.table_maintenance_latest_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + paths.table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + ]; + let before = backend.read_object(RUSTFS_META_BUCKET, &sidecars[target]).await.unwrap(); + let later_before = if target < 3 { + backend.read_object(RUSTFS_META_BUCKET, &sidecars[3]).await.unwrap() + } else { + None + }; + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &sidecars[target]).await; + let mut changed_config = config; + changed_config.worker_paused = true; + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.worker_id = Some("new-worker".to_string()); + let operation = async { + if target == 0 { + sidecar + .put_table_maintenance_config_for_entry_with_publication(&entry, changed_config, Some(&publication)) + .await + .map(|_| ()) + } else { + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await + } + }; + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + assert!(publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table)); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err(), "sidecar target {target} must reject a loss inside backend await"); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &sidecars[target]).await.unwrap(), before); + if target < 3 { + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &sidecars[3]).await.unwrap(), later_before); + } + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); } - async fn refresh(&self, id: &LockId) -> rustfs_lock::Result { - self.inner.refresh(id).await + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_await_loss_prevents_metadata_and_data_delete() { + for data_orphan in [false, true] { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + report, + orphan, + } = maintenance_mutation_fixture(data_orphan).await; + let before = backend.read_object(&entry.table_bucket, &orphan).await.unwrap(); + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let pause = backend.pause_next_delete(&entry.table_bucket, &orphan).await; + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + let operation = sidecar.finish_table_metadata_maintenance_run_for_entry_with_publication( + &entry, + true, + &effective, + report, + &publication, + ); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + assert!(publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table)); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(backend.read_object(&entry.table_bucket, &orphan).await.unwrap(), before); + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); + assert!( + sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &Namespace::parse(&entry.namespace).unwrap(), + &IdentifierSegment::parse(&entry.table).unwrap(), + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await + .unwrap() + .is_none() + ); } - async fn force_release(&self, id: &LockId) -> rustfs_lock::Result { - self.inner.force_release(id).await + } + + #[tokio::test] + async fn table_maintenance_sidecar_cas_preserves_concurrent_replacement() { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + .. + } = maintenance_mutation_fixture(false).await; + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + let path = TableCatalogObjectPaths::default().table_maintenance_config_path( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + ); + let publication = TableCommitLockPublication::new(&backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &path).await; + let mut changed_config = config.clone(); + changed_config.worker_paused = true; + let mut replacement = config; + replacement.delete_enabled = false; + let bytes = serde_json::to_vec(&replacement).unwrap(); + let operation = + sidecar.put_table_maintenance_config_for_entry_with_publication(&entry, changed_config, Some(&publication)); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + backend.seed_object(RUSTFS_META_BUCKET, &path, bytes.clone()).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(matches!(result, Err(TableCatalogStoreError::Conflict(_)))); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &path).await.unwrap().unwrap().data, bytes); + } + + #[tokio::test] + async fn table_maintenance_delete_cas_preserves_changed_candidates() { + for data_orphan in [false, true] { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + report, + orphan, + } = maintenance_mutation_fixture(data_orphan).await; + let publication = TableCommitLockPublication::new(&backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + let pause = backend.pause_next_delete(&entry.table_bucket, &orphan).await; + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + let operation = sidecar.finish_table_metadata_maintenance_run_for_entry_with_publication( + &entry, + true, + &effective, + report, + &publication, + ); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + backend + .seed_object(&entry.table_bucket, &orphan, b"replacement".to_vec()) + .await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(matches!(result, Err(TableCatalogStoreError::Conflict(_)))); + assert_eq!( + backend.read_object(&entry.table_bucket, &orphan).await.unwrap().unwrap().data, + b"replacement".to_vec() + ); + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); } - async fn check_status(&self, id: &LockId) -> rustfs_lock::Result> { - self.inner.check_status(id).await + } + + fn maintenance_rename_destination(entry: &TableEntry) -> (TableEntry, String, String) { + let mut destination = entry.clone(); + destination.table = "tablearchived".to_string(); + let paths = TableCatalogObjectPaths::default(); + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let source_path = paths.table_maintenance_config_path( + &entry.table_bucket, + &namespace, + &IdentifierSegment::parse(&entry.table).unwrap(), + &entry.table_id, + ); + let destination_path = paths.table_maintenance_config_path( + &destination.table_bucket, + &namespace, + &IdentifierSegment::parse(&destination.table).unwrap(), + &destination.table_id, + ); + (destination, source_path, destination_path) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_rename_await_loss_prevents_destination_copy() { + let MaintenanceMutationFixture { + backend, sidecar, entry, .. + } = maintenance_mutation_fixture(false).await; + let (destination, source_path, destination_path) = maintenance_rename_destination(&entry); + let source_before = backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(); + assert!(source_before.is_some()); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &destination_path).await.unwrap()); + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let fence = publication.object_mutation_fence().unwrap(); + let pause = backend.pause_next_put(RUSTFS_META_BUCKET, &destination_path).await; + let operation = sidecar.prepare_table_maintenance_rename(&entry, &destination, "source-snapshot-etag", &fence); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(), source_before); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &destination_path).await.unwrap()); + assert!( + sidecar + .read_table_maintenance_rename_intent(&entry.table_bucket) + .await + .unwrap() + .is_some() + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn table_maintenance_rename_await_loss_prevents_source_cleanup() { + let MaintenanceMutationFixture { + backend, sidecar, entry, .. + } = maintenance_mutation_fixture(false).await; + let (destination, source_path, destination_path) = maintenance_rename_destination(&entry); + let source_before = backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(); + assert!(source_before.is_some()); + let (publication, client) = maintenance_loss_publication(&backend, &entry).await; + let fence = publication.object_mutation_fence().unwrap(); + let (intent, intent_etag) = sidecar + .prepare_table_maintenance_rename(&entry, &destination, "source-snapshot-etag", &fence) + .await + .unwrap() + .expect("source maintenance config must be prepared for rename"); + let destination_before = backend.read_object(RUSTFS_META_BUCKET, &destination_path).await.unwrap(); + assert!(destination_before.is_some()); + let pause = backend.pause_next_delete(RUSTFS_META_BUCKET, &source_path).await; + let operation = sidecar.finish_table_maintenance_rename(&intent, &intent_etag, true, &fence); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + lose_maintenance_lease(&client, &publication).await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(result.is_err()); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, &source_path).await.unwrap(), source_before); + assert_eq!( + backend.read_object(RUSTFS_META_BUCKET, &destination_path).await.unwrap(), + destination_before + ); + assert!( + sidecar + .read_table_maintenance_rename_intent(&entry.table_bucket) + .await + .unwrap() + .is_some() + ); + } + + #[tokio::test] + async fn table_maintenance_rename_delete_cas_preserves_same_content_replacement() { + for committed in [true, false] { + let MaintenanceMutationFixture { + backend, sidecar, entry, .. + } = maintenance_mutation_fixture(false).await; + let (destination, source_path, destination_path) = maintenance_rename_destination(&entry); + let fence = TableCatalogObjectMutationFence::default(); + let (intent, intent_etag) = sidecar + .prepare_table_maintenance_rename(&entry, &destination, "source-snapshot-etag", &fence) + .await + .unwrap() + .expect("source maintenance config must be prepared for rename"); + let (target_path, other_path) = if committed { + (&source_path, &destination_path) + } else { + (&destination_path, &source_path) + }; + let target_before = backend.read_object(RUSTFS_META_BUCKET, target_path).await.unwrap().unwrap(); + let other_before = backend.read_object(RUSTFS_META_BUCKET, other_path).await.unwrap(); + assert!(other_before.is_some()); + let pause = backend.pause_next_delete(RUSTFS_META_BUCKET, target_path).await; + let operation = sidecar.finish_table_maintenance_rename(&intent, &intent_etag, committed, &fence); + let interfere = async { + tokio::time::timeout(Duration::from_secs(5), pause.wait_started()) + .await + .unwrap(); + backend + .seed_object(RUSTFS_META_BUCKET, target_path, target_before.data.clone()) + .await; + pause.release(); + }; + let (result, ()) = tokio::join!(operation, interfere); + assert!(matches!(result, Err(TableCatalogStoreError::Conflict(_)))); + let replacement = backend.read_object(RUSTFS_META_BUCKET, target_path).await.unwrap().unwrap(); + assert_eq!(replacement.data, target_before.data); + assert_ne!(replacement.etag, target_before.etag); + assert_eq!(backend.read_object(RUSTFS_META_BUCKET, other_path).await.unwrap(), other_before); + assert!( + sidecar + .read_table_maintenance_rename_intent(&entry.table_bucket) + .await + .unwrap() + .is_some() + ); } - async fn get_stats(&self) -> rustfs_lock::Result { - self.inner.get_stats().await - } - async fn close(&self) -> rustfs_lock::Result<()> { - self.inner.close().await - } - async fn is_online(&self) -> bool { - self.inner.is_online().await - } - async fn is_local(&self) -> bool { - self.inner.is_local().await + } + + #[tokio::test] + async fn table_maintenance_delete_respects_original_plan_watermark() { + for data_orphan in [false, true] { + let MaintenanceMutationFixture { + backend, + sidecar, + entry, + config, + mut report, + orphan, + } = maintenance_mutation_fixture(data_orphan).await; + let now = OffsetDateTime::now_utc(); + let window = time::Duration::seconds(TABLE_METADATA_CLEANUP_SAFETY_WINDOW_SECONDS); + report.job.cleanup_watermark_unix_seconds = (now - window * 3_i32).unix_timestamp(); + backend + .seed_object_with_mod_time( + &entry.table_bucket, + &orphan, + b"changed since plan".to_vec(), + Some(now - window * 2_i32), + ) + .await; + let before = backend.read_object(&entry.table_bucket, &orphan).await.unwrap(); + let publication = TableCommitLockPublication::new(&backend); + publication + .attach_catalog_migration_read_guard(TableCatalogLockGuard::stable(())) + .unwrap(); + publication.begin_table_bucket(&entry.table_bucket).await.unwrap(); + publication + .prepare(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(); + let effective = TableMaintenanceEffectiveConfig { + config, + source: TableMaintenanceConfigSource::TableOverride, + }; + let completed = sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, true, &effective, report, &publication) + .await + .unwrap(); + assert_eq!(completed.job.deleted_metadata_file_count, 0); + assert_eq!(completed.job.deleted_object_count, 0); + assert_eq!(backend.read_object(&entry.table_bucket, &orphan).await.unwrap(), before); + assert!( + backend + .object_exists(&entry.table_bucket, &entry.metadata_location) + .await + .unwrap() + ); } } diff --git a/rustfs/src/table_catalog/store/object.rs b/rustfs/src/table_catalog/store/object.rs index f62b2b93f..01690c7bc 100644 --- a/rustfs/src/table_catalog/store/object.rs +++ b/rustfs/src/table_catalog/store/object.rs @@ -14,6 +14,161 @@ use super::*; +const TABLE_MAINTENANCE_RENAME_MAX_OBJECTS: usize = 4096; +const TABLE_MAINTENANCE_SCHEDULER_HISTORY_MAX_OBJECTS: usize = 4096; + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct TableMaintenanceRenameObject { + etag: String, + data: serde_json::Value, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct TableMaintenanceRenameIntent { + version: u16, + pub(super) source: TableEntry, + pub(super) destination: TableEntry, + pub(super) source_snapshot_etag: String, + source_objects: BTreeMap, + destination_etags: BTreeMap, +} + +fn maintenance_rename_intent_path(table_bucket: &str) -> String { + format!( + "{}maintenance-rename-intent.json", + TableCatalogObjectPaths::default().table_bucket_root_prefix(table_bucket) + ) +} + +pub(super) fn maintenance_rename_recovery_error(mut error: TableCatalogStoreError) -> TableCatalogStoreError { + let message = match &mut error { + TableCatalogStoreError::NotFound(message) + | TableCatalogStoreError::NamespaceNotFound(message) + | TableCatalogStoreError::TableNotFound(message) + | TableCatalogStoreError::AlreadyExists(message) + | TableCatalogStoreError::Conflict(message) + | TableCatalogStoreError::Invalid(message) + | TableCatalogStoreError::Unsupported(message) + | TableCatalogStoreError::Unavailable(message) + | TableCatalogStoreError::LockUnavailable { message, .. } + | TableCatalogStoreError::Internal(message) => message, + }; + *message = format!( + "maintenance rename recovery required; retry the same rename after transient errors, or repair invalid persisted state before retrying: {message}" + ); + error +} + +fn maintenance_rename_prefix(entry: &TableEntry) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let path = TableCatalogObjectPaths::default().table_maintenance_config_path( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + ); + Ok(path.trim_end_matches(MAINTENANCE_CONFIG_FILE).to_string()) +} + +fn maintenance_rename_suffix_is_valid(suffix: &str) -> bool { + matches!( + suffix, + MAINTENANCE_CONFIG_FILE | MAINTENANCE_CURRENT_JOB_FILE | MAINTENANCE_LATEST_JOB_FILE + ) || suffix + .strip_prefix(&format!("{MAINTENANCE_JOB_ROOT}/")) + .and_then(|suffix| suffix.strip_suffix(".json")) + .is_some_and(|hash| { + hash.len() == 64 + && hash + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + }) +} + +fn maintenance_rename_data( + entry: &TableEntry, + destination: &TableEntry, + suffix: &str, + data: &serde_json::Value, +) -> TableCatalogStoreResult { + if !maintenance_rename_suffix_is_valid(suffix) { + return Err(TableCatalogStoreError::Invalid( + "unexpected table maintenance rename object path".to_string(), + )); + } + if suffix == MAINTENANCE_CONFIG_FILE { + let config = serde_json::from_value::(data.clone()) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename config: {err}")))?; + validate_table_maintenance_config(&config)?; + return Ok(data.clone()); + } + let report = serde_json::from_value::(data.clone()) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename report: {err}")))?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + validate_table_maintenance_report_owner(&report, &entry.table_bucket, &namespace, &table, &entry.table_id)?; + if suffix.starts_with(&format!("{MAINTENANCE_JOB_ROOT}/")) + && suffix != format!("{MAINTENANCE_JOB_ROOT}/{}.json", table_catalog_path_hash(&report.job.job_id)) + { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename job does not match its object path".to_string(), + )); + } + // Preserve every report field, including audit records and unknown additive fields. + // Only catalog owner identifiers change; physical locations and generations do not. + let mut renamed = data.clone(); + for field in ["job", "snapshot-expiration", "compaction"] { + let Some(owner) = renamed.get_mut(field).filter(|value| !value.is_null()) else { + continue; + }; + if owner.get("table_bucket").and_then(serde_json::Value::as_str) != Some(entry.table_bucket.as_str()) + || owner.get("namespace").and_then(serde_json::Value::as_str) != Some(entry.namespace.as_str()) + || owner.get("table").and_then(serde_json::Value::as_str) != Some(entry.table.as_str()) + || owner.get("table_id").and_then(serde_json::Value::as_str) != Some(entry.table_id.as_str()) + { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename nested report owner does not match its table".to_string(), + )); + } + owner["namespace"] = serde_json::Value::String(destination.namespace.clone()); + owner["table"] = serde_json::Value::String(destination.table.clone()); + } + Ok(renamed) +} + +fn ensure_table_maintenance_publication_held( + publication: Option<&(dyn TableCommitPublication + Sync)>, + entry: &TableEntry, + stage: &str, +) -> TableCatalogStoreResult<()> { + if publication.is_some_and(|publication| { + !publication.holds_table_bucket(&entry.table_bucket) + || !publication.holds_table(&entry.table_bucket, &entry.namespace, &entry.table) + }) { + return Err(TableCatalogStoreError::Internal(format!( + "table maintenance publication fence was lost {stage}" + ))); + } + Ok(()) +} + +fn table_maintenance_mutation_fence( + publication: Option<&(dyn TableCommitPublication + Sync)>, + entry: &TableEntry, +) -> TableCatalogStoreResult> { + ensure_table_maintenance_publication_held(publication, entry, "before storage mutation")?; + publication + .map(|publication| { + publication.object_mutation_fence().ok_or_else(|| { + TableCatalogStoreError::Internal("table maintenance requires a storage mutation fence".to_string()) + }) + }) + .transpose() +} + pub(super) fn validate_table_bucket_entry_object( paths: &TableCatalogObjectPaths, object: &str, @@ -673,6 +828,299 @@ where Ok(entries) } + async fn collect_table_maintenance_rename_objects( + &self, + entry: &TableEntry, + ) -> TableCatalogStoreResult> { + let prefix = maintenance_rename_prefix(entry)?; + let limit = NonZeroUsize::new(TABLE_CATALOG_LIST_MAX_KEYS) + .ok_or_else(|| TableCatalogStoreError::Internal("maintenance rename page limit must be positive".to_string()))?; + let mut cursor: Option = None; + let mut objects = BTreeMap::new(); + let mut total_bytes = 0usize; + loop { + let page = self + .backend + .list_objects_page(self.catalog_bucket(), &prefix, cursor.as_deref(), limit) + .await?; + if page.objects.len() > limit.get() || (page.is_truncated && page.objects.is_empty()) { + return Err(TableCatalogStoreError::Invalid("invalid maintenance rename listing page".to_string())); + } + for object in page.objects { + if cursor.as_ref().is_some_and(|cursor| object <= *cursor) { + return Err(TableCatalogStoreError::Invalid("maintenance rename listing did not advance".to_string())); + } + let suffix = object + .strip_prefix(&prefix) + .filter(|suffix| maintenance_rename_suffix_is_valid(suffix)) + .ok_or_else(|| { + TableCatalogStoreError::Invalid("maintenance rename listing escaped its owner paths".to_string()) + })?; + if objects.len() >= TABLE_MAINTENANCE_RENAME_MAX_OBJECTS { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename object count exceeds its limit".to_string(), + )); + } + let value = self + .backend + .read_object_limited(self.catalog_bucket(), &object, TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_OBJECT_SIZE) + .await? + .ok_or_else(|| { + TableCatalogStoreError::Conflict("maintenance rename object disappeared during collection".to_string()) + })?; + total_bytes = total_bytes.saturating_add(value.data.len()); + if total_bytes > TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_BYTES { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename content exceeds its byte limit".to_string(), + )); + } + let etag = value + .etag + .filter(|etag| !etag.is_empty()) + .ok_or_else(|| TableCatalogStoreError::Conflict("maintenance rename object has no ETag".to_string()))?; + let data = serde_json::from_slice(&value.data) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename JSON: {err}")))?; + maintenance_rename_data(entry, entry, suffix, &data)?; + objects.insert(suffix.to_string(), TableMaintenanceRenameObject { etag, data }); + cursor = Some(object); + } + if !page.is_truncated { + return Ok(objects); + } + } + } + + pub(super) async fn read_table_maintenance_rename_intent( + &self, + table_bucket: &str, + ) -> TableCatalogStoreResult> { + let object = maintenance_rename_intent_path(table_bucket); + let Some(value) = self + .backend + .read_object_limited(self.catalog_bucket(), &object, TABLE_CATALOG_BACKUP_MAX_SIZE) + .await? + else { + return Ok(None); + }; + let intent: TableMaintenanceRenameIntent = serde_json::from_slice(&value.data) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename intent: {err}")))?; + let etag = value + .etag + .filter(|etag| !etag.is_empty()) + .ok_or_else(|| TableCatalogStoreError::Conflict("maintenance rename intent has no ETag".to_string()))?; + validate_table_entry_version_and_id(&intent.source)?; + validate_table_entry_version_and_id(&intent.destination)?; + let mut expected_destination = intent.source.clone(); + expected_destination.namespace.clone_from(&intent.destination.namespace); + expected_destination.table.clone_from(&intent.destination.table); + if intent.version != 1 + || intent.source_snapshot_etag.is_empty() + || intent.source.table_bucket != table_bucket + || intent.source.state != TableCatalogEntryState::Active + || intent.destination != expected_destination + || (intent.source.namespace == intent.destination.namespace && intent.source.table == intent.destination.table) + || intent.source_objects.len() > TABLE_MAINTENANCE_RENAME_MAX_OBJECTS + || intent.destination_etags.len() > TABLE_MAINTENANCE_RENAME_MAX_OBJECTS + { + return Err(TableCatalogStoreError::Invalid( + "invalid maintenance rename intent identity, version, or count".to_string(), + )); + } + maintenance_rename_prefix(&intent.source)?; + maintenance_rename_prefix(&intent.destination)?; + let mut total_bytes = 0usize; + for (suffix, value) in &intent.source_objects { + let size = serde_json::to_vec(&value.data) + .map_err(|err| TableCatalogStoreError::Invalid(format!("invalid maintenance rename data: {err}")))? + .len(); + total_bytes = total_bytes.saturating_add(size); + if value.etag.is_empty() + || size > TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_OBJECT_SIZE + || total_bytes > TABLE_CATALOG_BACKUP_MAX_MAINTENANCE_BYTES + { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename intent object exceeds its limits".to_string(), + )); + } + maintenance_rename_data(&intent.source, &intent.destination, suffix, &value.data)?; + } + if intent + .destination_etags + .iter() + .any(|(suffix, etag)| !maintenance_rename_suffix_is_valid(suffix) || etag.is_empty()) + { + return Err(TableCatalogStoreError::Invalid( + "invalid maintenance rename destination watermark".to_string(), + )); + } + Ok(Some((intent, etag))) + } + + pub(super) async fn prepare_table_maintenance_rename( + &self, + source: &TableEntry, + destination: &TableEntry, + source_snapshot_etag: &str, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult> { + fence.ensure_held()?; + let source_objects = self.collect_table_maintenance_rename_objects(source).await?; + let destination_objects = self.collect_table_maintenance_rename_objects(destination).await?; + if source_objects.is_empty() && destination_objects.is_empty() { + return Ok(None); + } + let intent = TableMaintenanceRenameIntent { + version: 1, + source: source.clone(), + destination: destination.clone(), + source_snapshot_etag: source_snapshot_etag.to_string(), + source_objects, + destination_etags: destination_objects + .into_iter() + .map(|(key, value)| (key, value.etag)) + .collect(), + }; + let object = maintenance_rename_intent_path(&source.table_bucket); + let data = serde_json::to_vec(&intent) + .map_err(|err| TableCatalogStoreError::Internal(format!("failed to serialize maintenance rename intent: {err}")))?; + if data.len() > TABLE_CATALOG_BACKUP_MAX_SIZE { + return Err(TableCatalogStoreError::Invalid( + "maintenance rename intent exceeds its byte limit".to_string(), + )); + } + self.backend + .put_object_fenced(self.catalog_bucket(), &object, data, TableCatalogPutPrecondition::IfAbsent, fence) + .await?; + let (persisted_intent, intent_etag) = self + .read_table_maintenance_rename_intent(&source.table_bucket) + .await? + .ok_or_else(|| TableCatalogStoreError::Conflict("maintenance rename intent disappeared".to_string()))?; + if persisted_intent != intent { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename intent changed during preparation".to_string(), + )); + } + let destination_prefix = maintenance_rename_prefix(destination)?; + for (suffix, value) in &intent.source_objects { + let renamed = maintenance_rename_data(source, destination, suffix, &value.data)?; + let data = serde_json::to_vec(&renamed).map_err(|err| { + TableCatalogStoreError::Internal(format!("failed to serialize renamed maintenance object: {err}")) + })?; + let precondition = intent + .destination_etags + .get(suffix) + .map_or(TableCatalogPutPrecondition::IfAbsent, |etag| { + TableCatalogPutPrecondition::IfMatch(etag.clone()) + }); + self.backend + .put_object_fenced(self.catalog_bucket(), &format!("{destination_prefix}{suffix}"), data, precondition, fence) + .await?; + } + // Inactive leftovers from a former visit to this name must not resurrect jobs or overrides. + for (suffix, etag) in &intent.destination_etags { + if !intent.source_objects.contains_key(suffix) { + self.backend + .delete_object_if_match_fenced(self.catalog_bucket(), &format!("{destination_prefix}{suffix}"), etag, fence) + .await?; + } + } + let current_source = self.collect_table_maintenance_rename_objects(source).await?; + if current_source.len() != intent.source_objects.len() + || current_source + .iter() + .any(|(suffix, value)| intent.source_objects.get(suffix).is_none_or(|saved| saved.etag != value.etag)) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename source changed before catalog publication".to_string(), + )); + } + let current_destination = self.collect_table_maintenance_rename_objects(destination).await?; + if current_destination.len() != intent.source_objects.len() { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename destination is not a complete mirror".to_string(), + )); + } + for (suffix, value) in &intent.source_objects { + if current_destination.get(suffix).map(|value| &value.data) + != Some(&maintenance_rename_data(source, destination, suffix, &value.data)?) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename destination changed before publication".to_string(), + )); + } + } + fence.ensure_held()?; + Ok(Some((intent, intent_etag))) + } + + pub(super) async fn finish_table_maintenance_rename( + &self, + intent: &TableMaintenanceRenameIntent, + intent_etag: &str, + committed: bool, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult<()> { + fence.ensure_held()?; + let authoritative = if committed { &intent.destination } else { &intent.source }; + let authoritative_objects = self.collect_table_maintenance_rename_objects(authoritative).await?; + if intent + .source_objects + .keys() + .any(|suffix| !authoritative_objects.contains_key(suffix)) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename authoritative records are incomplete; repair persisted state before retrying rename" + .to_string(), + )); + } + let entry = if committed { &intent.source } else { &intent.destination }; + let prefix = maintenance_rename_prefix(entry)?; + let current = self.collect_table_maintenance_rename_objects(entry).await?; + // Validate the whole cleanup set before the first delete. A failed or interrupted copy + // is recognizable by its exact planned JSON; every deletion still uses the observed ETag. + for (suffix, value) in ¤t { + let allowed = if committed { + intent + .source_objects + .get(suffix) + .is_some_and(|saved| saved.etag == value.etag) + } else if intent.destination_etags.get(suffix) == Some(&value.etag) { + true + } else if let Some(saved) = intent.source_objects.get(suffix) { + value.data == maintenance_rename_data(&intent.source, &intent.destination, suffix, &saved.data)? + } else { + false + }; + if !allowed { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename cleanup encountered a changed object watermark".to_string(), + )); + } + } + for (suffix, value) in current { + self.backend + .delete_object_if_match_fenced(self.catalog_bucket(), &format!("{prefix}{suffix}"), &value.etag, fence) + .await?; + } + let object = maintenance_rename_intent_path(&intent.source.table_bucket); + match self + .backend + .delete_object_if_match_fenced(self.catalog_bucket(), &object, intent_etag, fence) + .await + { + Ok(()) => Ok(()), + Err(err) => { + fence.ensure_held()?; + // A lost delete response after removal must not strand a committed rename. + if self.backend.object_metadata(self.catalog_bucket(), &object).await?.is_none() { + Ok(()) + } else { + Err(err) + } + } + } + } + pub(in crate::table_catalog) async fn read_entry( &self, bucket: &str, @@ -741,6 +1189,34 @@ where self.backend.put_object(bucket, object, data, precondition).await } + async fn write_maintenance_entry( + &self, + object: &str, + value: &T, + entry: &TableEntry, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult<()> + where + T: Serialize, + { + let Some(fence) = table_maintenance_mutation_fence(publication, entry)? else { + return self + .write_entry(self.catalog_bucket(), object, value, TableCatalogPutPrecondition::Any) + .await; + }; + let precondition = match self.backend.object_metadata(self.catalog_bucket(), object).await? { + None => TableCatalogPutPrecondition::IfAbsent, + Some(current) => TableCatalogPutPrecondition::IfMatch(current.etag.ok_or_else(|| { + TableCatalogStoreError::Conflict(format!("maintenance sidecar has no conditional-write ETag: {object}")) + })?), + }; + let data = serde_json::to_vec(value) + .map_err(|err| TableCatalogStoreError::Internal(format!("failed to serialize catalog entry {object}: {err}")))?; + self.backend + .put_object_fenced(self.catalog_bucket(), object, data, precondition, &fence) + .await + } + async fn write_entry_unlocked( &self, bucket: &str, @@ -2660,9 +3136,19 @@ where ))); }; + self.get_table_maintenance_config_for_entry(&entry).await + } + + pub(super) async fn get_table_maintenance_config_for_entry( + &self, + entry: &TableEntry, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let config_path = self .paths - .table_maintenance_config_path(table_bucket, &namespace, &table, &entry.table_id); + .table_maintenance_config_path(&entry.table_bucket, &namespace, &table, &entry.table_id); let config = self .read_entry::(self.catalog_bucket(), &config_path) .await? @@ -2711,7 +3197,7 @@ where .await } - async fn get_effective_table_maintenance_config_for_entry_unlocked( + pub(super) async fn get_effective_table_maintenance_config_for_entry_unlocked( &self, table_bucket: &str, namespace: &Namespace, @@ -2770,11 +3256,34 @@ where ))); }; + self.put_table_maintenance_config_for_entry(&entry, config).await + } + + pub(super) async fn put_table_maintenance_config_for_entry( + &self, + entry: &TableEntry, + config: TableMaintenanceConfig, + ) -> TableCatalogStoreResult { + self.put_table_maintenance_config_for_entry_with_publication(entry, config, None) + .await + } + + pub(super) async fn put_table_maintenance_config_for_entry_with_publication( + &self, + entry: &TableEntry, + config: TableMaintenanceConfig, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult { + validate_table_maintenance_config(&config)?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; let config_path = self .paths - .table_maintenance_config_path(table_bucket, &namespace, &table, &entry.table_id); - self.write_entry(self.catalog_bucket(), &config_path, &config, TableCatalogPutPrecondition::Any) + .table_maintenance_config_path(&entry.table_bucket, &namespace, &table, &entry.table_id); + ensure_table_maintenance_publication_held(publication, entry, "before config write")?; + self.write_maintenance_entry(&config_path, &config, entry, publication) .await?; + ensure_table_maintenance_publication_held(publication, entry, "after config write")?; Ok(config) } @@ -2808,10 +3317,20 @@ where self.put_table_metadata_maintenance_report_for_entry(report, &entry).await } - async fn put_table_metadata_maintenance_report_for_entry( + pub(super) async fn put_table_metadata_maintenance_report_for_entry( &self, report: &TableMetadataMaintenanceReport, entry: &TableEntry, + ) -> TableCatalogStoreResult<()> { + self.put_table_metadata_maintenance_report_for_entry_with_publication(report, entry, None) + .await + } + + pub(super) async fn put_table_metadata_maintenance_report_for_entry_with_publication( + &self, + report: &TableMetadataMaintenanceReport, + entry: &TableEntry, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult<()> { let report = table_maintenance_report_with_recommended_actions(report.clone()); let namespace = parse_namespace_for_store(&entry.namespace)?; @@ -2826,12 +3345,17 @@ where let current_job_path = self.paths .table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id); - self.write_entry(self.catalog_bucket(), &job_path, &report, TableCatalogPutPrecondition::Any) + ensure_table_maintenance_publication_held(publication, entry, "before job report write")?; + self.write_maintenance_entry(&job_path, &report, entry, publication).await?; + ensure_table_maintenance_publication_held(publication, entry, "after job report write")?; + ensure_table_maintenance_publication_held(publication, entry, "before latest report write")?; + self.write_maintenance_entry(&latest_job_path, &report, entry, publication) .await?; - self.write_entry(self.catalog_bucket(), &latest_job_path, &report, TableCatalogPutPrecondition::Any) + ensure_table_maintenance_publication_held(publication, entry, "after latest report write")?; + ensure_table_maintenance_publication_held(publication, entry, "before current report write")?; + self.write_maintenance_entry(¤t_job_path, &report, entry, publication) .await?; - self.write_entry(self.catalog_bucket(), ¤t_job_path, &report, TableCatalogPutPrecondition::Any) - .await + ensure_table_maintenance_publication_held(publication, entry, "after current report write") } pub(crate) async fn get_table_metadata_maintenance_report( @@ -2856,7 +3380,7 @@ where .await } - async fn get_table_metadata_maintenance_report_for_entry_unlocked( + pub(super) async fn get_table_metadata_maintenance_report_for_entry_unlocked( &self, table_bucket: &str, namespace: &Namespace, @@ -2925,20 +3449,45 @@ where table.as_str() ))); }; + self.get_table_maintenance_scheduler_report_for_entry_at(&entry, now).await + } + + pub(super) async fn get_table_maintenance_scheduler_report_for_entry_at( + &self, + entry: &TableEntry, + now: OffsetDateTime, + ) -> TableCatalogStoreResult { + self.get_table_maintenance_scheduler_report_with_audit_reports_for_entry_at(entry, now, None) + .await + } + + pub(super) async fn get_table_maintenance_scheduler_report_with_audit_reports_for_entry_at( + &self, + entry: &TableEntry, + now: OffsetDateTime, + reports: Option>, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; let effective = self - .get_effective_table_maintenance_config(table_bucket, &namespace.public_name(), table.as_str()) + .get_effective_table_maintenance_config_for_entry_unlocked(&entry.table_bucket, &namespace, &table, entry) .await?; let current = self - .get_table_metadata_maintenance_report( - table_bucket, - &namespace.public_name(), - table.as_str(), + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, MAINTENANCE_JOB_ALIAS_CURRENT, ) .await?; - let reports = self - .list_table_metadata_maintenance_audit_reports(table_bucket, &namespace, &table, &entry.table_id) - .await?; + let reports = match reports { + Some(reports) => reports, + None => { + self.list_table_metadata_maintenance_audit_reports(&entry.table_bucket, &namespace, &table, &entry.table_id) + .await? + } + }; let quarantine = table_maintenance_scheduler_quarantine_boundary(&effective.config, &reports); let mut recommended_actions = Vec::new(); @@ -2978,10 +3527,10 @@ where }; Ok(TableMaintenanceSchedulerReport { - table_bucket: table_bucket.to_string(), + table_bucket: entry.table_bucket.clone(), namespace: namespace.public_name(), table: table.as_str().to_string(), - table_id: entry.table_id, + table_id: entry.table_id.clone(), status, config_source: effective.source, background_enabled: effective.config.background_enabled, @@ -3016,6 +3565,19 @@ where table: &str, scheduler_id: String, now: OffsetDateTime, + ) -> TableCatalogStoreResult { + self.run_table_maintenance_scheduler_once_with_guard_at(table_bucket, namespace, table, scheduler_id, now, None) + .await + } + + pub(super) async fn run_table_maintenance_scheduler_once_with_guard_at( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + scheduler_id: String, + now: OffsetDateTime, + periodic_guard: Option<&PeriodicTableMaintenanceQueueGuard<'_>>, ) -> TableCatalogStoreResult { let namespace = parse_namespace_for_store(namespace)?; let table = parse_table_for_store(table)?; @@ -3035,6 +3597,19 @@ where let effective = self .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) .await?; + if let Some(periodic_guard) = periodic_guard { + self.validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } self.table_metadata_maintenance_scheduler_preflight( TableMaintenancePreflightContext { table_bucket, @@ -3081,6 +3656,19 @@ where let effective = self .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) .await?; + if let Some(periodic_guard) = periodic_guard { + self.validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } match self .table_metadata_maintenance_scheduler_preflight( TableMaintenancePreflightContext { @@ -3278,26 +3866,105 @@ where let jobs_prefix = self .paths .table_maintenance_jobs_prefix(table_bucket, namespace, table, table_id); + let objects = self.backend.list_objects(self.catalog_bucket(), &jobs_prefix).await?; + self.read_table_metadata_maintenance_audit_reports(table_bucket, namespace, table, table_id, &objects, false) + .await + } + + pub(super) async fn list_table_metadata_maintenance_audit_objects_for_entry( + &self, + entry: &TableEntry, + ) -> TableCatalogStoreResult> { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let prefix = self + .paths + .table_maintenance_jobs_prefix(&entry.table_bucket, &namespace, &table, &entry.table_id); + let mut objects = Vec::new(); + let mut cursor: Option = None; + loop { + let remaining = TABLE_MAINTENANCE_SCHEDULER_HISTORY_MAX_OBJECTS.saturating_sub(objects.len()); + let limit = NonZeroUsize::new(TABLE_CATALOG_LIST_MAX_KEYS.min(remaining + 1)).ok_or_else(|| { + TableCatalogStoreError::Internal("maintenance scheduler page limit must be positive".to_string()) + })?; + let page = self + .backend + .list_objects_page(self.catalog_bucket(), &prefix, cursor.as_deref(), limit) + .await?; + if page.objects.len() > limit.get() || (page.is_truncated && page.objects.is_empty()) { + return Err(TableCatalogStoreError::Invalid( + "invalid maintenance scheduler history listing page".to_string(), + )); + } + for object in page.objects { + if !object.starts_with(&prefix) || cursor.as_ref().is_some_and(|cursor| object <= *cursor) { + return Err(TableCatalogStoreError::Invalid( + "maintenance scheduler history listing escaped its prefix or did not advance".to_string(), + )); + } + if objects.len() == TABLE_MAINTENANCE_SCHEDULER_HISTORY_MAX_OBJECTS { + return Err(TableCatalogStoreError::Unavailable( + "maintenance scheduler history exceeds the 4096-object read budget".to_string(), + )); + } + cursor = Some(object.clone()); + objects.push(object); + } + if !page.is_truncated { + return Ok(objects); + } + } + } + + pub(super) async fn read_table_metadata_maintenance_audit_reports_for_entry( + &self, + entry: &TableEntry, + objects: &[String], + ) -> TableCatalogStoreResult> { + self.read_table_metadata_maintenance_audit_reports( + &entry.table_bucket, + &parse_namespace_for_store(&entry.namespace)?, + &parse_table_for_store(&entry.table)?, + &entry.table_id, + objects, + true, + ) + .await + } + + async fn read_table_metadata_maintenance_audit_reports( + &self, + table_bucket: &str, + namespace: &Namespace, + table: &IdentifierSegment, + table_id: &str, + objects: &[String], + require_present: bool, + ) -> TableCatalogStoreResult> { let mut reports = Vec::new(); - for object in self.backend.list_objects(self.catalog_bucket(), &jobs_prefix).await? { + for object in objects { if !object.ends_with(".json") { continue; } if let Some((report, _)) = self - .read_entry::(self.catalog_bucket(), &object) + .read_entry::(self.catalog_bucket(), object) .await? { validate_table_maintenance_report_owner(&report, table_bucket, namespace, table, table_id)?; if self .paths .table_maintenance_job_path(table_bucket, namespace, table, table_id, &report.job.job_id) - != object + != *object { return Err(TableCatalogStoreError::Invalid( "table maintenance report identity does not match its object path".to_string(), )); } reports.push(table_maintenance_report_with_recommended_actions(report)); + } else if require_present { + return Err(TableCatalogStoreError::Unavailable( + "maintenance scheduler history changed during the read; retry the scheduler report".to_string(), + )); } } reports.sort_by(|left, right| { @@ -3309,42 +3976,115 @@ where Ok(reports) } - async fn table_metadata_maintenance_scheduler_preflight( + pub(super) async fn validate_periodic_table_maintenance_queue_guard( + &self, + context: TableMaintenancePreflightContext<'_>, + effective: &TableMaintenanceEffectiveConfig, + expected: &PeriodicTableMaintenanceQueueGuard<'_>, + ) -> TableCatalogStoreResult<()> { + if context.entry.table_id != expected.table_id { + return Err(TableCatalogStoreError::Conflict( + "table identity changed before periodic maintenance claim".to_string(), + )); + } + if !effective.config.background_enabled || effective.config.worker_paused { + return Err(TableCatalogStoreError::Conflict("periodic maintenance is disabled or paused".to_string())); + } + let current = self + .get_table_metadata_maintenance_report_for_entry_unlocked( + context.table_bucket, + context.namespace, + context.table, + &context.entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + if current.as_ref().map(table_maintenance_scheduler_job_summary).as_ref() != expected.current_job + || current + .as_ref() + .is_some_and(|report| report.job.status != TableMetadataMaintenanceJobStatus::Successful) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance job changed or requires explicit operator action before periodic scheduling".to_string(), + )); + } + let latest = self + .get_table_metadata_maintenance_report_for_entry_unlocked( + context.table_bucket, + context.namespace, + context.table, + &context.entry.table_id, + MAINTENANCE_JOB_ALIAS_LATEST, + ) + .await?; + // Report publication writes latest before current. A partial write must + // not let an automatic pass replace an unresolved job or quarantine. + if latest != current + || current + .as_ref() + .is_some_and(|report| report.job.quarantine_enabled && report.job.quarantined_object_count > 0) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance report publication or quarantine requires recovery before periodic scheduling".to_string(), + )); + } + Ok(()) + } + + pub(super) async fn table_metadata_maintenance_scheduler_preflight( &self, context: TableMaintenancePreflightContext<'_>, scheduler_id: &str, now: OffsetDateTime, effective: TableMaintenanceEffectiveConfig, + ) -> TableCatalogStoreResult { + self.table_metadata_maintenance_scheduler_preflight_with_publication(context, scheduler_id, now, effective, None) + .await + } + + pub(super) async fn table_metadata_maintenance_scheduler_preflight_with_publication( + &self, + context: TableMaintenancePreflightContext<'_>, + scheduler_id: &str, + now: OffsetDateTime, + effective: TableMaintenanceEffectiveConfig, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { if !effective.config.background_enabled { let report = self - .put_table_metadata_maintenance_scheduler_control_report(TableMaintenanceSchedulerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - scheduler_id: scheduler_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Disabled, - reason: "background maintenance is disabled", - now, - }) + .put_table_metadata_maintenance_scheduler_control_report( + TableMaintenanceSchedulerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + scheduler_id: scheduler_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Disabled, + reason: "background maintenance is disabled", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceSchedulerPreflight::Complete(Box::new(report))); } if effective.config.worker_paused { let report = self - .put_table_metadata_maintenance_scheduler_control_report(TableMaintenanceSchedulerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - scheduler_id: scheduler_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Paused, - reason: "background maintenance worker is paused", - now, - }) + .put_table_metadata_maintenance_scheduler_control_report( + TableMaintenanceSchedulerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + scheduler_id: scheduler_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Paused, + reason: "background maintenance worker is paused", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceSchedulerPreflight::Complete(Box::new(report))); } @@ -3369,6 +4109,7 @@ where now, "maintenance worker lease expired", TableMaintenanceAuditAction::WorkerLeaseExpired, + publication, ) .await?; } else if matches!(current.job.status, TableMetadataMaintenanceJobStatus::Queued) { @@ -3381,6 +4122,7 @@ where now, "maintenance scheduler lease expired", TableMaintenanceAuditAction::SchedulerLeaseExpired, + publication, ) .await?; } else if table_maintenance_job_retry_is_pending(¤t.job, now) { @@ -3391,6 +4133,169 @@ where Ok(TableMaintenanceSchedulerPreflight::Ready(effective)) } + pub(crate) async fn run_table_metadata_maintenance_worker_once_conditionally( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + ) -> TableCatalogStoreResult { + self.run_table_metadata_maintenance_worker_once_conditionally_with_clock( + table_bucket, + namespace, + table, + worker_id, + condition, + OffsetDateTime::now_utc, + ) + .await + } + + pub(in crate::table_catalog) async fn run_table_metadata_maintenance_worker_once_conditionally_with_clock( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + clock: impl FnOnce() -> OffsetDateTime + Send, + ) -> TableCatalogStoreResult { + validate_table_maintenance_worker_condition(&worker_id, condition)?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = TableCommitLockPublication::new(&self.backend); + publication.attach_catalog_migration_read_guard(self.acquire_object_backed_catalog_write_permit(table_bucket).await?)?; + publication.begin_table_bucket(table_bucket).await?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + self.ensure_no_active_table_rename(table_bucket).await?; + let table_path = self.paths.table_entry_path(table_bucket, &namespace, &table); + let guard = self.backend.acquire_write_lock(self.catalog_bucket(), &table_path).await?; + publication.attach_catalog_write_guard(guard)?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let Some((entry, _)) = self.read_table_with_etag_unlocked(table_bucket, &namespace, &table).await? else { + return Err(TableCatalogStoreError::NotFound(format!( + "table {}/{}/{}", + table_bucket, + namespace.public_name(), + table.as_str() + ))); + }; + self.run_conditional_table_maintenance_worker_for_entry(&entry, worker_id, condition, clock, &publication) + .await + } + + pub(super) async fn run_conditional_table_maintenance_worker_for_entry( + &self, + entry: &TableEntry, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + clock: impl FnOnce() -> OffsetDateTime + Send, + publication: &(dyn TableCommitPublication + Sync), + ) -> TableCatalogStoreResult { + validate_table_maintenance_worker_condition(&worker_id, condition)?; + ensure_table_maintenance_publication_held(Some(publication), entry, "before conditional worker claim")?; + if entry.state != TableCatalogEntryState::Active { + return Err(TableCatalogStoreError::Conflict( + "conditional maintenance requires an active table".to_string(), + )); + } + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let effective = self + .get_effective_table_maintenance_config_for_entry_unlocked(&entry.table_bucket, &namespace, &table, entry) + .await?; + let mut reports = Vec::with_capacity(3); + for job_id in [ + condition.expected_job_id.as_str(), + MAINTENANCE_JOB_ALIAS_LATEST, + MAINTENANCE_JOB_ALIAS_CURRENT, + ] { + let report = self.get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, &namespace, &table, &entry.table_id, job_id, + ).await?.ok_or_else(|| TableCatalogStoreError::Conflict( + "conditional maintenance requires fully published named, latest, and current reports; inspect the persisted job".to_string(), + ))?; + reports.push(report); + } + let mut report = reports.remove(0); + // Lock acquisition and report reads can outlast the queued job's lease. + // Sample the claim time only after those waits have completed. + let now = clock(); + let replay = validate_conditional_table_maintenance_claim(&report, entry, &effective, &worker_id, condition, now)?; + if reports.iter().any(|alias| alias != &report) { + // A complete terminal receipt can finish publishing aliases after a + // lost response. Never resume a partial Running claim or rerun deletes. + if !replay + || !matches!( + report.job.status, + TableMetadataMaintenanceJobStatus::Successful | TableMetadataMaintenanceJobStatus::Failed + ) + || reports + .iter() + .any(|alias| alias != &report && !maintenance_terminal_report_continues_running(&report, alias)) + { + return Err(TableCatalogStoreError::Conflict( + "maintenance report publication is incomplete; inspect the named job and recover before another claim" + .to_string(), + )); + } + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, Some(publication)) + .await?; + return Ok(report); + } + if replay { + return Ok(report); + } + + // Backoff delays follow the validated current policy; its attempt limit + // must never replace the budget persisted when this job was scheduled. + let mut retry_effective = effective; + retry_effective.config.max_retry_attempts = report.job.max_retry_attempts; + validate_table_maintenance_config(&retry_effective.config)?; + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + let started_at = maintenance_timestamp(now); + report.job.attempt = report + .job + .attempt + .checked_add(1) + .ok_or_else(|| TableCatalogStoreError::Invalid("maintenance attempt exceeds its supported range".to_string()))?; + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.failure_reason = None; + report.job.worker_id = Some(worker_id); + report.job.lease_id = Uuid::new_v4().to_string(); + report.job.next_retry_after = None; + report.job.heartbeat_at = Some(started_at.clone()); + report.job.started_at = Some(started_at); + report.job.finished_at = None; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerStarted, + None, + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, Some(publication)) + .await?; + + let delete = matches!(report.job.operation, TableMetadataMaintenanceOperation::Delete); + self.finish_table_metadata_maintenance_run_for_entry_with_optional_publication( + entry, + delete, + &retry_effective, + report, + Some(publication), + true, + ) + .await + } + pub(crate) async fn run_table_metadata_maintenance_worker_once( &self, table_bucket: &str, @@ -3551,42 +4456,60 @@ where .await } - async fn table_metadata_maintenance_worker_preflight( + pub(super) async fn table_metadata_maintenance_worker_preflight( &self, context: TableMaintenancePreflightContext<'_>, worker_id: &str, now: OffsetDateTime, effective: TableMaintenanceEffectiveConfig, + ) -> TableCatalogStoreResult { + self.table_metadata_maintenance_worker_preflight_with_publication(context, worker_id, now, effective, None) + .await + } + + pub(super) async fn table_metadata_maintenance_worker_preflight_with_publication( + &self, + context: TableMaintenancePreflightContext<'_>, + worker_id: &str, + now: OffsetDateTime, + effective: TableMaintenanceEffectiveConfig, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { if !effective.config.background_enabled { let report = self - .put_table_metadata_maintenance_worker_control_report(TableMaintenanceWorkerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - worker_id: worker_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Disabled, - reason: "background maintenance is disabled", - now, - }) + .put_table_metadata_maintenance_worker_control_report( + TableMaintenanceWorkerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + worker_id: worker_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Disabled, + reason: "background maintenance is disabled", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceWorkerPreflight::Complete(Box::new(report))); } if effective.config.worker_paused { let report = self - .put_table_metadata_maintenance_worker_control_report(TableMaintenanceWorkerControlReport { - table_bucket: context.table_bucket, - namespace: context.namespace, - table: context.table, - entry: context.entry, - worker_id: worker_id.to_string(), - effective: &effective, - status: TableMetadataMaintenanceJobStatus::Paused, - reason: "background maintenance worker is paused", - now, - }) + .put_table_metadata_maintenance_worker_control_report( + TableMaintenanceWorkerControlReport { + table_bucket: context.table_bucket, + namespace: context.namespace, + table: context.table, + entry: context.entry, + worker_id: worker_id.to_string(), + effective: &effective, + status: TableMetadataMaintenanceJobStatus::Paused, + reason: "background maintenance worker is paused", + now, + }, + publication, + ) .await?; return Ok(TableMaintenanceWorkerPreflight::Complete(Box::new(report))); } @@ -3611,6 +4534,7 @@ where now, "maintenance worker lease expired", TableMaintenanceAuditAction::WorkerLeaseExpired, + publication, ) .await?; } else if matches!(current.job.status, TableMetadataMaintenanceJobStatus::Queued) { @@ -3626,6 +4550,7 @@ where now, "maintenance scheduler lease expired", TableMaintenanceAuditAction::SchedulerLeaseExpired, + publication, ) .await?; } else if table_maintenance_job_retry_is_pending(¤t.job, now) { @@ -3682,9 +4607,32 @@ where table.as_str() ))); }; + self.heartbeat_table_metadata_maintenance_job_for_entry(&entry, heartbeat, now) + .await + } + + pub(super) async fn heartbeat_table_metadata_maintenance_job_for_entry( + &self, + entry: &TableEntry, + heartbeat: TableMaintenanceHeartbeatRef<'_>, + now: OffsetDateTime, + ) -> TableCatalogStoreResult { + self.heartbeat_table_metadata_maintenance_job_for_entry_with_publication(entry, heartbeat, now, None) + .await + } + + pub(super) async fn heartbeat_table_metadata_maintenance_job_for_entry_with_publication( + &self, + entry: &TableEntry, + heartbeat: TableMaintenanceHeartbeatRef<'_>, + now: OffsetDateTime, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; let Some(mut report) = self .get_table_metadata_maintenance_report_for_entry_unlocked( - heartbeat.table_bucket, + &entry.table_bucket, &namespace, &table, &entry.table_id, @@ -3725,7 +4673,8 @@ where Some(TableMetadataMaintenanceJobStatus::Running), before_quarantined_object_count, ); - self.put_table_metadata_maintenance_report_for_entry(&report, &entry).await?; + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; Ok(report) } @@ -3736,6 +4685,7 @@ where now: OffsetDateTime, reason: &str, action: TableMaintenanceAuditAction, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { let before_status = Some(report.job.status.clone()); let before_quarantined_object_count = Some(report.job.quarantined_object_count); @@ -3752,13 +4702,15 @@ where before_status, before_quarantined_object_count, ); - self.put_table_metadata_maintenance_report_for_entry(&report, entry).await?; + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; Ok(report) } async fn put_table_metadata_maintenance_scheduler_control_report( &self, control: TableMaintenanceSchedulerControlReport<'_>, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { let timestamp = maintenance_timestamp(control.now); let cleanup_watermark_unix_seconds = @@ -3829,7 +4781,7 @@ where None, None, ); - self.put_table_metadata_maintenance_report_for_entry(&report, control.entry) + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, control.entry, publication) .await?; Ok(report) } @@ -3837,6 +4789,7 @@ where async fn put_table_metadata_maintenance_worker_control_report( &self, control: TableMaintenanceWorkerControlReport<'_>, + publication: Option<&(dyn TableCommitPublication + Sync)>, ) -> TableCatalogStoreResult { let timestamp = maintenance_timestamp(control.now); let cleanup_watermark_unix_seconds = @@ -3907,7 +4860,7 @@ where None, None, ); - self.put_table_metadata_maintenance_report_for_entry(&report, control.entry) + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, control.entry, publication) .await?; Ok(report) } @@ -3951,7 +4904,30 @@ where )); } - let Some(current_metadata) = read_table_metadata_value(metadata_backend, table_bucket, &entry.metadata_location).await? + self.plan_table_snapshot_expiration_for_entry_with_backend(metadata_backend, &entry, config) + .await + } + + pub(super) async fn plan_table_snapshot_expiration_for_entry_with_backend

( + &self, + metadata_backend: &P, + entry: &TableEntry, + config: TableSnapshotExpirationConfig, + ) -> TableCatalogStoreResult + where + P: TableCatalogObjectBackend, + { + validate_table_snapshot_expiration_config(&config)?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + if !is_valid_table_metadata_location_for_entry(entry, &entry.metadata_location) { + return Err(TableCatalogStoreError::Invalid( + "current metadata location must be inside the table metadata directory".to_string(), + )); + } + + let Some(current_metadata) = + read_table_metadata_value(metadata_backend, &entry.table_bucket, &entry.metadata_location).await? else { return Err(TableCatalogStoreError::NotFound(format!( "current metadata object {}", @@ -3960,10 +4936,10 @@ where }; Ok(table_snapshot_expiration_report( - table_bucket, + &entry.table_bucket, &namespace, &table, - &entry, + entry, ¤t_metadata, config, OffsetDateTime::now_utc(), @@ -3994,7 +4970,25 @@ where )); } - let Some(current_metadata) = read_table_metadata_value(&self.backend, table_bucket, &entry.metadata_location).await? + self.plan_table_compaction_for_entry(&entry, config).await + } + + pub(super) async fn plan_table_compaction_for_entry( + &self, + entry: &TableEntry, + config: TableCompactionPlanningConfig, + ) -> TableCatalogStoreResult { + validate_table_compaction_planning_config(&config)?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + if !is_valid_table_metadata_location_for_entry(entry, &entry.metadata_location) { + return Err(TableCatalogStoreError::Invalid( + "current metadata location must be inside the table metadata directory".to_string(), + )); + } + + let Some(current_metadata) = + read_table_metadata_value(&self.backend, &entry.table_bucket, &entry.metadata_location).await? else { return Err(TableCatalogStoreError::NotFound(format!( "current metadata object {}", @@ -4002,7 +4996,8 @@ where ))); }; - table_compaction_planning_report(&self.backend, table_bucket, &namespace, &table, &entry, ¤t_metadata, config).await + table_compaction_planning_report(&self.backend, &entry.table_bucket, &namespace, &table, entry, ¤t_metadata, config) + .await } #[allow( @@ -4301,7 +5296,36 @@ where table.as_str() ))); }; - if !is_valid_table_metadata_location(&namespace, &table, &entry.metadata_location) { + self.plan_table_metadata_maintenance_for_entry_with_scope(&entry, retain_recent_metadata_files, false) + .await + } + + pub(super) async fn plan_table_metadata_maintenance_for_entry( + &self, + entry: &TableEntry, + retain_recent_metadata_files: usize, + ) -> TableCatalogStoreResult { + self.plan_table_metadata_maintenance_for_entry_with_scope(entry, retain_recent_metadata_files, true) + .await + } + + async fn plan_table_metadata_maintenance_for_entry_with_scope( + &self, + entry: &TableEntry, + retain_recent_metadata_files: usize, + entry_aware: bool, + ) -> TableCatalogStoreResult { + let table_bucket = &entry.table_bucket; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let is_valid_location = |location: &str| { + if entry_aware { + is_valid_table_metadata_location_for_entry(entry, location) + } else { + is_valid_table_metadata_location(&namespace, &table, location) + } + }; + if !is_valid_location(&entry.metadata_location) { return Err(TableCatalogStoreError::Invalid( "current metadata location must be inside the table metadata directory".to_string(), )); @@ -4317,7 +5341,7 @@ where let mut retained = BTreeSet::new(); let mut maintenance_reasons = BTreeMap::>::new(); - for metadata_location in metadata_log_locations(¤t_metadata, table_bucket, &namespace, &table) { + for metadata_location in metadata_log_locations_matching(¤t_metadata, table_bucket, is_valid_location) { retained.insert(metadata_location.clone()); insert_metadata_maintenance_reason( &mut maintenance_reasons, @@ -4333,9 +5357,19 @@ where ); let mut metadata_locations = Vec::new(); - let metadata_prefix = format!("{}/", default_table_metadata_dir_path(&namespace, &table)); + let metadata_dir = if entry_aware { + table_metadata_dir_path_for_entry(entry)? + } else { + default_table_metadata_dir_path(&namespace, &table) + }; + let metadata_prefix = format!("{metadata_dir}/"); for object in self.backend.list_objects(table_bucket, &metadata_prefix).await? { - if let Some(metadata_location) = metadata_location_from_metadata_file_path(&namespace, &table, &object) { + let metadata_location = if entry_aware { + is_valid_location(&object).then_some(object) + } else { + metadata_location_from_metadata_file_path(&namespace, &table, &object) + }; + if let Some(metadata_location) = metadata_location { metadata_locations.push(metadata_location); } } @@ -4353,16 +5387,27 @@ where ); } } - for metadata_location in metadata_locations_for_protected_snapshot_refs( - &self.backend, - table_bucket, - &namespace, - &table, - ¤t_metadata, - &metadata_locations, - ) - .await? - { + let protected_refs = if entry_aware { + metadata_locations_for_protected_snapshot_refs_matching( + &self.backend, + table_bucket, + ¤t_metadata, + &metadata_locations, + is_valid_location, + ) + .await? + } else { + metadata_locations_for_protected_snapshot_refs( + &self.backend, + table_bucket, + &namespace, + &table, + ¤t_metadata, + &metadata_locations, + ) + .await? + }; + for metadata_location in protected_refs { retained.insert(metadata_location.clone()); insert_metadata_maintenance_reason( &mut maintenance_reasons, @@ -4408,33 +5453,55 @@ where ); } } - let warehouse_object_prefix = table_warehouse_object_prefix(&entry).ok(); - let current_metadata_location = entry.metadata_location; + let warehouse_object_prefix = table_warehouse_object_prefix(entry).ok(); + let current_metadata_location = entry.metadata_location.clone(); let retained_metadata_locations = retained.into_iter().collect::>(); let object_reports = metadata_maintenance_object_reports(maintenance_reasons); - let referenced_object_reports = metadata_maintenance_referenced_object_reports( - &self.backend, - table_bucket, - &namespace, - &table, - warehouse_object_prefix.as_deref(), - ¤t_metadata, - &retained_metadata_locations, - ) - .await?; - let reachability_graph = - metadata_maintenance_reachability_graph_report(planned_metadata_file_count, &referenced_object_reports); - let (planned_object_file_count, cleanup_object_candidate_locations, deletable_object_locations, object_cleanup_reports) = - metadata_maintenance_object_cleanup_reports( + let referenced_object_reports = if entry_aware { + metadata_maintenance_referenced_object_reports_for_entry( + &self.backend, + entry, + warehouse_object_prefix.as_deref(), + ¤t_metadata, + &retained_metadata_locations, + ) + .await? + } else { + metadata_maintenance_referenced_object_reports( &self.backend, table_bucket, &namespace, &table, warehouse_object_prefix.as_deref(), - &referenced_object_reports, - now, + ¤t_metadata, + &retained_metadata_locations, ) - .await?; + .await? + }; + let reachability_graph = + metadata_maintenance_reachability_graph_report(planned_metadata_file_count, &referenced_object_reports); + let (planned_object_file_count, cleanup_object_candidate_locations, deletable_object_locations, object_cleanup_reports) = + if entry_aware { + metadata_maintenance_object_cleanup_reports_for_entry( + &self.backend, + entry, + warehouse_object_prefix.as_deref(), + &referenced_object_reports, + now, + ) + .await? + } else { + metadata_maintenance_object_cleanup_reports( + &self.backend, + table_bucket, + &namespace, + &table, + warehouse_object_prefix.as_deref(), + &referenced_object_reports, + now, + ) + .await? + }; let mut report = table_maintenance_report_with_recommended_actions(TableMetadataMaintenanceReport { job: TableMetadataMaintenanceJob { @@ -4442,7 +5509,7 @@ where table_bucket: table_bucket.to_string(), namespace: namespace.public_name(), table: table.as_str().to_string(), - table_id: entry.table_id, + table_id: entry.table_id.clone(), operation: TableMetadataMaintenanceOperation::DryRun, status: TableMetadataMaintenanceJobStatus::Successful, failure_reason: None, @@ -4707,6 +5774,134 @@ where Ok(report) } + pub(super) async fn finish_table_metadata_maintenance_run_for_entry_with_publication( + &self, + entry: &TableEntry, + delete: bool, + effective: &TableMaintenanceEffectiveConfig, + report: TableMetadataMaintenanceReport, + publication: &(dyn TableCommitPublication + Sync), + ) -> TableCatalogStoreResult { + self.finish_table_metadata_maintenance_run_for_entry_with_optional_publication( + entry, + delete, + effective, + report, + Some(publication), + false, + ) + .await + } + + async fn finish_table_metadata_maintenance_run_for_entry_with_optional_publication( + &self, + entry: &TableEntry, + delete: bool, + effective: &TableMaintenanceEffectiveConfig, + mut report: TableMetadataMaintenanceReport, + publication: Option<&(dyn TableCommitPublication + Sync)>, + preserve_candidates: bool, + ) -> TableCatalogStoreResult { + if delete && !effective.config.delete_enabled { + let finished_at = OffsetDateTime::now_utc(); + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + report.job.status = TableMetadataMaintenanceJobStatus::Failed; + report.job.failure_reason = Some(TABLE_MAINTENANCE_DELETE_DISABLED_REASON.to_string()); + apply_maintenance_retry_after(&mut report.job, &effective.config, finished_at); + report.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerFailed, + Some(TABLE_MAINTENANCE_DELETE_DISABLED_REASON.to_string()), + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; + return Ok(report); + } + + if delete { + let running_report = report.clone(); + let mut deleted = match self + .delete_table_metadata_maintenance_report_for_entry_with_publication(entry, report, publication) + .await + { + Ok(report) => report, + Err(err) => { + let finished_at = OffsetDateTime::now_utc(); + let mut failed = running_report; + let before_status = Some(failed.job.status.clone()); + let before_quarantined_object_count = Some(failed.job.quarantined_object_count); + let reason = err.to_string(); + failed.job.status = TableMetadataMaintenanceJobStatus::Failed; + failed.job.failure_reason = Some(reason.clone()); + apply_maintenance_retry_after(&mut failed.job, &effective.config, finished_at); + failed.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut failed); + push_table_maintenance_audit_event( + &mut failed, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerFailed, + Some(reason), + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&failed, entry, publication) + .await?; + return Err(err); + } + }; + if preserve_candidates { + deleted.cleanup_candidate_locations = running_report.cleanup_candidate_locations; + deleted.deletable_metadata_locations = running_report.deletable_metadata_locations; + deleted.cleanup_object_candidate_locations = running_report.cleanup_object_candidate_locations; + deleted.deletable_object_locations = running_report.deletable_object_locations; + } + let finished_at = OffsetDateTime::now_utc(); + let before_status = Some(TableMetadataMaintenanceJobStatus::Running); + let before_quarantined_object_count = Some(0); + deleted.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut deleted); + push_table_maintenance_audit_event( + &mut deleted, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerSucceeded, + None, + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&deleted, entry, publication) + .await?; + return Ok(deleted); + } + + let finished_at = OffsetDateTime::now_utc(); + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + report.job.status = TableMetadataMaintenanceJobStatus::Successful; + report.job.finished_at = Some(maintenance_timestamp(finished_at)); + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + finished_at, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerSucceeded, + None, + before_status, + before_quarantined_object_count, + ); + self.put_table_metadata_maintenance_report_for_entry_with_publication(&report, entry, publication) + .await?; + Ok(report) + } + #[allow( dead_code, reason = "exercised by table_catalog/tests.rs; the lib target cannot see test-only consumers (backlog#1823)" @@ -4750,12 +5945,47 @@ where table.as_str() ))); }; + self.delete_table_metadata_maintenance_report_for_entry(&entry, report).await + } + + pub(super) async fn delete_table_metadata_maintenance_report_for_entry( + &self, + entry: &TableEntry, + report: TableMetadataMaintenanceReport, + ) -> TableCatalogStoreResult { + self.delete_table_metadata_maintenance_report_for_entry_with_publication(entry, report, None) + .await + } + + async fn delete_table_metadata_maintenance_report_for_entry_with_publication( + &self, + entry: &TableEntry, + report: TableMetadataMaintenanceReport, + publication: Option<&(dyn TableCommitPublication + Sync)>, + ) -> TableCatalogStoreResult { + let mutation_fence = table_maintenance_mutation_fence(publication, entry)?; + let table_bucket = &entry.table_bucket; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let is_valid_location = |location: &str| { + if publication.is_some() { + is_valid_table_metadata_location_for_entry(entry, location) + } else { + is_valid_table_metadata_location(&namespace, &table, location) + } + }; + if !is_valid_location(&report.current_metadata_location) { + return Err(TableCatalogStoreError::Invalid( + "maintenance report current metadata location must be inside the table metadata directory".to_string(), + )); + } + if entry.metadata_location != report.current_metadata_location { return Err(TableCatalogStoreError::Conflict( "current metadata location changed before maintenance delete".to_string(), )); } - let warehouse_object_prefix = table_warehouse_object_prefix(&entry).ok(); + let warehouse_object_prefix = table_warehouse_object_prefix(entry).ok(); let Some(current_metadata) = read_table_metadata_value(&self.backend, table_bucket, &entry.metadata_location).await? else { @@ -4765,17 +5995,16 @@ where ))); }; - let mut protected = metadata_log_locations(¤t_metadata, table_bucket, &namespace, &table); + let mut protected = metadata_log_locations_matching(¤t_metadata, table_bucket, is_valid_location); protected.insert(entry.metadata_location.clone()); protected.extend(report.retained_metadata_locations.iter().cloned()); protected.extend( - metadata_locations_for_protected_snapshot_refs( + metadata_locations_for_protected_snapshot_refs_matching( &self.backend, table_bucket, - &namespace, - &table, ¤t_metadata, &report.cleanup_candidate_locations, + is_valid_location, ) .await?, ); @@ -4783,9 +6012,10 @@ where let cleanup_candidate_count = report.cleanup_candidate_locations.len(); let planned_deletable_locations = report.deletable_metadata_locations.iter().cloned().collect::>(); let mut cleanup_candidate_locations = BTreeSet::new(); + let mut delete_etags = BTreeMap::new(); let now = OffsetDateTime::now_utc(); for metadata_location in &report.cleanup_candidate_locations { - if !is_valid_table_metadata_location(&namespace, &table, metadata_location) { + if !is_valid_location(metadata_location) { return Err(TableCatalogStoreError::Invalid(format!( "cleanup candidate {metadata_location} must be inside the table metadata directory" ))); @@ -4804,25 +6034,58 @@ where if !metadata_candidate_is_past_safety_window(candidate_object.mod_time, now) { continue; } + if mutation_fence.is_some() { + if candidate_object + .mod_time + .is_none_or(|modified| modified.unix_timestamp() > report.job.cleanup_watermark_unix_seconds) + { + continue; + } + let etag = candidate_object.etag.ok_or_else(|| { + TableCatalogStoreError::Conflict(format!( + "maintenance candidate has no conditional-delete ETag: {metadata_location}" + )) + })?; + delete_etags.insert(metadata_location.clone(), etag); + } cleanup_candidate_locations.insert(metadata_location.clone()); } let cleanup_candidate_locations = cleanup_candidate_locations.into_iter().collect::>(); let deleted_locations = cleanup_candidate_locations.iter().cloned().collect::>(); for metadata_location in &cleanup_candidate_locations { - self.backend.delete_object(table_bucket, metadata_location).await?; + ensure_table_maintenance_publication_held(publication, entry, "before metadata deletion")?; + if let Some(fence) = &mutation_fence { + self.backend + .delete_object_if_match_fenced(table_bucket, metadata_location, &delete_etags[metadata_location], fence) + .await?; + } else { + self.backend.delete_object(table_bucket, metadata_location).await?; + } + ensure_table_maintenance_publication_held(publication, entry, "after metadata deletion")?; } - let referenced_object_reports = metadata_maintenance_referenced_object_reports( - &self.backend, - table_bucket, - &namespace, - &table, - warehouse_object_prefix.as_deref(), - ¤t_metadata, - &report.retained_metadata_locations, - ) - .await?; + let referenced_object_reports = if publication.is_some() { + metadata_maintenance_referenced_object_reports_for_entry( + &self.backend, + entry, + warehouse_object_prefix.as_deref(), + ¤t_metadata, + &report.retained_metadata_locations, + ) + .await? + } else { + metadata_maintenance_referenced_object_reports( + &self.backend, + table_bucket, + &namespace, + &table, + warehouse_object_prefix.as_deref(), + ¤t_metadata, + &report.retained_metadata_locations, + ) + .await? + }; let referenced_object_locations = if referenced_object_reports .iter() .any(|report| report.state == TableMetadataMaintenanceObjectState::ManualReviewRequired) @@ -4841,9 +6104,12 @@ where .any(|report| report.state == TableMetadataMaintenanceObjectState::ManualReviewRequired) { for object_location in &report.cleanup_object_candidate_locations { - if table_maintenance_object_kind(&namespace, &table, warehouse_object_prefix.as_deref(), object_location) - .is_none() - { + let object_kind = if publication.is_some() { + table_maintenance_object_kind_for_entry(entry, warehouse_object_prefix.as_deref(), object_location) + } else { + table_maintenance_object_kind(&namespace, &table, warehouse_object_prefix.as_deref(), object_location) + }; + if object_kind.is_none() { return Err(TableCatalogStoreError::Invalid(format!( "cleanup object candidate {object_location} must be inside table metadata, data, or delete directories" ))); @@ -4862,6 +6128,20 @@ where if !metadata_candidate_is_past_safety_window(candidate_object.mod_time, now) { continue; } + if mutation_fence.is_some() { + if candidate_object + .mod_time + .is_none_or(|modified| modified.unix_timestamp() > report.job.cleanup_watermark_unix_seconds) + { + continue; + } + let etag = candidate_object.etag.ok_or_else(|| { + TableCatalogStoreError::Conflict(format!( + "maintenance candidate has no conditional-delete ETag: {object_location}" + )) + })?; + delete_etags.insert(object_location.clone(), etag); + } cleanup_object_candidate_locations.insert(object_location.clone()); } } @@ -4869,7 +6149,15 @@ where let cleanup_object_candidate_locations = cleanup_object_candidate_locations.into_iter().collect::>(); let deleted_object_locations = cleanup_object_candidate_locations.iter().cloned().collect::>(); for object_location in &cleanup_object_candidate_locations { - self.backend.delete_object(table_bucket, object_location).await?; + ensure_table_maintenance_publication_held(publication, entry, "before object deletion")?; + if let Some(fence) = &mutation_fence { + self.backend + .delete_object_if_match_fenced(table_bucket, object_location, &delete_etags[object_location], fence) + .await?; + } else { + self.backend.delete_object(table_bucket, object_location).await?; + } + ensure_table_maintenance_publication_held(publication, entry, "after object deletion")?; } let retained_metadata_locations = protected.into_iter().collect::>(); @@ -4891,7 +6179,7 @@ where Ok(table_maintenance_report_with_recommended_actions(TableMetadataMaintenanceReport { job, - current_metadata_location: entry.metadata_location, + current_metadata_location: entry.metadata_location.clone(), retained_metadata_locations, cleanup_candidate_locations: cleanup_candidate_locations.clone(), deletable_metadata_locations: cleanup_candidate_locations, @@ -5875,7 +7163,7 @@ where )), ); } - if !is_valid_table_metadata_location(&namespace, &table, &request.new_metadata_location) { + if !is_valid_table_metadata_location_for_entry(¤t, &request.new_metadata_location) { return table_commit_result( &request.table_bucket, &request.namespace, @@ -6410,3 +7698,22 @@ where .await } } + +#[cfg(test)] +#[test] +fn catalog_lock_context_preserves_original_network_cause() { + let error = super::catalog_lock_acquisition_error( + "acquire catalog migration lock", + rustfs_lock::LockError::network("peer unavailable", std::io::Error::from(std::io::ErrorKind::TimedOut)), + ); + let error = maintenance_rename_recovery_error(error); + assert!(error.is_unavailable()); + assert!(error.to_string().contains("maintenance rename recovery required")); + let source = std::error::Error::source(&error).expect("retain the typed lock cause"); + assert!(source.is::()); + let io = source + .source() + .and_then(|cause| cause.downcast_ref::()) + .expect("retain the original network cause without cloning it"); + assert_eq!(io.kind(), std::io::ErrorKind::TimedOut); +} diff --git a/rustfs/src/table_catalog/store/strong.rs b/rustfs/src/table_catalog/store/strong.rs index f70f180fe..4412cad9a 100644 --- a/rustfs/src/table_catalog/store/strong.rs +++ b/rustfs/src/table_catalog/store/strong.rs @@ -321,6 +321,127 @@ impl StrongTableCatalogStore where B: TableCatalogObjectBackend, { + async fn table_maintenance_publication(&self) -> TableCatalogStoreResult> { + let publication = TableCommitLockPublication::new(&self.object_backend); + let migration_guard = self.acquire_snapshot_write_permit().await?; + // The publication checks must cover migration lease loss throughout sidecar writes and cleanup. + publication.attach_catalog_migration_read_guard(migration_guard)?; + Ok(publication) + } + + async fn table_maintenance_read_guards( + &self, + table_bucket: &str, + ) -> TableCatalogStoreResult<(TableCatalogLockGuard, TableCatalogLockGuard)> { + // Keep the catalog identity and its sidecars in one publication interval, + // including an A -> B -> A rename that would evade an entry-only recheck. + let migration = self.acquire_snapshot_write_permit().await?; + let publication = self + .object_backend + .acquire_read_lock(table_bucket, &default_table_bucket_publication_lock_path()) + .await?; + let guards = (migration, publication); + Self::ensure_table_maintenance_read_guards(&guards)?; + Ok(guards) + } + + fn ensure_table_maintenance_read_guards( + guards: &(TableCatalogLockGuard, TableCatalogLockGuard), + ) -> TableCatalogStoreResult<()> { + if guards.0.is_lock_lost() || guards.1.is_lock_lost() { + return Err(TableCatalogStoreError::Unavailable( + "table maintenance read lost its migration or publication fence; retry the read".to_string(), + )); + } + Ok(()) + } + + fn require_table_maintenance_publication( + publication: &(dyn TableCommitPublication + Sync), + table_bucket: &str, + namespace: &str, + table: &str, + operation: &str, + ) -> TableCatalogStoreResult<()> { + if !publication.holds_table_bucket(table_bucket) || !publication.holds_table(table_bucket, namespace, table) { + return Err(TableCatalogStoreError::Internal(format!( + "table maintenance {operation} publication fence was lost" + ))); + } + Ok(()) + } + + // Call only with migration -> bucket publication -> local write locks held. + // Reads remain side-effect free: a prepared rename never removes its source sidecars, + // and a published rename already has a complete destination before this cleanup. + pub(super) async fn recover_table_maintenance_rename_unlocked( + &self, + table_bucket: &str, + fence: Option<&TableCatalogObjectMutationFence>, + ) -> TableCatalogStoreResult> { + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let Some((intent, etag)) = sidecar + .read_table_maintenance_rename_intent(table_bucket) + .await + .map_err(object::maintenance_rename_recovery_error)? + else { + return Ok(None); + }; + let fence = fence.ok_or_else(|| { + TableCatalogStoreError::Conflict("maintenance rename recovery requires a storage mutation fence".to_string()) + })?; + fence.ensure_held()?; + let source_key = Self::table_key( + table_bucket, + &parse_namespace_for_store(&intent.source.namespace)?, + &parse_table_for_store(&intent.source.table)?, + ); + let destination_key = Self::table_key( + table_bucket, + &parse_namespace_for_store(&intent.destination.namespace)?, + &parse_table_for_store(&intent.destination.table)?, + ); + let committed = { + let state = self.state.lock().await; + let source = state.tables.get(&source_key); + let destination = state.tables.get(&destination_key); + let owned_tables = state + .tables + .values() + .filter(|entry| entry.table_bucket == table_bucket && entry.table_id == intent.source.table_id) + .count(); + if owned_tables != 1 { + return Err(TableCatalogStoreError::Conflict( + "maintenance rename recovery table identity is not unique; repair persisted state before retrying rename" + .to_string(), + )); + } + let identity_matches = |entry: &TableEntry| { + entry.table_id == intent.source.table_id + && entry.table_uuid == intent.source.table_uuid + && entry.state == TableCatalogEntryState::Active + }; + if source.is_some_and(identity_matches) && destination.is_none() { + false + } else if source.is_none() + && destination.is_some_and(identity_matches) + && state + .snapshot_etag + .as_deref() + .is_some_and(|etag| etag != intent.source_snapshot_etag) + { + true + } else { + return Err(TableCatalogStoreError::Conflict("maintenance rename recovery does not match the catalog publication state; repair persisted state before retrying rename".to_string())); + } + }; + sidecar + .finish_table_maintenance_rename(&intent, &etag, committed, fence) + .await + .map_err(object::maintenance_rename_recovery_error)?; + Ok(Some((intent, committed))) + } + pub fn new(object_backend: B) -> Self { Self::new_with_snapshot_requirement(object_backend, false) } @@ -1881,6 +2002,1022 @@ where Ok(TableCommitResult { table: next, commit_log }) } + async fn maintenance_table_entry( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + self.hydrate_state().await?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let key = Self::table_key(table_bucket, &namespace, &table); + let state = self.state.lock().await; + Self::ensure_identifier_is_unambiguous_locked(&state, &key)?; + state + .tables + .get(&key) + .filter(|entry| entry.state == TableCatalogEntryState::Active) + .cloned() + .ok_or_else(|| { + TableCatalogStoreError::NotFound(format!("table {}/{}/{}", table_bucket, namespace.public_name(), table.as_str())) + }) + } + + fn maintenance_cleanup_roots(entry: &TableEntry) -> TableCatalogStoreResult> { + let metadata_dir = table_metadata_dir_path_for_entry(entry)?; + let table_root = metadata_dir + .strip_suffix(&format!("/{METADATA_DIR}")) + .ok_or_else(|| TableCatalogStoreError::Invalid("maintenance metadata directory has no table root".to_string()))?; + let warehouse = table_warehouse_object_prefix(entry)?; + Ok(vec![ + format!("{metadata_dir}/"), + format!("{table_root}/{DATA_DIR}/"), + format!("{table_root}/{DELETE_DIR}/"), + format!("{warehouse}{METADATA_DIR}/"), + format!("{warehouse}{DATA_DIR}/"), + format!("{warehouse}{DELETE_DIR}/"), + ]) + } + + async fn maintenance_cleanup_entry( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + let roots = Self::maintenance_cleanup_roots(&entry)?; + let state = self.state.lock().await; + for other in state.tables.values().filter(|other| { + other.table_bucket == table_bucket + && other.state != TableCatalogEntryState::Deleted + && (other.namespace != entry.namespace || other.table != entry.table) + }) { + let other_roots = Self::maintenance_cleanup_roots(other)?; + if roots.iter().any(|root| { + other_roots + .iter() + .any(|other_root| warehouse_object_prefixes_overlap(root, other_root)) + }) { + return Err(TableCatalogStoreError::Conflict(format!( + "table maintenance requires exclusive metadata and warehouse roots; shared with {}/{}", + other.namespace, other.table + ))); + } + } + Ok(entry) + } + + pub(crate) async fn get_table_maintenance_config( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + let guards = self.table_maintenance_read_guards(table_bucket).await?; + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + let result = ObjectTableCatalogStore::new(self.object_backend.clone()) + .get_table_maintenance_config_for_entry(&entry) + .await; + Self::ensure_table_maintenance_read_guards(&guards)?; + result + } + + pub(crate) async fn put_table_maintenance_config( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + config: TableMaintenanceConfig, + ) -> TableCatalogStoreResult { + validate_table_maintenance_config(&config)?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "config update", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_table_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + ObjectTableCatalogStore::new(self.object_backend.clone()) + .put_table_maintenance_config_for_entry_with_publication(&entry, config, Some(&publication)) + .await + } + + pub(crate) async fn get_table_metadata_maintenance_report( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + job_id: &str, + ) -> TableCatalogStoreResult> { + let guards = self.table_maintenance_read_guards(table_bucket).await?; + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let result = ObjectTableCatalogStore::new(self.object_backend.clone()) + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + job_id, + ) + .await; + Self::ensure_table_maintenance_read_guards(&guards)?; + result + } + + pub(crate) async fn get_table_maintenance_scheduler_report( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + ) -> TableCatalogStoreResult { + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let (entry, objects) = { + let guards = self.table_maintenance_read_guards(table_bucket).await?; + let entry = self.maintenance_table_entry(table_bucket, namespace, table).await?; + // Fix membership while rename cannot remove the source directory. The + // bounded listing has no per-job reads under the publication lock. + let objects = sidecar + .list_table_metadata_maintenance_audit_objects_for_entry(&entry) + .await?; + Self::ensure_table_maintenance_read_guards(&guards)?; + (entry, objects) + }; + // History is observational, not a transactionally atomic snapshot. Read + // every fixed member outside the bucket lock, but never skip a member + // that disappeared during rename (including an A -> B -> A round trip). + let reports = sidecar + .read_table_metadata_maintenance_audit_reports_for_entry(&entry, &objects) + .await?; + + let guards = self.table_maintenance_read_guards(table_bucket).await?; + if self.maintenance_table_entry(table_bucket, namespace, table).await? != entry { + return Err(TableCatalogStoreError::Unavailable( + "table changed during the maintenance scheduler read; retry the scheduler report".to_string(), + )); + } + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let paths = TableCatalogObjectPaths::default(); + let selected_objects = reports + .iter() + .map(|report| paths.table_maintenance_job_path(table_bucket, &namespace, &table, &entry.table_id, &report.job.job_id)) + .collect::>(); + // Only the fixed-size audit window, effective config and current alias + // are refreshed under the final read fence. + let reports = sidecar + .read_table_metadata_maintenance_audit_reports_for_entry(&entry, &selected_objects) + .await?; + let result = sidecar + .get_table_maintenance_scheduler_report_with_audit_reports_for_entry_at( + &entry, + OffsetDateTime::now_utc(), + Some(reports), + ) + .await; + Self::ensure_table_maintenance_read_guards(&guards)?; + result + } + + // A dry-run may publish its audit record, but must not replace pending work + // or hide an incomplete sidecar publication. + async fn maintenance_planning_predecessor( + sidecar: &ObjectTableCatalogStore, + entry: &TableEntry, + ) -> TableCatalogStoreResult> { + let namespace = parse_namespace_for_store(&entry.namespace)?; + let table = parse_table_for_store(&entry.table)?; + let current = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + let latest = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_LATEST, + ) + .await?; + if current != latest { + return Err(TableCatalogStoreError::Conflict( + "maintenance report publication requires recovery before planning".to_string(), + )); + } + if let Some(report) = ¤t { + let persisted = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + &entry.table_bucket, + &namespace, + &table, + &entry.table_id, + &report.job.job_id, + ) + .await?; + if persisted.as_ref() != Some(report) { + return Err(TableCatalogStoreError::Conflict( + "maintenance job publication requires recovery before planning".to_string(), + )); + } + if matches!( + report.job.status, + TableMetadataMaintenanceJobStatus::Queued + | TableMetadataMaintenanceJobStatus::Running + | TableMetadataMaintenanceJobStatus::Failed + ) || report.job.quarantined_object_count > 0 + { + return Err(TableCatalogStoreError::Conflict( + "pending maintenance work or quarantine must be resolved before planning".to_string(), + )); + } + } + Ok(current) + } + + pub(crate) async fn run_table_metadata_maintenance_planning( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + retain_recent_metadata_files: usize, + snapshot_expiration: Option, + compaction: Option, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "planning", + )?; + let _completion = TableCommitPublicationCompletion::new(&publication); + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let planned_effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + let planned_job = Self::maintenance_planning_predecessor(&sidecar, &entry).await?; + let mut effective = planned_effective.clone(); + if retain_recent_metadata_files > 0 { + effective.config.retain_recent_metadata_files = retain_recent_metadata_files; + } + + // Keep the migration permit while reading potentially large manifests; + // unrelated catalog commits need not wait for the object traversal. + publication.release_publication_locks(); + let mut report = sidecar + .plan_table_metadata_maintenance_for_entry(&entry, effective.config.retain_recent_metadata_files) + .await?; + report.snapshot_expiration = match snapshot_expiration { + Some(config) => Some( + sidecar + .plan_table_snapshot_expiration_for_entry_with_backend(&self.object_backend, &entry, config) + .await?, + ), + None => None, + }; + report.compaction = match compaction { + Some(config) => Some(sidecar.plan_table_compaction_for_entry(&entry, config).await?), + None => None, + }; + + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "planning revalidation", + )?; + if self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await? + != entry + { + return Err(TableCatalogStoreError::Conflict("table changed during maintenance planning".to_string())); + } + let current_effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if current_effective != planned_effective { + return Err(TableCatalogStoreError::Conflict("maintenance config changed during planning".to_string())); + } + if Self::maintenance_planning_predecessor(&sidecar, &entry).await? != planned_job { + return Err(TableCatalogStoreError::Conflict( + "current maintenance job changed during planning".to_string(), + )); + } + + let now = OffsetDateTime::now_utc(); + let timestamp = maintenance_timestamp(now); + let before_status = Some(report.job.status.clone()); + report.job.operation = TableMetadataMaintenanceOperation::DryRun; + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.failure_reason = None; + report.job.config_source = effective.source; + report.job.worker_id = Some("rustfs-admin".to_string()); + report.job.lease_id = Uuid::new_v4().to_string(); + report.job.attempt = 1; + report.job.max_retry_attempts = effective.config.max_retry_attempts; + report.job.next_retry_after = None; + report.job.quarantine_enabled = effective.config.quarantine_enabled; + report.job.quarantine_retention_seconds = effective.config.quarantine_retention_seconds; + report.job.heartbeat_at = Some(timestamp.clone()); + report.job.started_at = Some(timestamp); + report.job.finished_at = None; + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerStarted, + None, + before_status, + Some(0), + ); + // Planning has no object side effects, so publish only the completed + // report. There is no Running lease to strand if the request is lost. + sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, false, &effective, report, &publication) + .await + } + + pub(crate) async fn run_table_maintenance_scheduler_once( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + scheduler_id: String, + ) -> TableCatalogStoreResult { + self.run_table_maintenance_scheduler_once_with_guard(table_bucket, namespace, table, scheduler_id, None) + .await + } + + pub(super) async fn run_table_maintenance_scheduler_once_with_guard( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + scheduler_id: String, + periodic_guard: Option<&PeriodicTableMaintenanceQueueGuard<'_>>, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "scheduler", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if let Some(periodic_guard) = periodic_guard { + sidecar + .validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } + let preflight = sidecar + .table_metadata_maintenance_scheduler_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &scheduler_id, + now, + effective, + Some(&publication), + ) + .await?; + let effective = match preflight { + TableMaintenanceSchedulerPreflight::Ready(effective) => effective, + TableMaintenanceSchedulerPreflight::Complete(report) => { + publication.release_publication_locks(); + let scheduler = sidecar + .get_table_maintenance_scheduler_report_for_entry_at(&entry, now) + .await?; + return Ok(TableMaintenanceSchedulerRunResult { + report: *report, + scheduler, + }); + } + }; + + // Planning only reads objects. Keep the migration permit, not the bucket-wide writer. + let planned_effective = effective; + publication.release_publication_locks(); + let mut report = sidecar + .plan_table_metadata_maintenance_for_entry(&entry, planned_effective.config.retain_recent_metadata_files) + .await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "scheduler revalidation", + )?; + let current = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + if current != entry { + return Err(TableCatalogStoreError::Conflict("table changed during maintenance planning".to_string())); + } + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if effective.config != planned_effective.config || effective.source != planned_effective.source { + return Err(TableCatalogStoreError::Conflict("maintenance config changed during planning".to_string())); + } + if let Some(periodic_guard) = periodic_guard { + sidecar + .validate_periodic_table_maintenance_queue_guard( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &effective, + periodic_guard, + ) + .await?; + } + match sidecar + .table_metadata_maintenance_scheduler_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &scheduler_id, + now, + effective, + Some(&publication), + ) + .await? + { + TableMaintenanceSchedulerPreflight::Ready(effective) => { + if report.job.retain_recent_metadata_files != effective.config.retain_recent_metadata_files { + return Err(TableCatalogStoreError::Conflict( + "maintenance config changed before scheduler claim".to_string(), + )); + } + if entry.metadata_location != report.current_metadata_location { + return Err(TableCatalogStoreError::Conflict( + "current metadata location changed before maintenance scheduler claim".to_string(), + )); + } + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + let scheduled_at = maintenance_timestamp(now); + report.job.operation = if effective.config.delete_enabled { + TableMetadataMaintenanceOperation::Delete + } else { + TableMetadataMaintenanceOperation::DryRun + }; + report.job.status = TableMetadataMaintenanceJobStatus::Queued; + report.job.failure_reason = None; + report.job.config_source = effective.source; + report.job.scheduler_id = Some(scheduler_id); + report.job.scheduler_lease_id = Uuid::new_v4().to_string(); + report.job.scheduled_at = Some(scheduled_at); + report.job.worker_id = None; + report.job.lease_id = String::new(); + report.job.attempt = 0; + report.job.max_retry_attempts = effective.config.max_retry_attempts; + report.job.next_retry_after = None; + report.job.quarantine_enabled = effective.config.quarantine_enabled; + report.job.quarantine_retention_seconds = effective.config.quarantine_retention_seconds; + report.job.heartbeat_at = None; + report.job.started_at = None; + report.job.finished_at = None; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Scheduler, + TableMaintenanceAuditAction::SchedulerQueued, + None, + before_status, + before_quarantined_object_count, + ); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "scheduler report", + )?; + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await?; + } + TableMaintenanceSchedulerPreflight::Complete(completed) => report = *completed, + } + + publication.release_publication_locks(); + let scheduler = sidecar + .get_table_maintenance_scheduler_report_for_entry_at(&entry, now) + .await?; + Ok(TableMaintenanceSchedulerRunResult { report, scheduler }) + } + + pub(crate) async fn run_table_metadata_maintenance_worker_once_conditionally( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + ) -> TableCatalogStoreResult { + self.run_table_metadata_maintenance_worker_once_conditionally_with_clock( + table_bucket, + namespace, + table, + worker_id, + condition, + OffsetDateTime::now_utc, + ) + .await + } + + pub(in crate::table_catalog) async fn run_table_metadata_maintenance_worker_once_conditionally_with_clock( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + condition: &TableMaintenanceWorkerCondition, + clock: impl FnOnce() -> OffsetDateTime + Send, + ) -> TableCatalogStoreResult { + validate_table_maintenance_worker_condition(&worker_id, condition)?; + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "conditional worker", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + ObjectTableCatalogStore::new(self.object_backend.clone()) + .run_conditional_table_maintenance_worker_for_entry(&entry, worker_id, condition, clock, &publication) + .await + } + + pub(crate) async fn run_table_metadata_maintenance_worker_once( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + worker_id: String, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let entry = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + let (effective, queued) = match sidecar + .table_metadata_maintenance_worker_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &worker_id, + now, + effective, + Some(&publication), + ) + .await? + { + TableMaintenanceWorkerPreflight::Ready { effective, queued } => (effective, queued), + TableMaintenanceWorkerPreflight::Complete(report) => return Ok(*report), + }; + + let mut report = if let Some(queued) = queued { + *queued + } else { + let planned_job = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + // Only fresh planning is read-only; queued execution and deletion keep publication held. + publication.release_publication_locks(); + let report = sidecar + .plan_table_metadata_maintenance_for_entry(&entry, effective.config.retain_recent_metadata_files) + .await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker revalidation", + )?; + let current = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + if current != entry { + return Err(TableCatalogStoreError::Conflict( + "table changed during maintenance worker planning".to_string(), + )); + } + let current_effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + if current_effective != effective { + return Err(TableCatalogStoreError::Conflict( + "maintenance config changed during worker planning".to_string(), + )); + } + let current_job = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + table_bucket, + &namespace, + &table, + &entry.table_id, + MAINTENANCE_JOB_ALIAS_CURRENT, + ) + .await?; + if current_job != planned_job { + return Err(TableCatalogStoreError::Conflict( + "current maintenance job changed during planning".to_string(), + )); + } + report + }; + let now = OffsetDateTime::now_utc(); + let effective = sidecar + .get_effective_table_maintenance_config_for_entry_unlocked(table_bucket, &namespace, &table, &entry) + .await?; + let (effective, queued) = match sidecar + .table_metadata_maintenance_worker_preflight_with_publication( + TableMaintenancePreflightContext { + table_bucket, + namespace: &namespace, + table: &table, + entry: &entry, + }, + &worker_id, + now, + effective, + Some(&publication), + ) + .await? + { + TableMaintenanceWorkerPreflight::Ready { effective, queued } => (effective, queued), + TableMaintenanceWorkerPreflight::Complete(report) => return Ok(*report), + }; + if let Some(queued) = queued { + if queued.job.job_id != report.job.job_id { + return Err(TableCatalogStoreError::Conflict( + "queued maintenance job changed before worker claim".to_string(), + )); + } + report = *queued; + } + if report.job.retain_recent_metadata_files != effective.config.retain_recent_metadata_files { + return Err(TableCatalogStoreError::Conflict( + "maintenance config changed before worker claim".to_string(), + )); + } + if entry.metadata_location != report.current_metadata_location || entry.generation != report.job.current_generation { + return Err(TableCatalogStoreError::Conflict( + "current metadata location changed before maintenance worker claim".to_string(), + )); + } + + let current = self + .maintenance_cleanup_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + if current != entry { + return Err(TableCatalogStoreError::Conflict( + "table changed before maintenance worker claim".to_string(), + )); + } + let was_queued_claim = matches!(report.job.status, TableMetadataMaintenanceJobStatus::Queued); + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + let started_at = maintenance_timestamp(now); + if !was_queued_claim { + report.job.operation = if effective.config.delete_enabled { + TableMetadataMaintenanceOperation::Delete + } else { + TableMetadataMaintenanceOperation::DryRun + }; + } + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.failure_reason = None; + report.job.config_source = effective.source; + report.job.worker_id = Some(worker_id); + report.job.lease_id = Uuid::new_v4().to_string(); + report.job.attempt = 1; + report.job.max_retry_attempts = effective.config.max_retry_attempts; + report.job.next_retry_after = None; + report.job.quarantine_enabled = effective.config.quarantine_enabled; + report.job.quarantine_retention_seconds = effective.config.quarantine_retention_seconds; + report.job.heartbeat_at = Some(started_at.clone()); + report.job.started_at = Some(started_at); + report.job.finished_at = None; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + now, + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerStarted, + None, + before_status, + before_quarantined_object_count, + ); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker report", + )?; + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await?; + + let delete = matches!(report.job.operation, TableMetadataMaintenanceOperation::Delete); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "worker finish", + )?; + sidecar + .finish_table_metadata_maintenance_run_for_entry_with_publication(&entry, delete, &effective, report, &publication) + .await + } + + pub(crate) async fn heartbeat_table_metadata_maintenance_job( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + job_id: &str, + lease_id: &str, + worker_id: &str, + ) -> TableCatalogStoreResult { + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "heartbeat", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_table_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + sidecar + .heartbeat_table_metadata_maintenance_job_for_entry_with_publication( + &entry, + TableMaintenanceHeartbeatRef { + table_bucket, + namespace: &namespace.public_name(), + table: table.as_str(), + job_id, + lease_id, + worker_id, + }, + OffsetDateTime::now_utc(), + Some(&publication), + ) + .await + } + + pub(crate) async fn apply_table_maintenance_quarantine_operation( + &self, + table_bucket: &str, + namespace: &str, + table: &str, + job_id: &str, + request: TableMaintenanceQuarantineOperationRequest, + ) -> TableCatalogStoreResult { + let action = request.action.clone(); + let namespace = parse_namespace_for_store(namespace)?; + let table = parse_table_for_store(table)?; + let publication = self.table_maintenance_publication().await?; + publication.begin_table_bucket(table_bucket).await?; + publication + .prepare(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "quarantine", + )?; + let _publication_completion = TableCommitPublicationCompletion::new(&publication); + let entry = self + .maintenance_table_entry(table_bucket, &namespace.public_name(), table.as_str()) + .await?; + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let report_lookup = if matches!(action, TableMaintenanceQuarantineAction::Inspect) { + job_id + } else { + MAINTENANCE_JOB_ALIAS_CURRENT + }; + let mut report = sidecar + .get_table_metadata_maintenance_report_for_entry_unlocked( + table_bucket, + &namespace, + &table, + &entry.table_id, + report_lookup, + ) + .await? + .ok_or_else(|| { + TableCatalogStoreError::NotFound(format!( + "maintenance job {}/{}/{}/{}", + table_bucket, + namespace.public_name(), + table.as_str(), + job_id + )) + })?; + + if !matches!(action, TableMaintenanceQuarantineAction::Inspect) { + if report.job.job_id != job_id { + return Err(TableCatalogStoreError::Conflict("maintenance job is not current".to_string())); + } + if !matches!(report.job.status, TableMetadataMaintenanceJobStatus::Failed) { + return Err(TableCatalogStoreError::Conflict( + "maintenance quarantine operation requires a failed job".to_string(), + )); + } + if !report.job.quarantine_enabled || report.job.quarantined_object_count == 0 { + return Err(TableCatalogStoreError::Conflict( + "maintenance job has no active quarantine boundary".to_string(), + )); + } + let before_status = Some(report.job.status.clone()); + let before_quarantined_object_count = Some(report.job.quarantined_object_count); + report.job.quarantined_object_count = 0; + let audit_action = match &action { + TableMaintenanceQuarantineAction::Inspect => unreachable!(), + TableMaintenanceQuarantineAction::Release => { + report.job.failure_reason = + Some(table_maintenance_quarantine_operator_reason("released", request.reason.as_deref())); + TableMaintenanceAuditAction::QuarantineRelease + } + TableMaintenanceQuarantineAction::Retry => { + report.job.next_retry_after = None; + report.job.failure_reason = Some(table_maintenance_quarantine_operator_reason( + "released for retry", + request.reason.as_deref(), + )); + TableMaintenanceAuditAction::QuarantineRetry + } + TableMaintenanceQuarantineAction::Abandon => { + report.job.next_retry_after = None; + report.job.failure_reason = + Some(table_maintenance_quarantine_operator_reason("abandoned", request.reason.as_deref())); + TableMaintenanceAuditAction::QuarantineAbandon + } + }; + refresh_table_maintenance_report_recommended_actions(&mut report); + push_table_maintenance_audit_event( + &mut report, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Operator, + audit_action, + request.reason, + before_status, + before_quarantined_object_count, + ); + Self::require_table_maintenance_publication( + &publication, + table_bucket, + &namespace.public_name(), + table.as_str(), + "quarantine update", + )?; + sidecar + .put_table_metadata_maintenance_report_for_entry_with_publication(&report, &entry, Some(&publication)) + .await?; + } + + let scheduler = sidecar + .get_table_maintenance_scheduler_report_for_entry_at(&entry, OffsetDateTime::now_utc()) + .await?; + Ok(TableMaintenanceQuarantineOperationResult { + action, + report, + scheduler, + }) + } + pub(crate) async fn plan_table_commit_recovery( &self, table_bucket: &str, @@ -2439,6 +3576,14 @@ where self.hydrate_state().await?; let key = Self::table_key(&entry.table_bucket, &namespace, &table); let publication_identity = (entry.table_bucket.clone(), entry.namespace.clone(), entry.table.clone()); + let mut recovery_fence = publication.object_mutation_fence(); + if let Some(fence) = recovery_fence.as_mut() { + fence + .signals + .extend(_migration_guard.as_ref().and_then(TableCatalogLockGuard::lock_lost_signal)); + } + self.recover_table_maintenance_rename_unlocked(&entry.table_bucket, recovery_fence.as_ref()) + .await?; let (snapshot, precondition, postcondition) = { let state = self.state.lock().await; Self::require_table_bucket_in_state(&state, &entry.table_bucket)?; @@ -2573,9 +3718,9 @@ where destination_namespace: &str, destination_table: &str, ) -> TableCatalogStoreResult<()> { - let publication = TableCommitLockPublication::new(&self.object_backend); - let _migration_guard = self.acquire_snapshot_write_permit().await?; + let publication = self.table_maintenance_publication().await?; publication.begin_table_bucket(table_bucket).await?; + publication.prepare(table_bucket, source_namespace, source_table).await?; if !publication.holds_table_bucket(table_bucket) { return Err(TableCatalogStoreError::Internal( "table rename requires a table-bucket publication fence".to_string(), @@ -2591,8 +3736,21 @@ where let destination_table = parse_table_for_store(destination_table)?; let source_key = Self::table_key(table_bucket, &source_namespace, &source_table); let destination_key = Self::table_key(table_bucket, &destination_namespace, &destination_table); + let fence = publication + .object_mutation_fence() + .ok_or_else(|| TableCatalogStoreError::Internal("table rename requires a storage mutation fence".to_string()))?; + if let Some((recovered, true)) = self + .recover_table_maintenance_rename_unlocked(table_bucket, Some(&fence)) + .await? + && recovered.source.namespace == source_namespace.public_name() + && recovered.source.table == source_table.as_str() + && recovered.destination.namespace == destination_namespace.public_name() + && recovered.destination.table == destination_table.as_str() + { + return Ok(()); + } - let (snapshot, precondition, postcondition) = { + let (snapshot, precondition, postcondition, source, destination) = { let state = self.state.lock().await; Self::require_table_bucket_in_state(&state, table_bucket)?; Self::ensure_identifier_is_unambiguous_locked(&state, &source_key)?; @@ -2637,7 +3795,7 @@ where let (precondition, mut draft_state) = Self::snapshot_draft_context_locked(&state); draft_state.tables.remove(&source_key); - let mut destination = source; + let mut destination = source.clone(); destination.namespace = destination_namespace.public_name(); destination.table = destination_table.as_str().to_string(); draft_state.tables.insert(destination_key.clone(), destination.clone()); @@ -2647,11 +3805,31 @@ where StrongSnapshotWritePostcondition::TableRenamed { source_key, destination_key, - table_id: destination.table_id, + table_id: destination.table_id.clone(), }, + source, + destination, ) }; - self.finalize_snapshot_write(snapshot, precondition, postcondition).await + let TableCatalogPutPrecondition::IfMatch(source_snapshot_etag) = &precondition else { + return Err(TableCatalogStoreError::Internal( + "table rename requires an existing snapshot watermark".to_string(), + )); + }; + let sidecar = ObjectTableCatalogStore::new(self.object_backend.clone()); + let pending = sidecar + .prepare_table_maintenance_rename(&source, &destination, source_snapshot_etag, &fence) + .await + .map_err(object::maintenance_rename_recovery_error)?; + self.finalize_snapshot_write_with_fence(snapshot, precondition, postcondition, Some(&fence)) + .await?; + if let Some((intent, etag)) = pending { + sidecar + .finish_table_maintenance_rename(&intent, &etag, true, &fence) + .await + .map_err(object::maintenance_rename_recovery_error)?; + } + Ok(()) } async fn resolve_table_data_plane_resource( @@ -2913,8 +4091,7 @@ where } Err(err) => Err(err), }; - let cas_result = result.as_ref().map(|_| ()).map_err(Clone::clone); - record_table_commit_cas_result(&request.operation, cas_started, &cas_result); + record_table_commit_cas_result(&request.operation, cas_started, &result); table_commit_result( &request.table_bucket, &request.namespace, @@ -2927,8 +4104,7 @@ where } async fn drop_table(&self, table_bucket: &str, namespace: &str, table: &str) -> TableCatalogStoreResult<()> { - let publication = TableCommitLockPublication::new(&self.object_backend); - let _migration_guard = self.acquire_snapshot_write_permit().await?; + let publication = self.table_maintenance_publication().await?; publication.begin_table_bucket(table_bucket).await?; publication.prepare(table_bucket, namespace, table).await?; if !publication.holds_table_bucket(table_bucket) || !publication.holds_table(table_bucket, namespace, table) { @@ -2939,6 +4115,9 @@ where let _publication_completion = TableCommitPublicationCompletion::new(&publication); let _write_guard = self.write_lock.lock().await; self.hydrate_state().await?; + let recovery_fence = publication.object_mutation_fence(); + self.recover_table_maintenance_rename_unlocked(table_bucket, recovery_fence.as_ref()) + .await?; let namespace = parse_namespace_for_store(namespace)?; let table = parse_table_for_store(table)?; let key = Self::table_key(table_bucket, &namespace, &table); diff --git a/rustfs/src/table_catalog/test_support.rs b/rustfs/src/table_catalog/test_support.rs index 3c0878b1c..5972c60df 100644 --- a/rustfs/src/table_catalog/test_support.rs +++ b/rustfs/src/table_catalog/test_support.rs @@ -22,12 +22,54 @@ //! fixed values (sequence 7 / snapshot 20), which keeps every produced byte //! identical to the pre-extraction fixtures. +use rustfs_lock::{LockClient, LockId, LockInfo, LockRequest, LockResponse, LockStats}; use std::collections::{BTreeMap, BTreeSet}; -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use time::OffsetDateTime; use super::*; +#[derive(Debug)] +pub(crate) struct RecordingLeaseClient { + pub(crate) inner: rustfs_lock::LocalClient, + pub(crate) acquired: Mutex>, +} + +#[async_trait::async_trait] +impl LockClient for RecordingLeaseClient { + async fn acquire_lock(&self, request: &LockRequest) -> rustfs_lock::Result { + let response = self.inner.acquire_lock(request).await?; + if let Some(info) = response.lock_info.as_ref() { + *self.acquired.lock().expect("record lease") = Some(info.id.clone()); + } + Ok(response) + } + async fn release(&self, id: &LockId) -> rustfs_lock::Result { + self.inner.release(id).await + } + async fn refresh(&self, id: &LockId) -> rustfs_lock::Result { + self.inner.refresh(id).await + } + async fn force_release(&self, id: &LockId) -> rustfs_lock::Result { + self.inner.force_release(id).await + } + async fn check_status(&self, id: &LockId) -> rustfs_lock::Result> { + self.inner.check_status(id).await + } + async fn get_stats(&self) -> rustfs_lock::Result { + self.inner.get_stats().await + } + async fn close(&self) -> rustfs_lock::Result<()> { + self.inner.close().await + } + async fn is_online(&self) -> bool { + self.inner.is_online().await + } + async fn is_local(&self) -> bool { + self.inner.is_local().await + } +} + pub(crate) fn table_metadata_json(table_uuid: &str, location: &str) -> serde_json::Value { serde_json::json!({ "format-version": 2, @@ -495,6 +537,7 @@ pub(crate) struct TestCatalogObjectState { pub(crate) pause_put_attempts: BTreeMap<(String, String), BTreeMap>, pub(crate) fail_delete_attempts: BTreeMap<(String, String), BTreeSet>, pub(crate) fail_after_delete_attempts: BTreeMap<(String, String), BTreeSet>, + pub(crate) pause_delete_attempts: BTreeMap<(String, String), BTreeMap>, pub(crate) put_attempts: BTreeMap<(String, String), usize>, pub(crate) delete_attempts: BTreeMap<(String, String), usize>, pub(crate) write_lock_acquisitions: BTreeMap<(String, String), usize>, @@ -689,6 +732,27 @@ impl TestCatalogObjectBackend { pause } + pub(crate) async fn pause_next_delete(&self, bucket: &str, object: &str) -> TestCatalogObjectPause { + let mut state = self.state.lock().await; + let key = (bucket.to_string(), object.to_string()); + let next_attempt = state.delete_attempts.get(&key).copied().unwrap_or_default() + 1; + Self::pause_delete_attempt_unlocked(&mut state, key, next_attempt) + } + + fn pause_delete_attempt_unlocked( + state: &mut TestCatalogObjectState, + key: (String, String), + attempt: usize, + ) -> TestCatalogObjectPause { + let pause = TestCatalogObjectPause::default(); + state + .pause_delete_attempts + .entry(key) + .or_default() + .insert(attempt, pause.clone()); + pause + } + pub(crate) async fn put_attempt_count(&self, bucket: &str, object: &str) -> usize { self.state .lock() @@ -700,6 +764,188 @@ impl TestCatalogObjectBackend { } } +impl TestCatalogObjectBackend { + async fn put_object_with_fence( + &self, + bucket: &str, + object: &str, + data: Vec, + precondition: TableCatalogPutPrecondition, + fence: Option<&TableCatalogObjectMutationFence>, + ) -> TableCatalogStoreResult<()> { + if let Some(fence) = fence { + fence.ensure_held()?; + } + let mut fail_put_object_path = self.fail_put_object_path.lock().await; + if fail_put_object_path.as_deref() == Some(object) { + fail_put_object_path.take(); + return Err(TableCatalogStoreError::Internal("injected metadata write failure".to_string())); + } + drop(fail_put_object_path); + + let mut corrupt_put_object_path = self.corrupt_put_object_path.lock().await; + let data = if corrupt_put_object_path.as_deref() == Some(object) { + corrupt_put_object_path.take(); + b"{}".to_vec() + } else { + data + }; + drop(corrupt_put_object_path); + + let key = (bucket.to_string(), object.to_string()); + let (attempt, pause) = { + let mut state = self.state.lock().await; + let attempt = { + let attempts = state.put_attempts.entry(key.clone()).or_default(); + *attempts += 1; + *attempts + }; + if state + .fail_put_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + return Err(TableCatalogStoreError::Internal(format!( + "injected put failure for {object} attempt {attempt}" + ))); + } + let pause = state + .pause_put_attempts + .get_mut(&key) + .and_then(|attempts| attempts.remove(&attempt)); + (attempt, pause) + }; + if let Some(pause) = pause { + pause.started.notify_one(); + pause.release.notified().await; + } + + let result = { + let mut state = self.state.lock().await; + let precondition_failure = match &precondition { + TableCatalogPutPrecondition::IfAbsent if state.objects.contains_key(&key) => { + Some(TableCatalogStoreError::Conflict(format!("object already exists: {object}"))) + } + TableCatalogPutPrecondition::IfMatch(expected) => match state.objects.get(&key) { + None => Some(TableCatalogStoreError::Conflict(format!("object is missing: {object}"))), + Some(current) if ¤t.etag != expected => { + Some(TableCatalogStoreError::Conflict(format!("object changed: {object}"))) + } + Some(_) => None, + }, + _ => None, + }; + if let Some(err) = precondition_failure { + Err(err) + } else { + if let Some(fence) = fence { + fence.ensure_held()?; + } + let etag = if self.content_addressed_etags { + content_etag(&data) + } else { + state.next_etag() + }; + state.objects.insert( + key.clone(), + TestCatalogObjectRecord { + data, + etag, + mod_time: Some(OffsetDateTime::now_utc()), + }, + ); + if state + .fail_after_put_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + Err(TableCatalogStoreError::Internal(format!( + "injected post-commit put failure for {object} attempt {attempt}" + ))) + } else { + Ok(()) + } + } + }; + if let Some(barrier) = &self.put_object_barrier { + barrier.wait().await; + } + result?; + if let Some(fence) = fence { + fence.ensure_held()?; + } + Ok(()) + } + + async fn delete_object_if_match_with_fence( + &self, + bucket: &str, + object: &str, + expected_etag: &str, + fence: Option<&TableCatalogObjectMutationFence>, + ) -> TableCatalogStoreResult<()> { + if let Some(fence) = fence { + fence.ensure_held()?; + } + let mut state = self.state.lock().await; + let key = (bucket.to_string(), object.to_string()); + let attempt = { + let attempts = state.delete_attempts.entry(key.clone()).or_default(); + *attempts += 1; + *attempts + }; + if state + .fail_delete_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + return Err(TableCatalogStoreError::Internal(format!( + "injected delete failure for {object} attempt {attempt}" + ))); + } + let pause = state + .pause_delete_attempts + .get_mut(&key) + .and_then(|attempts| attempts.remove(&attempt)); + if let Some(pause) = pause { + drop(state); + pause.started.notify_one(); + pause.release.notified().await; + state = self.state.lock().await; + } + match state.objects.get(&key) { + None => { + return Err(TableCatalogStoreError::Conflict(format!( + "conditionally delete catalog object: {bucket}/{object} disappeared before cleanup" + ))); + } + Some(record) if record.etag != expected_etag => { + return Err(TableCatalogStoreError::Conflict(format!( + "conditionally delete catalog object: {bucket}/{object} changed before cleanup" + ))); + } + Some(_) => {} + } + if let Some(fence) = fence { + fence.ensure_held()?; + } + state.objects.remove(&key); + if state + .fail_after_delete_attempts + .get(&key) + .is_some_and(|attempts| attempts.contains(&attempt)) + { + return Err(TableCatalogStoreError::Internal(format!( + "injected post-commit delete failure for {object} attempt {attempt}" + ))); + } + if let Some(fence) = fence { + fence.ensure_held()?; + } + Ok(()) + } +} + impl TestCatalogObjectState { pub(crate) fn next_etag(&mut self) -> String { self.next_etag += 1; @@ -835,98 +1081,19 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend { data: Vec, precondition: TableCatalogPutPrecondition, ) -> TableCatalogStoreResult<()> { - let mut fail_put_object_path = self.fail_put_object_path.lock().await; - if fail_put_object_path.as_deref() == Some(object) { - fail_put_object_path.take(); - return Err(TableCatalogStoreError::Internal("injected metadata write failure".to_string())); - } - drop(fail_put_object_path); + self.put_object_with_fence(bucket, object, data, precondition, None).await + } - let mut corrupt_put_object_path = self.corrupt_put_object_path.lock().await; - let data = if corrupt_put_object_path.as_deref() == Some(object) { - corrupt_put_object_path.take(); - b"{}".to_vec() - } else { - data - }; - drop(corrupt_put_object_path); - - let key = (bucket.to_string(), object.to_string()); - let (attempt, pause) = { - let mut state = self.state.lock().await; - let attempt = { - let attempts = state.put_attempts.entry(key.clone()).or_default(); - *attempts += 1; - *attempts - }; - if state - .fail_put_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - return Err(TableCatalogStoreError::Internal(format!( - "injected put failure for {object} attempt {attempt}" - ))); - } - let pause = state - .pause_put_attempts - .get_mut(&key) - .and_then(|attempts| attempts.remove(&attempt)); - (attempt, pause) - }; - if let Some(pause) = pause { - pause.started.notify_one(); - pause.release.notified().await; - } - - let result = { - let mut state = self.state.lock().await; - let precondition_failure = match &precondition { - TableCatalogPutPrecondition::IfAbsent if state.objects.contains_key(&key) => { - Some(TableCatalogStoreError::Conflict(format!("object already exists: {object}"))) - } - TableCatalogPutPrecondition::IfMatch(expected) => match state.objects.get(&key) { - None => Some(TableCatalogStoreError::Conflict(format!("object is missing: {object}"))), - Some(current) if ¤t.etag != expected => { - Some(TableCatalogStoreError::Conflict(format!("object changed: {object}"))) - } - Some(_) => None, - }, - _ => None, - }; - if let Some(err) = precondition_failure { - Err(err) - } else { - let etag = if self.content_addressed_etags { - content_etag(&data) - } else { - state.next_etag() - }; - state.objects.insert( - key.clone(), - TestCatalogObjectRecord { - data, - etag, - mod_time: Some(OffsetDateTime::now_utc()), - }, - ); - if state - .fail_after_put_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - Err(TableCatalogStoreError::Internal(format!( - "injected post-commit put failure for {object} attempt {attempt}" - ))) - } else { - Ok(()) - } - } - }; - if let Some(barrier) = &self.put_object_barrier { - barrier.wait().await; - } - result + async fn put_object_fenced( + &self, + bucket: &str, + object: &str, + data: Vec, + precondition: TableCatalogPutPrecondition, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult<()> { + self.put_object_with_fence(bucket, object, data, precondition, Some(fence)) + .await } async fn delete_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<()> { @@ -960,46 +1127,19 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend { } async fn delete_object_if_match(&self, bucket: &str, object: &str, expected_etag: &str) -> TableCatalogStoreResult<()> { - let mut state = self.state.lock().await; - let key = (bucket.to_string(), object.to_string()); - let attempt = { - let attempts = state.delete_attempts.entry(key.clone()).or_default(); - *attempts += 1; - *attempts - }; - if state - .fail_delete_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - return Err(TableCatalogStoreError::Internal(format!( - "injected delete failure for {object} attempt {attempt}" - ))); - } - match state.objects.get(&key) { - None => { - return Err(TableCatalogStoreError::Conflict(format!( - "conditionally delete catalog object: {bucket}/{object} disappeared before cleanup" - ))); - } - Some(record) if record.etag != expected_etag => { - return Err(TableCatalogStoreError::Conflict(format!( - "conditionally delete catalog object: {bucket}/{object} changed before cleanup" - ))); - } - Some(_) => {} - } - state.objects.remove(&key); - if state - .fail_after_delete_attempts - .get(&key) - .is_some_and(|attempts| attempts.contains(&attempt)) - { - return Err(TableCatalogStoreError::Internal(format!( - "injected post-commit delete failure for {object} attempt {attempt}" - ))); - } - Ok(()) + self.delete_object_if_match_with_fence(bucket, object, expected_etag, None) + .await + } + + async fn delete_object_if_match_fenced( + &self, + bucket: &str, + object: &str, + expected_etag: &str, + fence: &TableCatalogObjectMutationFence, + ) -> TableCatalogStoreResult<()> { + self.delete_object_if_match_with_fence(bucket, object, expected_etag, Some(fence)) + .await } async fn list_objects(&self, bucket: &str, prefix: &str) -> TableCatalogStoreResult> { diff --git a/rustfs/src/table_catalog/tests.rs b/rustfs/src/table_catalog/tests.rs index 2f61241c3..dbc8b4a0f 100644 --- a/rustfs/src/table_catalog/tests.rs +++ b/rustfs/src/table_catalog/tests.rs @@ -18,6 +18,9 @@ use datafusion::{ use std::assert_matches; use std::sync::Arc; +mod strong_maintenance_planning; +mod strong_maintenance_planning_paths; + const TABLE_CATALOG_TEST_TIMEOUT: StdDuration = StdDuration::from_secs(30); #[test] @@ -71,7 +74,7 @@ fn catalog_lock_authority_failures_are_typed_as_unavailable() { ] { assert_matches!( super::store::catalog_lock_acquisition_error("acquire catalog table lock", error), - TableCatalogStoreError::Unavailable(_) + TableCatalogStoreError::LockUnavailable { .. } ); } @@ -131,7 +134,7 @@ fn catalog_storage_retryable_lock_failures_are_typed_as_unavailable() { ] { assert_matches!( storage_error_to_catalog("stat catalog object", StorageError::Lock(error)), - TableCatalogStoreError::Unavailable(_) + TableCatalogStoreError::LockUnavailable { .. } ); } @@ -734,6 +737,79 @@ async fn strong_table_registration_and_drop_acquire_migration_read_lock_before_p drop_task.await.expect("drop task should join").expect("drop should succeed"); } +#[tokio::test] +async fn strong_table_maintenance_acquires_migration_read_lock_before_publication() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").expect("namespace should parse"); + let table = IdentifierSegment::parse("orders").expect("table should parse"); + let metadata_location = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry(bucket, &namespace, &table, metadata_location)) + .await + .unwrap(); + + let publication_lock = default_table_bucket_publication_lock_path(); + let migration_lock = TableCatalogObjectPaths::default().backing_migration_global_fence_lock_path(); + let publication_guard = backend + .acquire_write_lock(bucket, &publication_lock) + .await + .expect("publication lock should be acquired"); + let publication_attempts = backend.write_lock_acquisition_count(bucket, &publication_lock).await; + let migration_reads = backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_lock).await; + let maintenance_store = store.clone(); + let migration_guard = backend + .acquire_write_lock(RUSTFS_META_BUCKET, &migration_lock) + .await + .expect("migration lock should be acquired"); + let maintenance = tokio::spawn(async move { + maintenance_store + .put_table_maintenance_config( + bucket, + "sales", + "orders", + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + ..Default::default() + }, + ) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.read_lock_acquisition_count(RUSTFS_META_BUCKET, &migration_lock).await == migration_reads { + tokio::task::yield_now().await; + } + }) + .await + .expect("maintenance should request the migration read lock first"); + assert_eq!( + backend.write_lock_acquisition_count(bucket, &publication_lock).await, + publication_attempts, + "maintenance must not request publication while waiting for the migration fence" + ); + drop(migration_guard); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.write_lock_acquisition_count(bucket, &publication_lock).await == publication_attempts { + tokio::task::yield_now().await; + } + }) + .await + .expect("maintenance should request publication after the migration fence"); + assert!(!maintenance.is_finished()); + drop(publication_guard); + maintenance + .await + .expect("maintenance task should join") + .expect("maintenance config should persist"); +} + async fn assert_direct_drop_uses_publication_locks(store: S, backend: &TestCatalogObjectBackend) where S: TableCatalogStore + Clone + Send + Sync + 'static, @@ -1527,9 +1603,9 @@ fn iceberg_metadata_validation_enforces_schema_evolution() { let mut incompatible = promoted; incompatible["schemas"][1]["fields"][0]["type"] = serde_json::Value::from("string"); let error = validate_supported_table_metadata(&incompatible).expect_err("field IDs must retain compatible types"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("schema field 1 has an incompatible type evolution".to_string()) + TableCatalogStoreError::Invalid(message) if message == "schema field 1 has an incompatible type evolution" ); let mut moved_into_collection = table_metadata_json_for_validation(); @@ -1571,20 +1647,18 @@ fn iceberg_metadata_validation_enforces_schema_evolution() { ]); reused["current-schema-id"] = serde_json::Value::from(2); let error = validate_supported_table_metadata(&reused).expect_err("removed Iceberg field IDs must never be reused"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("schema field 1 cannot be reused after removal".to_string()) + TableCatalogStoreError::Invalid(message) if message == "schema field 1 cannot be reused after removal" ); let mut stale_last_column_id = table_metadata_json_for_validation(); stale_last_column_id["last-column-id"] = serde_json::Value::from(0); let error = validate_supported_table_metadata(&stale_last_column_id) .expect_err("last-column-id must cover nested and top-level assigned field IDs"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid( - "last-column-id must be non-negative and cover every assigned schema field id".to_string() - ) + TableCatalogStoreError::Invalid(message) if message == "last-column-id must be non-negative and cover every assigned schema field id" ); } @@ -1596,18 +1670,18 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor lower_column_watermark["last-column-id"] = serde_json::Value::from(0); let error = validate_table_metadata_transition(¤t, &lower_column_watermark) .expect_err("last-column-id must not decrease across commits"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("last-column-id must not decrease across table metadata commits".to_string()) + TableCatalogStoreError::Invalid(message) if message == "last-column-id must not decrease across table metadata commits" ); let mut lower_partition_watermark = current.clone(); lower_partition_watermark["last-partition-id"] = serde_json::Value::from(998); let error = validate_table_metadata_transition(¤t, &lower_partition_watermark) .expect_err("last-partition-id must not decrease across commits"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("last-partition-id must not decrease across table metadata commits".to_string()) + TableCatalogStoreError::Invalid(message) if message == "last-partition-id must not decrease across table metadata commits" ); let mut current_sequence = current.clone(); @@ -1616,9 +1690,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor lower_sequence_watermark["last-sequence-number"] = serde_json::Value::from(1); let error = validate_table_metadata_transition(¤t_sequence, &lower_sequence_watermark) .expect_err("last-sequence-number must not decrease across commits"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("last-sequence-number must not decrease across table metadata commits".to_string()) + TableCatalogStoreError::Invalid(message) if message == "last-sequence-number must not decrease across table metadata commits" ); let mut current_partitioned = current.clone(); @@ -1636,9 +1710,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor modified_partition["partition-specs"][0]["fields"][0]["name"] = serde_json::Value::from("renamed_id"); let error = validate_table_metadata_transition(¤t_partitioned, &modified_partition) .expect_err("published partition specs must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing partition spec 0 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing partition spec 0 must not be modified" ); let mut current_sorted = current.clone(); @@ -1659,9 +1733,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor modified_sort["sort-orders"][1]["fields"][0]["direction"] = serde_json::Value::from("desc"); let error = validate_table_metadata_transition(¤t_sorted, &modified_sort).expect_err("published sort orders must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing sort order 1 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing sort order 1 must not be modified" ); let mut current_with_snapshot = current.clone(); @@ -1677,18 +1751,18 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor modified_snapshot["snapshots"][0]["timestamp-ms"] = serde_json::Value::from(2); let error = validate_table_metadata_transition(¤t_with_snapshot, &modified_snapshot) .expect_err("published snapshots must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing snapshot 10 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing snapshot 10 must not be modified" ); let mut modified_history = current.clone(); modified_history["schemas"][0]["fields"][0]["type"] = serde_json::Value::from("string"); let error = validate_table_metadata_transition(¤t, &modified_history) .expect_err("published schema definitions must be immutable"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("existing schema 0 must not be modified".to_string()) + TableCatalogStoreError::Invalid(message) if message == "existing schema 0 must not be modified" ); let mut current_with_retired_id = current.clone(); @@ -1713,9 +1787,9 @@ fn iceberg_metadata_transition_preserves_assignment_watermarks_and_schema_histor reused_id["current-schema-id"] = serde_json::Value::from(2); let error = validate_table_metadata_transition(¤t_with_retired_id, &reused_id) .expect_err("new schemas must not reuse a previously assigned field ID"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("schema field 1 cannot reuse a previously assigned field id".to_string()) + TableCatalogStoreError::Invalid(message) if message == "schema field 1 cannot reuse a previously assigned field id" ); let mut valid = current.clone(); @@ -2057,9 +2131,9 @@ fn iceberg_metadata_validation_rejects_duplicate_snapshot_statistics() { metadata["statistics"] = serde_json::json!([statistics.clone(), statistics]); let error = validate_supported_table_metadata(&metadata).expect_err("a snapshot must not have duplicate statistics entries"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics contains duplicate entries for snapshot 10".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics contains duplicate entries for snapshot 10" ); } @@ -2118,9 +2192,9 @@ async fn snapshot_validation_bounds_changed_statistics_object_fanout() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("statistics object fanout must be bounded before storage lookups"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics object count exceeds the commit limit".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics object count exceeds the commit limit" ); } @@ -2179,9 +2253,9 @@ async fn snapshot_validation_bounds_changed_statistics_object_bytes() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("statistics bytes must be bounded before storage lookups"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics bytes exceed the commit validation limit".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics bytes exceed the commit validation limit" ); } @@ -2215,9 +2289,9 @@ async fn snapshot_validation_rechecks_retained_statistics_locations() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("retained statistics must remain inside the table warehouse"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics object is outside the table warehouse".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics object is outside the table warehouse" ); } @@ -2255,9 +2329,9 @@ async fn snapshot_validation_rejects_non_puffin_table_statistics() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("table statistics must be a Puffin file"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("table statistics object is not a Puffin file".to_string()) + TableCatalogStoreError::Invalid(message) if message == "table statistics object is not a Puffin file" ); } @@ -2300,9 +2374,9 @@ async fn snapshot_validation_accepts_non_parquet_partition_statistics() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("partition statistics must use a supported table data-file envelope"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("partition statistics object is not a supported table data file".to_string()) + TableCatalogStoreError::Invalid(message) if message == "partition statistics object is not a supported table data file" ); } @@ -2340,9 +2414,9 @@ async fn snapshot_validation_rejects_statistics_size_mismatch() { let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("statistics lengths must match the published object"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("statistics file-size-in-bytes does not match the object".to_string()) + TableCatalogStoreError::Invalid(message) if message == "statistics file-size-in-bytes does not match the object" ); } @@ -2358,11 +2432,9 @@ fn iceberg_metadata_validation_bounds_snapshot_sequence_numbers() { "summary": {"operation": "append"} }]); let error = validate_supported_table_metadata(&v2).expect_err("a v2 snapshot sequence must not exceed the table sequence"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid( - "Iceberg v2 snapshot sequence-number must be between zero and last-sequence-number".to_string() - ) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v2 snapshot sequence-number must be between zero and last-sequence-number" ); v2["last-sequence-number"] = serde_json::Value::from(0); v2["snapshots"][0] @@ -2370,9 +2442,9 @@ fn iceberg_metadata_validation_bounds_snapshot_sequence_numbers() { .expect("snapshot should be an object") .remove("sequence-number"); let error = validate_supported_table_metadata(&v2).expect_err("Iceberg v2 snapshots must include sequence-number"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("Iceberg v2 snapshot sequence-number is required".to_string()) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v2 snapshot sequence-number is required" ); let mut v1 = serde_json::json!({ @@ -2391,9 +2463,9 @@ fn iceberg_metadata_validation_bounds_snapshot_sequence_numbers() { }] }); let error = validate_supported_table_metadata(&v1).expect_err("v1 metadata must not carry a non-zero snapshot sequence"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("Iceberg v1 snapshot sequence-number must be zero when present".to_string()) + TableCatalogStoreError::Invalid(message) if message == "Iceberg v1 snapshot sequence-number must be zero when present" ); v1["snapshots"][0]["sequence-number"] = serde_json::Value::from(0); validate_supported_table_metadata(&v1).expect("a zero v1 compatibility sequence should validate"); @@ -2434,7 +2506,9 @@ fn metadata_log_locations_normalize_same_bucket_s3_uris() { }); assert_eq!( - metadata_log_locations(&metadata, "warehouse", &namespace, &table), + metadata_log_locations_matching(&metadata, "warehouse", |location| { + is_valid_table_metadata_location(&namespace, &table, location) + }), BTreeSet::from([metadata_location]) ); } @@ -2601,9 +2675,9 @@ async fn iceberg_snapshot_graph_rejects_unknown_partition_specs() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("manifest partition specs absent from table metadata must be rejected"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest references missing partition spec 7".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest references missing partition spec 7" ); } @@ -2684,9 +2758,9 @@ async fn iceberg_snapshot_graph_revalidates_unchanged_snapshots_after_spec_remov let error = validate_table_snapshot_changes(&context, Some(¤t), &target) .await .expect_err("removing a spec referenced by an unchanged snapshot must fail"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest references missing partition spec 7".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest references missing partition spec 7" ); } @@ -2904,9 +2978,9 @@ async fn iceberg_snapshot_registration_validates_all_retained_snapshots() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("registration must validate every retained snapshot"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest-list object is missing".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest-list object is missing" ); } @@ -3016,9 +3090,9 @@ async fn iceberg_snapshot_graph_revalidates_cached_manifest_declarations() { let error = validate_table_snapshot_changes(&context, Some(¤t_metadata), &metadata) .await .expect_err("every manifest-list declaration must match the cached manifest object"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest-list manifest_length does not match the manifest object".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest-list manifest_length does not match the manifest object" ); backend @@ -3036,9 +3110,9 @@ async fn iceberg_snapshot_graph_revalidates_cached_manifest_declarations() { let error = validate_table_snapshot_changes(&context, Some(¤t_metadata), &metadata) .await .expect_err("every cached manifest must match each manifest-list partition spec declaration"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest partition-spec-id does not match its manifest-list entry".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest partition-spec-id does not match its manifest-list entry" ); } @@ -3090,9 +3164,9 @@ async fn iceberg_snapshot_graph_bounds_shared_manifest_traversals() { let error = validate_table_snapshot_changes(&context, Some(¤t_metadata), &metadata) .await .expect_err("logical manifest traversals must remain bounded across shared snapshots"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest traversal count exceeds the commit limit".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest traversal count exceeds the commit limit" ); } @@ -3184,9 +3258,9 @@ async fn iceberg_v2_snapshot_graph_rejects_embedded_v2_manifests() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("new v2 snapshots must use a manifest list"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("new Iceberg v2 snapshots require manifest-list".to_string()) + TableCatalogStoreError::Invalid(message) if message == "new Iceberg v2 snapshots require manifest-list" ); } @@ -3229,9 +3303,9 @@ async fn iceberg_snapshot_graph_rejects_delete_files_in_data_manifest() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("a data manifest must not contain delete files"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest-list content does not match manifest file content".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest-list content does not match manifest file content" ); } @@ -3315,9 +3389,9 @@ async fn iceberg_snapshot_graph_rejects_manifest_length_mismatch() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("manifest-list lengths must match the published manifest object"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest-list manifest_length does not match the manifest object".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest-list manifest_length does not match the manifest object" ); } @@ -3360,9 +3434,9 @@ async fn iceberg_snapshot_graph_rejects_manifest_partition_spec_mismatch() { let error = validate_table_snapshot_changes(&context, None, &metadata) .await .expect_err("manifest headers must agree with their manifest-list entry"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("manifest partition-spec-id does not match its manifest-list entry".to_string()) + TableCatalogStoreError::Invalid(message) if message == "manifest partition-spec-id does not match its manifest-list entry" ); } @@ -3574,7 +3648,7 @@ fn parquet_i32_values(data: Vec) -> Vec { values } -fn test_bucket_entry(bucket: &str) -> TableBucketEntry { +pub(super) fn test_bucket_entry(bucket: &str) -> TableBucketEntry { TableBucketEntry { version: TABLE_CATALOG_ENTRY_VERSION, table_bucket: bucket.to_string(), @@ -3588,7 +3662,7 @@ fn test_bucket_entry(bucket: &str) -> TableBucketEntry { } } -fn test_namespace_entry(bucket: &str, namespace: &Namespace) -> NamespaceEntry { +pub(super) fn test_namespace_entry(bucket: &str, namespace: &Namespace) -> NamespaceEntry { NamespaceEntry { version: TABLE_CATALOG_ENTRY_VERSION, table_bucket: bucket.to_string(), @@ -3601,7 +3675,12 @@ fn test_namespace_entry(bucket: &str, namespace: &Namespace) -> NamespaceEntry { } } -fn test_table_entry(bucket: &str, namespace: &Namespace, table: &IdentifierSegment, metadata_location: String) -> TableEntry { +pub(super) fn test_table_entry( + bucket: &str, + namespace: &Namespace, + table: &IdentifierSegment, + metadata_location: String, +) -> TableEntry { TableEntry { version: TABLE_CATALOG_ENTRY_VERSION, table_bucket: bucket.to_string(), @@ -4145,9 +4224,9 @@ async fn durable_catalog_backup_rejects_incomplete_snapshot_graph() { .create_durable_catalog_backup(bucket, None) .await .expect_err("backup must reject a snapshot without a manifest graph"); - assert_eq!( + assert_matches!( error, - TableCatalogStoreError::Invalid("snapshot manifest-list object is missing".to_string()) + TableCatalogStoreError::Invalid(message) if message == "snapshot manifest-list object is missing" ); } @@ -4620,6 +4699,2159 @@ async fn durable_catalog_restore_retries_after_ambiguous_maintenance_replacement assert_eq!(restored.data, serde_json::to_vec(&backup_report).unwrap()); } +#[tokio::test] +async fn durable_strong_table_maintenance_uses_strong_entry_and_recoverable_sidecars() { + let backend = TestCatalogObjectBackend::default(); + let store = ConfiguredTableCatalogStore::new_for_test(backend.clone(), TableCatalogBackingMode::DurableStrong); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_location = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let old_metadata_location = default_table_metadata_file_path(&namespace, &table, "00000.metadata.json"); + + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry(bucket, &namespace, &table, metadata_location.clone())) + .await + .unwrap(); + backend + .seed_object(bucket, &metadata_location, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object(bucket, &old_metadata_location, b"{}".to_vec()).await; + + store + .put_table_maintenance_config( + bucket, + &namespace.public_name(), + table.as_str(), + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!( + store + .get_table_maintenance_config(bucket, &namespace.public_name(), table.as_str()) + .await + .unwrap() + .background_enabled + ); + + let scheduled = store + .run_table_maintenance_scheduler_once(bucket, &namespace.public_name(), table.as_str(), "scheduler-1".to_string()) + .await + .unwrap(); + assert_eq!(scheduled.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(scheduled.report.job.operation, TableMetadataMaintenanceOperation::Delete); + + let completed = store + .run_table_metadata_maintenance_worker_once(bucket, &namespace.public_name(), table.as_str(), "worker-1".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.table_id, "table-id"); + assert_eq!(completed.current_metadata_location, metadata_location); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert!(!backend.object_exists(bucket, &old_metadata_location).await.unwrap()); + assert!( + store + .get_table_metadata_maintenance_report(bucket, &namespace.public_name(), table.as_str(), &completed.job.job_id) + .await + .unwrap() + .is_some() + ); + + let mut running = completed.clone(); + running.job.status = TableMetadataMaintenanceJobStatus::Running; + running.job.worker_id = Some("worker-heartbeat".to_string()); + running.job.lease_id = "lease-heartbeat".to_string(); + running.job.heartbeat_at = None; + running.job.finished_at = None; + let paths = TableCatalogObjectPaths::default(); + let current_job_path = paths.table_maintenance_current_job_path(bucket, &namespace, &table, &running.job.table_id); + backend + .seed_object(RUSTFS_META_BUCKET, ¤t_job_path, serde_json::to_vec(&running).unwrap()) + .await; + let heartbeat = store + .heartbeat_table_metadata_maintenance_job( + bucket, + &namespace.public_name(), + table.as_str(), + &running.job.job_id, + "lease-heartbeat", + "worker-heartbeat", + ) + .await + .unwrap(); + assert_eq!(heartbeat.job.status, TableMetadataMaintenanceJobStatus::Running); + assert!(heartbeat.job.heartbeat_at.is_some()); + assert!( + heartbeat + .audit_events + .iter() + .any(|event| event.action == TableMaintenanceAuditAction::WorkerHeartbeat) + ); + + let reloaded = ConfiguredTableCatalogStore::new_for_test(backend.clone(), TableCatalogBackingMode::DurableStrong); + assert!( + reloaded + .get_table_maintenance_config(bucket, &namespace.public_name(), table.as_str()) + .await + .unwrap() + .background_enabled + ); + let reloaded_report = reloaded + .get_table_metadata_maintenance_report(bucket, &namespace.public_name(), table.as_str(), &completed.job.job_id) + .await + .unwrap() + .expect("maintenance report should survive strong catalog reload"); + assert_eq!(reloaded_report.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!( + reloaded + .get_table_maintenance_scheduler_report(bucket, &namespace.public_name(), table.as_str()) + .await + .unwrap() + .current_job + .expect("reloaded scheduler report should expose the current job") + .job_id, + completed.job.job_id + ); +} + +async fn seed_strong_maintenance_table( + store: &StrongTableCatalogStore, + backend: &TestCatalogObjectBackend, +) -> TableEntry { + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00003.metadata.json"); + let entry = test_table_entry(bucket, &namespace, &table, current.clone()); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store.create_table(entry.clone()).await.unwrap(); + backend + .seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store + .put_table_maintenance_config( + bucket, + "sales", + "orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }, + ) + .await + .unwrap(); + entry +} + +async fn maintenance_test_object_versions(backend: &TestCatalogObjectBackend) -> BTreeMap<(String, String), TableCatalogObject> { + backend + .state + .lock() + .await + .objects + .iter() + .map(|(key, object)| { + ( + key.clone(), + TableCatalogObject { + data: object.data.clone(), + etag: Some(object.etag.clone()), + mod_time: object.mod_time, + }, + ) + }) + .collect() +} + +#[tokio::test] +async fn durable_strong_maintenance_follows_renamed_metadata_and_manifest_roots() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let source = IdentifierSegment::parse("orders").unwrap(); + let destination = IdentifierSegment::parse("archived_orders").unwrap(); + let metadata_dir = default_table_metadata_dir_path(&namespace, &source); + let logged = format!("{metadata_dir}/00002.metadata.json"); + let orphan = format!("{metadata_dir}/00001.metadata.json"); + let manifest_list = format!("{metadata_dir}/snap-10.avro"); + let manifest = format!("{metadata_dir}/manifest-10.avro"); + let orphan_manifest = format!("{metadata_dir}/manifest-orphan.avro"); + let data = format!("{}/referenced.parquet", default_table_data_dir_path(&namespace, &source)); + let delete = format!("{}/referenced.parquet", default_table_delete_dir_path(&namespace, &source)); + let orphan_data = format!("{}/orphan.parquet", default_table_data_dir_path(&namespace, &source)); + let unrelated = default_table_metadata_file_path(&namespace, &destination, "00000.metadata.json"); + let manifest_bytes = manifest_avro_bytes(&[(&data, 0), (&delete, 1)]); + backend + .seed_object(bucket, &manifest_list, manifest_list_avro_bytes(&[(&manifest, manifest_bytes.len())])) + .await; + backend.seed_object(bucket, &manifest, manifest_bytes).await; + for location in [&logged, &orphan, &orphan_manifest, &data, &delete, &orphan_data, &unrelated] { + backend.seed_object(bucket, location, b"{}".to_vec()).await; + } + backend + .seed_object( + bucket, + &entry.metadata_location, + serde_json::to_vec(&serde_json::json!({ + "metadata-log": [{"metadata-file": format!("s3://{bucket}/{logged}")}], + "snapshots": [{"snapshot-id": 10, "manifest-list": manifest_list}] + })) + .unwrap(), + ) + .await; + store + .rename_table(bucket, "sales", "orders", "sales", "archived_orders") + .await + .unwrap(); + store + .put_table_maintenance_config( + bucket, + "sales", + "archived_orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + }, + ) + .await + .unwrap(); + + let queued = store + .run_table_maintenance_scheduler_once(bucket, "sales", "archived_orders", "rename-scheduler".to_string()) + .await + .unwrap(); + assert_eq!(queued.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert!(queued.report.retained_metadata_locations.contains(&logged)); + assert_eq!(queued.report.deletable_metadata_locations, vec![orphan.clone()]); + assert_eq!(queued.report.reachability_graph.status, TableMaintenanceReachabilityGraphStatus::Complete); + for location in [&manifest_list, &manifest, &data, &delete] { + assert!(queued.report.referenced_object_reports.iter().any(|report| { + &report.object_location == location && report.state == TableMetadataMaintenanceObjectState::Retained + })); + } + for location in [&orphan_manifest, &orphan_data] { + assert!(queued.report.deletable_object_locations.contains(location)); + } + let completed = store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "archived_orders", "rename-worker".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert_eq!(completed.job.deleted_object_count, 2); + for location in [ + &entry.metadata_location, + &logged, + &manifest_list, + &manifest, + &data, + &delete, + &unrelated, + ] { + assert!(backend.object_exists(bucket, location).await.unwrap(), "retained object {location}"); + } + for location in [&orphan, &orphan_manifest, &orphan_data] { + assert!(!backend.object_exists(bucket, location).await.unwrap(), "orphan object {location}"); + } + + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let renamed = reloaded + .load_table(bucket, "sales", "archived_orders") + .await + .unwrap() + .unwrap(); + assert_eq!(renamed.table_id, entry.table_id); + assert_eq!(renamed.metadata_location, entry.metadata_location); + assert!(reloaded.load_table(bucket, "sales", "orders").await.unwrap().is_none()); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report(bucket, "sales", "archived_orders", &completed.job.job_id) + .await + .unwrap() + .unwrap(), + completed + ); +} + +#[tokio::test] +async fn durable_strong_maintenance_rejects_shared_roots_before_successful_job_preflight() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let source = IdentifierSegment::parse("orders").unwrap(); + let destination = IdentifierSegment::parse("archived_orders").unwrap(); + store + .rename_table(bucket, "sales", "orders", "sales", "archived_orders") + .await + .unwrap(); + let config = TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + }; + store + .put_table_maintenance_config(bucket, "sales", "archived_orders", config.clone()) + .await + .unwrap(); + let completed = store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "archived_orders", "before-share".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + + let replacement_metadata = default_table_metadata_file_path(&namespace, &source, "00004.metadata.json"); + let mut replacement = test_table_entry(bucket, &namespace, &source, replacement_metadata.clone()); + replacement.table_id = "replacement-table-id".to_string(); + replacement.table_uuid = "replacement-table-uuid".to_string(); + replacement.warehouse_location = format!("s3://{bucket}/tables/replacement-table-id"); + backend + .seed_object(bucket, &replacement_metadata, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store.create_table(replacement.clone()).await.unwrap(); + store + .put_table_maintenance_config(bucket, "sales", "orders", config) + .await + .unwrap(); + let renamed = store.load_table(bucket, "sales", "archived_orders").await.unwrap().unwrap(); + assert_ne!(renamed.table_id, replacement.table_id); + assert_ne!(renamed.warehouse_location, replacement.warehouse_location); + assert_eq!( + table_metadata_dir_path_for_entry(&renamed).unwrap(), + table_metadata_dir_path_for_entry(&replacement).unwrap() + ); + + let paths = TableCatalogObjectPaths::default(); + let mut replacement_report = completed.clone(); + replacement_report.job.table = "orders".to_string(); + replacement_report.job.table_id = replacement.table_id.clone(); + replacement_report.job.current_metadata_location = replacement_metadata.clone(); + replacement_report.current_metadata_location = replacement_metadata; + for path in [ + paths.table_maintenance_current_job_path(bucket, &namespace, &source, &replacement.table_id), + paths.table_maintenance_latest_job_path(bucket, &namespace, &source, &replacement.table_id), + paths.table_maintenance_job_path(bucket, &namespace, &source, &replacement.table_id, &replacement_report.job.job_id), + ] { + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(&replacement_report).unwrap()) + .await; + } + for status in [ + TableMetadataMaintenanceJobStatus::Successful, + TableMetadataMaintenanceJobStatus::Running, + ] { + for (table, report) in [(&source, &replacement_report), (&destination, &completed)] { + let mut current = report.clone(); + current.job.status = status.clone(); + if status == TableMetadataMaintenanceJobStatus::Running { + current.job.worker_id = Some("active-worker".to_string()); + current.job.lease_id = "active-lease".to_string(); + current.job.heartbeat_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + current.job.finished_at = None; + } + let path = paths.table_maintenance_current_job_path(bucket, &namespace, table, ¤t.job.table_id); + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(¤t).unwrap()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + for table in [&source, &destination] { + assert_matches!( + store.run_table_maintenance_scheduler_once(bucket, "sales", table.as_str(), "shared-scheduler".to_string()).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata and warehouse roots") + ); + assert_matches!( + store.run_table_metadata_maintenance_worker_once(bucket, "sales", table.as_str(), "shared-worker".to_string()).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata and warehouse roots") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } + } +} + +#[tokio::test] +async fn durable_strong_maintenance_rejects_queued_cleanup_after_rename_and_recreate() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let source = IdentifierSegment::parse("orders").unwrap(); + let orphan = default_table_metadata_file_path(&namespace, &source, "00001.metadata.json"); + backend.seed_object(bucket, &orphan, b"{}".to_vec()).await; + let queued = store + .run_table_maintenance_scheduler_once(bucket, "sales", "orders", "before-rename".to_string()) + .await + .unwrap(); + assert_eq!(queued.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.report.deletable_metadata_locations, vec![orphan.clone()]); + store + .rename_table(bucket, "sales", "orders", "sales", "archived_orders") + .await + .unwrap(); + store + .put_table_maintenance_config( + bucket, + "sales", + "archived_orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + ..Default::default() + }, + ) + .await + .unwrap(); + let renamed_queued = store + .run_table_maintenance_scheduler_once(bucket, "sales", "archived_orders", "before-recreate".to_string()) + .await + .unwrap(); + assert_eq!(renamed_queued.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + let mut replacement = test_table_entry( + bucket, + &namespace, + &source, + default_table_metadata_file_path(&namespace, &source, "00004.metadata.json"), + ); + replacement.table_id = "replacement-table-id".to_string(); + replacement.table_uuid = "replacement-table-uuid".to_string(); + replacement.warehouse_location = format!("s3://{bucket}/tables/replacement-table-id"); + backend + .seed_object(bucket, &replacement.metadata_location, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store.create_table(replacement).await.unwrap(); + let before = maintenance_test_object_versions(&backend).await; + for table in ["orders", "archived_orders"] { + assert_matches!( + store.run_table_metadata_maintenance_worker_once(bucket, "sales", table, "after-share".to_string()).await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata and warehouse roots") + ); + } + assert_eq!(maintenance_test_object_versions(&backend).await, before); + assert!(backend.object_exists(bucket, &orphan).await.unwrap()); + let queued_path = + TableCatalogObjectPaths::default().table_maintenance_current_job_path(bucket, &namespace, &source, &entry.table_id); + assert!(backend.read_object(RUSTFS_META_BUCKET, &queued_path).await.unwrap().is_none()); + assert_eq!(renamed_queued.report.job.job_id, queued.report.job.job_id); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "archived_orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + renamed_queued.report + ); +} + +#[tokio::test] +async fn durable_strong_scheduler_plans_without_bucket_lock_and_revalidates_full_config() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let other = IdentifierSegment::parse("payments").unwrap(); + let other_current = default_table_metadata_file_path(&namespace, &other, "00001.metadata.json"); + let other_next = default_table_metadata_file_path(&namespace, &other, "00002.metadata.json"); + let mut other_entry = test_table_entry(bucket, &namespace, &other, other_current.clone()); + other_entry.table_id = "payments-id".to_string(); + other_entry.table_uuid = "payments-uuid".to_string(); + other_entry.warehouse_location = format!("s3://{bucket}/tables/payments-id"); + store.create_table(other_entry).await.unwrap(); + backend.seed_object(bucket, &other_next, b"{}".to_vec()).await; + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let scheduler_store = Arc::clone(&store); + let scheduler = tokio::spawn(async move { + scheduler_store + .run_table_maintenance_scheduler_once(bucket, "sales", "orders", "paused-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("planner must reach current metadata read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "payments".to_string(), + commit_id: "unrelated-during-maintenance".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: "token-v1".to_string(), + expected_metadata_location: other_current, + new_metadata_location: other_next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("planning must release the bucket publication writer") + .unwrap(); + assert_eq!(committed.table.metadata_location, other_next); + assert!(!scheduler.is_finished()); + let mut changed = store.get_table_maintenance_config(bucket, "sales", "orders").await.unwrap(); + changed.max_retry_attempts = 1; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.put_table_maintenance_config(bucket, "sales", "orders", changed), + ) + .await + .expect("planning must release the table publication writer") + .unwrap(); + let winning_claim = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_maintenance_scheduler_once(bucket, "sales", "orders", "new-config-claim".to_string()), + ) + .await + .expect("a second scheduler can claim while the first planner is paused") + .unwrap(); + assert_eq!(winning_claim.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(winning_claim.report.job.max_retry_attempts, 1); + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, scheduler).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("config changed during planning") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + winning_claim.report + ); +} + +#[tokio::test] +async fn durable_strong_scheduler_revalidates_entry_after_unlocked_planning() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let next = default_table_metadata_file_path(&namespace, &table, "00004.metadata.json"); + backend.seed_object(bucket, &next, b"{}".to_vec()).await; + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let scheduler_store = Arc::clone(&store); + let scheduler = tokio::spawn(async move { + scheduler_store + .run_table_maintenance_scheduler_once(bucket, "sales", "orders", "paused-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("planner must reach current metadata read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "target-during-maintenance".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: entry.version_token, + expected_metadata_location: entry.metadata_location, + new_metadata_location: next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("planning must release the table publication writer") + .unwrap(); + assert_eq!(committed.table.metadata_location, next); + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, scheduler).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("table changed during maintenance planning") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); +} + +#[tokio::test] +async fn durable_strong_rename_preserves_complete_maintenance_state_across_namespaces_and_reload() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let curated = Namespace::parse("curated").unwrap(); + store + .create_namespace(test_namespace_entry(&source.table_bucket, &curated)) + .await + .unwrap(); + let mut current = source.clone(); + for (namespace, table) in [("curated", "orders_v2"), ("curated", "orders_v3"), ("sales", "orders")] { + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded + .rename_table(&source.table_bucket, ¤t.namespace, ¤t.table, namespace, table) + .await + .unwrap(); + assert!(rename_maintenance_objects(&backend, ¤t).await.is_empty()); + assert!( + reloaded + .load_table(&source.table_bucket, ¤t.namespace, ¤t.table) + .await + .unwrap() + .is_none() + ); + current = reloaded + .load_table(&source.table_bucket, namespace, table) + .await + .unwrap() + .unwrap(); + assert_eq!(current.table_id, source.table_id); + assert_eq!(current.table_uuid, source.table_uuid); + assert_renamed_maintenance_objects(&backend, ¤t, &expected).await; + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path(&source.table_bucket)) + .await + .unwrap() + ); + } + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!( + reloaded + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap() + .max_retry_attempts, + 2 + ); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap() + .job + .job_id, + "running-rename" + ); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_LATEST) + .await + .unwrap() + .unwrap() + .job + .job_id, + "quarantined-rename" + ); +} + +fn rename_maintenance_root(entry: &TableEntry) -> String { + let config = TableCatalogObjectPaths::default().table_maintenance_config_path( + &entry.table_bucket, + &Namespace::parse(&entry.namespace).unwrap(), + &IdentifierSegment::parse(&entry.table).unwrap(), + &entry.table_id, + ); + config.strip_suffix(MAINTENANCE_CONFIG_FILE).unwrap().to_string() +} + +fn rename_maintenance_intent_path(bucket: &str) -> String { + let entry_path = TableCatalogObjectPaths::default().table_bucket_entry_path(bucket); + format!( + "{}maintenance-rename-intent.json", + entry_path.strip_suffix(TABLE_BUCKET_ENTRY_FILE).unwrap() + ) +} + +async fn rename_maintenance_objects( + backend: &TestCatalogObjectBackend, + entry: &TableEntry, +) -> BTreeMap { + let prefix = rename_maintenance_root(entry); + backend + .state + .lock() + .await + .objects + .iter() + .filter_map(|((bucket, path), object)| { + if bucket != RUSTFS_META_BUCKET { + return None; + } + path.strip_prefix(&prefix) + .map(|suffix| (suffix.to_string(), serde_json::from_slice(&object.data).unwrap())) + }) + .collect() +} + +async fn assert_renamed_maintenance_objects( + backend: &TestCatalogObjectBackend, + entry: &TableEntry, + expected: &BTreeMap, +) { + let renamed = expected + .iter() + .map(|(suffix, value)| { + let mut value = value.clone(); + for field in ["job", "snapshot-expiration", "compaction"] { + if let Some(owner) = value.get_mut(field).and_then(serde_json::Value::as_object_mut) { + owner.insert("namespace".to_string(), serde_json::Value::String(entry.namespace.clone())); + owner.insert("table".to_string(), serde_json::Value::String(entry.table.clone())); + } + } + (suffix.clone(), value) + }) + .collect::>(); + assert_eq!(rename_maintenance_objects(backend, entry).await, renamed); +} + +async fn seed_strong_rename_maintenance_state( + store: &StrongTableCatalogStore, + backend: &TestCatalogObjectBackend, +) -> TableEntry { + let entry = seed_strong_maintenance_table(store, backend).await; + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + store + .put_table_maintenance_config( + &entry.table_bucket, + &entry.namespace, + &entry.table, + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + max_retry_attempts: 2, + quarantine_enabled: true, + quarantine_retention_seconds: 86_400, + ..Default::default() + }, + ) + .await + .unwrap(); + let mut successful = store + .run_table_metadata_maintenance_worker_once( + &entry.table_bucket, + &entry.namespace, + &entry.table, + "seed-rename".to_string(), + ) + .await + .unwrap(); + let metadata = serde_json::json!({"metadata-log": []}); + successful.snapshot_expiration = Some(table_snapshot_expiration_report( + &entry.table_bucket, + &namespace, + &table, + &entry, + &metadata, + TableSnapshotExpirationConfig { + min_snapshots_to_keep: 1, + max_snapshot_age_ms: 86_400_000, + }, + OffsetDateTime::now_utc(), + )); + successful.compaction = Some( + table_compaction_planning_report( + backend, + &entry.table_bucket, + &namespace, + &table, + &entry, + &metadata, + TableCompactionPlanningConfig { + target_file_size_bytes: 512 * 1024 * 1024, + small_file_threshold_bytes: 128 * 1024 * 1024, + min_input_files: 2, + max_rewrite_bytes_per_job: 1024 * 1024 * 1024, + }, + ) + .await + .unwrap(), + ); + let mut queued = successful.clone(); + queued.job.job_id = "queued-rename".to_string(); + queued.job.status = TableMetadataMaintenanceJobStatus::Queued; + queued.job.scheduler_id = Some("scheduler-rename".to_string()); + queued.job.scheduler_lease_id = "scheduler-lease-rename".to_string(); + queued.job.scheduled_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + queued.job.finished_at = None; + push_table_maintenance_audit_event( + &mut queued, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Scheduler, + TableMaintenanceAuditAction::SchedulerQueued, + Some("preserve queued audit".to_string()), + None, + None, + ); + let mut running = queued.clone(); + running.job.job_id = "running-rename".to_string(); + running.job.status = TableMetadataMaintenanceJobStatus::Running; + running.job.worker_id = Some("worker-rename".to_string()); + running.job.lease_id = "worker-lease-rename".to_string(); + running.job.heartbeat_at = Some(maintenance_timestamp(OffsetDateTime::now_utc())); + push_table_maintenance_audit_event( + &mut running, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Worker, + TableMaintenanceAuditAction::WorkerHeartbeat, + Some("preserve heartbeat audit".to_string()), + None, + None, + ); + let mut quarantined = running.clone(); + quarantined.job.job_id = "quarantined-rename".to_string(); + quarantined.job.status = TableMetadataMaintenanceJobStatus::Failed; + quarantined.job.failure_reason = Some("preserve quarantine details".to_string()); + quarantined.job.quarantined_object_count = 3; + quarantined.job.next_retry_after = Some(maintenance_timestamp(OffsetDateTime::now_utc() + Duration::hours(1))); + push_table_maintenance_audit_event( + &mut quarantined, + OffsetDateTime::now_utc(), + TableMaintenanceAuditActor::Operator, + TableMaintenanceAuditAction::QuarantineRetry, + Some("preserve operator audit".to_string()), + None, + Some(3), + ); + let paths = TableCatalogObjectPaths::default(); + for report in [&successful, &queued, &running, &quarantined] { + let path = paths.table_maintenance_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id, &report.job.job_id); + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).unwrap()) + .await; + } + for (path, report) in [ + ( + paths.table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + &running, + ), + ( + paths.table_maintenance_latest_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + &quarantined, + ), + ] { + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).unwrap()) + .await; + } + assert_eq!(rename_maintenance_objects(backend, &entry).await.len(), 7); + entry +} + +#[tokio::test] +async fn durable_strong_maintenance_gets_hold_publication_read_lock_through_sidecar_reads() { + for read_kind in ["config", "report"] { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected_objects = rename_maintenance_objects(&backend, &source).await; + let expected = match read_kind { + "config" => { + let config = store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + assert_ne!(config, TableMaintenanceConfig::default()); + serde_json::to_value(config).unwrap() + } + "report" => { + let report = store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(); + assert_eq!(report.job.job_id, "running-rename"); + serde_json::to_value(report).unwrap() + } + _ => unreachable!(), + }; + let suffix = if read_kind == "report" { + MAINTENANCE_CURRENT_JOB_FILE + } else { + MAINTENANCE_CONFIG_FILE + }; + let pause = backend + .pause_next_read(RUSTFS_META_BUCKET, &format!("{}{suffix}", rename_maintenance_root(&source))) + .await; + let read_store = Arc::clone(&store); + let read = tokio::spawn(async move { + match read_kind { + "config" => serde_json::to_value( + read_store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(), + ) + .unwrap(), + "report" => serde_json::to_value( + read_store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + ) + .unwrap(), + _ => unreachable!(), + } + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("GET must reach its sidecar read"); + let publication_path = default_table_bucket_publication_lock_path(); + let attempts = backend.write_lock_acquisition_count("analytics", &publication_path).await; + let rename_store = Arc::clone(&store); + let mut rename = tokio::spawn(async move { + rename_store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.write_lock_acquisition_count("analytics", &publication_path).await == attempts { + tokio::task::yield_now().await; + } + }) + .await + .expect("rename must attempt the bucket publication write lock"); + assert!( + tokio::time::timeout(StdDuration::from_millis(50), &mut rename).await.is_err(), + "rename must wait for the {read_kind} GET" + ); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected_objects); + pause.release(); + assert_eq!(tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, read).await.unwrap().unwrap(), expected); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, rename) + .await + .unwrap() + .unwrap() + .unwrap(); + assert!(store.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + let destination = store.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected_objects).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + } +} + +#[tokio::test] +async fn durable_strong_scheduler_history_scan_allows_same_bucket_commit() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let namespace = Namespace::parse("sales").unwrap(); + let payments = IdentifierSegment::parse("payments").unwrap(); + let payments_current = default_table_metadata_file_path(&namespace, &payments, "00001.metadata.json"); + let payments_next = default_table_metadata_file_path(&namespace, &payments, "00002.metadata.json"); + let mut payments_entry = test_table_entry("analytics", &namespace, &payments, payments_current.clone()); + payments_entry.table_id = "payments-id".to_string(); + payments_entry.table_uuid = "payments-uuid".to_string(); + payments_entry.warehouse_location = "s3://analytics/tables/payments-id".to_string(); + store.create_table(payments_entry).await.unwrap(); + backend.seed_object("analytics", &payments_next, b"{}".to_vec()).await; + let expected = store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + .unwrap(); + assert_eq!(expected.audit_timeline.len(), 4); + let job_path = TableCatalogObjectPaths::default().table_maintenance_job_path( + "analytics", + &namespace, + &IdentifierSegment::parse("orders").unwrap(), + &source.table_id, + "queued-rename", + ); + let pause = backend.pause_next_read(RUSTFS_META_BUCKET, &job_path).await; + let read_store = Arc::clone(&store); + let read = tokio::spawn(async move { + read_store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("scheduler GET must reach a history read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: "analytics".to_string(), + namespace: "sales".to_string(), + table: "payments".to_string(), + commit_id: "payments-during-scheduler-history".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: "token-v1".to_string(), + expected_metadata_location: payments_current, + new_metadata_location: payments_next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("history scanning must not hold the bucket publication reader") + .unwrap(); + assert_eq!(committed.table.metadata_location, payments_next); + assert!(!read.is_finished()); + let before_read_completion = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_eq!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, read) + .await + .unwrap() + .unwrap() + .unwrap(), + expected + ); + assert_eq!( + store.load_table("analytics", "sales", "payments").await.unwrap().unwrap(), + committed.table + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_read_completion); +} + +#[tokio::test] +async fn durable_strong_scheduler_history_missing_during_rename_rejects_even_after_content_etag_aba() { + for round_trip in [false, true] { + let backend = TestCatalogObjectBackend::content_addressed(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected_objects = rename_maintenance_objects(&backend, &source).await; + let root = rename_maintenance_root(&source); + for (suffix, value) in &expected_objects { + backend + .put_json(RUSTFS_META_BUCKET, &format!("{root}{suffix}"), value.clone()) + .await; + } + let snapshot_path = StrongTableCatalogStore::::snapshot_object_path(); + let snapshot_before = backend + .read_object(RUSTFS_META_BUCKET, &snapshot_path) + .await + .unwrap() + .unwrap(); + let job_path = TableCatalogObjectPaths::default().table_maintenance_job_path( + "analytics", + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders").unwrap(), + &source.table_id, + "queued-rename", + ); + let job_before = backend.read_object(RUSTFS_META_BUCKET, &job_path).await.unwrap().unwrap(); + let before_read = backend.pause_before_next_read(RUSTFS_META_BUCKET, &job_path).await; + let after_read = backend.pause_next_read(RUSTFS_META_BUCKET, &job_path).await; + let read_store = Arc::clone(&store); + let read = tokio::spawn(async move { + read_store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, before_read.wait_started()) + .await + .expect("fixed history membership must precede reading its job"); + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.rename_table("analytics", "sales", "orders", "sales", "orders_v2"), + ) + .await + .expect("rename must proceed while history reading is paused") + .unwrap(); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &job_path).await.unwrap()); + before_read.release(); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, after_read.wait_started()) + .await + .expect("history read must capture the missing object before returning"); + if round_trip { + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.rename_table("analytics", "sales", "orders_v2", "sales", "orders"), + ) + .await + .expect("reverse rename must proceed before the missing read is returned") + .unwrap(); + assert_eq!(store.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + let snapshot_after = backend + .read_object(RUSTFS_META_BUCKET, &snapshot_path) + .await + .unwrap() + .unwrap(); + assert_eq!(snapshot_after.data, snapshot_before.data); + assert_eq!(snapshot_after.etag, snapshot_before.etag); + let job_after = backend.read_object(RUSTFS_META_BUCKET, &job_path).await.unwrap().unwrap(); + assert_eq!(job_after.data, job_before.data); + assert_eq!(job_after.etag, job_before.etag); + assert_renamed_maintenance_objects(&backend, &source, &expected_objects).await; + } + let before_read_completion = maintenance_test_object_versions(&backend).await; + after_read.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, read).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Unavailable(message)) if message.contains("maintenance scheduler history changed") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_read_completion); + } +} + +#[tokio::test] +async fn durable_strong_scheduler_history_membership_budget_precedes_report_reads() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_maintenance_table(&store, &backend).await; + let prefix = TableCatalogObjectPaths::default().table_maintenance_jobs_prefix( + "analytics", + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders").unwrap(), + &source.table_id, + ); + for index in 0..4097 { + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{prefix}{index:064x}.json"), b"not-json".to_vec()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store.get_table_maintenance_scheduler_report("analytics", "sales", "orders").await, + Err(TableCatalogStoreError::Unavailable(message)) if message.contains("maintenance scheduler history exceeds") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); +} + +#[tokio::test] +async fn durable_strong_maintenance_gets_preserve_defaults_without_sidecars() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let entry = test_table_entry("analytics", &namespace, &table, metadata); + store.put_table_bucket(test_bucket_entry("analytics")).await.unwrap(); + store + .create_namespace(test_namespace_entry("analytics", &namespace)) + .await + .unwrap(); + store.create_table(entry.clone()).await.unwrap(); + let before = maintenance_test_object_versions(&backend).await; + assert_eq!( + store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(), + TableMaintenanceConfig::default() + ); + assert!( + store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); + let scheduler = store + .get_table_maintenance_scheduler_report("analytics", "sales", "orders") + .await + .unwrap(); + assert_eq!(scheduler.status, TableMaintenanceSchedulerStatus::Disabled); + assert_eq!(scheduler.config_source, TableMaintenanceConfigSource::Default); + assert!(!scheduler.background_enabled); + assert!(!scheduler.delete_enabled); + assert!(scheduler.current_job.is_none()); + assert!(scheduler.audit_timeline.is_empty()); + assert!(rename_maintenance_objects(&backend, &entry).await.is_empty()); + assert_eq!(maintenance_test_object_versions(&backend).await, before); +} + +#[tokio::test] +async fn durable_strong_rename_replaces_same_identity_destination_sidecars_and_removes_extra_history() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let root = rename_maintenance_root(&destination); + for (suffix, value) in &expected { + let mut value = value.clone(); + if let Some(job) = value.get_mut("job") { + job["table"] = serde_json::json!("orders_v2"); + job["failure_reason"] = serde_json::json!("stale destination state"); + for field in ["snapshot-expiration", "compaction"] { + value[field]["table"] = serde_json::json!("orders_v2"); + } + } else { + value["delete-enabled"] = serde_json::json!(false); + } + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{root}{suffix}"), serde_json::to_vec(&value).unwrap()) + .await; + } + let mut extra = expected[MAINTENANCE_CURRENT_JOB_FILE].clone(); + extra["job"]["job_id"] = serde_json::json!("extra-destination-history"); + for field in ["job", "snapshot-expiration", "compaction"] { + extra[field]["table"] = serde_json::json!("orders_v2"); + } + let extra_path = TableCatalogObjectPaths::default().table_maintenance_job_path( + "analytics", + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders_v2").unwrap(), + &source.table_id, + "extra-destination-history", + ); + backend + .seed_object(RUSTFS_META_BUCKET, &extra_path, serde_json::to_vec(&extra).unwrap()) + .await; + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!(!backend.object_exists(RUSTFS_META_BUCKET, &extra_path).await.unwrap()); +} + +async fn assert_strong_rename_recovers_each_sidecar_copy_failure(after_write: bool) { + for failed_index in 0..7 { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let suffix = expected.keys().nth(failed_index).unwrap(); + let target = format!("{}{suffix}", rename_maintenance_root(&destination)); + if after_write { + backend.fail_after_next_put(RUSTFS_META_BUCKET, &target).await; + } else { + backend.fail_next_put(RUSTFS_META_BUCKET, &target).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err(), + "copy failure {suffix}" + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + assert!( + reloaded + .load_table("analytics", "sales", "orders_v2") + .await + .unwrap() + .is_none() + ); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected); + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_recovers_each_sidecar_copy_failure() { + assert_strong_rename_recovers_each_sidecar_copy_failure(false).await; +} + +#[tokio::test] +async fn durable_strong_rename_recovers_each_sidecar_copy_response_loss() { + assert_strong_rename_recovers_each_sidecar_copy_failure(true).await; +} + +#[tokio::test] +async fn durable_strong_rename_recovers_snapshot_failure_and_response_loss() { + for after_write in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let snapshot = StrongTableCatalogStore::::snapshot_object_path(); + if after_write { + backend.fail_after_next_put(RUSTFS_META_BUCKET, &snapshot).await; + } else { + backend.fail_next_put(RUSTFS_META_BUCKET, &snapshot).await; + } + let result = store.rename_table("analytics", "sales", "orders", "sales", "orders_v2").await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + if after_write { + assert!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + let destination = reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + if result.is_err() { + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + } + } else { + assert!(result.is_err()); + assert_eq!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected); + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + } + reloaded + .rename_table("analytics", "sales", "orders_v2", "sales", "orders_v3") + .await + .unwrap(); + let final_entry = reloaded.load_table("analytics", "sales", "orders_v3").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &final_entry, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_retries_source_cleanup_failure_and_response_loss() { + for after_delete in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let path = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + if after_delete { + backend.fail_after_next_delete(RUSTFS_META_BUCKET, &path).await; + } else { + let attempt = backend + .state + .lock() + .await + .delete_attempts + .get(&(RUSTFS_META_BUCKET.to_string(), path.clone())) + .copied() + .unwrap_or_default() + + 1; + backend.fail_delete_attempt(RUSTFS_META_BUCKET, &path, attempt).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + let destination = reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let recovered_destination = if after_delete { + reloaded + .rename_table("analytics", "sales", "orders_v2", "sales", "orders_v3") + .await + .unwrap(); + assert!(rename_maintenance_objects(&backend, &destination).await.is_empty()); + reloaded.load_table("analytics", "sales", "orders_v3").await.unwrap().unwrap() + } else { + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + destination + }; + assert_renamed_maintenance_objects(&backend, &recovered_destination, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_rejects_corrupt_recovery_journal_without_mutation() { + for corruption in ["json", "version", "owner", "path", "report-owner", "unknown-field"] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let target = format!("{}{MAINTENANCE_CONFIG_FILE}", rename_maintenance_root(&destination)); + backend.fail_next_put(RUSTFS_META_BUCKET, &target).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let journal_path = rename_maintenance_intent_path("analytics"); + let journal = backend.read_object(RUSTFS_META_BUCKET, &journal_path).await.unwrap().unwrap(); + let mut value: serde_json::Value = serde_json::from_slice(&journal.data).unwrap(); + match corruption { + "json" => {} + "version" => value["version"] = serde_json::json!(999), + "owner" => value["destination"]["table_uuid"] = serde_json::json!("other-table-uuid"), + "path" => { + let objects = value["source_objects"].as_object_mut().unwrap(); + let object = objects.remove(MAINTENANCE_CONFIG_FILE).unwrap(); + objects.insert("../outside.json".to_string(), object); + } + "report-owner" => { + value["source_objects"][MAINTENANCE_CURRENT_JOB_FILE]["data"]["job"]["namespace"] = serde_json::json!("other") + } + "unknown-field" => value["source_objects"][MAINTENANCE_CONFIG_FILE]["unexpected"] = serde_json::json!(true), + _ => unreachable!(), + } + let data = if corruption == "json" { + b"not json".to_vec() + } else { + serde_json::to_vec(&value).unwrap() + }; + backend.seed_object(RUSTFS_META_BUCKET, &journal_path, data).await; + let before = maintenance_test_object_versions(&backend).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_matches!( + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await, + Err(TableCatalogStoreError::Invalid(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before, "corruption {corruption}"); + } +} + +#[tokio::test] +async fn durable_strong_create_table_recovers_an_uncommitted_maintenance_rename() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let mut inactive_destination = source.clone(); + inactive_destination.table = "orders_v2".to_string(); + let target = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&inactive_destination)); + backend.fail_after_next_put(RUSTFS_META_BUCKET, &target).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + assert!(!rename_maintenance_objects(&backend, &inactive_destination).await.is_empty()); + + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders_v2").unwrap(); + let metadata = default_table_metadata_file_path(&namespace, &table, "recreated.metadata.json"); + let mut recreated = test_table_entry("analytics", &namespace, &table, metadata.clone()); + recreated.table_id = "recreated-table-id".to_string(); + recreated.table_uuid = "recreated-table-uuid".to_string(); + recreated.warehouse_location = "s3://analytics/tables/recreated-table-id".to_string(); + backend.seed_object("analytics", &metadata, b"{}".to_vec()).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded.create_table(recreated.clone()).await.unwrap(); + assert_eq!(reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(), recreated); + assert_eq!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), source); + assert_eq!(rename_maintenance_objects(&backend, &source).await, expected); + assert!(rename_maintenance_objects(&backend, &inactive_destination).await.is_empty()); + assert_eq!( + reloaded + .get_table_maintenance_config("analytics", "sales", "orders_v2") + .await + .unwrap(), + TableMaintenanceConfig::default() + ); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn durable_strong_drop_table_recovers_committed_maintenance_rename_cleanup() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let expected = rename_maintenance_objects(&backend, &source).await; + let current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + backend.fail_after_next_delete(RUSTFS_META_BUCKET, ¤t).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let destination = reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + reloaded.drop_table("analytics", "sales", "orders_v2").await.unwrap(); + assert!(reloaded.load_table("analytics", "sales", "orders").await.unwrap().is_none()); + assert!( + reloaded + .load_table("analytics", "sales", "orders_v2") + .await + .unwrap() + .is_none() + ); + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert_renamed_maintenance_objects(&backend, &destination, &expected).await; + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn durable_catalog_backup_recovers_pending_maintenance_rename_before_collecting_sidecars() { + for committed in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_maintenance_table(&store, &backend).await; + backend + .seed_object( + "analytics", + &source.metadata_location, + serde_json::to_vec(&table_metadata_json_for_backup("analytics", &source.table_id, &source.table_uuid)).unwrap(), + ) + .await; + let completed = store + .run_table_metadata_maintenance_worker_once("analytics", "sales", "orders", "backup-recovery".to_string()) + .await + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + let expected = rename_maintenance_objects(&backend, &source).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + if committed { + let path = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + backend.fail_after_next_delete(RUSTFS_META_BUCKET, &path).await; + } else { + let path = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&destination)); + backend.fail_after_next_put(RUSTFS_META_BUCKET, &path).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let backup = reloaded.create_durable_catalog_backup("analytics", None).await.unwrap(); + assert_eq!(backup.maintenance_object_count, expected.len()); + let (active, inactive) = if committed { + (&destination, &source) + } else { + (&source, &destination) + }; + assert_eq!( + reloaded + .load_table("analytics", &active.namespace, &active.table) + .await + .unwrap() + .unwrap(), + *active + ); + assert!( + reloaded + .load_table("analytics", &inactive.namespace, &inactive.table) + .await + .unwrap() + .is_none() + ); + assert_renamed_maintenance_objects(&backend, active, &expected).await; + assert!(rename_maintenance_objects(&backend, inactive).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_retry_preserves_source_commit_and_config_progress() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let destination_current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&destination)); + backend.fail_after_next_put(RUSTFS_META_BUCKET, &destination_current).await; + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let next = default_table_metadata_file_path(&namespace, &table, "00004.metadata.json"); + backend.seed_object("analytics", &next, b"{}".to_vec()).await; + let committed = store + .commit_table(TableCommitRequest { + table_bucket: "analytics".to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "source-progress-during-rename-recovery".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: source.version_token.clone(), + expected_metadata_location: source.metadata_location.clone(), + new_metadata_location: next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }) + .await + .unwrap(); + assert_eq!(committed.table.metadata_location, next); + let mut config = store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts = 1; + store + .put_table_maintenance_config("analytics", "sales", "orders", config.clone()) + .await + .unwrap(); + let expected = rename_maintenance_objects(&backend, &source).await; + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + let mut expected_entry = committed.table; + expected_entry.table = "orders_v2".to_string(); + assert_eq!( + reloaded.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(), + expected_entry + ); + assert_eq!( + reloaded + .get_table_maintenance_config("analytics", "sales", "orders_v2") + .await + .unwrap(), + config + ); + assert_renamed_maintenance_objects(&backend, &expected_entry, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn durable_strong_rename_recovery_validates_authoritative_sidecars_before_cleanup() { + for committed in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let mut destination = source.clone(); + destination.table = "orders_v2".to_string(); + let source_current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&source)); + let destination_current = format!("{}{MAINTENANCE_CURRENT_JOB_FILE}", rename_maintenance_root(&destination)); + if committed { + backend.fail_after_next_delete(RUSTFS_META_BUCKET, &source_current).await; + } else { + backend.fail_after_next_put(RUSTFS_META_BUCKET, &destination_current).await; + } + assert!( + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .is_err() + ); + assert!( + backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + let authoritative_current = if committed { &destination_current } else { &source_current }; + let object = backend + .read_object(RUSTFS_META_BUCKET, authoritative_current) + .await + .unwrap() + .unwrap(); + let mut report: serde_json::Value = serde_json::from_slice(&object.data).unwrap(); + report["job"]["table_id"] = serde_json::json!("wrong-authoritative-owner"); + backend + .seed_object(RUSTFS_META_BUCKET, authoritative_current, serde_json::to_vec(&report).unwrap()) + .await; + let before = maintenance_test_object_versions(&backend).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + let retry = reloaded + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await; + if committed { + assert_matches!(retry, Err(TableCatalogStoreError::Invalid(_) | TableCatalogStoreError::Conflict(_))); + } else { + assert_matches!(retry, Err(TableCatalogStoreError::Invalid(_))); + } + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } +} + +#[tokio::test] +async fn durable_strong_rename_rejects_unrecognized_or_misowned_sidecars_before_journaling() { + for invalid_path in [true, false] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_rename_maintenance_state(&store, &backend).await; + let root = rename_maintenance_root(&source); + if invalid_path { + backend + .seed_object(RUSTFS_META_BUCKET, &format!("{root}unexpected.json"), b"{}".to_vec()) + .await; + } else { + let path = format!("{root}{MAINTENANCE_CURRENT_JOB_FILE}"); + let mut report = rename_maintenance_objects(&backend, &source).await[MAINTENANCE_CURRENT_JOB_FILE].clone(); + report["snapshot-expiration"]["table_id"] = serde_json::json!("other-table-id"); + backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(&report).unwrap()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store.rename_table("analytics", "sales", "orders", "sales", "orders_v2").await, + Err(TableCatalogStoreError::Invalid(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + assert!( + !backend + .object_exists(RUSTFS_META_BUCKET, &rename_maintenance_intent_path("analytics")) + .await + .unwrap() + ); + } +} + +#[tokio::test] +async fn durable_strong_rename_keeps_maintenance_backup_and_restore_complete() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let source = seed_strong_maintenance_table(&store, &backend).await; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_dir = default_table_metadata_dir_path(&namespace, &table); + let manifest_list = format!("{metadata_dir}/snap-20.avro"); + let manifest = format!("{metadata_dir}/manifest-20.avro"); + let data_file = format!("{}/part-00001.parquet", default_table_data_dir_path(&namespace, &table)); + let manifest_bytes = manifest_avro_bytes(&[(&data_file, 0)]); + backend + .seed_object( + "analytics", + &manifest_list, + manifest_list_avro_bytes(&[(&manifest, manifest_bytes.len())]), + ) + .await; + backend.seed_object("analytics", &manifest, manifest_bytes).await; + backend.seed_object("analytics", &data_file, b"d".to_vec()).await; + let mut metadata = table_metadata_json_for_backup("analytics", &source.table_id, &source.table_uuid); + metadata["last-sequence-number"] = serde_json::json!(7); + metadata["current-snapshot-id"] = serde_json::json!(20); + metadata["snapshots"] = serde_json::json!([{ + "snapshot-id": 20, "sequence-number": 7, "timestamp-ms": 1, + "manifest-list": format!("s3://analytics/{manifest_list}"), "summary": {"operation": "append"} + }]); + metadata["snapshot-log"] = serde_json::json!([{"timestamp-ms": 1, "snapshot-id": 20}]); + metadata["refs"] = serde_json::json!({"main": {"type": "branch", "snapshot-id": 20}}); + backend + .seed_object("analytics", &source.metadata_location, serde_json::to_vec(&metadata).unwrap()) + .await; + store + .run_table_metadata_maintenance_worker_once("analytics", "sales", "orders", "backup-rename".to_string()) + .await + .unwrap(); + let expected = rename_maintenance_objects(&backend, &source).await; + let before_rename = store.create_durable_catalog_backup("analytics", None).await.unwrap(); + assert_eq!(before_rename.object_count, 4); + store + .rename_table("analytics", "sales", "orders", "sales", "orders_v2") + .await + .unwrap(); + let renamed = store.load_table("analytics", "sales", "orders_v2").await.unwrap().unwrap(); + let backup = store.create_durable_catalog_backup("analytics", None).await.unwrap(); + assert_eq!(backup.object_count, before_rename.object_count); + assert_eq!(backup.maintenance_object_count, expected.len()); + store + .put_table_maintenance_config("analytics", "sales", "orders_v2", TableMaintenanceConfig::default()) + .await + .unwrap(); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + reloaded + .restore_durable_catalog_backup("analytics", &backup.backup_id, None, true) + .await + .unwrap(); + assert_renamed_maintenance_objects(&backend, &renamed, &expected).await; + assert!(rename_maintenance_objects(&backend, &source).await.is_empty()); +} + +#[tokio::test] +async fn durable_strong_direct_worker_plans_without_bucket_publication_lock() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let other = IdentifierSegment::parse("payments").unwrap(); + let other_current = default_table_metadata_file_path(&namespace, &other, "00001.metadata.json"); + let other_next = default_table_metadata_file_path(&namespace, &other, "00002.metadata.json"); + let mut other_entry = test_table_entry(bucket, &namespace, &other, other_current.clone()); + other_entry.table_id = "payments-id".to_string(); + other_entry.table_uuid = "payments-uuid".to_string(); + other_entry.warehouse_location = format!("s3://{bucket}/tables/payments-id"); + store.create_table(other_entry).await.unwrap(); + backend.seed_object(bucket, &other_next, b"{}".to_vec()).await; + assert!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let worker_store = Arc::clone(&store); + let worker = tokio::spawn(async move { + worker_store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "direct-paused-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("direct worker planner must reach current metadata read"); + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "payments".to_string(), + commit_id: "unrelated-during-direct-worker".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: "token-v1".to_string(), + expected_metadata_location: other_current, + new_metadata_location: other_next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("direct worker planning must release the bucket publication writer") + .unwrap(); + assert_eq!(committed.table.metadata_location, other_next); + assert!(!worker.is_finished()); + pause.release(); + let completed = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker) + .await + .expect("direct worker must finish after planning resumes") + .unwrap() + .unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.worker_id.as_deref(), Some("direct-paused-plan")); + assert_eq!(completed.current_metadata_location, entry.metadata_location); + assert_eq!(store.load_table(bucket, "sales", "payments").await.unwrap().unwrap(), committed.table); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + completed + ); +} + +#[tokio::test] +async fn durable_strong_direct_worker_does_not_overwrite_new_scheduler_claim() { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let previous = store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "previous-worker".to_string()) + .await + .unwrap(); + assert_eq!(previous.job.status, TableMetadataMaintenanceJobStatus::Successful); + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let worker_store = Arc::clone(&store); + let worker = tokio::spawn(async move { + worker_store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "stale-direct-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("direct worker planner must reach current metadata read"); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + previous + ); + let winning_claim = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_maintenance_scheduler_once(bucket, "sales", "orders", "winning-scheduler".to_string()), + ) + .await + .expect("scheduler must be able to claim while the direct worker plans") + .unwrap(); + assert_eq!(winning_claim.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_ne!(winning_claim.report.job.job_id, previous.job.job_id); + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains("current maintenance job changed during planning") + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .unwrap(), + winning_claim.report + ); + assert_eq!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", &previous.job.job_id) + .await + .unwrap() + .unwrap(), + previous + ); +} + +#[tokio::test] +async fn durable_strong_direct_worker_revalidates_entry_and_full_config_after_planning() { + for change_config in [false, true] { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = "analytics"; + let pause = backend.pause_next_read(bucket, &entry.metadata_location).await; + let worker_store = Arc::clone(&store); + let worker = tokio::spawn(async move { + worker_store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "stale-direct-plan".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("direct worker planner must reach current metadata read"); + let expected_conflict = if change_config { + let mut config = store.get_table_maintenance_config(bucket, "sales", "orders").await.unwrap(); + config.max_retry_attempts = 1; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.put_table_maintenance_config(bucket, "sales", "orders", config), + ) + .await + .expect("direct worker planning must permit config changes") + .unwrap(); + "maintenance config changed during worker planning" + } else { + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let next = default_table_metadata_file_path(&namespace, &table, "00004.metadata.json"); + backend.seed_object(bucket, &next, b"{}".to_vec()).await; + let committed = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: bucket.to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "target-during-direct-worker".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: entry.version_token, + expected_metadata_location: entry.metadata_location, + new_metadata_location: next.clone(), + requirements: Vec::new(), + writer: Some("maintenance-test".to_string()), + }), + ) + .await + .expect("direct worker planning must permit target table commits") + .unwrap(); + assert_eq!(committed.table.metadata_location, next); + "table changed during maintenance worker planning" + }; + let before_claim = maintenance_test_object_versions(&backend).await; + pause.release(); + assert_matches!( + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker).await.unwrap().unwrap(), + Err(TableCatalogStoreError::Conflict(message)) if message.contains(expected_conflict) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before_claim); + assert!( + store + .get_table_metadata_maintenance_report(bucket, "sales", "orders", MAINTENANCE_JOB_ALIAS_CURRENT) + .await + .unwrap() + .is_none() + ); + } +} + +#[tokio::test] +async fn durable_strong_quarantine_mutation_rejects_non_current_job() { + let backend = TestCatalogObjectBackend::default(); + let object_store = ObjectTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_location = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + + seed_table_for_metadata_maintenance(&object_store, bucket, &namespace, &table, metadata_location.clone()).await; + let store = ConfiguredTableCatalogStore::new_for_test(backend.clone(), TableCatalogBackingMode::DurableStrong); + store.put_table_bucket(test_bucket_entry(bucket)).await.unwrap(); + store + .create_namespace(test_namespace_entry(bucket, &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry(bucket, &namespace, &table, metadata_location.clone())) + .await + .unwrap(); + backend + .seed_object(bucket, &metadata_location, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store + .put_table_maintenance_config( + bucket, + &namespace.public_name(), + table.as_str(), + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + quarantine_enabled: true, + quarantine_retention_seconds: 86_400, + ..Default::default() + }, + ) + .await + .unwrap(); + + let mut old = object_store + .plan_table_metadata_maintenance(bucket, &namespace.public_name(), table.as_str(), 0) + .await + .unwrap(); + old.job.status = TableMetadataMaintenanceJobStatus::Failed; + old.job.quarantine_enabled = true; + old.job.quarantined_object_count = 1; + object_store.put_table_metadata_maintenance_report(&old).await.unwrap(); + + let mut current = object_store + .plan_table_metadata_maintenance(bucket, &namespace.public_name(), table.as_str(), 0) + .await + .unwrap(); + current.job.status = TableMetadataMaintenanceJobStatus::Failed; + current.job.quarantine_enabled = true; + current.job.quarantined_object_count = 1; + object_store.put_table_metadata_maintenance_report(¤t).await.unwrap(); + + assert_matches!( + store + .apply_table_maintenance_quarantine_operation( + bucket, + &namespace.public_name(), + table.as_str(), + &old.job.job_id, + TableMaintenanceQuarantineOperationRequest { + action: TableMaintenanceQuarantineAction::Release, + reason: None, + }, + ) + .await, + Err(TableCatalogStoreError::Conflict(message)) if message.contains("not current") + ); + + let released = store + .apply_table_maintenance_quarantine_operation( + bucket, + &namespace.public_name(), + table.as_str(), + ¤t.job.job_id, + TableMaintenanceQuarantineOperationRequest { + action: TableMaintenanceQuarantineAction::Release, + reason: Some("strong quarantine release".to_string()), + }, + ) + .await + .unwrap(); + assert_eq!(released.report.job.quarantined_object_count, 0); + assert!( + released + .report + .job + .failure_reason + .as_deref() + .is_some_and(|reason| reason.contains("strong quarantine release")) + ); +} + #[tokio::test] async fn object_table_catalog_store_persists_view_entries_and_blocks_non_empty_namespace_drop() { let backend = TestCatalogObjectBackend { @@ -7908,6 +10140,67 @@ async fn maintenance_worker_run_honors_paused_config() { assert!(backend.object_exists(bucket, &old).await.unwrap()); } +#[tokio::test] +async fn object_maintenance_worker_delete_waits_for_table_publication() { + let backend = TestCatalogObjectBackend::default(); + let store = ObjectTableCatalogStore::new(backend.clone()); + let bucket = "analytics"; + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let old = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let current = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json"); + seed_table_for_metadata_maintenance(&store, bucket, &namespace, &table, current.clone()).await; + backend.seed_object(bucket, &old, b"{}".to_vec()).await; + backend + .seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + store + .put_table_maintenance_config( + bucket, + "sales", + "orders", + TableMaintenanceConfig { + version: TABLE_MAINTENANCE_CONFIG_VERSION, + background_enabled: true, + delete_enabled: true, + retain_recent_metadata_files: 0, + ..Default::default() + }, + ) + .await + .unwrap(); + let table_path = store.paths.table_entry_path(bucket, &namespace, &table); + let table_locks = backend.write_lock_acquisition_count(RUSTFS_META_BUCKET, &table_path).await; + let publication_path = default_table_publication_lock_path(&namespace, &table); + let publication_guard = backend.acquire_write_lock(bucket, &publication_path).await.unwrap(); + let publication_locks = backend.write_lock_acquisition_count(bucket, &publication_path).await; + let worker = tokio::spawn(async move { + store + .run_table_metadata_maintenance_worker_once(bucket, "sales", "orders", "worker-a".to_string()) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while backend.write_lock_acquisition_count(bucket, &publication_path).await == publication_locks { + tokio::task::yield_now().await; + } + }) + .await + .expect("object maintenance deletion must acquire the table publication lock"); + assert!(!worker.is_finished()); + assert_eq!( + backend.write_lock_acquisition_count(RUSTFS_META_BUCKET, &table_path).await, + table_locks + 3 + ); + assert!(backend.object_exists(bucket, &old).await.unwrap()); + assert!(backend.object_exists(bucket, ¤t).await.unwrap()); + drop(publication_guard); + let report = worker.await.unwrap().unwrap(); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.deleted_metadata_file_count, 1); + assert!(!backend.object_exists(bucket, &old).await.unwrap()); + assert!(backend.object_exists(bucket, ¤t).await.unwrap()); +} + #[tokio::test] async fn maintenance_worker_run_claims_table_entry_write_lock() { let backend = TestCatalogObjectBackend::default(); @@ -8123,6 +10416,927 @@ async fn maintenance_scheduler_run_reuses_active_queued_job() { assert_eq!(second.scheduler.audit_timeline.len(), 1); } +struct ConditionalMaintenanceFixture { + backend: TestCatalogObjectBackend, + store: ConfiguredTableCatalogStore, + queued: TableMetadataMaintenanceReport, + entry: TableEntry, + old_metadata: String, + now: OffsetDateTime, +} + +impl ConditionalMaintenanceFixture { + async fn new(mode: TableCatalogBackingMode, max_retry_attempts: u16) -> Self { + let backend = TestCatalogObjectBackend::default(); + let store = ConfiguredTableCatalogStore::new_for_test(backend.clone(), mode); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let current = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json"); + let old_metadata = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + store.put_table_bucket(test_bucket_entry("analytics")).await.unwrap(); + store + .create_namespace(test_namespace_entry("analytics", &namespace)) + .await + .unwrap(); + store + .create_table(test_table_entry("analytics", &namespace, &table, current.clone())) + .await + .unwrap(); + let entry = store.load_table("analytics", "sales", "orders").await.unwrap().unwrap(); + backend + .seed_object("analytics", ¤t, br#"{"metadata-log":[]}"#.to_vec()) + .await; + backend.seed_object("analytics", &old_metadata, b"{}".to_vec()).await; + store + .put_table_maintenance_config( + "analytics", + "sales", + "orders", + TableMaintenanceConfig { + background_enabled: true, + delete_enabled: true, + max_retry_attempts, + retry_initial_backoff_seconds: 1, + retry_max_backoff_seconds: 1, + ..Default::default() + }, + ) + .await + .unwrap(); + let queued = store + .run_table_maintenance_scheduler_once("analytics", "sales", "orders", "scheduler-a".to_string()) + .await + .unwrap() + .report; + assert_eq!(queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(queued.deletable_metadata_locations, vec![old_metadata.clone()]); + Self { + backend, + store, + queued, + entry, + old_metadata, + now: OffsetDateTime::now_utc(), + } + } + + async fn run( + &self, + worker: &str, + attempt: u16, + now: OffsetDateTime, + ) -> TableCatalogStoreResult { + let condition = TableMaintenanceWorkerCondition { + expected_job_id: self.queued.job.job_id.clone(), + expected_attempt: attempt, + }; + match &self.store { + ConfiguredTableCatalogStore::ObjectBacked(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + worker.to_string(), + &condition, + || now, + ) + .await + } + ConfiguredTableCatalogStore::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + worker.to_string(), + &condition, + || now, + ) + .await + } + } + } + + fn report_paths(&self) -> [String; 3] { + let paths = TableCatalogObjectPaths::default(); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + [ + paths.table_maintenance_job_path("analytics", &namespace, &table, &self.entry.table_id, &self.queued.job.job_id), + paths.table_maintenance_latest_job_path("analytics", &namespace, &table, &self.entry.table_id), + paths.table_maintenance_current_job_path("analytics", &namespace, &table, &self.entry.table_id), + ] + } + + async fn seed_report(&self, report: &TableMetadataMaintenanceReport) { + for path in self.report_paths() { + self.backend + .seed_object(RUSTFS_META_BUCKET, &path, serde_json::to_vec(report).unwrap()) + .await; + } + } + + async fn report(&self, id: &str) -> TableMetadataMaintenanceReport { + self.store + .get_table_metadata_maintenance_report("analytics", "sales", "orders", id) + .await + .unwrap() + .unwrap() + } + + async fn assert_report_views(&self, expected: &TableMetadataMaintenanceReport) { + for path in self.report_paths() { + let object = self.backend.read_object(RUSTFS_META_BUCKET, &path).await.unwrap().unwrap(); + let actual: TableMetadataMaintenanceReport = serde_json::from_slice(&object.data).unwrap(); + assert_eq!(&actual, expected, "report view {path}"); + } + assert_eq!( + self.store.load_table("analytics", "sales", "orders").await.unwrap(), + Some(self.entry.clone()) + ); + assert!( + self.backend + .object_exists("analytics", &self.entry.metadata_location) + .await + .unwrap() + ); + } + + async fn delete_attempts(&self) -> usize { + self.backend + .state + .lock() + .await + .delete_attempts + .get(&("analytics".to_string(), self.old_metadata.clone())) + .copied() + .unwrap_or_default() + } + + async fn fail_first_attempt(&self) -> TableMetadataMaintenanceReport { + self.backend.fail_delete_attempt("analytics", &self.old_metadata, 1).await; + self.run("worker-a", 0, self.now) + .await + .expect_err("injected delete failure must be surfaced"); + let failed = self.report(&self.queued.job.job_id).await; + assert_eq!(failed.job.status, TableMetadataMaintenanceJobStatus::Failed); + assert_eq!(failed.job.attempt, 1); + self.assert_report_views(&failed).await; + failed + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_claims_queued_job_and_replays_without_writes() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let fixture = ConditionalMaintenanceFixture::new(mode, 0).await; + let completed = fixture.run("worker-a", 0, fixture.now).await.unwrap(); + assert_eq!(completed.job.job_id, fixture.queued.job.job_id); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.max_retry_attempts, 0); + assert_eq!(completed.job.worker_id.as_deref(), Some("worker-a")); + assert_eq!(completed.job.scheduler_id, fixture.queued.job.scheduler_id); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert_eq!(completed.cleanup_candidate_locations, fixture.queued.cleanup_candidate_locations); + assert_eq!( + completed.job.cleanup_watermark_unix_seconds, + fixture.queued.job.cleanup_watermark_unix_seconds + ); + assert!( + !fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + fixture.assert_report_views(&completed).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + let replayed = fixture.run("worker-a", 0, fixture.now + Duration::seconds(1)).await.unwrap(); + assert_eq!(replayed, completed); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + assert_eq!( + completed + .audit_events + .iter() + .filter(|event| event.action == TableMaintenanceAuditAction::WorkerStarted) + .count(), + 1 + ); + assert_eq!( + completed + .audit_events + .iter() + .filter(|event| event.action == TableMaintenanceAuditAction::WorkerSucceeded) + .count(), + 1 + ); + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_samples_time_after_publication_wait_and_report_reads() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for expired in [false, true] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + let deadline = fixture.now; + let lease_timeout = Duration::seconds(i64::try_from(config.worker_lease_timeout_seconds).unwrap()); + fixture.queued.job.scheduled_at = Some(maintenance_timestamp(deadline - lease_timeout)); + fixture.seed_report(&fixture.queued).await; + let fixture = Arc::new(fixture); + let before = maintenance_test_object_versions(&fixture.backend).await; + let publication_path = default_table_bucket_publication_lock_path(); + let publication_guard = fixture + .backend + .acquire_write_lock("analytics", &publication_path) + .await + .unwrap(); + let attempts = fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await; + let final_read = fixture + .backend + .pause_next_read(RUSTFS_META_BUCKET, &fixture.report_paths()[2]) + .await; + let clock = Arc::new(std::sync::Mutex::new((deadline - Duration::seconds(2), 0_usize))); + let worker_clock = Arc::clone(&clock); + let worker_fixture = Arc::clone(&fixture); + let worker = tokio::spawn(async move { + let condition = TableMaintenanceWorkerCondition { + expected_job_id: worker_fixture.queued.job.job_id.clone(), + expected_attempt: 0, + }; + let clock = move || { + let mut state = worker_clock.lock().unwrap(); + state.1 += 1; + state.0 + }; + match &worker_fixture.store { + ConfiguredTableCatalogStore::ObjectBacked(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + "worker-a".to_string(), + &condition, + clock, + ) + .await + } + ConfiguredTableCatalogStore::DurableStrong(store) => { + store + .run_table_metadata_maintenance_worker_once_conditionally_with_clock( + "analytics", + "sales", + "orders", + "worker-a".to_string(), + &condition, + clock, + ) + .await + } + } + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + while fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await + == attempts + { + tokio::task::yield_now().await; + } + }) + .await + .expect("conditional worker must reach publication lock contention"); + assert!(!worker.is_finished(), "worker must wait for publication ownership"); + assert_eq!(clock.lock().unwrap().1, 0, "claim time must not precede publication ownership"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let claim_time = if expired { + deadline + } else { + deadline - Duration::nanoseconds(1) + }; + clock.lock().unwrap().0 = claim_time; + drop(publication_guard); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, final_read.wait_started()) + .await + .expect("conditional worker must read its final persisted report"); + assert_eq!(clock.lock().unwrap().1, 0, "claim time must follow every persisted report read"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + final_read.release(); + let result = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, worker) + .await + .expect("conditional worker must finish after publication and report reads") + .unwrap(); + assert_eq!(clock.lock().unwrap().1, 1, "claim validation must sample the clock once"); + if expired { + assert_matches!(result, Err(TableCatalogStoreError::Conflict(reason)) + if reason == "maintenance job is not eligible for a conditional claim; inspect its status and persisted attempt budget"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + fixture.assert_report_views(&fixture.queued).await; + assert_eq!(fixture.queued.job.status, TableMetadataMaintenanceJobStatus::Queued); + assert_eq!(fixture.queued.job.attempt, 0); + assert_eq!(fixture.delete_attempts().await, 0); + assert!( + fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + } else { + let completed = result.expect("a queued lease remains claimable immediately before its deadline"); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.started_at, Some(maintenance_timestamp(claim_time))); + assert_eq!(completed.job.heartbeat_at, Some(maintenance_timestamp(claim_time))); + assert_eq!( + completed + .audit_events + .iter() + .filter(|event| event.action == TableMaintenanceAuditAction::WorkerStarted) + .map(|event| event.timestamp.clone()) + .collect::>(), + vec![maintenance_timestamp(claim_time)], + ); + assert_eq!(completed.job.deleted_metadata_file_count, 1); + assert_eq!(fixture.delete_attempts().await, 1); + assert!( + !fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + fixture.assert_report_views(&completed).await; + } + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_retries_failed_job_with_persisted_plan_and_budget() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let failed = fixture.fail_first_attempt().await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_eq!(fixture.run("worker-a", 0, fixture.now).await.unwrap(), failed); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let retry_at = OffsetDateTime::parse( + failed.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + assert_matches!( + fixture.run("worker-b", 1, retry_at - Duration::seconds(1)).await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let mut config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts = 10; + fixture + .store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .unwrap(); + fixture.store = ConfiguredTableCatalogStore::new_for_test(fixture.backend.clone(), mode); + let completed = fixture.run("worker-b", 1, retry_at).await.unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.job_id, failed.job.job_id); + assert_eq!(completed.job.attempt, 2); + assert_eq!(completed.job.max_retry_attempts, 2); + assert_ne!(completed.job.lease_id, failed.job.lease_id); + assert_eq!(completed.cleanup_candidate_locations, failed.cleanup_candidate_locations); + assert_eq!(completed.deletable_metadata_locations, failed.deletable_metadata_locations); + assert_eq!(completed.job.cleanup_watermark_unix_seconds, failed.job.cleanup_watermark_unix_seconds); + assert_eq!(fixture.delete_attempts().await, 2); + fixture.assert_report_views(&completed).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_eq!(fixture.run("worker-b", 1, retry_at).await.unwrap(), completed); + assert_matches!(fixture.run("worker-a", 0, retry_at).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!( + fixture + .store + .heartbeat_table_metadata_maintenance_job( + "analytics", + "sales", + "orders", + &failed.job.job_id, + &failed.job.lease_id, + "worker-a", + ) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_does_not_reset_exhausted_budget_after_reload() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for budget in [0, 1, 2] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, budget).await; + let mut failed = fixture.fail_first_attempt().await; + if budget == 2 { + fixture + .backend + .fail_delete_attempt("analytics", &fixture.old_metadata, 2) + .await; + let due = OffsetDateTime::parse( + failed.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + fixture + .run("worker-b", 1, due) + .await + .expect_err("second delete failure must be surfaced"); + failed = fixture.report(&failed.job.job_id).await; + assert_eq!(failed.job.attempt, 2); + } + assert_eq!(failed.job.next_retry_after, None); + let mut config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts = 10; + fixture + .store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .unwrap(); + fixture.store = ConfiguredTableCatalogStore::new_for_test(fixture.backend.clone(), mode); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture + .run("worker-c", failed.job.attempt, fixture.now + Duration::hours(1)) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, usize::from(budget.max(1))); + fixture.assert_report_views(&failed).await; + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_rejects_unsafe_persisted_states_without_effects() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for scenario in [ + "job", + "attempt", + "expired-queued", + "expired-running", + "quarantine", + "pointer", + "generation", + "retention", + "paused", + "disabled", + "delete-disabled", + ] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let mut report = fixture.queued.clone(); + let mut attempt = 0; + match scenario { + "job" => report.job.job_id = "different-job".to_string(), + "attempt" => attempt = 2, + "expired-queued" => report.job.scheduled_at = Some(maintenance_timestamp(fixture.now - Duration::hours(1))), + "expired-running" => { + report.job.status = TableMetadataMaintenanceJobStatus::Running; + report.job.attempt = 1; + report.job.worker_id = Some("worker-a".to_string()); + report.job.lease_id = "expired-lease".to_string(); + report.job.heartbeat_at = Some(maintenance_timestamp(fixture.now - Duration::hours(1))); + } + "quarantine" => report.job.quarantined_object_count = 1, + "pointer" => report.job.current_metadata_location.push_str(".stale"), + "generation" => report.job.current_generation += 1, + "retention" => report.job.retain_recent_metadata_files += 1, + "paused" | "disabled" | "delete-disabled" => { + let mut config = fixture + .store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + match scenario { + "paused" => config.worker_paused = true, + "disabled" => config.background_enabled = false, + _ => config.delete_enabled = false, + } + fixture + .store + .put_table_maintenance_config("analytics", "sales", "orders", config) + .await + .unwrap(); + } + _ => unreachable!(), + } + fixture.seed_report(&report).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture.run("worker-a", attempt, fixture.now).await, + Err(TableCatalogStoreError::Conflict(_)), + "{mode:?}: {scenario}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before, "{mode:?}: {scenario}"); + assert_eq!(fixture.delete_attempts().await, 0, "{mode:?}: {scenario}"); + assert_eq!( + fixture.store.load_table("analytics", "sales", "orders").await.unwrap(), + Some(fixture.entry) + ); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_claim_publication_failures_do_not_repeat_effects() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for target in 0..3 { + for after_write in [false, true] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let path = &fixture.report_paths()[target]; + if after_write { + fixture.backend.fail_after_next_put(RUSTFS_META_BUCKET, path).await; + } else { + fixture.backend.fail_next_put(RUSTFS_META_BUCKET, path).await; + } + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("claim publication fault must be surfaced"); + assert_eq!(fixture.delete_attempts().await, 0); + let before = maintenance_test_object_versions(&fixture.backend).await; + let replay = fixture.run("worker-a", 0, fixture.now).await; + if target == 0 && !after_write { + let completed = replay.expect("unpublished claim can safely retry"); + assert_eq!(completed.job.attempt, 1); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&completed).await; + } else if target == 2 && after_write { + let running = replay.expect("fully published running claim can be replayed"); + assert_eq!(running.job.attempt, 1); + assert_eq!(running.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + fixture.assert_report_views(&running).await; + } else { + assert_matches!(replay, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 0); + } + } + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_terminal_publication_faults_never_repeat_deletion() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for target in 0..3 { + for after_write in [false, true] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let path = fixture.report_paths()[target].clone(); + let attempt = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &path).await + 2; + if after_write { + fixture + .backend + .state + .lock() + .await + .fail_after_put_attempts + .entry((RUSTFS_META_BUCKET.to_string(), path)) + .or_default() + .insert(attempt); + } else { + fixture.backend.fail_put_attempt(RUSTFS_META_BUCKET, &path, attempt).await; + } + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("terminal publication fault must be surfaced"); + assert_eq!(fixture.delete_attempts().await, 1); + assert!( + !fixture + .backend + .object_exists("analytics", &fixture.old_metadata) + .await + .unwrap() + ); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!(fixture.run("worker-b", 0, fixture.now).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!(fixture.run("worker-a", 1, fixture.now).await, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + let replay = fixture.run("worker-a", 0, fixture.now).await.unwrap(); + assert_eq!(replay.job.job_id, fixture.queued.job.job_id); + assert_eq!(replay.job.attempt, 1); + if target == 0 && !after_write { + assert_eq!(replay.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + } else { + assert_eq!(replay.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(replay.job.deleted_metadata_file_count, 1); + } + fixture.assert_report_views(&replay).await; + let repaired = maintenance_test_object_versions(&fixture.backend).await; + assert_eq!(fixture.run("worker-a", 0, fixture.now).await.unwrap(), replay); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, repaired); + assert_eq!(fixture.delete_attempts().await, 1); + } + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_terminal_repair_requires_exact_running_predecessor() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for mismatch in ["lease", "attempt", "worker", "candidate", "audit", "terminal"] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let paths = fixture.report_paths(); + let attempt = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &paths[1]).await + 2; + fixture.backend.fail_put_attempt(RUSTFS_META_BUCKET, &paths[1], attempt).await; + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("latest terminal write must fail"); + let object = fixture + .backend + .read_object(RUSTFS_META_BUCKET, &paths[2]) + .await + .unwrap() + .unwrap(); + let mut running: TableMetadataMaintenanceReport = serde_json::from_slice(&object.data).unwrap(); + assert_eq!(running.job.status, TableMetadataMaintenanceJobStatus::Running); + match mismatch { + "lease" => running.job.lease_id = "other-lease".to_string(), + "attempt" => running.job.attempt += 1, + "worker" => running.job.worker_id = Some("worker-b".to_string()), + "candidate" => running.cleanup_candidate_locations.clear(), + "audit" => { + running.audit_events.pop(); + } + "terminal" => running.job.status = TableMetadataMaintenanceJobStatus::Failed, + _ => unreachable!(), + } + fixture + .backend + .seed_object(RUSTFS_META_BUCKET, &paths[2], serde_json::to_vec(&running).unwrap()) + .await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture.run("worker-a", 0, fixture.now).await, + Err(TableCatalogStoreError::Conflict(_)), + "{mode:?}: {mismatch}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_failed_terminal_publication_can_be_repaired_and_retried() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let latest = fixture.report_paths()[1].clone(); + let attempt = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &latest).await + 2; + fixture.backend.fail_put_attempt(RUSTFS_META_BUCKET, &latest, attempt).await; + fixture + .backend + .fail_delete_attempt("analytics", &fixture.old_metadata, 1) + .await; + fixture + .run("worker-a", 0, fixture.now) + .await + .expect_err("delete and terminal publication faults must surface"); + let repaired = fixture.run("worker-a", 0, fixture.now).await.unwrap(); + assert_eq!(repaired.job.status, TableMetadataMaintenanceJobStatus::Failed); + assert_eq!(repaired.job.attempt, 1); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&repaired).await; + let due = OffsetDateTime::parse( + repaired.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + let completed = fixture.run("worker-b", 1, due).await.unwrap(); + assert_eq!(completed.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(completed.job.attempt, 2); + assert_eq!(fixture.delete_attempts().await, 2); + fixture.assert_report_views(&completed).await; + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_serializes_competing_claims_and_same_worker_replays() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for same_worker in [false, true] { + let fixture = Arc::new(ConditionalMaintenanceFixture::new(mode, 2).await); + let second_worker = if same_worker { "worker-a" } else { "worker-b" }; + let named_path = fixture.report_paths()[0].clone(); + let before = maintenance_test_object_versions(&fixture.backend).await; + let puts_before = fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &named_path).await; + let pause = fixture.backend.pause_next_put(RUSTFS_META_BUCKET, &named_path).await; + let first_fixture = Arc::clone(&fixture); + let first = tokio::spawn(async move { first_fixture.run("worker-a", 0, first_fixture.now).await }); + let (first, second) = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, async { + pause.wait_started().await; + // The first Running write has not committed: both workers must + // contend while every durable report still describes Queued. + let publication_path = default_table_bucket_publication_lock_path(); + let attempts = fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await; + let second_fixture = Arc::clone(&fixture); + let second = tokio::spawn(async move { second_fixture.run(second_worker, 0, second_fixture.now).await }); + while fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await + == attempts + { + tokio::task::yield_now().await; + } + assert_eq!( + fixture + .backend + .write_lock_acquisition_count("analytics", &publication_path) + .await, + attempts + 1, + ); + assert!(!first.is_finished(), "the first claim must remain paused before publication"); + assert!(!second.is_finished(), "the competing claim must wait for publication ownership"); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.backend.put_attempt_count(RUSTFS_META_BUCKET, &named_path).await, puts_before + 1); + assert_eq!(fixture.delete_attempts().await, 0); + pause.release(); + (first.await.unwrap(), second.await.unwrap()) + }) + .await + .expect("concurrent claims must complete without deadlock"); + let winner = if same_worker { + let first = first.unwrap(); + assert_eq!(second.unwrap(), first); + first + } else { + match (first, second) { + (Ok(report), Err(TableCatalogStoreError::Conflict(_))) + | (Err(TableCatalogStoreError::Conflict(_)), Ok(report)) => report, + other => panic!("exactly one competing claim must win: {other:?}"), + } + }; + assert_eq!(winner.job.attempt, 1); + assert_eq!(winner.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&winner).await; + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_running_retry_rejects_previous_lease_and_takeover() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let failed = fixture.fail_first_attempt().await; + let due = OffsetDateTime::parse( + failed.job.next_retry_after.as_deref().unwrap(), + &time::format_description::well_known::Rfc3339, + ) + .unwrap(); + let current = fixture.report_paths()[2].clone(); + fixture.backend.fail_after_next_put(RUSTFS_META_BUCKET, ¤t).await; + fixture + .run("worker-b", 1, due) + .await + .expect_err("response after complete Running claim is lost"); + let running = fixture.run("worker-b", 1, due).await.unwrap(); + assert_eq!(running.job.status, TableMetadataMaintenanceJobStatus::Running); + assert_eq!(running.job.attempt, 2); + assert_ne!(running.job.lease_id, failed.job.lease_id); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture + .store + .heartbeat_table_metadata_maintenance_job( + "analytics", + "sales", + "orders", + &failed.job.job_id, + &failed.job.lease_id, + "worker-a", + ) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_matches!(fixture.run("worker-a", 1, due).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!(fixture.run("worker-c", 2, due).await, Err(TableCatalogStoreError::Conflict(_))); + assert_matches!( + fixture.run("worker-b", 1, due + Duration::hours(1)).await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + fixture.assert_report_views(&running).await; + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_rejects_invalid_conditions_before_writes() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for invalid in ["empty-worker", "overflow", "current", "latest", "../job", ""] { + let mut fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let mut worker = "worker-a"; + let mut attempt = 0; + match invalid { + "empty-worker" => worker = " \t ", + "overflow" => attempt = u16::MAX, + job_id => fixture.queued.job.job_id = job_id.to_string(), + } + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture.run(worker, attempt, fixture.now).await, + Err(TableCatalogStoreError::Invalid(_)), + "{mode:?}: {invalid}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 0); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_requires_every_persisted_report_view() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for missing in 0..3 { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + fixture + .backend + .state + .lock() + .await + .objects + .remove(&(RUSTFS_META_BUCKET.to_string(), fixture.report_paths()[missing].clone())); + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!(fixture.run("worker-a", 0, fixture.now).await, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 0); + } + } +} + +#[tokio::test] +async fn conditional_maintenance_worker_rejects_invalid_failed_retry_state() { + for mode in [TableCatalogBackingMode::ObjectBacked, TableCatalogBackingMode::DurableStrong] { + for invalid in [ + "missing-backoff", + "malformed-backoff", + "zero-attempt", + "invalid-budget", + "exhausted", + ] { + let fixture = ConditionalMaintenanceFixture::new(mode, 2).await; + let mut failed = fixture.fail_first_attempt().await; + match invalid { + "missing-backoff" => failed.job.next_retry_after = None, + "malformed-backoff" => failed.job.next_retry_after = Some("not-a-timestamp".to_string()), + "zero-attempt" => failed.job.attempt = 0, + "invalid-budget" => failed.job.max_retry_attempts = 11, + "exhausted" => failed.job.attempt = failed.job.max_retry_attempts, + _ => unreachable!(), + } + fixture.seed_report(&failed).await; + let before = maintenance_test_object_versions(&fixture.backend).await; + assert_matches!( + fixture + .run("worker-b", failed.job.attempt, fixture.now + Duration::hours(1)) + .await, + Err(TableCatalogStoreError::Conflict(_)), + "{mode:?}: {invalid}" + ); + assert_eq!(maintenance_test_object_versions(&fixture.backend).await, before); + assert_eq!(fixture.delete_attempts().await, 1); + } + } +} + #[tokio::test] async fn maintenance_worker_claims_queued_job_before_running_it() { let backend = TestCatalogObjectBackend::default(); @@ -8677,7 +11891,10 @@ async fn maintenance_quarantine_rejects_mutating_non_current_job() { .await .expect_err("mutating a non-current quarantine job should fail"); - assert_eq!(error, TableCatalogStoreError::Conflict("maintenance job is not current".to_string())); + assert_matches!( + error, + TableCatalogStoreError::Conflict(message) if message == "maintenance job is not current" + ); } #[tokio::test] diff --git a/rustfs/src/table_catalog/tests/strong_maintenance_planning.rs b/rustfs/src/table_catalog/tests/strong_maintenance_planning.rs new file mode 100644 index 000000000..d0f864017 --- /dev/null +++ b/rustfs/src/table_catalog/tests/strong_maintenance_planning.rs @@ -0,0 +1,363 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; + +fn planning_report_paths(entry: &TableEntry, job_id: &str) -> [String; 3] { + let paths = TableCatalogObjectPaths::default(); + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + [ + paths.table_maintenance_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id, job_id), + paths.table_maintenance_current_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + paths.table_maintenance_latest_job_path(&entry.table_bucket, &namespace, &table, &entry.table_id), + ] +} + +async fn assert_only_planning_sidecars_changed( + backend: &TestCatalogObjectBackend, + before: BTreeMap<(String, String), TableCatalogObject>, + entry: &TableEntry, + report: &TableMetadataMaintenanceReport, +) { + let permitted = planning_report_paths(entry, &report.job.job_id) + .map(|path| (RUSTFS_META_BUCKET.to_string(), path)) + .into_iter() + .collect::>(); + let after = maintenance_test_object_versions(backend).await; + for key in &permitted { + let persisted = after.get(key).expect("all three report views must be published"); + assert_eq!( + serde_json::from_slice::(&persisted.data).unwrap(), + *report + ); + } + let unchanged_before = before + .into_iter() + .filter(|(key, _)| !permitted.contains(key)) + .collect::>(); + let unchanged_after = after + .into_iter() + .filter(|(key, _)| !permitted.contains(key)) + .collect::>(); + assert_eq!( + unchanged_after, unchanged_before, + "planning must not change catalog, metadata, manifests or data" + ); +} + +#[tokio::test] +async fn durable_strong_planning_uses_authoritative_entry_and_persists_only_audited_sidecars() { + for requested_retention in [0, 2] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let namespace = Namespace::parse(&entry.namespace).unwrap(); + let table = IdentifierSegment::parse(&entry.table).unwrap(); + let orphan = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json"); + let recent = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json"); + for path in [&orphan, &recent] { + backend + .seed_object(&entry.table_bucket, path, br#"{"metadata-log":[]}"#.to_vec()) + .await; + } + let config = TableMaintenanceConfig { + retain_recent_metadata_files: 3, + delete_enabled: true, + max_retry_attempts: 2, + ..Default::default() + }; + store + .put_table_maintenance_config(&entry.table_bucket, &entry.namespace, &entry.table, config.clone()) + .await + .unwrap(); + assert!( + ObjectTableCatalogStore::new(backend.clone()) + .load_table(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap() + .is_none() + ); + let before = maintenance_test_object_versions(&backend).await; + let report = store + .run_table_metadata_maintenance_planning( + &entry.table_bucket, + &entry.namespace, + &entry.table, + requested_retention, + None, + None, + ) + .await + .unwrap(); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(report.job.table_id, entry.table_id); + assert_eq!(report.current_metadata_location, entry.metadata_location); + assert_eq!(report.job.current_generation, entry.generation); + assert_eq!(report.job.config_source, TableMaintenanceConfigSource::TableOverride); + assert_eq!(report.job.retain_recent_metadata_files, if requested_retention == 0 { 3 } else { 2 }); + assert_eq!(report.job.max_retry_attempts, 2); + assert_eq!(report.job.worker_id.as_deref(), Some("rustfs-admin")); + assert_eq!(report.job.attempt, 1); + assert!(report.job.started_at.is_some() && report.job.finished_at.is_some()); + assert_eq!(report.job.deleted_metadata_file_count, 0); + assert_eq!(report.job.deleted_object_count, 0); + assert_eq!( + report.cleanup_candidate_locations, + if requested_retention == 0 { vec![] } else { vec![orphan] } + ); + for action in [ + TableMaintenanceAuditAction::Planned, + TableMaintenanceAuditAction::WorkerStarted, + TableMaintenanceAuditAction::WorkerSucceeded, + ] { + assert!(report.audit_events.iter().any(|event| event.action == action)); + } + assert_only_planning_sidecars_changed(&backend, before, &entry, &report).await; + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!( + reloaded + .load_table(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap() + .unwrap(), + entry + ); + assert_eq!( + reloaded + .get_table_maintenance_config(&entry.table_bucket, &entry.namespace, &entry.table) + .await + .unwrap(), + config + ); + assert_eq!( + reloaded + .get_table_metadata_maintenance_report(&entry.table_bucket, &entry.namespace, &entry.table, &report.job.job_id) + .await + .unwrap() + .unwrap(), + report + ); + } +} + +#[tokio::test] +async fn durable_strong_planning_rejects_pending_work_quarantine_and_incomplete_report_views() { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let previous = store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await + .unwrap(); + let paths = planning_report_paths(&entry, &previous.job.job_id); + for (status, quarantined) in [ + (TableMetadataMaintenanceJobStatus::Queued, 0), + (TableMetadataMaintenanceJobStatus::Running, 0), + (TableMetadataMaintenanceJobStatus::Failed, 0), + (TableMetadataMaintenanceJobStatus::Successful, 1), + ] { + let mut pending = previous.clone(); + pending.job.status = status; + pending.job.quarantined_object_count = quarantined; + pending.job.quarantine_enabled = quarantined > 0; + pending.job.quarantine_retention_seconds = if quarantined > 0 { 86_400 } else { 0 }; + for path in &paths { + backend + .seed_object(RUSTFS_META_BUCKET, path, serde_json::to_vec(&pending).unwrap()) + .await; + } + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } + for mismatched_view in [0, 2] { + for path in &paths { + backend + .seed_object(RUSTFS_META_BUCKET, path, serde_json::to_vec(&previous).unwrap()) + .await; + } + let mut inconsistent = previous.clone(); + inconsistent.job.attempt += 1; + backend + .seed_object(RUSTFS_META_BUCKET, &paths[mismatched_view], serde_json::to_vec(&inconsistent).unwrap()) + .await; + let before = maintenance_test_object_versions(&backend).await; + assert_matches!( + store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await, + Err(TableCatalogStoreError::Conflict(_)) + ); + assert_eq!(maintenance_test_object_versions(&backend).await, before); + } +} + +#[tokio::test] +async fn durable_strong_planning_revalidates_config_entry_and_job_after_unlocked_reads() { + for change in ["config", "entry", "job", "completed-job"] { + let backend = TestCatalogObjectBackend::default(); + let store = Arc::new(StrongTableCatalogStore::new(backend.clone())); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let pause = backend.pause_next_read("analytics", &entry.metadata_location).await; + let planning_store = Arc::clone(&store); + let planning = tokio::spawn(async move { + planning_store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await + }); + tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, pause.wait_started()) + .await + .expect("planning must reach metadata read"); + match change { + "config" => { + let mut config = store + .get_table_maintenance_config("analytics", "sales", "orders") + .await + .unwrap(); + config.max_retry_attempts += 1; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.put_table_maintenance_config("analytics", "sales", "orders", config), + ) + .await + .expect("planning must release config publication lock") + .unwrap(); + } + "entry" => { + let next = default_table_metadata_file_path( + &Namespace::parse("sales").unwrap(), + &IdentifierSegment::parse("orders").unwrap(), + "00004.metadata.json", + ); + backend.seed_object("analytics", &next, b"{}".to_vec()).await; + tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.commit_table(TableCommitRequest { + table_bucket: "analytics".to_string(), + namespace: "sales".to_string(), + table: "orders".to_string(), + commit_id: "concurrent-planning-commit".to_string(), + idempotency_key: None, + operation: "append".to_string(), + expected_version_token: entry.version_token.clone(), + expected_metadata_location: entry.metadata_location.clone(), + new_metadata_location: next, + requirements: Vec::new(), + writer: Some("planning-test".to_string()), + }), + ) + .await + .expect("planning must release catalog publication lock") + .unwrap(); + } + "job" => { + let winning = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_maintenance_scheduler_once( + "analytics", + "sales", + "orders", + "concurrent-scheduler".to_string(), + ), + ) + .await + .expect("planning must release job publication lock") + .unwrap(); + assert_eq!(winning.report.job.status, TableMetadataMaintenanceJobStatus::Queued); + } + "completed-job" => { + let winner = tokio::time::timeout( + TABLE_CATALOG_TEST_TIMEOUT, + store.run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None), + ) + .await + .expect("a second planner must complete while the first reads metadata") + .unwrap(); + assert_eq!(winner.job.status, TableMetadataMaintenanceJobStatus::Successful); + } + _ => unreachable!(), + } + let before_publish = maintenance_test_object_versions(&backend).await; + pause.release(); + let result = tokio::time::timeout(TABLE_CATALOG_TEST_TIMEOUT, planning) + .await + .unwrap() + .unwrap(); + if change == "completed-job" { + assert_matches!(result, Err(TableCatalogStoreError::Conflict(reason)) + if reason == "current maintenance job changed during planning"); + } else { + assert_matches!(result, Err(TableCatalogStoreError::Conflict(_))); + } + assert_eq!( + maintenance_test_object_versions(&backend).await, + before_publish, + "stale {change} planning must publish nothing" + ); + } +} + +#[tokio::test] +async fn durable_strong_planning_partial_terminal_publication_preserves_safe_retry_boundary() { + // A named-only completed plan is harmless history. A published latest alias + // without its current alias is observable incomplete publication and conflicts. + for failed_view in [1, 2] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let paths = planning_report_paths(&entry, "unused"); + let before = maintenance_test_object_versions(&backend).await; + backend.fail_next_put(RUSTFS_META_BUCKET, &paths[failed_view]).await; + let result = store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await; + assert!(result.is_err(), "an interrupted report publication must not return success"); + let interrupted = maintenance_test_object_versions(&backend).await; + for (key, value) in &before { + assert_eq!(interrupted.get(key), Some(value)); + } + let (named_key, named_value) = interrupted + .iter() + .find(|(key, _)| !before.contains_key(*key) && key.1 != paths[1] && key.1 != paths[2]) + .expect("the completed named report must be retained"); + let original: TableMetadataMaintenanceReport = serde_json::from_slice(&named_value.data).unwrap(); + assert_eq!(original.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(original.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(original.job.deleted_object_count, 0); + assert_eq!(original.job.deleted_metadata_file_count, 0); + let retry = store + .run_table_metadata_maintenance_planning("analytics", "sales", "orders", 0, None, None) + .await; + if failed_view == 2 { + assert_eq!(interrupted.len(), before.len() + 1); + let retried = retry.expect("named-only completed dry-run history permits safe replanning"); + assert_ne!(retried.job.job_id, original.job.job_id); + assert_eq!(retried.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(maintenance_test_object_versions(&backend).await.get(named_key), Some(named_value)); + assert_only_planning_sidecars_changed(&backend, interrupted.clone(), &entry, &retried).await; + } else { + assert_eq!(interrupted.len(), before.len() + 2); + assert_matches!(retry, Err(TableCatalogStoreError::Conflict(_))); + assert_eq!(maintenance_test_object_versions(&backend).await, interrupted); + } + assert_eq!(store.load_table("analytics", "sales", "orders").await.unwrap().unwrap(), entry); + } +} diff --git a/rustfs/src/table_catalog/tests/strong_maintenance_planning_paths.rs b/rustfs/src/table_catalog/tests/strong_maintenance_planning_paths.rs new file mode 100644 index 000000000..5116c495b --- /dev/null +++ b/rustfs/src/table_catalog/tests/strong_maintenance_planning_paths.rs @@ -0,0 +1,166 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; + +#[tokio::test] +async fn durable_strong_planning_renamed_paths_preserve_identity_and_fail_closed() { + for (target_namespace, delete_content, foreign_manifest) in [ + ("sales", None, false), + ("archive", None, false), + ("archive", Some(1), false), + ("archive", Some(2), false), + ("archive", None, true), + ] { + let backend = TestCatalogObjectBackend::default(); + let store = StrongTableCatalogStore::new(backend.clone()); + let entry = seed_strong_maintenance_table(&store, &backend).await; + let bucket = entry.table_bucket.as_str(); + let namespace = Namespace::parse("sales").unwrap(); + let table = IdentifierSegment::parse("orders").unwrap(); + let metadata_dir = default_table_metadata_dir_path(&namespace, &table); + let data_dir = default_table_data_dir_path(&namespace, &table); + let manifest_list = format!("{metadata_dir}/snap-20.avro"); + let manifest = if foreign_manifest { + default_table_metadata_file_path(&namespace, &IdentifierSegment::parse("foreign").unwrap(), "manifest.avro") + } else { + format!("{metadata_dir}/manifest-20.avro") + }; + let left = format!("{data_dir}/left.parquet"); + let right = format!("{data_dir}/right.parquet"); + let delete = format!("{}/delete.parquet", default_table_delete_dir_path(&namespace, &table)); + let mut files = vec![(left.as_str(), 0), (right.as_str(), 0)]; + if let Some(content) = delete_content { + files.push((delete.as_str(), content)); + backend.seed_object(bucket, &delete, b"delete fixture".to_vec()).await; + } + let manifest_bytes = manifest_avro_bytes(&files); + backend + .seed_object(bucket, &manifest_list, manifest_list_avro_bytes(&[(&manifest, manifest_bytes.len())])) + .await; + backend.seed_object(bucket, &manifest, manifest_bytes).await; + backend.seed_object(bucket, &left, parquet_i32_bytes(&[1])).await; + backend.seed_object(bucket, &right, parquet_i32_bytes(&[2])).await; + backend + .seed_object( + bucket, + &entry.metadata_location, + serde_json::to_vec(&serde_json::json!({ + "format-version": 2, + "table-uuid": entry.table_uuid, + "location": entry.warehouse_location, + "metadata-log": [], + "snapshots": [ + {"snapshot-id": 10, "timestamp-ms": 1000, "manifest-list": manifest_list}, + {"snapshot-id": 20, "timestamp-ms": 2000, "manifest-list": manifest_list} + ], + "current-snapshot-id": 20, + "refs": {"main": {"snapshot-id": 20, "type": "branch"}} + })) + .unwrap(), + ) + .await; + if target_namespace != "sales" { + store + .create_namespace(test_namespace_entry(bucket, &Namespace::parse(target_namespace).unwrap())) + .await + .unwrap(); + } + store + .rename_table(bucket, "sales", "orders", target_namespace, "renamed") + .await + .unwrap(); + let renamed = store.load_table(bucket, target_namespace, "renamed").await.unwrap().unwrap(); + let before = maintenance_test_object_versions(&backend).await; + let report = store + .run_table_metadata_maintenance_planning( + bucket, + target_namespace, + "renamed", + 0, + Some(TableSnapshotExpirationConfig { + min_snapshots_to_keep: 1, + max_snapshot_age_ms: 1000, + }), + Some(TableCompactionPlanningConfig { + target_file_size_bytes: 1024 * 1024, + small_file_threshold_bytes: 64 * 1024, + min_input_files: 2, + max_rewrite_bytes_per_job: 2 * 1024 * 1024, + }), + ) + .await + .unwrap(); + assert_eq!(report.job.operation, TableMetadataMaintenanceOperation::DryRun); + assert_eq!(report.job.status, TableMetadataMaintenanceJobStatus::Successful); + assert_eq!(report.job.deleted_metadata_file_count, 0); + assert_eq!(report.job.deleted_object_count, 0); + let expiration = report.snapshot_expiration.as_ref().unwrap(); + assert_eq!(expiration.namespace, target_namespace); + assert_eq!(expiration.table, "renamed"); + assert_eq!(expiration.table_id, entry.table_id); + assert_eq!(expiration.current_metadata_location, entry.metadata_location); + assert_eq!(expiration.expiration_candidate_count, 1); + assert!(expiration.committed_metadata_location.is_none()); + let compaction = report.compaction.as_ref().unwrap(); + assert_eq!(compaction.namespace, target_namespace); + assert_eq!(compaction.table, "renamed"); + assert_eq!(compaction.table_id, entry.table_id); + assert_eq!(compaction.current_metadata_location, entry.metadata_location); + assert_eq!(compaction.current_snapshot_id, Some(20)); + assert!(compaction.committed_metadata_location.is_none()); + if delete_content.is_some() || foreign_manifest { + assert_eq!(compaction.status, TableCompactionPlanningStatus::ManualReviewRequired); + assert!(compaction.rewrite_groups.is_empty()); + assert!(compaction.manual_review_count > 0); + if delete_content.is_some() { + assert_eq!(compaction.row_level_planning.delete_file_count, 1); + } + } else { + assert_eq!(compaction.status, TableCompactionPlanningStatus::RewriteCandidates); + assert_eq!(compaction.candidate_file_count, 2); + assert_eq!(compaction.rewrite_group_count, 1); + assert_eq!(compaction.rewrite_groups[0].input_file_locations, vec![left, right]); + assert!(compaction.rewrite_groups[0].output_file_location.is_none()); + } + let after = maintenance_test_object_versions(&backend).await; + for (key, value) in &before { + assert_eq!(after.get(key), Some(value), "planning changed an existing object: {key:?}"); + } + assert_eq!(after.len(), before.len() + 3, "only named/current/latest reports may be added"); + let reloaded = StrongTableCatalogStore::new(backend.clone()); + assert_eq!( + reloaded + .load_table(bucket, target_namespace, "renamed") + .await + .unwrap() + .unwrap(), + renamed + ); + for job in [ + report.job.job_id.as_str(), + MAINTENANCE_JOB_ALIAS_CURRENT, + MAINTENANCE_JOB_ALIAS_LATEST, + ] { + assert_eq!( + reloaded + .get_table_metadata_maintenance_report(bucket, target_namespace, "renamed", job) + .await + .unwrap() + .unwrap(), + report + ); + } + } +} diff --git a/rustfs/tests/admin_rebalance_status.rs b/rustfs/tests/admin_rebalance_status.rs new file mode 100644 index 000000000..c246038a4 --- /dev/null +++ b/rustfs/tests/admin_rebalance_status.rs @@ -0,0 +1,190 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![recursion_limit = "256"] + +use reqwest::{Client, StatusCode}; +use rustfs::app::context::get_global_app_context; +use rustfs::embedded::{RustFSServerBuilder, find_available_port}; +use rustfs_ecstore::api::{config::com::save_config, rebalance::RebalanceMeta}; +use s3s_sigv4::{ + AmzDate, EMPTY_STRING_SHA256_HASH, Payload, calculate_signature, create_canonical_request, create_string_to_sign, +}; +use std::{path::Path, sync::Arc, time::Duration}; + +mod common; + +const ACCESS_KEY: &str = "rebalance-status-test"; +const SECRET_KEY: &str = "rebalance-status-test-secret"; +const DENIED_KEY: &str = "rebalance-status-denied"; +const STATUS_PATH: &str = "/rustfs/admin/v3/rebalance/status"; +const IDLE_XML: &[u8] = b"NoSuchResourcepool rebalance is not started"; + +fn signed_status_request( + client: &Client, + endpoint: &str, + path: &str, + access_key: &str, + secret_key: &str, +) -> reqwest::RequestBuilder { + let host = endpoint.strip_prefix("http://").expect("local HTTP endpoint"); + let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%SZ").to_string(); + let date = AmzDate::parse(×tamp).expect("valid signing date"); + let canonical = create_canonical_request( + "GET", + path, + &[] as &[(&str, &str)], + [ + ("host", host), + ("x-amz-content-sha256", EMPTY_STRING_SHA256_HASH), + ("x-amz-date", timestamp.as_str()), + ], + Payload::SingleChunk(EMPTY_STRING_SHA256_HASH), + ); + let string_to_sign = create_string_to_sign(&canonical, &date, "us-east-1", "s3"); + let signature = calculate_signature(&string_to_sign, secret_key, &date, "us-east-1", "s3"); + client + .get(format!("{endpoint}{path}")) + .header("host", host) + .header("x-amz-date", ×tamp) + .header("x-amz-content-sha256", EMPTY_STRING_SHA256_HASH) + .header( + "authorization", + format!( + "AWS4-HMAC-SHA256 Credential={access_key}/{}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature={signature}", + date.fmt_date() + ), + ) +} + +async fn status_response_with_logs( + request: reqwest::RequestBuilder, + log_path: &Path, +) -> (StatusCode, reqwest::header::HeaderMap, bytes::Bytes, String) { + let before = usize::try_from(std::fs::metadata(log_path).expect("captured log metadata").len()).expect("log size fits usize"); + let response = request.send().await.expect("send status request"); + let status = response.status(); + let headers = response.headers().clone(); + let body = response.bytes().await.expect("read response body"); + let logs = std::fs::read_to_string(log_path).expect("read captured log"); + (status, headers, body, logs[before..].to_string()) +} + +#[test] +fn rebalance_status_preserves_http_contract_and_log_severity() { + let log = tempfile::NamedTempFile::new().expect("create captured log"); + tracing_subscriber::fmt() + .with_ansi(false) + .without_time() + .with_env_filter( + tracing_subscriber::EnvFilter::new("off") + .add_directive("s3s::ops=debug".parse().expect("S3 route log filter")) + .add_directive( + "rustfs::admin::handlers::rebalance=debug" + .parse() + .expect("rebalance log filter"), + ), + ) + .with_writer(log.reopen().expect("open captured log writer")) + .try_init() + .expect("install request log capture"); + + common::run_embedded_test(move || async move { + let port = find_available_port().expect("find local test port"); + let server = RustFSServerBuilder::new() + .address(format!("127.0.0.1:{port}")) + .access_key(ACCESS_KEY) + .secret_key(SECRET_KEY) + .build() + .await + .expect("start embedded RustFS"); + let endpoint = server.endpoint(); + let client = Client::builder().no_proxy().timeout(Duration::from_secs(30)).build().unwrap(); + + for path in [STATUS_PATH, "/minio/admin/v3/rebalance/status"] { + for _ in 0..2 { + let (status, headers, body, logs) = status_response_with_logs( + signed_status_request(&client, &endpoint, path, ACCESS_KEY, SECRET_KEY), + log.path(), + ) + .await; + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!(headers[reqwest::header::CONTENT_TYPE], "application/xml"); + assert_eq!(body.as_ref(), IDLE_XML); + assert!(!logs.contains("ERROR"), "idle polls must not log errors: {logs}"); + assert!(!logs.contains("WARN"), "idle polls must not be reported as rejected requests: {logs}"); + } + } + + let (status, _, body, _) = status_response_with_logs( + signed_status_request(&client, &endpoint, STATUS_PATH, ACCESS_KEY, "incorrect-secret"), + log.path(), + ) + .await; + assert_eq!(status, StatusCode::FORBIDDEN); + assert!(String::from_utf8_lossy(&body).contains("SignatureDoesNotMatch")); + + let context = get_global_app_context().expect("embedded server context"); + context + .iam() + .handle() + .create_user( + DENIED_KEY, + &rustfs_madmin::AddOrUpdateUserReq { + secret_key: SECRET_KEY.to_string(), + policy: None, + status: rustfs_madmin::AccountStatus::Enabled, + }, + ) + .await + .expect("create user without rebalance permission"); + let (status, _, body, _) = + status_response_with_logs(signed_status_request(&client, &endpoint, STATUS_PATH, DENIED_KEY, SECRET_KEY), log.path()) + .await; + assert_eq!(status, StatusCode::FORBIDDEN); + assert!(String::from_utf8_lossy(&body).contains("AccessDenied")); + + let store = context.object_store(); + let meta = RebalanceMeta { + id: "status-test-run".to_string(), + pool_stats: vec![Default::default()], + ..Default::default() + }; + meta.save(Arc::clone(&store.pools[0])) + .await + .expect("persist valid rebalance metadata"); + let (status, headers, body, logs) = + status_response_with_logs(signed_status_request(&client, &endpoint, STATUS_PATH, ACCESS_KEY, SECRET_KEY), log.path()) + .await; + assert_eq!(status, StatusCode::OK); + assert_eq!(headers[reqwest::header::CONTENT_TYPE], "application/json"); + let status: serde_json::Value = serde_json::from_slice(&body).expect("status JSON"); + assert_eq!(status["id"], "status-test-run"); + assert!(!logs.contains("ERROR"), "valid metadata must not log errors: {logs}"); + + save_config(store, "rebalance.bin", b"bad".to_vec()) + .await + .expect("inject corrupt rebalance metadata"); + let (status, _, body, logs) = + status_response_with_logs(signed_status_request(&client, &endpoint, STATUS_PATH, ACCESS_KEY, SECRET_KEY), log.path()) + .await; + assert_eq!(status, StatusCode::INTERNAL_SERVER_ERROR); + assert!(String::from_utf8_lossy(&body).contains("InternalError")); + assert!( + logs.contains("ERROR") && logs.contains("custom route returns error"), + "unexpected failure must retain diagnostics: {logs}" + ); + server.shutdown().await; + }); +} diff --git a/rustfs/tests/connect_license.rs b/rustfs/tests/connect_license.rs index d5ad32bed..bbe55e583 100644 --- a/rustfs/tests/connect_license.rs +++ b/rustfs/tests/connect_license.rs @@ -126,8 +126,8 @@ fn tamper_foreign_scope_and_expiry_are_structured_and_preserve_installed_state() let payload = URL_SAFE_NO_PAD .decode_to_vec(envelope["payload"].as_str().expect("payload").as_bytes()) .expect("decode payload"); - let mut claims: Value = serde_json::from_slice(&payload).expect("decode claims"); - claims["deployment"] = Value::String(OTHER_DEPLOYMENT.to_owned()); + let mut claims: LicenseClaims = serde_json::from_slice(&payload).expect("decode claims"); + claims.deployment = OTHER_DEPLOYMENT.to_owned(); envelope["payload"] = Value::String(URL_SAFE_NO_PAD.encode_to_string(serde_json::to_vec(&claims).unwrap())); let tampered = write_artifact(temporary.path(), "tampered.json", &serde_json::to_string(&envelope).unwrap()); let error = apply_license_artifact(&tampered, &state, &valid).expect_err("tampered artifact must fail"); diff --git a/rustfs/tests/connect_offline_bundle.rs b/rustfs/tests/connect_offline_bundle.rs index 9146e25b1..88cb394bf 100644 --- a/rustfs/tests/connect_offline_bundle.rs +++ b/rustfs/tests/connect_offline_bundle.rs @@ -155,7 +155,7 @@ fn connect_offline_bundle_is_deterministic_bounded_and_signed_over_exact_manifes assert_eq!(archive.len(), PATHS.len()); for (index, expected_path) in PATHS.into_iter().enumerate() { let file = archive.by_index(index).expect("archive member"); - assert_eq!(file.name(), expected_path); + assert_eq!(file.name().expect("archive entry name"), expected_path); assert_eq!(file.compression(), CompressionMethod::Stored); assert_eq!(file.last_modified(), Some(zip::DateTime::DEFAULT)); assert_eq!(file.unix_mode().map(|mode| mode & 0o777), Some(0o600)); diff --git a/rustfs/tests/connect_perf_drive.rs b/rustfs/tests/connect_perf_drive.rs index 43a3ba919..a3c88ec81 100644 --- a/rustfs/tests/connect_perf_drive.rs +++ b/rustfs/tests/connect_perf_drive.rs @@ -389,9 +389,15 @@ async fn successful_result_has_signed_bounded_private_offline_export() { let mut archive = zip::ZipArchive::new(Cursor::new(&export.archive_bytes)).expect("three-file archive"); assert_eq!(archive.len(), 3); - assert_eq!(archive.by_index(0).expect("envelope").name(), "envelope.json"); - assert_eq!(archive.by_index(1).expect("signature").name(), "envelope.sig"); - assert_eq!(archive.by_index(2).expect("result").name(), "result.json"); + assert_eq!( + archive.by_index(0).expect("envelope").name().expect("archive entry name"), + "envelope.json" + ); + assert_eq!( + archive.by_index(1).expect("signature").name().expect("archive entry name"), + "envelope.sig" + ); + assert_eq!(archive.by_index(2).expect("result").name().expect("archive entry name"), "result.json"); let envelope: serde_json::Value = serde_json::from_slice(&export.envelope_json).expect("envelope JSON"); assert_eq!(envelope["toolId"], "performance.drive"); @@ -535,7 +541,14 @@ fn production_cli_measures_and_signs_exact_binary_provenance() { let archive_bytes = fs::read(&output).expect("saved archive"); let mut archive = zip::ZipArchive::new(Cursor::new(archive_bytes.as_slice())).expect("signed archive"); let names = (0..archive.len()) - .map(|index| archive.by_index(index).expect("archive member").name().to_owned()) + .map(|index| { + archive + .by_index(index) + .expect("archive member") + .name() + .expect("archive entry name") + .into_owned() + }) .collect::>(); assert_eq!(names, ["envelope.json", "envelope.sig", "result.json"]); let envelope_bytes = read_archive_member(&mut archive, "envelope.json"); diff --git a/rustfs/tests/connect_perf_network.rs b/rustfs/tests/connect_perf_network.rs index 4136a9ca8..6ada066b8 100644 --- a/rustfs/tests/connect_perf_network.rs +++ b/rustfs/tests/connect_perf_network.rs @@ -669,9 +669,26 @@ async fn successful_result_has_signed_bounded_private_offline_export() { assert_eq!(export.archive_sha256, hex(&Sha256::digest(&export.archive_bytes))); let mut archive = zip::ZipArchive::new(Cursor::new(&export.archive_bytes)).expect("three-file archive"); assert_eq!(archive.len(), 3); - assert_eq!(archive.by_index(0).expect("envelope entry").name(), "envelope.json"); - assert_eq!(archive.by_index(1).expect("signature entry").name(), "envelope.sig"); - assert_eq!(archive.by_index(2).expect("result entry").name(), "result.json"); + assert_eq!( + archive + .by_index(0) + .expect("envelope entry") + .name() + .expect("archive entry name"), + "envelope.json" + ); + assert_eq!( + archive + .by_index(1) + .expect("signature entry") + .name() + .expect("archive entry name"), + "envelope.sig" + ); + assert_eq!( + archive.by_index(2).expect("result entry").name().expect("archive entry name"), + "result.json" + ); let envelope: serde_json::Value = serde_json::from_slice(&export.envelope_json).expect("envelope JSON"); assert_eq!(envelope["toolId"], "performance.network"); diff --git a/rustfs/tests/connect_perf_object.rs b/rustfs/tests/connect_perf_object.rs index 62dd99d18..292d74634 100644 --- a/rustfs/tests/connect_perf_object.rs +++ b/rustfs/tests/connect_perf_object.rs @@ -19,20 +19,22 @@ use std::os::unix::fs::PermissionsExt as _; use std::path::Path; use std::process::Command; use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; use std::time::{Duration, SystemTime, UNIX_EPOCH}; use base64_simd::URL_SAFE_NO_PAD; use p256::ecdsa::{Signature, VerifyingKey, signature::Verifier as _}; use p256::pkcs8::DecodePublicKey as _; use rustfs::connect::{ - DeviceIdentity, LocalObjectConsent, OBJECT_CAPABILITY, ObjectOperation, ObjectOutcome, ObjectPerformanceError, - ObjectPerformanceRequest, ObjectProbe, ObjectProbeError, ObjectProbeFuture, ObjectProbeMeasurement, ObjectProvenance, - ObjectReasonCode, ObjectTargetReasonCode, S3ObjectProbe, measure_object, save_signed_object_export, sign_object_export, - validate_object_limits, + DeviceIdentity, LocalObjectConsent, OBJECT_CAPABILITY, ObjectMeasurement, ObjectOperation, ObjectOutcome, + ObjectPerformanceError, ObjectPerformanceRequest, ObjectProbe, ObjectProbeError, ObjectProbeFuture, ObjectProbeMeasurement, + ObjectProvenance, ObjectReasonCode, ObjectTargetReasonCode, S3ObjectProbe, measure_object, save_signed_object_export, + sign_object_export, validate_object_limits, }; use rustfs::embedded::{RustFSServerBuilder, find_available_port}; use sha2::{Digest as _, Sha256}; use tokio_util::sync::CancellationToken; +use tracing::instrument::WithSubscriber as _; use zeroize::Zeroizing; mod common; @@ -500,9 +502,9 @@ async fn cleanup_uses_a_budget_independent_of_the_measurement_window() { "target result: {:?}", measurement.target ); - // The cleanup budget is two sequential requests, each allowed to use the - // probe's bounded cleanup timeout. Leave scheduler and CI runner headroom - // here while keeping the assertion independent from the one-second + // Both sequential cleanup requests share one bounded cleanup deadline. + // Leave scheduler and CI runner headroom while keeping this assertion + // independent from the one-second // measurement window above. tokio::time::timeout(Duration::from_secs(10), server) .await @@ -510,6 +512,251 @@ async fn cleanup_uses_a_budget_independent_of_the_measurement_window() { .expect("response server"); } +#[derive(Clone, Default)] +struct CleanupLog(Arc>>); + +impl std::io::Write for CleanupLog { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.0.lock().expect("cleanup log").extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +#[derive(Clone, Copy)] +enum CleanupReply { + Status(u16), + Disconnect, + Delay(Duration), +} + +struct CleanupObservation { + measurement: ObjectMeasurement, + log: String, + deletes: Vec, + elapsed: Duration, +} + +async fn observe_cleanup(replies: [CleanupReply; 2]) -> CleanupObservation { + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + + const PRIVATE_BODY: &str = "private-cleanup-response-body"; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("cleanup listener"); + let endpoint = format!("http://{}", listener.local_addr().expect("listener address")); + let deletes = Arc::new(Mutex::new(Vec::new())); + let seen_deletes = Arc::clone(&deletes); + let server = tokio::spawn(async move { + for index in 0..4 { + let (mut socket, _) = listener.accept().await.expect("probe connection"); + let mut bytes = Vec::new(); + let (header_end, content_length) = loop { + let mut buffer = [0_u8; 4096]; + let count = socket.read(&mut buffer).await.expect("request headers"); + assert!(count > 0, "complete request headers"); + bytes.extend_from_slice(&buffer[..count]); + assert!(bytes.len() <= 64 * 1024, "bounded request"); + if let Some(offset) = bytes.windows(4).position(|window| window == b"\r\n\r\n") { + let headers = std::str::from_utf8(&bytes[..offset]).expect("ASCII headers"); + let content_length = headers + .lines() + .filter_map(|line| line.split_once(':')) + .find(|(name, _)| name.eq_ignore_ascii_case("content-length")) + .map_or(0, |(_, value)| value.trim().parse::().expect("content length")); + break (offset + 4, content_length); + } + }; + assert!(content_length <= 4, "bounded probe body"); + while bytes.len() < header_end + content_length { + let mut buffer = [0_u8; 4]; + let count = socket.read(&mut buffer).await.expect("request body"); + assert!(count > 0, "complete request body"); + bytes.extend_from_slice(&buffer[..count]); + } + let request_line = std::str::from_utf8(&bytes[..header_end]) + .expect("ASCII request") + .lines() + .next() + .expect("request line") + .to_owned(); + let reply = if index < 2 { + assert!(request_line.starts_with("PUT ")); + CleanupReply::Status(200) + } else { + assert!(request_line.starts_with("DELETE ")); + seen_deletes.lock().expect("seen deletes").push(request_line); + replies[index - 2] + }; + let status = match reply { + CleanupReply::Status(status) => status, + CleanupReply::Disconnect => return, + CleanupReply::Delay(delay) => { + tokio::time::sleep(delay).await; + 204 + } + }; + let body = if status >= 400 { PRIVATE_BODY } else { "" }; + let response = format!( + "HTTP/1.1 {status} Fixture\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + socket.write_all(response.as_bytes()).await.expect("fixture response"); + } + }); + let probe = S3ObjectProbe::new( + &endpoint, + None, + None, + Zeroizing::new("private-cleanup-access".to_owned()), + Zeroizing::new("private-cleanup-secret".to_owned()), + Zeroizing::new("private-cleanup-token".to_owned()), + Duration::from_secs(1), + ) + .expect("cleanup probe"); + let mut request = request(ObjectOperation::PutObject); + request.traffic_bytes = 4; + let log = CleanupLog::default(); + let writer = log.clone(); + let subscriber = tracing_subscriber::fmt() + .with_max_level(tracing::Level::ERROR) + .with_ansi(false) + .without_time() + .with_target(false) + .with_writer(move || writer.clone()) + .finish(); + let started = std::time::Instant::now(); + let measurement = tokio::time::timeout( + Duration::from_secs(8), + measure_object(&request, &probe, &CancellationToken::new()).with_subscriber(subscriber), + ) + .await; + let elapsed = started.elapsed(); + // Reap the fixture before asserting, including stalled or early-failure cases. + server.abort(); + if let Err(error) = server.await { + assert!(error.is_cancelled(), "cleanup fixture failed: {error}"); + } + let measurement = measurement.expect("bounded cleanup probe").expect("typed cleanup result"); + let log = String::from_utf8(log.0.lock().expect("cleanup log").clone()).expect("UTF-8 cleanup log"); + for sensitive in [ + endpoint.as_str(), + "127.0.0.1", + "private-cleanup-access", + "private-cleanup-secret", + "private-cleanup-token", + PRIVATE_BODY, + "rustfs-connect-perf-", + "synthetic-object", + "Authorization", + "X-Amz-Security-Token", + ] { + assert!(!log.contains(sensitive), "cleanup diagnostic exposed private context"); + } + let deletes = deletes.lock().expect("seen deletes").clone(); + CleanupObservation { + measurement, + log, + deletes, + elapsed, + } +} + +fn assert_cleanup_failure(observed: &CleanupObservation, stage: &str, reason: &str, status: Option) { + assert_eq!(observed.measurement.result.outcome(), ObjectOutcome::Failed); + assert_eq!(observed.measurement.target.reason_code, ObjectTargetReasonCode::CleanupFailed); + let events: Vec<_> = observed + .log + .lines() + .filter(|line| line.contains("event=\"connect_object_cleanup_failed\"")) + .collect(); + assert_eq!(events.len(), 1, "one cleanup diagnostic: {}", observed.log); + let event = events[0]; + for field in [ + "component=\"connect\"".to_owned(), + "subsystem=\"performance_object\"".to_owned(), + "result=\"failed\"".to_owned(), + format!("stage=\"{stage}\""), + format!("reason={reason}"), + ] { + assert!(event.contains(&field), "missing {field}: {event}"); + } + match status { + Some(status) => assert!(event.contains(&format!("http_status={status}")), "{event}"), + None => assert!(!event.contains("http_status="), "transport failure has no HTTP status: {event}"), + } + for field in ["elapsed_ms=", "remaining_ms="] { + let value = event + .split_whitespace() + .find_map(|part| part.strip_prefix(field)) + .expect("bounded timing field") + .parse::() + .expect("integer milliseconds"); + if field == "remaining_ms=" { + assert!(value <= 5_000, "cleanup deadline must not be extended"); + } + } +} + +#[tokio::test] +async fn cleanup_diagnostics_identify_each_failed_http_stage_without_private_context() { + let _guard = TEST_LOCK.lock().await; + for (replies, stage, status) in [ + ([CleanupReply::Status(503), CleanupReply::Status(204)], "object_delete", 503), + ([CleanupReply::Status(204), CleanupReply::Status(403)], "bucket_delete", 403), + ([CleanupReply::Status(204), CleanupReply::Status(404)], "bucket_delete", 404), + ] { + let observed = observe_cleanup(replies).await; + assert_cleanup_failure(&observed, stage, "CleanupFailed", Some(status)); + assert_eq!(observed.deletes.len(), if stage == "object_delete" { 1 } else { 2 }); + } +} + +#[tokio::test] +async fn cleanup_diagnostics_identify_each_disconnected_transport_stage() { + let _guard = TEST_LOCK.lock().await; + for (replies, stage) in [ + ([CleanupReply::Disconnect, CleanupReply::Status(204)], "object_delete"), + ([CleanupReply::Status(204), CleanupReply::Disconnect], "bucket_delete"), + ] { + let observed = observe_cleanup(replies).await; + assert_cleanup_failure(&observed, stage, "ProtocolFailure", None); + } +} + +#[tokio::test] +async fn cleanup_accepts_missing_object_and_still_deletes_the_bucket() { + let _guard = TEST_LOCK.lock().await; + let observed = observe_cleanup([CleanupReply::Status(404), CleanupReply::Status(204)]).await; + assert_eq!(observed.measurement.result.outcome(), ObjectOutcome::Succeeded); + assert_eq!(observed.measurement.target.reason_code, ObjectTargetReasonCode::Complete); + assert_eq!(observed.deletes.len(), 2); + assert!(observed.deletes[0].ends_with("/synthetic-object HTTP/1.1")); + assert!(observed.deletes[1].ends_with("/ HTTP/1.1")); + assert!(!observed.log.contains("connect_object_cleanup_failed")); +} + +#[tokio::test] +async fn cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline() { + let _guard = TEST_LOCK.lock().await; + for (replies, stage) in [ + ([CleanupReply::Delay(Duration::from_secs(6)), CleanupReply::Status(204)], "object_delete"), + ( + [ + CleanupReply::Delay(Duration::from_secs(3)), + CleanupReply::Delay(Duration::from_secs(3)), + ], + "bucket_delete", + ), + ] { + let observed = observe_cleanup(replies).await; + assert_cleanup_failure(&observed, stage, "TimedOut", None); + assert!(observed.elapsed >= Duration::from_secs(5)); + assert!(observed.log.contains("remaining_ms=0"), "{}", observed.log); + } +} + #[tokio::test] async fn signed_result_is_saved_without_overwrite() { let _guard = TEST_LOCK.lock().await; @@ -610,6 +857,14 @@ async fn real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put_bod request.artifact_uid = "019e3ae0-0000-7000-8000-000000000016".to_owned(); } let measurement = measure_object(&request, &probe, &CancellationToken::new()) + .with_subscriber( + tracing_subscriber::fmt() + .with_max_level(tracing::Level::ERROR) + .with_ansi(false) + .without_time() + .with_writer(std::io::stderr) + .finish(), + ) .await .expect("real object measurement"); assert_eq!( diff --git a/rustfs/tests/connect_top_net.rs b/rustfs/tests/connect_top_net.rs index 294cd3642..0da443a39 100644 --- a/rustfs/tests/connect_top_net.rs +++ b/rustfs/tests/connect_top_net.rs @@ -399,7 +399,14 @@ fn production_cli_exports_top_net_and_fails_closed_for_unavailable_and_invalid_r assert_eq!(fs::metadata(&net_output).expect("output metadata").permissions().mode() & 0o777, 0o600); let mut archive = ZipArchive::new(Cursor::new(bytes)).expect("top.net archive"); let names = (0..archive.len()) - .map(|index| archive.by_index(index).expect("archive member").name().to_owned()) + .map(|index| { + archive + .by_index(index) + .expect("archive member") + .name() + .expect("archive entry name") + .into_owned() + }) .collect::>(); assert_eq!(names, ["envelope.json", "envelope.sig", "result.json"]); diff --git a/scripts/check_doc_paths.sh b/scripts/check_doc_paths.sh index 0e7772046..143c795b7 100755 --- a/scripts/check_doc_paths.sh +++ b/scripts/check_doc_paths.sh @@ -1,10 +1,10 @@ #!/usr/bin/env bash # # Fail when an agent-instruction or architecture document references a -# repository file path that no longer exists. Keeps CLAUDE.md / AGENTS.md / +# repository file path that no longer exists. Keeps AGENTS.md / # ARCHITECTURE.md / docs/ honest after refactors move code. # -# Checked files: all tracked AGENTS.md, CLAUDE.md, ARCHITECTURE.md, and every +# Checked files: all tracked AGENTS.md, ARCHITECTURE.md, and every # Markdown file under docs/ (architecture, operations, testing, and the index). # # A reference is any token starting with crates/, rustfs/, scripts/, docs/, @@ -21,7 +21,7 @@ cd "$ROOT_DIR" FAILURES=0 doc_files() { - git ls-files 'AGENTS.md' '*/AGENTS.md' 'CLAUDE.md' 'ARCHITECTURE.md' 'docs/*.md' 'docs/architecture/*.md' 'docs/operations/*.md' 'docs/postmortems/*.md' 'docs/testing/*.md' + git ls-files 'AGENTS.md' '*/AGENTS.md' 'ARCHITECTURE.md' 'docs/*.md' 'docs/architecture/*.md' 'docs/operations/*.md' 'docs/postmortems/*.md' 'docs/testing/*.md' } check_file() { diff --git a/scripts/check_offline_enrollment_e2e.sh b/scripts/check_offline_enrollment_e2e.sh index 1cbde5f95..8af74ad64 100755 --- a/scripts/check_offline_enrollment_e2e.sh +++ b/scripts/check_offline_enrollment_e2e.sh @@ -46,13 +46,17 @@ cleanup() trap cleanup EXIT trap 'exit 130' HUP INT TERM -export CARGO_TARGET_DIR="${task_root}/target" +# The binaries land in the workspace target directory so CI reuses its restored +# dependency cache. The E2E root is isolated by the feature gate and the +# dedicated binary name, not by where the build happens; only the challenge, +# response and key files are task-local. +target_dir="${CARGO_TARGET_DIR:-${repo_root}/target}" feature=offline-enrollment-e2e-root cargo build --locked -p rustfs --bin rustfs-cli-e2e --features "${feature}" cargo build --locked -p rustfs --bin rustfs-cli --features "${feature}" cargo test --locked -p rustfs --test connect_offline_enrollment --features "${feature}" e2e_ -"${CARGO_TARGET_DIR}/debug/rustfs-cli-e2e" connect offline enroll \ +"${target_dir}/debug/rustfs-cli-e2e" connect offline enroll \ --challenge "${challenge_fixture}" \ --output "${task_root}/e2e-response.json" \ --key-dir "${task_root}/e2e-key" \ @@ -60,7 +64,7 @@ cargo test --locked -p rustfs --test connect_offline_enrollment --features "${fe test -s "${task_root}/e2e-response.json" \ || { echo 'offline enrollment E2E gate: dedicated CLI produced no response' >&2; exit 1; } -if "${CARGO_TARGET_DIR}/debug/rustfs-cli" connect offline enroll \ +if "${target_dir}/debug/rustfs-cli" connect offline enroll \ --challenge "${challenge_fixture}" \ --output "${task_root}/production-response.json" \ --key-dir "${task_root}/production-key" \ diff --git a/scripts/check_test_wiring.py b/scripts/check_test_wiring.py index 3a8251775..65899459c 100755 --- a/scripts/check_test_wiring.py +++ b/scripts/check_test_wiring.py @@ -699,6 +699,10 @@ def check_ilm_build_budget(root: Path) -> list[str]: if step is None: return [".github/workflows/ci.yml: missing ignored ILM integration step"] + # Measured budget: 3 build jobs on main pushes and dispatches, 2 on pull + # requests. A pull-request run at 3 (39193827281) spent 70 minutes with the + # pod's io.pressure "full" at 86-88% and timed out before any test started; + # the shared pod storage, not CPU, bounds the PR lane. budget = ( "CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || " "github.event_name == 'workflow_dispatch') && '3' || '2' }}" @@ -5596,6 +5600,8 @@ class SelfTests(unittest.TestCase): workflow.write_text(valid.replace(" CARGO_BUILD_JOBS:", " BUILD_JOBS:")) self.assertEqual(len(check_ilm_build_budget(root)), 1) for old, new in ( + ("&& '3' || '2' }}", "&& '3' || '3' }}"), + ("&& '3' || '2' }}", "&& '2' || '2' }}"), ("test-and-lint:", "test-ilm-integration-serial:"), ("outputs.mode == 'full'", "outputs.mode == 'docs'"), ("--profile default", "--profile ci"), diff --git a/scripts/ci_gate.py b/scripts/ci_gate.py index b99c10927..872e1bc0f 100644 --- a/scripts/ci_gate.py +++ b/scripts/ci_gate.py @@ -16,13 +16,14 @@ import unittest ROOT = Path(__file__).resolve().parent.parent ALWAYS_JOBS = ("classify-changes", "typos", "quick-checks") CODE_JOBS = ( - "test-and-lint", "offline-enrollment-e2e", "test-and-lint-rio-v2", - "connect-short-credential-boundary", "test-and-lint-protocols", - "build-rustfs-debug-binary", "uring-integration", "e2e-tests", - "s3-implemented-tests", "s3-lifecycle-behavior-tests", + "test-and-lint", "connect-feature-boundaries", "test-and-lint-rio-v2", + "test-and-lint-protocols", "build-rustfs-debug-binary", "uring-integration", + "e2e-tests", "s3-implemented-tests", "s3-lifecycle-behavior-tests", ) -# Full alternate-feature tests remain mandatory on main, merge groups, and -# scheduled/manual runs, without repeating four builds on every PR revision. +# Full alternate-feature tests run on merge groups and on scheduled/manual +# runs. Pull requests and main pushes skip them: the nightly run reports a +# feature regression within a day, and repeating four builds per PR revision +# or per merge bought no earlier signal. EXTENDED_FEATURE_JOBS = ("test-and-lint-rio-v2", "test-and-lint-protocols") # These jobs still exercise the server and its black-box harness when only E2E # Rust sources change. Production code, manifests and shared test configuration @@ -130,7 +131,7 @@ def expected_results(mode: str, event: str, ref: str) -> dict[str, str]: raise ValueError(f"invalid CI selection: {mode!r} for {event!r}") expected = {job: "success" for job in ALWAYS_JOBS} expected.update({job: "success" if mode == "full" or (mode == "e2e" and job in E2E_JOBS) else "skipped" for job in CODE_JOBS}) - if event == "pull_request": + if event in ("pull_request", "push"): expected.update({job: "skipped" for job in EXTENDED_FEATURE_JOBS}) rio = mode == "full" and event in ("schedule", "workflow_dispatch") expected.update({job: "success" if rio else "skipped" for job in OPTIONAL_JOBS[:2]}) @@ -222,12 +223,14 @@ def check_workflow(root: Path) -> list[str]: errors.append("required-checks must run after failed or skipped dependencies") for job in EXTENDED_FEATURE_JOBS: block = yaml_block(jobs, job, 2) or [] - if scalar(block, "if", 4) != "needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'": - errors.append(f"{job} must retain full validation on every non-PR event") + if scalar(block, "if", 4) != "needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('[\"merge_group\", \"schedule\", \"workflow_dispatch\"]'), github.event_name)": + errors.append(f"{job} must run on every merge-group, scheduled and manual event") workspace = "\n".join(yaml_block(jobs, "test-and-lint", 2) or []) clippy = re.search(r"^ - name: Run clippy lints\n(?P.*?)(?=^ - name:|\Z)", workspace, re.MULTILINE | re.DOTALL) - if clippy is None or scalar(clippy["body"].splitlines(), "if", 8) != "github.event_name != 'pull_request'": - errors.append("workspace Clippy must run on every non-PR event") + clippy_lines = clippy["body"].splitlines() if clippy else [] + if (clippy is None or any(re.match(r'''\s{8}["']?if["']?\s*:''', line) for line in clippy_lines) + or "cargo clippy --all-targets -- -D warnings" not in clippy["body"]): + errors.append("workspace Clippy must run on every event, pull requests included") if scalar(gate, "shell", 8) != "bash" or scalar(gate, "run", 8) != "python3 scripts/ci_gate.py verify" or scalar(gate, "CI_NEEDS", 10) != "${{ toJSON(needs) }}": errors.append("required-checks must verify the actual needs results") if any(re.match(r'''\s+(?:- )?(?:["']?continue-on-error["']?\s*:|["']?if["']?\s*:)''', line) and not line.startswith(" if:") for line in gate): @@ -335,7 +338,8 @@ class SelfTests(unittest.TestCase): self.assertEqual({job for job, state in e2e.items() if state == "success"}, set(ALWAYS_JOBS + E2E_JOBS)) for event in ("schedule", "workflow_dispatch", "merge_group", "push"): result = expected_results("full", event, "refs/heads/main") - self.assertTrue(all(result[job] == "success" for job in CODE_JOBS)) + for job in CODE_JOBS: + self.assertEqual(result[job], "skipped" if event == "push" and job in EXTENDED_FEATURE_JOBS else "success", (event, job)) self.assertEqual(result["e2e-full"], "skipped" if event == "schedule" else "success") self.assertEqual(result["e2e-tests-rio-v2"], "success" if event in ("schedule", "workflow_dispatch") else "skipped") with self.assertRaises(ValueError): @@ -391,15 +395,17 @@ class SelfTests(unittest.TestCase): (root / ".github/workflows").mkdir(parents=True) source = (ROOT / ".github/workflows/ci.yml").read_text() path = root / ".github/workflows/ci.yml" - for condition in ("false", "github.event_name == 'schedule'", "github.event_name == 'pull_request'"): - path.write_text(source.replace(" - name: Run clippy lints\n if: github.event_name != 'pull_request'", f" - name: Run clippy lints\n if: {condition}")) - self.assertIn("workspace Clippy must run on every non-PR event", check_workflow(root)) + for condition in ("false", "github.event_name == 'schedule'", "github.event_name != 'pull_request'"): + path.write_text(source.replace(" - name: Run clippy lints\n", f" - name: Run clippy lints\n if: {condition}\n", 1)) + self.assertIn("workspace Clippy must run on every event, pull requests included", check_workflow(root)) + path.write_text(source.replace("cargo clippy --all-targets -- -D warnings", "cargo clippy --all-targets", 1)) + self.assertIn("workspace Clippy must run on every event, pull requests included", check_workflow(root)) for job in EXTENDED_FEATURE_JOBS: before, block = source.split(f" {job}:\n", 1) for condition in ("false", "github.event_name == 'schedule'", "github.event_name != 'pull_request'"): changed = re.sub(r"^ if: .*", f" if: {condition}", block, count=1, flags=re.MULTILINE) path.write_text(before + f" {job}:\n" + changed) - self.assertIn(f"{job} must retain full validation on every non-PR event", check_workflow(root)) + self.assertIn(f"{job} must run on every merge-group, scheduled and manual event", check_workflow(root)) for job in ALWAYS_JOBS + CODE_JOBS + OPTIONAL_JOBS: before, gate = source.split(" required-checks:\n", 1) path.write_text(before + " required-checks:\n" + gate.replace(f" - {job}\n", "", 1)) diff --git a/scripts/error-other-format-baseline.txt b/scripts/error-other-format-baseline.txt index 7cbac441b..05c7212b2 100644 --- a/scripts/error-other-format-baseline.txt +++ b/scripts/error-other-format-baseline.txt @@ -68,4 +68,4 @@ 10|crates/ecstore/src/store/init.rs 3|crates/ecstore/src/store/multipart.rs 6|crates/ecstore/src/store/object.rs -5|crates/ecstore/src/store/rebalance/support.rs +3|crates/ecstore/src/store/rebalance/support.rs diff --git a/scripts/release/check_glibc.py b/scripts/release/check_glibc.py new file mode 100644 index 000000000..7bf618d36 --- /dev/null +++ b/scripts/release/check_glibc.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Reject GNU artifacts whose ELF version needs exceed the package ABI floor.""" + +from __future__ import annotations + +import argparse +import os +import re +import subprocess +from pathlib import Path + + +def version(value: str) -> tuple[int, ...]: + if not re.fullmatch(r"[0-9]+(?:\.[0-9]+)+", value): + raise ValueError(f"invalid glibc version: {value}") + parts = tuple(map(int, value.split("."))) + return parts + (0,) * max(0, 3 - len(parts)) + + +def check(binary: Path, baseline: str) -> str: + floor = version(baseline) + result = subprocess.run( + ["readelf", "--version-info", "--wide", str(binary)], + check=True, capture_output=True, text=True, env={**os.environ, "LC_ALL": "C"}, + ) + # Only loader requirements matter. Definitions and arbitrary strings in + # the executable do not describe the libc that it needs to run. + needs = result.stdout.partition("Version needs section")[2] + symbols = re.findall(r"\bName:\s+(GLIBC_\S+)", needs) + if not symbols: + raise ValueError(f"{binary}: no GLIBC version needs found in GNU artifact") + highest = "GLIBC_0.0" + for symbol in symbols: + required = version(symbol[len("GLIBC_"):]) + if required > floor: + raise ValueError(f"{binary}: requires {symbol}, above GLIBC_{baseline}") + if required > version(highest[len("GLIBC_"):]): + highest = symbol + return highest + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("baseline") + parser.add_argument("binaries", type=Path, nargs="+") + args = parser.parse_args() + try: + for binary in args.binaries: + print(f"{binary}: highest required symbol {check(binary, args.baseline)} <= GLIBC_{args.baseline}") + except (OSError, ValueError, subprocess.CalledProcessError) as error: + parser.exit(1, f"{error}\n") + + +if __name__ == "__main__": + main() diff --git a/scripts/s3-tests/README.md b/scripts/s3-tests/README.md index c7bebde87..ed6aefb12 100644 --- a/scripts/s3-tests/README.md +++ b/scripts/s3-tests/README.md @@ -260,6 +260,17 @@ server's default. Set both key IDs to keys that already exist, and set templates should include `kms_keyid`, `kms_keyid2`, and `kms_default_keyid`, or the commented placeholders from the bundled template. +### Conditional write contract + +The conditional-write patch keeps the upstream tests in the gate while aligning +their requests with the [S3 write contract](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html): +`If-None-Match` accepts `*`, and an existing current object returns 412. RustFS +returns 501 `NotImplemented` for unsupported ETag-valued `If-None-Match`; the S3 +documentation does not specify that error code for PutObject or +CompleteMultipartUpload. The patched tests assert rejection without replacing +object contents, successful creation after a delete marker, and the existing +`If-Match` checks. No conditional-write tests are excluded. + ### Custom Configuration Files ```bash diff --git a/scripts/s3-tests/patches/0006-conditional-write-wildcard-contract.patch b/scripts/s3-tests/patches/0006-conditional-write-wildcard-contract.patch new file mode 100644 index 000000000..7ab4f0139 --- /dev/null +++ b/scripts/s3-tests/patches/0006-conditional-write-wildcard-contract.patch @@ -0,0 +1,163 @@ +diff --git a/s3tests/functional/test_s3.py b/s3tests/functional/test_s3.py +index e28021325..b1187e44e 100644 +--- a/s3tests/functional/test_s3.py ++++ b/s3tests/functional/test_s3.py +@@ -3203,11 +3203,12 @@ def test_put_object_ifnonmatch_good(): + + lf = (lambda **kwargs: kwargs['params']['headers'].update({'If-None-Match': 'ABCORZ'})) + client.meta.events.register('before-call.s3.PutObject', lf) +- response = client.put_object(Bucket=bucket_name,Key='foo', Body='zar') ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key='foo', Body='zar') ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) + + response = client.get_object(Bucket=bucket_name, Key='foo') + body = _get_body(response) +- assert body == 'zar' ++ assert body == 'bar' + + @pytest.mark.fails_on_aws + @pytest.mark.fails_on_dbstore +@@ -3227,8 +3228,8 @@ def test_put_object_ifnonmatch_failed(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key='foo', Body='zar') + + status, error_code = _get_status_and_error_code(e.response) +- assert status == 412 +- assert error_code == 'PreconditionFailed' ++ assert status == 501 ++ assert error_code == 'NotImplemented' + + response = client.get_object(Bucket=bucket_name, Key='foo') + body = _get_body(response) +@@ -19498,10 +19499,9 @@ def test_put_object_if_match(): + + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch='*') + assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag) +- assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch='badetag') +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag, 'badetag'): ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=unsupported) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) + + client.put_object(Bucket=bucket, Key=key, IfMatch=etag) + +@@ -19512,7 +19512,9 @@ def test_put_object_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfMatch='badetag') + assert (404, 'NoSuchKey') == _get_status_and_error_code(e.response) + +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch=etag) ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) ++ response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + response = client.put_object(Bucket=bucket, Key=key, IfMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] +@@ -19560,11 +19562,8 @@ def test_multipart_put_object_if_match(): + etag = response['ETag'] + + failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch='*') +- failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch=etag) +- +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch='badetag') +- etag = response['ETag'] +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag, 'badetag'): ++ failing_conditional_multipart_upload((501, 'NotImplemented'), client, bucket, key, IfNoneMatch=unsupported) + + response = successful_conditional_multipart_upload(client, bucket, key, IfMatch=etag) + etag = response['ETag'] +@@ -19574,7 +19573,8 @@ def test_multipart_put_object_if_match(): + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='*') + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='badetag') + +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch=etag) ++ failing_conditional_multipart_upload((501, 'NotImplemented'), client, bucket, key, IfNoneMatch=etag) ++ response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch='*') + etag = response['ETag'] + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + response = successful_conditional_multipart_upload(client, bucket, key, IfMatch='*') +@@ -19602,16 +19602,16 @@ def test_put_current_object_if_none_match(): + + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch='*') + assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag2) +- assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) +- +- # we can't specify a version, so we only check against current object +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch=etag) +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] +- response = client.put_object(Bucket=bucket, Key=key, IfNoneMatch='badetag') +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag2, etag, 'badetag'): ++ e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=unsupported) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data2 + ++ # If-None-Match checks the current version; a delete marker permits creation. + client.delete_object(Bucket=bucket, Key=key) ++ response = client.put_object(Bucket=bucket, Key=key, Body=data1, IfNoneMatch='*') ++ assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data1 + + @pytest.mark.conditional_write + @pytest.mark.fails_on_dbstore +@@ -19630,15 +19630,15 @@ def test_multipart_put_current_object_if_none_match(): + etag2 = response['ETag'] + + failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch='*') +- failing_conditional_multipart_upload((412, 'PreconditionFailed'), client, bucket, key, IfNoneMatch=etag2) +- +- # we can't specify a version, so we only check against current object +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch=etag) +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] +- response = successful_conditional_multipart_upload(client, bucket, key, IfNoneMatch='badetag') +- assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ for unsupported in (etag2, etag, 'badetag'): ++ failing_conditional_multipart_upload((501, 'NotImplemented'), client, bucket, key, IfNoneMatch=unsupported) ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data2 + ++ # If-None-Match checks the current version; a delete marker permits creation. + client.delete_object(Bucket=bucket, Key=key) ++ response = successful_conditional_multipart_upload(client, bucket, key, data1, IfNoneMatch='*') ++ assert 200 == response['ResponseMetadata']['HTTPStatusCode'] ++ assert _get_body(client.get_object(Bucket=bucket, Key=key)) == data1 + + @pytest.mark.conditional_write + @pytest.mark.fails_on_dbstore +@@ -19656,7 +19656,7 @@ def test_put_current_object_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfMatch='badetag') + assert (404, 'NoSuchKey') == _get_status_and_error_code(e.response) + +- response = client.put_object(Bucket=bucket, Key=key, Body=data1, IfNoneMatch=etag) ++ response = client.put_object(Bucket=bucket, Key=key, Body=data1, IfNoneMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + etag = response['ETag'] + versionId = response['VersionId'] +@@ -19685,7 +19685,7 @@ def test_multipart_put_current_object_if_match(): + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='*') + failing_conditional_multipart_upload((404, 'NoSuchKey'), client, bucket, key, IfMatch='badetag') + +- response = successful_conditional_multipart_upload(client, bucket, key, data1, IfNoneMatch=etag) ++ response = successful_conditional_multipart_upload(client, bucket, key, data1, IfNoneMatch='*') + assert 200 == response['ResponseMetadata']['HTTPStatusCode'] + etag = response['ETag'] + versionId = response['VersionId'] +@@ -19711,7 +19711,7 @@ def test_put_object_current_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch='*') + assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfNoneMatch=etag) +- assert (412, 'PreconditionFailed') == _get_status_and_error_code(e.response) ++ assert (501, 'NotImplemented') == _get_status_and_error_code(e.response) + + client.put_object(Bucket=bucket, Key=key, IfMatch=etag) + +@@ -19723,7 +19723,7 @@ def test_put_object_current_if_match(): + e = assert_raises(ClientError, client.put_object, Bucket=bucket, Key=key, IfMatch='badetag') + assert (404, 'NoSuchKey') == _get_status_and_error_code(e.response) + +- client.put_object(Bucket=bucket, Key=key, IfNoneMatch=etag) ++ client.put_object(Bucket=bucket, Key=key, IfNoneMatch='*') + + @pytest.mark.fails_on_aws # only supported for directory buckets + @pytest.mark.conditional_write diff --git a/scripts/security/check_preview_release_workflow.sh b/scripts/security/check_preview_release_workflow.sh index 54bf2ef50..7f6b4d05b 100755 --- a/scripts/security/check_preview_release_workflow.sh +++ b/scripts/security/check_preview_release_workflow.sh @@ -32,10 +32,10 @@ require_absent() { fi } -require_line "$build_workflow" " cargo zigbuild --release --target \${{ matrix.target }} \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin \"\$binary\"" "cross builds must keep matrix features" -require_line "$build_workflow" " cargo build --release --target \${{ matrix.target }} \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin \"\$binary\"" "native builds must keep matrix features" -require_absent "$build_workflow" " cargo zigbuild --release --target \${{ matrix.target }} -p rustfs --bin \"\$binary\"" "cross builds must not drop matrix features" -require_absent "$build_workflow" " cargo build --release --target \${{ matrix.target }} -p rustfs --bin \"\$binary\"" "native builds must not drop matrix features" +require_line "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "cross builds must keep matrix features" +require_line "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "native builds must keep matrix features" +require_absent "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "cross builds must not drop matrix features" +require_absent "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "native builds must not drop matrix features" extract_job_if() { local file="$1" diff --git a/scripts/table-catalog/README.md b/scripts/table-catalog/README.md index b1b42df83..5a4a16244 100644 --- a/scripts/table-catalog/README.md +++ b/scripts/table-catalog/README.md @@ -55,6 +55,29 @@ metadata, mismatched expected and observed status, invalid row counts, or claim promotion beyond the client boundary are treated as evidence failures instead of support-matrix proof. +The smoke reads `rustfs.catalog-backing` from `/v1/config`, applying overrides +after defaults, before creating objects. `--catalog-backing` is an optional +expected value, not a way to relabel the deployment. A mismatch or an unknown +server backing aborts the run. Object mode uses the wire value `object`; the +legacy operator label `object-backed` is accepted as an expected-value alias. +Durable-strong must already be provisioned through the +[cutover procedure](../../docs/operations/s3-tables-cutover-runbook.md). +`--client-version` likewise checks the installed PyIceberg version rather than +overriding the version recorded in evidence. A mismatch aborts before setup. + +New PyIceberg evidence includes `catalog_probes`, `rest_signing_name`, and +`credential_vending_required`. Probe results distinguish `pass`, +`expected-unsupported`, and `skipped`. Both backings exercise maintenance +configuration, scheduler/worker dry-run, job inspection, quarantine inspection, +diagnostics, and export. Only the direct metadata-maintenance endpoint is +expected to reject durable-strong requests. A completed dry-run must retain its +queued job identity, report zero deletions, and leave the commit state unchanged; +the smoke restores its disabled, paused configuration after execution. +New full runs use `create-append-reload-scan-backing-contracts-v2`; historical +scenario names retain their original scope. +Explicitly skipping REST probes records only `create-append-reload-scan-core`; +it cannot establish extended REST compatibility. + The smoke test covers: - create or reuse the S3 bucket @@ -62,10 +85,18 @@ The smoke test covers: - load the PyIceberg REST catalog - create namespace and table - append two rows through PyIceberg -- reload and scan the table +- reload and compare the complete scanned rows, not only the row count - probe direct REST catalog endpoints for metadata-location, table refs, Iceberg views, maintenance config, metadata maintenance, worker run, and catalog diagnostics +- update namespace properties, reject overlapping update/removal keys without + changing persisted properties, and verify rename preserves table metadata +- round-trip an encoded multi-level namespace and reject double-encoded aliases +- check LoadTable does not vend credentials for missing or unknown delegation + tokens; when vending is required, check exact-token negotiation and scope +- on both backings, check a successful maintenance dry-run and export against + the current pointer, token, and generation; in durable-strong mode, also + verify the direct metadata-maintenance rejection leaves commit state unchanged - optionally drop the table and namespace The default profile uses the canonical RustFS catalog URI: @@ -139,7 +170,9 @@ temporary credentials: rejected. The direct REST catalog probes run by default after the PyIceberg append and -scan. For deployments that intentionally expose only the core Iceberg REST +scan. Both backings run maintenance control, diagnostics, and export probes. +Durable-strong checks the direct metadata-maintenance rejection separately. +For deployments that intentionally expose only the core Iceberg REST Catalog table path, skip those probes explicitly: ```bash diff --git a/scripts/table-catalog/duckdb_smoke.py b/scripts/table-catalog/duckdb_smoke.py index 5b89ce399..0459dd78c 100755 --- a/scripts/table-catalog/duckdb_smoke.py +++ b/scripts/table-catalog/duckdb_smoke.py @@ -41,6 +41,7 @@ class DuckDBSmokeResult: row_count: int cleanup_result: str checks: dict[str, str] + catalog_backing: str def parse_args(argv: list[str] | None = None) -> argparse.Namespace: @@ -335,6 +336,7 @@ def pyiceberg_args(args: argparse.Namespace) -> argparse.Namespace: require_vended_credentials=False, timeout=args.timeout, insecure=args.insecure, + catalog_backing=args.catalog_backing, ) @@ -417,6 +419,7 @@ def run_smoke(args: argparse.Namespace, deps: pyiceberg_smoke.RuntimeDeps) -> Du pyiceberg_smoke.ensure_local_proxy_bypass(endpoint) pyiceberg_smoke.ensure_aws_env(args.access_key, args.secret_key, args.region) iceberg_args = pyiceberg_args(args) + backing = pyiceberg_smoke.discover_catalog_backing(iceberg_args, deps) pyiceberg_smoke.ensure_bucket(iceberg_args, deps) pyiceberg_smoke.enable_table_bucket(iceberg_args, deps) @@ -581,7 +584,7 @@ def run_smoke(args: argparse.Namespace, deps: pyiceberg_smoke.RuntimeDeps) -> Du drop_namespace=not namespace_preexisting, ) - return DuckDBSmokeResult(client_version, metadata_location, 2, cleanup_result, checks) + return DuckDBSmokeResult(client_version, metadata_location, 2, cleanup_result, checks, backing) def current_utc_timestamp() -> str: @@ -598,7 +601,7 @@ def write_live_evidence(args: argparse.Namespace, result: DuckDBSmokeResult) -> scenario="rest-catalog-single-table-read-write-cross-engine-negative-boundaries", rustfs_build=args.rustfs_build, git_sha=args.git_sha, - catalog_backing=args.catalog_backing, + catalog_backing=result.catalog_backing, endpoint=args.endpoint, warehouse=args.bucket, rest_path="/iceberg", diff --git a/scripts/table-catalog/engine_compatibility.py b/scripts/table-catalog/engine_compatibility.py index 5b7de88c0..4743bcba0 100644 --- a/scripts/table-catalog/engine_compatibility.py +++ b/scripts/table-catalog/engine_compatibility.py @@ -18,6 +18,8 @@ DEFAULT_ICEBERG_VERSION = "1.7.1" DEFAULT_SCALA_VERSION = "2.12" DEFAULT_TRINO_VERSION = "477" DEFAULT_DUCKDB_VERSION = "1.5.5" +PYICEBERG_BACKING_SCENARIO = "create-append-reload-scan-backing-contracts-v2" +PYICEBERG_CORE_SCENARIO = "create-append-reload-scan-core" DEFAULT_SNOWFLAKE_CLIENT_VERSION = "operator-recorded" DEFAULT_DATABEND_VERSION = "operator-recorded" DEFAULT_TRINO_SERVER = "http://127.0.0.1:8080" @@ -738,6 +740,21 @@ def live_conformance_evidence( ) +def pyiceberg_catalog_probe_results(backing: str, *, skipped: bool, vended: bool) -> dict[str, str]: + if backing not in {"object", "durable-strong"}: + raise ValueError("PyIceberg probe evidence requires an observed RustFS backing") + if skipped: + return {"direct-rest": "skipped"} + probes = dict.fromkeys(("metadata-location", "refs", "views-and-namespace-properties", "rename", "load-table-delegation"), "pass") + probes["load-table-vended-credentials"] = "pass" if vended else "skipped" + probes.update(dict.fromkeys(( + "maintenance-config", "maintenance-scheduler", "maintenance-worker-dry-run", + "maintenance-job", "maintenance-quarantine-inspect", "diagnostics", "export", + ), "pass")) + probes["metadata-maintenance"] = "expected-unsupported" if backing == "durable-strong" else "pass" + return probes + + def live_conformance_evidence_schema() -> OrderedDict[str, Any]: return OrderedDict( [ @@ -879,6 +896,18 @@ def validate_live_conformance_evidence(record: dict[str, Any]) -> OrderedDict[st if expected_status == "pass" and client_name in LIVE_EVIDENCE_ROW_COUNT_CLIENTS and row_count != 2: raise ValueError("live conformance evidence row_count must be 2 for successful table smoke/read probes") + if client_name == "PyIceberg" and record["scenario"] in {PYICEBERG_CORE_SCENARIO, PYICEBERG_BACKING_SCENARIO}: + vended = record.get("credential_vending_required") + if not isinstance(vended, bool): + raise ValueError("PyIceberg probe evidence requires the credential negotiation mode") + expected_probes = pyiceberg_catalog_probe_results( + record["catalog_backing"], skipped=record["scenario"] == PYICEBERG_CORE_SCENARIO, vended=vended + ) + if record.get("catalog_probes") != expected_probes: + raise ValueError("PyIceberg probe evidence does not match the scenario and backing capabilities") + if record.get("rest_signing_name") not in {"s3", "s3tables"}: + raise ValueError("PyIceberg probe evidence requires the REST signing name") + return OrderedDict( [ ("status", "accepted"), diff --git a/scripts/table-catalog/pyiceberg_smoke.py b/scripts/table-catalog/pyiceberg_smoke.py index ebf7b5de2..911c994bc 100755 --- a/scripts/table-catalog/pyiceberg_smoke.py +++ b/scripts/table-catalog/pyiceberg_smoke.py @@ -27,6 +27,9 @@ import failure_coverage DEFAULT_PROFILE = "rustfs" CATALOG_VENDED_PROFILE = "rustfs-vended-credentials" VENDED_CREDENTIAL_PROFILES = {CATALOG_VENDED_PROFILE} +CATALOG_BACKING_KEY = "rustfs.catalog-backing" +CATALOG_BACKINGS = {"object", "durable-strong"} +SMOKE_ROWS = [{"id": 1, "payload": "alpha"}, {"id": 2, "payload": "beta"}] REQUIRED_STORAGE_CREDENTIAL_KEYS = ( "s3.access-key-id", "s3.secret-access-key", @@ -200,7 +203,7 @@ UNSUPPORTED_INVENTORY: list[dict[str, str]] = [ "status": "controlled-run-once-supported", "roadmap_area": "maintenance-worker", "catalog_endpoint": "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/maintenance/scheduler", - "expected_behavior": "background-enabled maintenance can be queued by the scheduler run endpoint, claimed by the worker run endpoint, and inspected through the scheduler status endpoint with disabled/paused state, queued-job handoff, current-job backpressure, retry deferral, quarantine boundary, audit timeline, per-job audit events, lease expiry recovery, heartbeat updates, and operator quarantine inspect/release/retry/abandon actions; built-in periodic scheduling is not claimed", + "expected_behavior": "background-enabled maintenance can be queued by the scheduler run endpoint, claimed by the worker run endpoint, and inspected through the scheduler status endpoint with disabled/paused state, queued-job handoff, current-job backpressure, retry deferral, quarantine boundary, audit timeline, per-job audit events, lease expiry recovery, heartbeat updates, and operator quarantine inspect/release/retry/abandon actions; opt-in periodic admission can queue jobs for allowlisted buckets, while workers still require an explicit operator invocation", }, { "capability": "manifest-data-reachability-cleanup", @@ -212,13 +215,13 @@ UNSUPPORTED_INVENTORY: list[dict[str, str]] = [ "capability": "snapshot-expiration", "status": "manual-maintenance-supported", "roadmap_area": "snapshot-maintenance", - "expected_behavior": "metadata maintenance can report snapshot expiration plans and manually commit safe snapshot expiration through the catalog; background scheduling and manifest/data cleanup are not claimed", + "expected_behavior": "metadata maintenance can report snapshot expiration plans and manually commit safe snapshot expiration through the catalog; automatic snapshot expiration and background manifest/data cleanup are not claimed", }, { "capability": "compaction-rewrite", "status": "controlled-run-once-supported", "roadmap_area": "snapshot-maintenance", - "expected_behavior": "metadata maintenance can plan binpack candidates and commit a safe partition-local and sort-order-preserving Parquet rewrite through the catalog; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; built-in periodic scheduling, delete-file rewrite, and row-level compaction execution are not claimed", + "expected_behavior": "metadata maintenance can plan binpack candidates and commit a safe partition-local and sort-order-preserving Parquet rewrite through the catalog; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; automatic compaction execution, delete-file rewrite, and row-level compaction execution are not claimed", }, { "capability": "row-level-delete-update-merge", @@ -283,6 +286,8 @@ class SmokeResult: row_count: int cleanup_result: str table_warehouse_location: str + catalog_backing: str + catalog_probes: dict[str, str] class RestRequestError(RuntimeError): @@ -536,7 +541,15 @@ def sign_rest_request(args: argparse.Namespace, deps: RuntimeDeps, request: Any) deps.botocore_auth(credentials, args.rest_signing_name, args.region).add_auth(request) -def signed_rest_request(args: argparse.Namespace, deps: RuntimeDeps, method: str, path: str, body: dict[str, Any] | None = None) -> dict[str, Any]: +def signed_rest_request( + args: argparse.Namespace, + deps: RuntimeDeps, + method: str, + path: str, + body: dict[str, Any] | None = None, + *, + access_delegation: str | None = None, +) -> dict[str, Any]: endpoint = normalized_endpoint(args.endpoint) url = f"{endpoint}{path}" payload = b"" if body is None else json.dumps(body, separators=(",", ":")).encode("utf-8") @@ -546,6 +559,8 @@ def signed_rest_request(args: argparse.Namespace, deps: RuntimeDeps, method: str } if body is not None: headers["content-type"] = "application/json" + if access_delegation is not None: + headers["X-Iceberg-Access-Delegation"] = access_delegation aws_request = deps.botocore_awsrequest(method=method, url=url, data=payload, headers=headers) sign_rest_request(args, deps, aws_request) @@ -586,6 +601,21 @@ def signed_rest_request_expect_error( raise RuntimeError(f"{method} {path} unexpectedly succeeded") +def discover_catalog_backing(args: argparse.Namespace, deps: RuntimeDeps) -> str: + query = urllib.parse.urlencode({"warehouse": profile_warehouse(args)}) + response = signed_rest_request(args, deps, "GET", f"{args.rest_path}/v1/config?{query}") + defaults, overrides = response.get("defaults"), response.get("overrides") + if not isinstance(defaults, dict) or not isinstance(overrides, dict): + raise RuntimeError("catalog config must return defaults and overrides") + backing = overrides.get(CATALOG_BACKING_KEY, defaults.get(CATALOG_BACKING_KEY)) + if not isinstance(backing, str) or backing not in CATALOG_BACKINGS: + raise RuntimeError("catalog config did not identify a supported RustFS backing") + expected = "object" if args.catalog_backing == "object-backed" else args.catalog_backing + if expected != "operator-recorded" and expected != backing: + raise RuntimeError("configured catalog backing does not match the server config") + return backing + + def ensure_bucket(args: argparse.Namespace, deps: RuntimeDeps) -> None: client = configured_s3_client(args, deps) try: @@ -972,12 +1002,13 @@ def current_utc_timestamp() -> str: def pyiceberg_client_version(args: argparse.Namespace) -> str: - if args.client_version: - return args.client_version try: - return importlib.metadata.version("pyiceberg") + version = importlib.metadata.version("pyiceberg") except importlib.metadata.PackageNotFoundError: - return "operator-recorded" + version = "operator-recorded" + if args.client_version and args.client_version != version: + raise RuntimeError(f"expected PyIceberg {args.client_version}, found {version}") + return version def redacted_command(argv: list[str]) -> str: @@ -1012,10 +1043,12 @@ def pyiceberg_live_evidence_record( operator: str, command: str, ) -> dict[str, Any]: - return engine_compatibility.live_conformance_evidence_record( + if catalog_backing != result.catalog_backing: + raise ValueError("evidence backing must match the observed server backing") + record = engine_compatibility.live_conformance_evidence_record( client_name="PyIceberg", client_version=client_version, - scenario="create-append-reload-scan-direct-rest-probes", + scenario=engine_compatibility.PYICEBERG_CORE_SCENARIO if args.skip_catalog_api_probes else engine_compatibility.PYICEBERG_BACKING_SCENARIO, rustfs_build=rustfs_build, git_sha=git_sha, catalog_backing=catalog_backing, @@ -1034,6 +1067,10 @@ def pyiceberg_live_evidence_record( claim="automated-smoke", command=command, ) + record["catalog_probes"] = result.catalog_probes.copy() + record["rest_signing_name"] = args.rest_signing_name + record["credential_vending_required"] = bool(args.require_vended_credentials) + return record def write_live_evidence(args: argparse.Namespace, result: SmokeResult) -> None: @@ -1045,7 +1082,7 @@ def write_live_evidence(args: argparse.Namespace, result: SmokeResult) -> None: client_version=pyiceberg_client_version(args), rustfs_build=args.rustfs_build, git_sha=args.git_sha, - catalog_backing=args.catalog_backing, + catalog_backing=result.catalog_backing, run_timestamp_utc=args.run_timestamp_utc or current_utc_timestamp(), operator=args.operator, command=redacted_command(sys.argv), @@ -1145,12 +1182,12 @@ def drop_table_if_present(catalog: Any, identifier: tuple[str, str]) -> None: def cleanup_catalog(catalog: Any, identifier: tuple[str, str]) -> None: drop_table_if_present(catalog, identifier) - try: + if catalog.table_exists(identifier): + raise RuntimeError("smoke table remains after cleanup") + if catalog.namespace_exists(identifier[0]): catalog.drop_namespace(identifier[0]) - except Exception as error: - message = str(error).lower() - if "not found" not in message and "does not exist" not in message: - print(f"warning: failed to drop namespace {identifier[0]}: {error}", file=sys.stderr) + if catalog.namespace_exists(identifier[0]): + raise RuntimeError("smoke namespace remains after cleanup") def current_snapshot_id_from_table_response(response: dict[str, Any]) -> int: @@ -1262,6 +1299,8 @@ def run_view_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: cleanup_namespace = False raise + run_namespace_properties_probe(args, deps, probe_namespace) + run_encoded_namespace_probe(args, deps, probe_namespace) view_path = view_endpoint_path(args, namespace=probe_namespace) for view_name in view_names: cleanup_views.append(view_name) @@ -1315,6 +1354,138 @@ def run_view_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: print(f"warning: failed to clean up view pagination probe: {error}", file=sys.stderr) +def run_encoded_namespace_probe(args: argparse.Namespace, deps: RuntimeDeps, parent: str) -> None: + segments = [parent, "encoded"] + path = namespace_endpoint_path(args, "\x1f".join(segments)) + signed_rest_request(args, deps, "POST", namespace_endpoint_path(args), {"namespace": segments}) + try: + response = signed_rest_request(args, deps, "GET", path) + if response.get("namespace") != segments: + raise RuntimeError("encoded namespace path did not preserve its component identity") + signed_rest_request_expect_error(args, deps, "GET", path.replace("%1F", "%251F"), expected_statuses={400}) + if signed_rest_request(args, deps, "GET", path).get("namespace") != segments: + raise RuntimeError("double-encoded lookup changed the namespace identity") + finally: + signed_rest_request(args, deps, "DELETE", path) + + +def run_namespace_properties_probe(args: argparse.Namespace, deps: RuntimeDeps, namespace: str) -> None: + path = namespace_endpoint_path(args, namespace) + before = signed_rest_request(args, deps, "GET", path) + properties = before.get("properties") + if not isinstance(properties, dict): + raise RuntimeError("namespace response did not include properties") + update = signed_rest_request( + args, deps, "POST", f"{path}/properties", {"updates": {"rustfs.smoke.updated": "true"}, "removals": ["rustfs.smoke"]} + ) + if update.get("updated") != ["rustfs.smoke.updated"] or update.get("removed") != ["rustfs.smoke"]: + raise RuntimeError("namespace property update did not report the applied keys") + expected = {**properties, "rustfs.smoke.updated": "true"} + expected.pop("rustfs.smoke", None) + if signed_rest_request(args, deps, "GET", path).get("properties") != expected: + raise RuntimeError("namespace property update was not persisted") + signed_rest_request_expect_error( + args, deps, "POST", f"{path}/properties", + {"updates": {"rustfs.smoke.updated": "false"}, "removals": ["rustfs.smoke.updated"]}, + expected_statuses={422}, + ) + if signed_rest_request(args, deps, "GET", path).get("properties") != expected: + raise RuntimeError("rejected namespace property update changed persisted properties") + + +def run_table_rename_probe(args: argparse.Namespace, deps: RuntimeDeps, backing: str) -> None: + if backing not in CATALOG_BACKINGS: + raise RuntimeError("table rename probe requires a known backing") + source = {"namespace": [args.namespace], "name": args.table} + destination = {"namespace": [args.namespace], "name": f"renamed-{uuid.uuid4().hex}"} + rename_path = f"{args.rest_path}/v1/{urllib.parse.quote(args.bucket, safe='')}/tables/rename" + source_path = table_endpoint_path(args) + destination_path = source_path.rsplit("/", 1)[0] + "/" + destination["name"] + before = signed_rest_request(args, deps, "GET", source_path) + signed_rest_request(args, deps, "POST", rename_path, {"source": source, "destination": destination}) + try: + signed_rest_request_expect_error(args, deps, "GET", source_path, expected_statuses={404}) + after = signed_rest_request(args, deps, "GET", destination_path) + for key in ("metadata-location", "metadata"): + if key not in before or after.get(key) != before[key]: + raise RuntimeError("table rename changed the Iceberg metadata or warehouse identity") + finally: + signed_rest_request(args, deps, "POST", rename_path, {"source": destination, "destination": source}) + restored = signed_rest_request(args, deps, "GET", source_path) + if any(restored.get(key) != before[key] for key in ("metadata-location", "metadata")): + raise RuntimeError("restored table rename changed the metadata or identity") + if backing == "object": + # Object-backed rename retains an inactive source entry. Consume only + # this probe's random identifier through normal create/drop APIs. + metadata = restored["metadata"] + schema = next((schema for schema in metadata.get("schemas", []) + if schema.get("schema-id") == metadata.get("current-schema-id")), None) + if schema is None: + raise RuntimeError("rename cleanup requires the original table schema") + pointer_path = table_endpoint_path(args, "/metadata-location") + pointer = signed_rest_request(args, deps, "GET", pointer_path) + if any(key not in pointer for key in ("metadata-location", "version-token", "generation")): + raise RuntimeError("rename cleanup requires the original table commit state") + replacement = signed_rest_request(args, deps, "POST", source_path.rsplit("/", 1)[0], + {"name": destination["name"], "schema": schema}) + fresh = replacement.get("metadata", {}) + if (any(not isinstance(fresh.get(key), str) or not fresh[key] or fresh[key] == metadata.get(key) + for key in ("table-uuid", "location")) + or fresh.get("snapshots", []) or fresh.get("current-snapshot-id") not in (None, -1)): + raise RuntimeError("rename cleanup replacement is not an isolated empty table") + signed_rest_request(args, deps, "DELETE", destination_path) + signed_rest_request_expect_error(args, deps, "GET", destination_path, expected_statuses={404}) + final = signed_rest_request(args, deps, "GET", source_path) + if (any(final.get(key) != restored[key] for key in ("metadata-location", "metadata")) + or signed_rest_request(args, deps, "GET", pointer_path) != pointer): + raise RuntimeError("rename cleanup changed the original table metadata or commit state") + + +def run_load_table_delegation_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: + path = table_endpoint_path(args) + for token in (None, "remote-signing", "not-vended-credentials", "VENDED-CREDENTIALS"): + response = signed_rest_request(args, deps, "GET", path, access_delegation=token) + config = response.get("config", {}) + if response.get("storage-credentials") or any(config.get(key) for key in REQUIRED_STORAGE_CREDENTIAL_KEYS): + raise RuntimeError("LoadTable returned credentials without exact delegation negotiation") + if args.require_vended_credentials: + response = signed_rest_request(args, deps, "GET", path, access_delegation="remote-signing, vended-credentials") + warehouse = table_warehouse_location(response.get("metadata")) + credentials = response.get("storage-credentials") + if not isinstance(credentials, list) or len(credentials) != 2 or any(not isinstance(item, dict) for item in credentials): + raise RuntimeError("LoadTable must vend exactly the warehouse and metadata scopes") + by_prefix = {item.get("prefix"): item for item in credentials} + warehouse_prefix = warehouse.rstrip("/") + "/" + metadata_location = response.get("metadata-location") + if not isinstance(metadata_location, str) or set(by_prefix) != {warehouse_prefix, metadata_location}: + raise RuntimeError("LoadTable vended scopes do not match the warehouse and current metadata") + credential = storage_credential_from_response({"storage-credentials": [by_prefix[warehouse_prefix]]}) + metadata_credential = storage_credential_from_response({"storage-credentials": [by_prefix[metadata_location]]}) + if any(credential.config[key] != metadata_credential.config[key] for key in REQUIRED_STORAGE_CREDENTIAL_KEYS): + raise RuntimeError("LoadTable scopes do not share the same temporary session") + verify_vended_credential_data_plane_scope(args, deps, credential, warehouse) + + +def run_strong_backing_boundaries(args: argparse.Namespace, deps: RuntimeDeps) -> None: + pointer_path = table_endpoint_path(args, "/metadata-location") + before = signed_rest_request(args, deps, "GET", pointer_path) + for key in ("metadata-location", "version-token", "generation"): + if key not in before: + raise RuntimeError("metadata-location response is missing the commit state") + path = table_endpoint_path(args, "/maintenance/metadata") + error = signed_rest_request_expect_error( + args, deps, "POST", path, + {"retain-recent-metadata-files": 1, "delete": False}, expected_statuses={400}, + ) + envelope = json.loads(error.response_body).get("error") + if not isinstance(envelope, dict) or (envelope.get("code"), envelope.get("type"), envelope.get("message")) != ( + 400, "BadRequestException", "operation is not supported with durable-strong table catalog backing" + ): + raise RuntimeError("strong metadata maintenance did not return the expected Iceberg error envelope") + if signed_rest_request(args, deps, "GET", pointer_path) != before: + raise RuntimeError("unsupported strong backing operation changed the table commit state") + + def maintenance_job_id(job: object) -> str | None: if not isinstance(job, dict): return None @@ -1324,93 +1495,125 @@ def maintenance_job_id(job: object) -> str | None: return value -def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None: +def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps, backing: str = "object") -> None: config_path = table_endpoint_path(args, "/maintenance/config") scheduler_path = table_endpoint_path(args, "/maintenance/scheduler") - scheduler_run_path = table_endpoint_path(args, "/maintenance/scheduler/run") - signed_rest_request(args, deps, "PUT", config_path, default_maintenance_config()) + pointer_path = table_endpoint_path(args, "/metadata-location") + before = signed_rest_request(args, deps, "GET", pointer_path) + for key in ("metadata-location", "version-token", "generation"): + if key not in before: + raise RuntimeError("metadata-location response is missing the commit state") + safe_config = default_maintenance_config() + signed_rest_request(args, deps, "PUT", config_path, safe_config) config = signed_rest_request(args, deps, "GET", config_path) - if config.get("version") != TABLE_MAINTENANCE_CONFIG_VERSION: + if any(config.get(key) != value for key, value in safe_config.items()): raise RuntimeError("maintenance config endpoint did not persist the smoke config") - report = signed_rest_request( - args, - deps, - "POST", - table_endpoint_path(args, "/maintenance/metadata"), - {"retain-recent-metadata-files": 1, "delete": False}, - ) - job = report.get("job") - job_id = maintenance_job_id(job) - if job_id is None: - raise RuntimeError("maintenance metadata endpoint did not return a job id") - audit_events = report.get("audit-events") - if not isinstance(audit_events, list) or not audit_events: - raise RuntimeError("maintenance metadata endpoint did not return audit events") - if not any(isinstance(event, dict) and event.get("action") == "PLANNED" for event in audit_events): - raise RuntimeError("maintenance metadata endpoint did not report a planning audit event") - job_report = signed_rest_request(args, deps, "GET", table_endpoint_path(args, f"/maintenance/jobs/{job_id}")) - if not isinstance(job_report.get("audit-events"), list): - raise RuntimeError("maintenance job endpoint did not return audit events") - quarantine = signed_rest_request( - args, - deps, - "POST", - table_endpoint_path(args, f"/maintenance/jobs/{job_id}/quarantine"), - {"action": "INSPECT"}, - ) - if quarantine.get("action") != "INSPECT" or not isinstance(quarantine.get("report"), dict): - raise RuntimeError("maintenance quarantine endpoint did not return an inspection report") + if backing == "durable-strong": + run_strong_backing_boundaries(args, deps) + elif backing == "object": + report = signed_rest_request( + args, deps, "POST", table_endpoint_path(args, "/maintenance/metadata"), + {"retain-recent-metadata-files": 1, "delete": False}, + ) + if maintenance_job_id(report.get("job")) is None: + raise RuntimeError("maintenance metadata endpoint did not return a job id") + audit_events = report.get("audit-events") + if not isinstance(audit_events, list) or not any( + isinstance(event, dict) and event.get("action") == "PLANNED" for event in audit_events + ): + raise RuntimeError("maintenance metadata endpoint did not report a planning audit event") + else: + raise RuntimeError("maintenance probes require a known backing") + scheduler = signed_rest_request(args, deps, "GET", scheduler_path) - expected_scheduler_statuses = {"READY", "QUEUED", "DISABLED", "PAUSED", "BACKPRESSURED", "RETRY_DEFERRED", "QUARANTINED"} - if scheduler.get("status") not in expected_scheduler_statuses: + expected_statuses = {"READY", "QUEUED", "DISABLED", "PAUSED", "BACKPRESSURED", "RETRY_DEFERRED", "QUARANTINED"} + if scheduler.get("status") not in expected_statuses: raise RuntimeError("maintenance scheduler endpoint did not return a stable scheduler status") - scheduler_timeline = scheduler.get("audit_timeline") - if not isinstance(scheduler_timeline, list): + timeline = scheduler.get("audit_timeline") + if not isinstance(timeline, list) or any( + not isinstance(item, dict) or not isinstance(item.get("audit-events"), list) for item in timeline + ): raise RuntimeError("maintenance scheduler endpoint did not return an audit timeline") - if scheduler_timeline and not isinstance(scheduler_timeline[0].get("audit-events"), list): - raise RuntimeError("maintenance scheduler audit timeline did not include job audit events") - scheduler_config = default_maintenance_config() - scheduler_config["background-enabled"] = True - scheduler_config["worker-paused"] = False - signed_rest_request(args, deps, "PUT", config_path, scheduler_config) - scheduler_run = signed_rest_request(args, deps, "POST", scheduler_run_path, {"scheduler-id": "pyiceberg-smoke-scheduler"}) - scheduler_run_job = scheduler_run.get("report", {}).get("job") - if not isinstance(scheduler_run_job, dict) or scheduler_run_job.get("status") != "QUEUED": - raise RuntimeError("maintenance scheduler run endpoint did not queue a maintenance job") - if scheduler_run_job.get("scheduler-id") != "pyiceberg-smoke-scheduler": - raise RuntimeError("maintenance scheduler run endpoint did not preserve scheduler identity") - scheduler_run_report = scheduler_run.get("scheduler") - if not isinstance(scheduler_run_report, dict) or scheduler_run_report.get("status") != "QUEUED": - raise RuntimeError("maintenance scheduler run endpoint did not return queued scheduler state") + scheduler_id = f"pyiceberg-smoke-{uuid.uuid4().hex}" + scheduler_config = {**safe_config, "background-enabled": True, "worker-paused": False} + try: + signed_rest_request(args, deps, "PUT", config_path, scheduler_config) + scheduler_run = signed_rest_request( + args, deps, "POST", f"{scheduler_path}/run", {"scheduler-id": scheduler_id} + ) + queued = scheduler_run.get("report", {}).get("job", {}) + job_id = maintenance_job_id(queued) + if job_id is None or queued.get("status") != "QUEUED" or queued.get("operation") != "DRY_RUN": + raise RuntimeError("maintenance scheduler run endpoint did not queue a dry-run job") + if queued.get("scheduler-id") != scheduler_id: + raise RuntimeError("maintenance scheduler run endpoint did not preserve scheduler identity") + if scheduler_run.get("scheduler", {}).get("status") != "QUEUED": + raise RuntimeError("maintenance scheduler run endpoint did not return queued scheduler state") + job_path = table_endpoint_path(args, f"/maintenance/jobs/{urllib.parse.quote(job_id, safe='')}") + job_report = signed_rest_request(args, deps, "GET", job_path) + if maintenance_job_id(job_report.get("job")) != job_id or not isinstance(job_report.get("audit-events"), list): + raise RuntimeError("maintenance job endpoint did not return the queued job and audit events") + quarantine = signed_rest_request(args, deps, "POST", f"{job_path}/quarantine", {"action": "INSPECT"}) + if quarantine.get("action") != "INSPECT" or maintenance_job_id(quarantine.get("report", {}).get("job")) != job_id: + raise RuntimeError("maintenance quarantine endpoint did not return the queued job inspection") - worker = signed_rest_request(args, deps, "POST", table_endpoint_path(args, "/maintenance/worker/run"), {}) - worker_job = worker.get("job") - expected_statuses = {"DISABLED", "PAUSED", "FAILED", "SUCCESSFUL", "RUNNING", "QUEUED"} - if not isinstance(worker_job, dict) or worker_job.get("status") not in expected_statuses: - raise RuntimeError("maintenance worker endpoint did not return a stable job status") - if not isinstance(worker.get("audit-events"), list): - raise RuntimeError("maintenance worker endpoint did not return audit events") + worker = signed_rest_request(args, deps, "POST", table_endpoint_path(args, "/maintenance/worker/run"), {}) + job = worker.get("job") + if not isinstance(job, dict) or job.get("status") != "SUCCESSFUL": + raise RuntimeError("maintenance worker did not complete the dry-run job") + if maintenance_job_id(job) != job_id or job.get("operation") != "DRY_RUN": + raise RuntimeError("maintenance worker changed the queued job identity or operation") + for key in ("deleted_metadata_file_count", "deleted_object_count"): + if type(job.get(key)) is not int or job[key] != 0: + raise RuntimeError("maintenance dry-run did not report zero deletions") + if not isinstance(worker.get("audit-events"), list) or not worker["audit-events"]: + raise RuntimeError("maintenance worker endpoint did not return audit events") + if signed_rest_request(args, deps, "GET", pointer_path) != before: + raise RuntimeError("maintenance dry-run changed the table commit state") + finally: + # Leave smoke-owned tables paused even if admission or execution fails. + signed_rest_request(args, deps, "PUT", config_path, safe_config) -def run_catalog_api_probes(args: argparse.Namespace, deps: RuntimeDeps) -> dict[str, Any]: +def run_catalog_api_probes(args: argparse.Namespace, deps: RuntimeDeps, backing: str) -> tuple[dict[str, Any], dict[str, str]]: + if backing not in CATALOG_BACKINGS: + raise RuntimeError("catalog probes require a known backing") table_response = signed_rest_request(args, deps, "GET", table_endpoint_path(args)) snapshot_id = current_snapshot_id_from_table_response(table_response) run_metadata_location_probe(args, deps, table_response) run_table_ref_probe(args, deps, snapshot_id) run_view_probe(args, deps) - run_maintenance_probe(args, deps) + run_table_rename_probe(args, deps, backing) + run_load_table_delegation_probe(args, deps) + run_maintenance_probe(args, deps, backing) diagnostics = signed_rest_request(args, deps, "GET", table_endpoint_path(args, "/catalog/diagnostics")) if not isinstance(diagnostics, dict) or not diagnostics: raise RuntimeError("catalog diagnostics endpoint returned an empty response") - return table_response + exported = signed_rest_request(args, deps, "GET", table_endpoint_path(args, "/catalog/export")) + pointer = signed_rest_request(args, deps, "GET", table_endpoint_path(args, "/metadata-location")) + entry = exported.get("table", {}) + location = entry.get("metadata_location") + if not isinstance(location, str) or not location: + raise RuntimeError("catalog export did not include its metadata location") + client_location = location if location.startswith("s3://") else f"s3://{args.bucket}/{location}" + if pointer.get("metadata-location") != client_location: + raise RuntimeError("catalog export does not match the current metadata pointer") + for public, stored in (("version-token", "version_token"), ("generation", "generation")): + if public not in pointer or entry.get(stored) != pointer[public]: + raise RuntimeError("catalog export does not match the current table commit state") + probes = engine_compatibility.pyiceberg_catalog_probe_results(backing, skipped=False, vended=bool(args.require_vended_credentials)) + return signed_rest_request(args, deps, "GET", table_endpoint_path(args)), probes def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: + if args.client_version: + pyiceberg_client_version(args) endpoint = normalized_endpoint(args.endpoint) ensure_local_proxy_bypass(endpoint) ensure_aws_env(args.access_key, args.secret_key, args.region) + backing = discover_catalog_backing(args, deps) print(f"[1/10] ensuring S3 bucket {args.bucket}") ensure_bucket(args, deps) @@ -1457,10 +1660,7 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: print(f"[8/10] appending rows through PyIceberg") table = catalog.load_table(identifier) rows = deps.pyarrow.Table.from_pylist( - [ - {"id": 1, "payload": "alpha"}, - {"id": 2, "payload": "beta"}, - ], + SMOKE_ROWS, schema=arrow_schema, ) table.append(rows) @@ -1470,9 +1670,12 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: scanned = loaded.scan().to_arrow() if scanned.num_rows != 2: raise RuntimeError(f"expected 2 rows after append, got {scanned.num_rows}") + if sorted(scanned.to_pylist(), key=lambda row: row["id"]) != SMOKE_ROWS: + raise RuntimeError("scan returned different data after append") loaded_table_location = table_warehouse_location(loaded) loaded_metadata_location = table_metadata_location(loaded) table_response = None + catalog_probes = {"direct-rest": "skipped"} if args.skip_catalog_api_probes: print("[10/10] skipping direct REST catalog API probes") @@ -1480,7 +1683,7 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: table_response = signed_rest_request(args, deps, "GET", table_endpoint_path(args)) else: print("[10/10] probing direct REST catalog APIs") - table_response = run_catalog_api_probes(args, deps) + table_response, catalog_probes = run_catalog_api_probes(args, deps, backing) metadata_location = loaded_metadata_location if isinstance(table_response, dict): @@ -1500,6 +1703,8 @@ def run_smoke(args: argparse.Namespace, deps: RuntimeDeps) -> SmokeResult: row_count=scanned.num_rows, cleanup_result="dropped-table-and-namespace" if args.cleanup else "not-requested", table_warehouse_location=loaded_table_location, + catalog_backing=backing, + catalog_probes=catalog_probes, ) diff --git a/scripts/table-catalog/test_duckdb_smoke.py b/scripts/table-catalog/test_duckdb_smoke.py index 768a95e8a..e5fb05afc 100644 --- a/scripts/table-catalog/test_duckdb_smoke.py +++ b/scripts/table-catalog/test_duckdb_smoke.py @@ -180,15 +180,28 @@ class DuckDBSmokeTest(unittest.TestCase): self.assertEqual(args.rest_path, "/iceberg") self.assertEqual(args.rest_signing_name, "s3") self.assertEqual(args.bucket, "rustfs-duckdb-smoke") + self.assertEqual(args.catalog_backing, "object") + + def test_backing_mismatch_prevents_duckdb_table_setup(self) -> None: + with mock.patch.object(duckdb_smoke, "duckdb_path", return_value="duckdb"), mock.patch.object( + duckdb_smoke, "duckdb_client_version", return_value="1.5.5" + ), mock.patch.object(duckdb_smoke.pyiceberg_smoke, "ensure_aws_env"), mock.patch.object( + duckdb_smoke.pyiceberg_smoke, "discover_catalog_backing", side_effect=RuntimeError("backing mismatch") + ), mock.patch.object(duckdb_smoke.pyiceberg_smoke, "ensure_bucket") as bucket: + with self.assertRaisesRegex(RuntimeError, "backing mismatch"): + duckdb_smoke.run_smoke(self.args(), mock.Mock()) + bucket.assert_not_called() def test_live_evidence_records_automated_duckdb_claim(self) -> None: args = self.args() + args.catalog_backing = "operator-recorded" result = duckdb_smoke.DuckDBSmokeResult( client_version="1.5.5", metadata_location="s3://rustfs-duckdb-smoke/metadata/00001.json", row_count=2, cleanup_result="dropped-tables-and-namespace", checks={"canonical_rest_catalog": "pass"}, + catalog_backing="durable-strong", ) with tempfile.TemporaryDirectory() as temp_dir: output = Path(temp_dir) / "evidence.json" @@ -200,6 +213,7 @@ class DuckDBSmokeTest(unittest.TestCase): evidence = document["live_conformance_evidence"] self.assertEqual(evidence["client_name"], "DuckDB Iceberg") self.assertEqual(evidence["claim"], "automated-rest-catalog-smoke") + self.assertEqual(evidence["catalog_backing"], "durable-strong") self.assertIn("--secret-key ''", evidence["command"]) self.assertNotIn("--secret-key secret", evidence["command"]) self.assertEqual(document["validation"]["status"], "accepted") diff --git a/scripts/table-catalog/test_engine_compatibility.py b/scripts/table-catalog/test_engine_compatibility.py index 65a766f40..27214e120 100644 --- a/scripts/table-catalog/test_engine_compatibility.py +++ b/scripts/table-catalog/test_engine_compatibility.py @@ -544,6 +544,53 @@ class EngineCompatibilityTest(unittest.TestCase): self.assertEqual(validation["claim"], "automated-smoke") self.assertEqual(validation["client_name"], "PyIceberg") + for legacy in ("create-append-reload-scan", "create-append-reload-scan-backing-contracts"): + with self.subTest(scenario=legacy): + historical = dict(record, scenario=legacy) + validated = engine_compatibility.validate_live_conformance_evidence(historical) + self.assertEqual(validated["scenario"], legacy) + self.assertNotIn("catalog_probes", historical) + + def test_backing_probe_evidence_rejects_missing_or_overstated_results(self) -> None: + for backing in ("object", "durable-strong"): + for skipped in (False, True): + for vended in (False, True): + with self.subTest(backing=backing, skipped=skipped, vended=vended): + scenario = (engine_compatibility.PYICEBERG_CORE_SCENARIO if skipped + else engine_compatibility.PYICEBERG_BACKING_SCENARIO) + record = engine_compatibility.live_conformance_evidence_record( + client_name="PyIceberg", client_version="0.11.1", scenario=scenario, + rustfs_build="test", git_sha="abc123", catalog_backing=backing, + endpoint="http://127.0.0.1:9000", warehouse="lake", rest_path="/iceberg", + namespace="sales", table="orders", metadata_location="s3://lake/metadata/v2.json", + run_timestamp_utc="2026-10-14T00:00:00Z", operator="test", + expected_status="pass", observed_status="pass", row_count=2, + cleanup_result="not-requested", claim="automated-smoke", command="smoke", + ) + probes = engine_compatibility.pyiceberg_catalog_probe_results( + backing, skipped=skipped, vended=vended + ) + record.update(catalog_probes=probes, rest_signing_name="s3", + credential_vending_required=vended) + validate = engine_compatibility.validate_live_conformance_evidence + self.assertEqual(validate(record)["status"], "accepted") + for changes in ( + {"catalog_probes": {}}, + {"catalog_probes": {**probes, "automatic-worker": "pass"}}, + {"rest_signing_name": "unknown"}, + {"credential_vending_required": "false"}, + ): + with self.subTest(changes=changes), self.assertRaises(ValueError): + validate({**record, **changes}) + if skipped: + with self.assertRaises(ValueError): + validate({**record, "scenario": engine_compatibility.PYICEBERG_BACKING_SCENARIO}) + else: + for name in ("metadata-maintenance", "maintenance-worker-dry-run", "diagnostics", "export"): + broken = {**probes, name: "skipped"} + with self.subTest(probe=name), self.assertRaises(ValueError): + validate({**record, "catalog_probes": broken}) + def test_live_conformance_evidence_rejects_missing_or_mismatched_results(self) -> None: record = engine_compatibility.live_conformance_evidence_record( client_name="Spark Iceberg REST catalog", diff --git a/scripts/table-catalog/test_pyiceberg_signing.py b/scripts/table-catalog/test_pyiceberg_signing.py index c326ea636..95ec7ebe2 100644 --- a/scripts/table-catalog/test_pyiceberg_signing.py +++ b/scripts/table-catalog/test_pyiceberg_signing.py @@ -104,6 +104,22 @@ class PyIcebergSigningTest(unittest.TestCase): SigV4Auth(Credentials(args.access_key, args.secret_key), args.rest_signing_name, args.region).add_auth(expected) self.assertEqual(request.headers["Authorization"], expected.headers["Authorization"]) + def test_delegation_header_is_included_in_the_signed_request(self) -> None: + args = self.args() + response = mock.MagicMock() + response.__enter__.return_value.read.return_value = b"{}" + with mock.patch.object(pyiceberg_smoke.urllib.request, "urlopen", return_value=response) as send: + pyiceberg_smoke.signed_rest_request( + args, self.deps, "GET", "/iceberg/v1/warehouse/namespaces/sales/tables/orders", + access_delegation="remote-signing, vended-credentials", + ) + wire = send.call_args.args[0] + request = Request(wire.method, wire.full_url, headers=dict(wire.header_items())).prepare() + self.assertEqual(request.headers["X-Iceberg-Access-Delegation"], "remote-signing, vended-credentials") + self.assertEqual(request.headers["Authorization"], self.expected_signature(request, args)) + request.headers["X-Iceberg-Access-Delegation"] = "remote-signing" + self.assertNotEqual(request.headers["Authorization"], self.expected_signature(request, args)) + def test_initial_config_and_recreated_sessions_are_signed(self) -> None: for profile in sorted(RUSTFS_PROFILES): args = self.args(profile) diff --git a/scripts/table-catalog/test_pyiceberg_smoke.py b/scripts/table-catalog/test_pyiceberg_smoke.py index 9c321f180..8f67684d6 100644 --- a/scripts/table-catalog/test_pyiceberg_smoke.py +++ b/scripts/table-catalog/test_pyiceberg_smoke.py @@ -3,12 +3,13 @@ from __future__ import annotations +import copy import json import os import re import sys import unittest -from contextlib import redirect_stdout +from contextlib import redirect_stderr, redirect_stdout from io import StringIO from types import SimpleNamespace from pathlib import Path @@ -286,7 +287,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json"}, ) - def test_view_probe_drops_smoke_view_after_load_failure(self) -> None: + @mock.patch.object(pyiceberg_smoke, "run_namespace_properties_probe") + @mock.patch.object(pyiceberg_smoke, "run_encoded_namespace_probe") + def test_view_probe_drops_smoke_view_after_load_failure(self, _encoded: object, _properties: object) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) probe_namespace = "smoke-sales-orders-views-12345678" namespace_path = pyiceberg_smoke.namespace_endpoint_path(args) @@ -331,7 +334,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIn(("DELETE", f"{view_path}/view-b", None), calls) self.assertIn(("DELETE", pyiceberg_smoke.namespace_endpoint_path(args, probe_namespace), None), calls) - def test_view_probe_cleans_candidate_after_create_timeout(self) -> None: + @mock.patch.object(pyiceberg_smoke, "run_namespace_properties_probe") + @mock.patch.object(pyiceberg_smoke, "run_encoded_namespace_probe") + def test_view_probe_cleans_candidate_after_create_timeout(self, _encoded: object, _properties: object) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) probe_namespace = "smoke-sales-orders-views-12345678" namespace_path = pyiceberg_smoke.namespace_endpoint_path(args) @@ -364,7 +369,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIn(("DELETE", f"{view_path}/view-a", None), calls) self.assertIn(("DELETE", pyiceberg_smoke.namespace_endpoint_path(args, probe_namespace), None), calls) - def test_view_probe_continues_cleanup_after_delete_failure(self) -> None: + @mock.patch.object(pyiceberg_smoke, "run_namespace_properties_probe") + @mock.patch.object(pyiceberg_smoke, "run_encoded_namespace_probe") + def test_view_probe_continues_cleanup_after_delete_failure(self, _encoded: object, _properties: object) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) probe_namespace = "smoke-sales-orders-views-12345678" namespace_path = pyiceberg_smoke.namespace_endpoint_path(args) @@ -440,47 +447,104 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): with self.assertRaisesRegex(RuntimeError, message): pyiceberg_smoke.paginated_identifier_names(args, mock.Mock(), view_path, page_size=1) - def test_maintenance_probe_rejects_unknown_worker_status(self) -> None: - args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) - config_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/config") - maintenance_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/metadata") - job_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1") - quarantine_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1/quarantine") - scheduler_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler") - scheduler_run_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler/run") - worker_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/worker/run") + def maintenance_responses(self, backing="object", *, worker_changes=None, mutate_pointer=False): + config = {} + worker_ran = False + pointer = {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "token-2", "generation": 2} + job = {"job_id": "job-2", "status": "QUEUED", "operation": "DRY_RUN"} + audit = [{"action": "PLANNED"}] - def fake_signed_request( - _args: object, - _deps: object, - method: str, - path: str, - body: object = None, - ) -> dict[str, object]: - if (method, path) == ("PUT", config_path): - return {} - if (method, path) == ("GET", config_path): - return {"version": 1} - if (method, path) == ("POST", maintenance_path): - return {"job": {"job_id": "job-1"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("GET", job_path): - return {"job": {"job-id": "job-1", "status": "SUCCESSFUL"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("POST", quarantine_path): - return {"action": "INSPECT", "report": {"job": {"job_id": "job-1"}}} - if (method, path) == ("GET", scheduler_path): - return {"status": "DISABLED", "audit_timeline": [{"job_id": "job-1", "audit-events": [{"action": "PLANNED"}]}]} - if (method, path) == ("POST", scheduler_run_path): - return { - "report": {"job": {"job-id": "job-2", "status": "QUEUED", "scheduler-id": "pyiceberg-smoke-scheduler"}}, - "scheduler": {"status": "QUEUED"}, - } - if (method, path) == ("POST", worker_path): - return {"job": {"status": "UNKNOWN"}, "audit-events": [{"action": "WORKER_CONTROL"}]} - raise AssertionError(f"unexpected REST request: {method} {path}") + def request(_args, _deps, method, path, body=None): + nonlocal config, worker_ran + if (method, path.rsplit("/", 1)[-1]) == ("GET", "metadata-location"): + return {**pointer, "generation": 3} if worker_ran and mutate_pointer else pointer.copy() + if path.endswith("/maintenance/config"): + if method == "PUT": + config = body.copy() + return config.copy() + if path.endswith("/maintenance/metadata"): + if backing == "durable-strong": + envelope = {"error": {"code": 400, "type": "BadRequestException", + "message": "operation is not supported with durable-strong table catalog backing"}} + raise pyiceberg_smoke.RestRequestError(method, path, 400, json.dumps(envelope)) + return {"job": {"job_id": "job-1"}, "audit-events": audit} + if path.endswith("/maintenance/scheduler"): + return {"status": "DISABLED", "audit_timeline": []} + if path.endswith("/maintenance/scheduler/run"): + return {"report": {"job": {**job, "scheduler-id": body["scheduler-id"]}}, + "scheduler": {"status": "QUEUED"}} + if path.endswith("/maintenance/jobs/job-2"): + return {"job": job.copy(), "audit-events": audit} + if path.endswith("/maintenance/jobs/job-2/quarantine"): + return {"action": "INSPECT", "report": {"job": job.copy()}} + if path.endswith("/maintenance/worker/run"): + worker_ran = True + completed = {**job, "status": "SUCCESSFUL", "deleted_metadata_file_count": 0, "deleted_object_count": 0} + completed.update(worker_changes or {}) + return {"job": completed, "audit-events": audit} + raise AssertionError(f"unexpected maintenance request: {method} {path}") - with mock.patch.object(pyiceberg_smoke, "signed_rest_request", side_effect=fake_signed_request): - with self.assertRaisesRegex(RuntimeError, "stable job status"): - pyiceberg_smoke.run_maintenance_probe(args, mock.Mock()) + return request + + def test_maintenance_dry_run_exercises_both_backings_and_leaves_worker_paused(self) -> None: + args = self.parse_with_args([]) + for backing in ("object", "durable-strong"): + with self.subTest(backing=backing), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=self.maintenance_responses(backing) + ) as request: + pyiceberg_smoke.run_maintenance_probe(args, mock.Mock(), backing) + paths = [call.args[3] for call in request.call_args_list] + for suffix in ("/maintenance/scheduler/run", "/maintenance/jobs/job-2", + "/maintenance/jobs/job-2/quarantine", "/maintenance/worker/run"): + self.assertTrue(any(path.endswith(suffix) for path in paths), suffix) + self.assertEqual(request.call_args.args[2], "PUT") + self.assertEqual(request.call_args.args[4], pyiceberg_smoke.default_maintenance_config()) + for call in request.call_args_list: + if call.args[2] == "PUT": + self.assertFalse(call.args[4]["delete-enabled"]) + + def test_maintenance_failure_never_counts_as_success_and_restores_safe_config(self) -> None: + args = self.parse_with_args([]) + failures = [ + ({"status": status}, "complete the dry-run") + for status in ("FAILED", "PAUSED", "RUNNING", "QUEUED", "DISABLED", "UNKNOWN", None) + ] + failures.extend([ + ({"job_id": "other-job"}, "job identity"), + ({"operation": "DELETE"}, "job identity"), + ({"deleted_object_count": 1}, "zero deletions"), + ({"deleted_metadata_file_count": 1}, "zero deletions"), + ({"deleted_object_count": None}, "zero deletions"), + ({"deleted_object_count": False}, "zero deletions"), + ]) + for backing in ("object", "durable-strong"): + for changes, message in failures: + with self.subTest(backing=backing, changes=changes), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", + side_effect=self.maintenance_responses(backing, worker_changes=changes) + ) as request: + with self.assertRaisesRegex(RuntimeError, message): + pyiceberg_smoke.run_maintenance_probe(args, mock.Mock(), backing) + self.assertEqual(request.call_args.args[4], pyiceberg_smoke.default_maintenance_config()) + + def test_maintenance_detects_pointer_advance_and_pauses_after_transport_failure(self) -> None: + args = self.parse_with_args([]) + for backing in ("object", "durable-strong"): + for transport_failure in (False, True): + base = self.maintenance_responses(backing, mutate_pointer=not transport_failure) + + def request(*args, **kwargs): + if transport_failure and args[3].endswith("/maintenance/worker/run"): + raise OSError("worker connection lost") + return base(*args, **kwargs) + + with self.subTest(backing=backing, transport_failure=transport_failure), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=request + ) as calls: + expected = OSError if transport_failure else RuntimeError + with self.assertRaises(expected): + pyiceberg_smoke.run_maintenance_probe(args, mock.Mock(), backing) + self.assertEqual(calls.call_args.args[4], pyiceberg_smoke.default_maintenance_config()) def test_catalog_api_probe_exercises_extended_rest_surfaces(self) -> None: args = self.parse_with_args(["--namespace", "sales", "--table", "orders"]) @@ -490,13 +554,12 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): metadata_location_path = pyiceberg_smoke.table_endpoint_path(args, "/metadata-location") refs_path = pyiceberg_smoke.table_ref_endpoint_path(args) ref_name = pyiceberg_smoke.safe_ref_segment(args.namespace, args.table) - config_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/config") - maintenance_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/metadata") quarantine_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1/quarantine") scheduler_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler") scheduler_run_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/scheduler/run") worker_path = pyiceberg_smoke.table_endpoint_path(args, "/maintenance/worker/run") diagnostics_path = pyiceberg_smoke.table_endpoint_path(args, "/catalog/diagnostics") + maintenance_request = self.maintenance_responses() def fake_signed_request( _args: object, @@ -509,7 +572,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): if (method, path) == ("GET", table_path): return {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "metadata": {"current-snapshot-id": 7}} if (method, path) == ("GET", metadata_location_path): - return {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "token-2"} + return {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "token-2", "generation": 2} + if (method, path) == ("GET", pyiceberg_smoke.table_endpoint_path(args, "/catalog/export")): + return {"table": {"metadata_location": "s3://lake/tables/id/metadata/v2.metadata.json", "version_token": "token-2", "generation": 2}} if method == "PUT" and path.startswith(f"{refs_path}/"): return {} if (method, path) == ("GET", refs_path): @@ -518,31 +583,16 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): return {"refs": {}} if method == "DELETE" and path.startswith(f"{refs_path}/"): return {} - if (method, path) == ("PUT", config_path): - return {} - if (method, path) == ("GET", config_path): - return {"version": 1} - if (method, path) == ("POST", maintenance_path): - return {"job": {"job_id": "job-1"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("GET", pyiceberg_smoke.table_endpoint_path(args, "/maintenance/jobs/job-1")): - return {"job": {"job-id": "job-1", "status": "SUCCESSFUL"}, "audit-events": [{"action": "PLANNED"}]} - if (method, path) == ("POST", quarantine_path): - return {"action": "INSPECT", "report": {"job": {"job_id": "job-1"}}} - if (method, path) == ("GET", scheduler_path): - return {"status": "DISABLED", "audit_timeline": [{"job_id": "job-1", "audit-events": [{"action": "PLANNED"}]}]} - if (method, path) == ("POST", scheduler_run_path): - return { - "report": {"job": {"job-id": "job-2", "status": "QUEUED", "scheduler-id": "pyiceberg-smoke-scheduler"}}, - "scheduler": {"status": "QUEUED"}, - } - if (method, path) == ("POST", worker_path): - return {"job": {"status": "PAUSED"}, "audit-events": [{"action": "WORKER_CONTROL"}]} + if "/maintenance/" in path: + return maintenance_request(_args, _deps, method, path, body) if (method, path) == ("GET", diagnostics_path): return {"status": "ok"} raise AssertionError(f"unexpected REST request: {method} {path}") with mock.patch.object(pyiceberg_smoke, "signed_rest_request", side_effect=fake_signed_request): - with mock.patch.object(pyiceberg_smoke, "run_view_probe") as view_probe: + with mock.patch.object(pyiceberg_smoke, "run_view_probe") as view_probe, mock.patch.object( + pyiceberg_smoke, "run_table_rename_probe" + ), mock.patch.object(pyiceberg_smoke, "run_load_table_delegation_probe"): with mock.patch.object( pyiceberg_smoke, "signed_rest_request_expect_error", @@ -553,8 +603,10 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): "snapshot ref has retention policy; force is required", ), ) as expect_error: - pyiceberg_smoke.run_catalog_api_probes(args, deps) + _, probes = pyiceberg_smoke.run_catalog_api_probes(args, deps, "object") + self.assertEqual(probes["maintenance-worker-dry-run"], "pass") + self.assertEqual(probes["diagnostics"], "pass") view_probe.assert_called_once_with(args, deps) expect_error.assert_called_once_with( args, @@ -567,8 +619,8 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIn(("GET", metadata_location_path, None), calls) self.assertIn(("GET", diagnostics_path, None), calls) self.assertIn(("GET", scheduler_path, None), calls) - self.assertIn(("POST", scheduler_run_path, {"scheduler-id": "pyiceberg-smoke-scheduler"}), calls) - self.assertIn(("POST", quarantine_path, {"action": "INSPECT"}), calls) + self.assertTrue(any(method == "POST" and path == scheduler_run_path for method, path, _ in calls)) + self.assertIn(("POST", quarantine_path.replace("job-1", "job-2"), {"action": "INSPECT"}), calls) self.assertIn(("POST", worker_path, {}), calls) def test_table_ref_probe_force_deletes_smoke_ref_after_validation_failure(self) -> None: @@ -601,12 +653,15 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): calls, ) - def run_smoke_with_fakes(self, args: object, probe_calls: list[str]) -> list[str]: + def run_smoke_with_fakes(self, args: object, probe_calls: list[str], scanned_rows: list[dict[str, object]] | None = None) -> list[str]: events: list[str] = [] class FakeArrowTable: num_rows = 2 + def to_pylist(self) -> list[dict[str, object]]: + return pyiceberg_smoke.SMOKE_ROWS if scanned_rows is None else scanned_rows + class FakePyArrowTableFactory: @staticmethod def from_pylist(rows: list[dict[str, object]], *, schema: object) -> object: @@ -665,7 +720,9 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): }, ) - with mock.patch.object(pyiceberg_smoke, "ensure_local_proxy_bypass"): + with mock.patch.object(pyiceberg_smoke, "ensure_local_proxy_bypass"), mock.patch.object( + pyiceberg_smoke, "discover_catalog_backing", return_value="object" + ): with mock.patch.object(pyiceberg_smoke, "ensure_aws_env"): with mock.patch.object(pyiceberg_smoke, "ensure_bucket"): with mock.patch.object(pyiceberg_smoke, "enable_table_bucket"): @@ -687,12 +744,20 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): side_effect=lambda *_args: ( events.append("catalog-probes"), probe_calls.append("catalog-probes"), - ), + ({}, {"direct-rest": "pass"}), + )[-1], ): with redirect_stdout(StringIO()): pyiceberg_smoke.run_smoke(args, deps) return events + def test_run_smoke_rejects_equal_row_count_with_different_data(self) -> None: + args = self.parse_with_args(["--bucket", "lake", "--namespace", "sales", "--table", "orders"]) + calls: list[str] = [] + with self.assertRaisesRegex(RuntimeError, "different data"): + self.run_smoke_with_fakes(args, calls, [{"id": 1, "payload": "wrong"}, {"id": 2, "payload": "beta"}]) + self.assertEqual(calls, []) + def test_run_smoke_probes_extended_catalog_apis_by_default(self) -> None: args = self.parse_with_args(["--bucket", "lake", "--namespace", "sales", "--table", "orders"]) probe_calls: list[str] = [] @@ -1052,6 +1117,8 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): row_count=2, cleanup_result="not-requested", table_warehouse_location="s3://lake/tables/table-id", + catalog_backing="durable-strong", + catalog_probes=pyiceberg_smoke.engine_compatibility.pyiceberg_catalog_probe_results("durable-strong", skipped=False, vended=False), ) record = pyiceberg_smoke.pyiceberg_live_evidence_record( @@ -1072,6 +1139,217 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertEqual(record["metadata_location"], "s3://lake/tables/table-id/metadata/v2.metadata.json") self.assertEqual(record["row_count"], 2) self.assertEqual(record["claim"], "automated-smoke") + self.assertEqual(record["catalog_probes"]["metadata-maintenance"], "expected-unsupported") + self.assertEqual(record["catalog_probes"]["maintenance-worker-dry-run"], "pass") + self.assertEqual(record["rest_signing_name"], "s3") + validate = pyiceberg_smoke.engine_compatibility.validate_live_conformance_evidence + self.assertEqual(validate(record)["status"], "accepted") + record["catalog_probes"]["metadata-maintenance"] = "pass" + with self.assertRaisesRegex(ValueError, "capabilities"): + validate(record) + del record["catalog_probes"] + with self.assertRaisesRegex(ValueError, "capabilities"): + validate(record) + + def test_backing_discovery_uses_overrides_and_rejects_mismatches(self) -> None: + args = self.parse_with_args(["--bucket", "lake", "--catalog-backing", "durable-strong"]) + config = {"defaults": {"rustfs.catalog-backing": "object"}, "overrides": {"rustfs.catalog-backing": "durable-strong"}} + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=config) as request: + self.assertEqual(pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()), "durable-strong") + self.assertEqual(request.call_args.args[2:4], ("GET", "/iceberg/v1/config?warehouse=lake")) + for response in ({}, {"defaults": {}, "overrides": {}}, {"defaults": {"rustfs.catalog-backing": "unknown"}, "overrides": {}}): + with self.subTest(response=response), mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=response): + with self.assertRaises(RuntimeError): + pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()) + config["overrides"] = {} + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=config): + with self.assertRaisesRegex(RuntimeError, "does not match"): + pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()) + args.catalog_backing = "object-backed" + self.assertEqual(pyiceberg_smoke.discover_catalog_backing(args, mock.Mock()), "object") + + def test_client_version_is_observed_and_mismatch_aborts_before_setup(self) -> None: + args = self.parse_with_args(["--client-version", "0.11.1"]) + with mock.patch.object(pyiceberg_smoke.importlib.metadata, "version", return_value="0.11.1"): + self.assertEqual(pyiceberg_smoke.pyiceberg_client_version(args), "0.11.1") + with mock.patch.object(pyiceberg_smoke.importlib.metadata, "version", return_value="0.10.0"), mock.patch.object( + pyiceberg_smoke, "ensure_bucket" + ) as bucket, mock.patch.object(pyiceberg_smoke, "discover_catalog_backing") as config: + with self.assertRaisesRegex(RuntimeError, "expected PyIceberg 0.11.1, found 0.10.0"): + pyiceberg_smoke.run_smoke(args, mock.Mock()) + bucket.assert_not_called() + config.assert_not_called() + + def test_backing_mismatch_fails_before_creating_any_objects(self) -> None: + args = self.parse_with_args([]) + with mock.patch.object(pyiceberg_smoke, "discover_catalog_backing", side_effect=RuntimeError("mismatch")), mock.patch.object( + pyiceberg_smoke, "ensure_bucket" + ) as bucket, mock.patch.object(pyiceberg_smoke, "ensure_aws_env"): + with self.assertRaisesRegex(RuntimeError, "mismatch"): + pyiceberg_smoke.run_smoke(args, mock.Mock()) + bucket.assert_not_called() + + def strong_probe_responses(self, *, status=400, error_type="BadRequestException", mutate=False): + pointer = {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "version-token": "t2", "generation": 2} + reads = 0 + + def request(_args, _deps, method, path, body=None): + nonlocal reads + if path.endswith("/metadata-location"): + reads += 1 + return {**pointer, "generation": 3} if mutate and reads > 1 else pointer.copy() + self.assertTrue(path.endswith("/maintenance/metadata"), path) + error = {"error": {"code": status, "type": error_type, + "message": "operation is not supported with durable-strong table catalog backing"}} + raise pyiceberg_smoke.RestRequestError(method, path, status, json.dumps(error)) + + return request + + def test_strong_metadata_boundary_checks_error_and_unchanged_pointer(self) -> None: + args = self.parse_with_args([]) + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", side_effect=self.strong_probe_responses()) as request: + pyiceberg_smoke.run_strong_backing_boundaries(args, mock.Mock()) + self.assertEqual(request.call_count, 3) + for kwargs, message in (({"status": 403}, "expected one of"), ({"status": 500}, "expected one of"), + ({"error_type": "RESTException"}, "error envelope"), ({"mutate": True}, "commit state")): + with self.subTest(kwargs=kwargs), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=self.strong_probe_responses(**kwargs) + ): + with self.assertRaisesRegex(RuntimeError, message): + pyiceberg_smoke.run_strong_backing_boundaries(args, mock.Mock()) + + def test_catalog_export_checks_pointer_token_and_generation_on_both_backings(self) -> None: + args = self.parse_with_args(["--bucket", "lake"]) + location = "s3://lake/tables/id/metadata/v2.metadata.json" + table = {"metadata-location": location, "metadata": {"current-snapshot-id": 7}} + pointer = {"metadata-location": location, "version-token": "token-2", "generation": 2} + entry = {"metadata_location": location, "version_token": "token-2", "generation": 2} + for backing in ("object", "durable-strong"): + for mismatch in (None, "metadata_location", "version_token", "generation"): + exported = {**entry, mismatch: "stale"} if mismatch else entry + with self.subTest(backing=backing, mismatch=mismatch), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", + side_effect=[table, {"status": "HEALTHY"}, {"table": exported}, pointer, table] + ), mock.patch.object(pyiceberg_smoke, "run_metadata_location_probe"), mock.patch.object( + pyiceberg_smoke, "run_table_ref_probe" + ), mock.patch.object(pyiceberg_smoke, "run_view_probe"), mock.patch.object( + pyiceberg_smoke, "run_table_rename_probe" + ), mock.patch.object(pyiceberg_smoke, "run_load_table_delegation_probe"), mock.patch.object( + pyiceberg_smoke, "run_maintenance_probe" + ) as maintenance: + if mismatch: + with self.assertRaisesRegex(RuntimeError, "does not match"): + pyiceberg_smoke.run_catalog_api_probes(args, mock.Mock(), backing) + else: + _, probes = pyiceberg_smoke.run_catalog_api_probes(args, mock.Mock(), backing) + self.assertEqual(probes["diagnostics"], "pass") + self.assertEqual(probes["export"], "pass") + self.assertEqual(probes["maintenance-worker-dry-run"], "pass") + self.assertEqual(probes["metadata-maintenance"], + "expected-unsupported" if backing == "durable-strong" else "pass") + self.assertEqual(maintenance.call_args.args[2], backing) + + def test_namespace_properties_detect_partial_or_rejected_writes(self) -> None: + args = self.parse_with_args([]) + before = {"properties": {"rustfs.smoke": "true", "owner": "unchanged"}} + after = {"properties": {"rustfs.smoke.updated": "true", "owner": "unchanged"}} + update = {"updated": ["rustfs.smoke.updated"], "removed": ["rustfs.smoke"]} + error = pyiceberg_smoke.RestRequestError("POST", "/properties", 422, "overlapping keys") + for final in (after, before): + with self.subTest(final=final), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[before, update, after, error, final] + ): + if final == after: + pyiceberg_smoke.run_namespace_properties_probe(args, mock.Mock(), "probe") + else: + with self.assertRaisesRegex(RuntimeError, "rejected namespace"): + pyiceberg_smoke.run_namespace_properties_probe(args, mock.Mock(), "probe") + + def test_encoded_namespace_probe_rejects_aliasing_and_cleans_up(self) -> None: + args = self.parse_with_args([]) + namespace = {"namespace": ["probe", "encoded"]} + error = pyiceberg_smoke.RestRequestError("GET", "/namespace", 400, "invalid namespace") + for double_encoded_response in (error, namespace): + with self.subTest(response=double_encoded_response), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{}, namespace, double_encoded_response, namespace, {}] + ) as request: + if isinstance(double_encoded_response, Exception): + pyiceberg_smoke.run_encoded_namespace_probe(args, mock.Mock(), "probe") + else: + with self.assertRaisesRegex(RuntimeError, "unexpectedly succeeded"): + pyiceberg_smoke.run_encoded_namespace_probe(args, mock.Mock(), "probe") + self.assertTrue(request.call_args_list[1].args[3].endswith("probe%1Fencoded")) + self.assertTrue(request.call_args_list[2].args[3].endswith("probe%251Fencoded")) + self.assertEqual(request.call_args.args[2], "DELETE") + + def test_rename_probe_restores_name_even_when_metadata_changes(self) -> None: + args = self.parse_with_args(["--bucket", "lake", "--namespace", "sales", "--table", "orders"]) + before = {"metadata-location": "s3://lake/tables/id/metadata/v2.metadata.json", "metadata": {"location": "s3://lake/tables/id", "table-uuid": "uuid"}} + missing = pyiceberg_smoke.RestRequestError("GET", "/table", 404, "missing") + for changed in (False, True): + after = {**before, "metadata-location": "other"} if changed else before + with self.subTest(changed=changed), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[before, {}, missing, after, {}, before] + ) as request: + if changed: + with self.assertRaisesRegex(RuntimeError, "warehouse identity"): + pyiceberg_smoke.run_table_rename_probe(args, mock.Mock(), "durable-strong") + else: + pyiceberg_smoke.run_table_rename_probe(args, mock.Mock(), "durable-strong") + renames = [call.args[4] for call in request.call_args_list if call.args[2] == "POST"] + self.assertEqual(renames[0]["source"], renames[1]["destination"]) + self.assertEqual(renames[0]["destination"], renames[1]["source"]) + + def test_delegation_requires_exact_token_and_warehouse_scope(self) -> None: + args = self.parse_with_args(["--require-vended-credentials"]) + credential = {"prefix": "s3://lake/tables/id/", "config": {key: "temporary" for key in pyiceberg_smoke.REQUIRED_STORAGE_CREDENTIAL_KEYS}} + metadata_credential = {"prefix": "s3://lake/catalog/current.metadata.json", "config": credential["config"].copy()} + response = {"metadata": {"location": "s3://lake/tables/id"}, "metadata-location": metadata_credential["prefix"], "storage-credentials": [credential, metadata_credential]} + for prefix in ("s3://lake/tables/id/", "s3://lake/tables/other/"): + credential["prefix"] = prefix + with self.subTest(prefix=prefix), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{"storage-credentials": []}] * 4 + [response] + ) as request, mock.patch.object(pyiceberg_smoke, "verify_vended_credential_data_plane_scope") as scope: + if prefix.endswith("/other/"): + with self.assertRaisesRegex(RuntimeError, "do not match"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_not_called() + else: + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_called_once() + self.assertEqual(scope.call_args.args[-1], "s3://lake/tables/id") + self.assertEqual(request.call_args.kwargs["access_delegation"], "remote-signing, vended-credentials") + with mock.patch.object(pyiceberg_smoke, "signed_rest_request", return_value=response): + with self.assertRaisesRegex(RuntimeError, "without exact"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + response["storage-credentials"] = [credential, metadata_credential, credential] + with mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{}] * 4 + [response] + ), mock.patch.object(pyiceberg_smoke, "verify_vended_credential_data_plane_scope") as scope: + with self.assertRaisesRegex(RuntimeError, "exactly the warehouse"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_not_called() + + credential["prefix"] = "s3://lake/tables/id/" + metadata_credential["config"]["s3.session-token"] = "different-session" + response["storage-credentials"] = [credential, metadata_credential] + with mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=[{}] * 4 + [response] + ), mock.patch.object(pyiceberg_smoke, "verify_vended_credential_data_plane_scope") as scope: + with self.assertRaisesRegex(RuntimeError, "same temporary session"): + pyiceberg_smoke.run_load_table_delegation_probe(args, mock.Mock()) + scope.assert_not_called() + + def test_skipped_probes_are_not_recorded_as_full_rest_evidence(self) -> None: + args = self.parse_with_args(["--skip-catalog-api-probes"]) + result = pyiceberg_smoke.SmokeResult("s3://lake/tables/id/metadata/v2.metadata.json", 2, "not-requested", "s3://lake/tables/id", "object", {"direct-rest": "skipped"}) + kwargs = dict(client_version="0.11.1", rustfs_build="test", git_sha="abc123", catalog_backing="object", run_timestamp_utc="2026-09-09T00:00:00Z", operator="test", command="smoke") + record = pyiceberg_smoke.pyiceberg_live_evidence_record(args, result, **kwargs) + self.assertEqual(record["scenario"], "create-append-reload-scan-core") + self.assertEqual(record["catalog_probes"], {"direct-rest": "skipped"}) + kwargs["catalog_backing"] = "durable-strong" + with self.assertRaisesRegex(ValueError, "observed server"): + pyiceberg_smoke.pyiceberg_live_evidence_record(args, result, **kwargs) def test_live_evidence_command_redacts_cli_secrets(self) -> None: command = pyiceberg_smoke.redacted_command( @@ -1129,5 +1407,206 @@ class PyIcebergSmokeConfigTest(unittest.TestCase): self.assertIsNone(INTERNAL_ROADMAP_LABEL_RE.search(str(pyiceberg_smoke.failure_coverage.production_failure_matrix()))) +class SmokeCleanupTest(unittest.TestCase): + def setUp(self) -> None: + self.args = SimpleNamespace(bucket="lake", namespace="sales", table="orders", rest_path="/iceberg") + self.original = { + "metadata-location": "s3://lake/original/metadata/v2.metadata.json", + "metadata": {"table-uuid": "original-id", "location": "s3://lake/original", + "current-schema-id": 0, "schemas": [{"schema-id": 0, "type": "struct", "fields": []}], + "current-snapshot-id": 7, "snapshots": [{"snapshot-id": 7}]}, + } + self.pointer = {"metadata-location": self.original["metadata-location"], "version-token": "token", "generation": 4} + self.replacement = {"metadata": {"table-uuid": "fresh-id", "location": "s3://lake/fresh", "snapshots": []}} + self.entries = {"orders": copy.deepcopy(self.original)} + self.namespace_present = True + self.calls = [] + self.backing = "object" + self.after_cleanup = None + self.after_cleanup_pointer = None + + def request(self, args: object, deps: object, method: str, path: str, body: object = None) -> dict[str, object]: + self.calls.append((method, path, body)) + if path.endswith("/tables/rename"): + source, destination = body["source"]["name"], body["destination"]["name"] + value = self.entries.pop(source) + if self.backing == "object": + self.entries[source] = None # Retained inactive entry, hidden from load/list. + self.entries[destination] = value + return {} + if path.endswith("/metadata-location"): + return copy.deepcopy(self.pointer) + if method == "POST" and path.endswith("/tables"): + self.entries[body["name"]] = copy.deepcopy(self.replacement) + return copy.deepcopy(self.replacement) + name = path.rsplit("/", 1)[-1] + if method == "GET": + value = self.entries.get(name) + if value is None: + raise pyiceberg_smoke.RestRequestError(method, path, 404, "missing") + return copy.deepcopy(value) + if method == "DELETE": + del self.entries[name] + if self.after_cleanup is not None: + self.entries["orders"] = self.after_cleanup + if self.after_cleanup_pointer is not None: + self.pointer = self.after_cleanup_pointer + return {} + raise AssertionError(f"unexpected request: {method} {path}") + + def rename(self) -> None: + with mock.patch.object(pyiceberg_smoke.uuid, "uuid4", return_value=SimpleNamespace(hex="owned")), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=self.request + ): + pyiceberg_smoke.run_table_rename_probe(self.args, mock.Mock(), self.backing) + + def catalog(self) -> object: + catalog = mock.Mock() + catalog.table_exists.side_effect = lambda identifier: self.entries.get(identifier[1]) is not None + catalog.drop_table.side_effect = lambda identifier: self.entries.pop(identifier[1]) + catalog.namespace_exists.side_effect = lambda namespace: self.namespace_present + + def drop_namespace(namespace: str) -> None: + if self.entries: + raise RuntimeError("NamespaceNotEmptyException") + self.namespace_present = False + + catalog.drop_namespace.side_effect = drop_namespace + return catalog + + def test_object_rename_consumes_only_its_tombstone_before_namespace_cleanup(self) -> None: + self.rename() + creates = [call for call in self.calls if call[0] == "POST" and call[1].endswith("/tables")] + deletes = [call for call in self.calls if call[0] == "DELETE"] + self.assertEqual(creates, [("POST", "/iceberg/v1/lake/namespaces/sales/tables", + {"name": "renamed-owned", "schema": self.original["metadata"]["schemas"][0]})]) + self.assertEqual(deletes, [("DELETE", "/iceberg/v1/lake/namespaces/sales/tables/renamed-owned", None)]) + self.assertEqual(self.entries, {"orders": self.original}) + self.assertEqual(self.pointer, {"metadata-location": self.original["metadata-location"], + "version-token": "token", "generation": 4}) + pyiceberg_smoke.cleanup_catalog(self.catalog(), ("sales", "orders")) + self.assertFalse(self.namespace_present) + self.assertEqual(self.entries, {}) + + def test_strong_rename_does_not_create_or_drop_a_cleanup_table(self) -> None: + self.backing = "durable-strong" + self.rename() + self.assertFalse(any(method == "DELETE" or (method == "POST" and path.endswith("/tables")) + for method, path, _ in self.calls)) + pyiceberg_smoke.cleanup_catalog(self.catalog(), ("sales", "orders")) + self.assertFalse(self.namespace_present) + + def test_rename_does_not_sweep_unrelated_inactive_entries(self) -> None: + self.entries["unrelated"] = None + self.rename() + self.assertIn("unrelated", self.entries) + with self.assertRaisesRegex(RuntimeError, "NamespaceNotEmptyException"): + pyiceberg_smoke.cleanup_catalog(self.catalog(), ("sales", "orders")) + self.assertTrue(self.namespace_present) + + def test_rename_rejects_unknown_backing_before_mutation(self) -> None: + self.backing = "unknown" + with self.assertRaisesRegex(RuntimeError, "known backing"): + self.rename() + self.assertEqual(self.calls, []) + + def test_replacement_identity_must_be_new_and_empty(self) -> None: + cases = [("table-uuid", "original-id"), ("table-uuid", ""), ("location", "s3://lake/original"), + ("location", None), ("snapshots", [{"snapshot-id": 7}]), ("current-snapshot-id", 7)] + for key, value in cases: + with self.subTest(key=key, value=value): + self.setUp() + self.replacement["metadata"][key] = value + with self.assertRaisesRegex(RuntimeError, "isolated empty table"): + self.rename() + self.assertFalse(any(method == "DELETE" for method, _, _ in self.calls)) + self.assertEqual(self.entries["orders"], self.original) + + def test_missing_schema_or_commit_state_fails_before_create(self) -> None: + for missing in ("schemas", "metadata-location", "version-token", "generation"): + with self.subTest(missing=missing): + self.setUp() + if missing == "schemas": + del self.entries["orders"]["metadata"][missing] + else: + del self.pointer[missing] + with self.assertRaisesRegex(RuntimeError, "rename cleanup requires"): + self.rename() + self.assertFalse(any(method == "POST" and path.endswith("/tables") for method, path, _ in self.calls)) + + def test_cleanup_detects_original_metadata_or_commit_change(self) -> None: + for changed in ("metadata", "metadata-location", "version-token", "generation"): + with self.subTest(changed=changed): + self.setUp() + if changed == "metadata": + self.after_cleanup = copy.deepcopy(self.original) + self.after_cleanup["metadata"]["table-uuid"] = "changed" + else: + self.after_cleanup_pointer = {**self.pointer, changed: "changed"} + with self.assertRaisesRegex(RuntimeError, "changed the original"): + self.rename() + + def test_cleanup_rejects_namespace_drop_failure_and_does_not_write_evidence(self) -> None: + errors = [RuntimeError("NamespaceNotEmptyException"), PermissionError("403"), + RuntimeError("500: object not found during namespace write"), TimeoutError("timeout")] + for error in errors: + with self.subTest(error=error): + catalog = mock.Mock() + catalog.table_exists.side_effect = [True, False] + catalog.namespace_exists.return_value = True + catalog.drop_namespace.side_effect = error + with mock.patch.object(pyiceberg_smoke, "parse_args", return_value=object()), mock.patch.object( + pyiceberg_smoke, "printed_metadata", return_value=False + ), mock.patch.object(pyiceberg_smoke, "load_runtime_deps"), mock.patch.object( + pyiceberg_smoke, "run_smoke", + side_effect=lambda *args: pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + ), mock.patch.object(pyiceberg_smoke, "write_live_evidence") as write, redirect_stderr(StringIO()): + self.assertEqual(pyiceberg_smoke.main(), 1) + write.assert_not_called() + + def test_cleanup_rejects_false_successful_table_or_namespace_drop(self) -> None: + for resource in ("table", "namespace"): + with self.subTest(resource=resource): + catalog = mock.Mock() + catalog.table_exists.side_effect = [True, resource == "table"] + catalog.namespace_exists.return_value = True + with self.assertRaisesRegex(RuntimeError, resource + " remains"): + pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + + def test_cleanup_accepts_observed_already_absent_resources(self) -> None: + catalog = mock.Mock() + catalog.table_exists.return_value = False + catalog.namespace_exists.return_value = False + pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + catalog.drop_table.assert_not_called() + catalog.drop_namespace.assert_not_called() + + def test_cleanup_read_failures_are_not_absence(self) -> None: + for check in ("table_exists", "namespace_exists"): + with self.subTest(check=check): + catalog = mock.Mock() + catalog.table_exists.return_value = False + getattr(catalog, check).side_effect = TimeoutError("read failed") + with self.assertRaises(TimeoutError): + pyiceberg_smoke.cleanup_catalog(catalog, ("sales", "orders")) + catalog.drop_namespace.assert_not_called() + + def test_rename_transport_and_drop_failure_propagate(self) -> None: + for method, suffix in (("POST", "/tables"), ("DELETE", "/renamed-owned")): + with self.subTest(method=method): + self.setUp() + + def fail(args: object, deps: object, verb: str, path: str, body: object = None) -> dict[str, object]: + if verb == method and path.endswith(suffix): + raise TimeoutError("cleanup interrupted") + return self.request(args, deps, verb, path, body) + + with mock.patch.object(pyiceberg_smoke.uuid, "uuid4", return_value=SimpleNamespace(hex="owned")), mock.patch.object( + pyiceberg_smoke, "signed_rest_request", side_effect=fail + ), self.assertRaises(TimeoutError): + pyiceberg_smoke.run_table_rename_probe(self.args, mock.Mock(), self.backing) + self.assertEqual(self.entries["orders"], self.original) + + if __name__ == "__main__": unittest.main() diff --git a/scripts/test/delete_recovery_docker.py b/scripts/test/delete_recovery_docker.py index c9bb96b32..07f581bc7 100644 --- a/scripts/test/delete_recovery_docker.py +++ b/scripts/test/delete_recovery_docker.py @@ -541,6 +541,8 @@ def mixed_versions(): for initiator in (0, 3): (code, body) = cluster.clients[initiator].request('DELETE', bucket + '/' + key) assert code >= 400, f'mixed initiator{initiator} must fail closed: {code} {body!r}' + if initiator == 0: + assert code == 503 and b'ServiceUnavailable' in body, 'modern mixed-version rejection must remain retryable' after = residue_inventory(cluster, bucket, key) (ROOT / f'mixed-initiator-{initiator}-inventory.json').write_text(json.dumps({'before': original, 'after': after}, indent=2)) assert after == original, 'mixed rejection must preserve exact metadata/shards after rollback' @@ -893,6 +895,50 @@ def deep_key_crash(large_stack=False): finally: cluster.remove() +def automatic_clean_restart(): + cluster = Cluster('automatic-clean-restart', drives=4) + try: + cluster.create() + client = cluster.clients[0] + bucket = 'automatic-clean-restart' + key = 'payload' + client.put_bucket(bucket) + client.put(bucket, key, b'automatic-recovery' * 150000) + nonce = arm_barrier(cluster, 0, bucket, key, 'DECISION') + thread, request = async_delete(client, bucket, key) + reached_barrier(cluster, 0, nonce) + deadline = time.monotonic() + 40 + while len(list(cluster.barriers[0].glob('reached.' + nonce + '.*.json'))) < 4: + if time.monotonic() > deadline: + raise TimeoutError('all four receipts must persist before injecting response errors') + time.sleep(0.02) + # Prevent the live retry worker from completing before the clean stop. + # The failure is in task-owned payload directories, never in the proof. + deny = "from pathlib import Path; import json,os; roots=list(Path('/data').glob('d*/%s/%s')); dirs=[p for root in roots for p in root.iterdir() if p.is_dir() and (p/'part.1').is_file()]; assert len(dirs)==4; [os.chmod(p,0) for p in dirs]; print(json.dumps({'protected_payload_dirs':len(dirs)}))" % (bucket, key) + cluster.inspect_data(deny) + (cluster.barriers[0] / ('release.' + nonce)).write_bytes(b'io_error') + thread.join(40) + assert not thread.is_alive(), 'uncertain publication must return to the client' + assert 'response' in request and request['response'][0] >= 400, request + clear_barriers(cluster) + before = residue_inventory(cluster, bucket, key) + assert all(any(path.endswith('delete-quorum-committed') for path in drive['files']) for node in before for drive in node) + docker('stop', '--time', '60', cluster.names[0]) + state = json.loads(docker('inspect', '--format', '{{json .State}}', cluster.names[0]).stdout) + assert state['ExitCode'] == 0 and not state['OOMKilled'], state + repair = "from pathlib import Path; import json,os; roots=list(Path('/data').glob('d*/%s/%s')); dirs=[p for root in roots for p in root.iterdir() if p.is_dir() and (p/'part.1').is_file()]; assert len(dirs)==4; [os.chmod(p,0o755) for p in dirs]; print(json.dumps({'restored_payload_dirs':len(dirs)}))" % (bucket, key) + cluster.inspect_data(repair) + cluster.restart() + # Only readiness and read-only inventory are polled after restart. + # No DELETE or administrator heal may manufacture a new retry hint. + wait_inventory(cluster, bucket, key, lambda inventory: all(not drive['exists'] for node in inventory for drive in node), timeout=90) + client.request('GET', bucket + '/' + key, expected=404) + assert client.list(bucket) == [] + client.delete_bucket(bucket) + return {'case': 'uncertain-publication-clean-restart-automatic-recovery', 'status': 'passed', 'clean_exit_code': state['ExitCode']} + finally: + cluster.remove() + def receipt_response_loss(): cluster = Cluster('receipt-response-loss', nodes=4, drives=2) try: @@ -1227,7 +1273,7 @@ def main(): existing = docker(resource, 'ls', '-q', '--filter', 'label=rustfs.delete-recovery-test=' + PREFIX).stdout.strip() if existing: parser.error('namespace already owns Docker resources; choose a new namespace') - cases = {'unknown-fresh': unknown_predecessor_fresh_delete, 'batch1000': large_batch_delete, 'versions3000': large_version_history, 'admin-smoke': admin_smoke, 'legacy-alias': legacy_transaction_data_dir_alias, 'missing-offline': ordinary_missing_with_offline_peer, 'inverse': retained_decision_newer_undecided, 'deep': deep_key_crash, 'deep-large-stack': lambda : deep_key_crash(True), 'admin-prefix': admin_prefix_scoped, 'response-loss': receipt_response_loss, 'admin-object': lambda : admin_legacy_dry_run(True), 'smoke': smoke, 'legacy1': lambda : legacy_retry(1), 'legacy4': lambda : legacy_retry(4), 'restore': rollback_crash, 'retention': protected_version, 'version-applied': lambda : versioned_crash('APPLIED'), 'version-decision': lambda : versioned_crash('DECISION'), 'returning': returning_drive_new_generation, 'untouched': untouched_returning_peer, 'missing-version': missing_version_returning_peer, 'mixed': mixed_versions, 'admin': admin_legacy_dry_run, 'prefix': uuid_prefix_crash, 'corruption': corruption_repair, 'permission': permission_fault, 'error-prepared': lambda : injected_error('PREPARED'), 'error-applied': lambda : injected_error('APPLIED'), 'error-decision': lambda : injected_error('DECISION')} + cases = {'automatic-clean-restart': automatic_clean_restart, 'unknown-fresh': unknown_predecessor_fresh_delete, 'batch1000': large_batch_delete, 'versions3000': large_version_history, 'admin-smoke': admin_smoke, 'legacy-alias': legacy_transaction_data_dir_alias, 'missing-offline': ordinary_missing_with_offline_peer, 'inverse': retained_decision_newer_undecided, 'deep': deep_key_crash, 'deep-large-stack': lambda : deep_key_crash(True), 'admin-prefix': admin_prefix_scoped, 'response-loss': receipt_response_loss, 'admin-object': lambda : admin_legacy_dry_run(True), 'smoke': smoke, 'legacy1': lambda : legacy_retry(1), 'legacy4': lambda : legacy_retry(4), 'restore': rollback_crash, 'retention': protected_version, 'version-applied': lambda : versioned_crash('APPLIED'), 'version-decision': lambda : versioned_crash('DECISION'), 'returning': returning_drive_new_generation, 'untouched': untouched_returning_peer, 'missing-version': missing_version_returning_peer, 'mixed': mixed_versions, 'admin': admin_legacy_dry_run, 'prefix': uuid_prefix_crash, 'corruption': corruption_repair, 'permission': permission_fault, 'error-prepared': lambda : injected_error('PREPARED'), 'error-applied': lambda : injected_error('APPLIED'), 'error-decision': lambda : injected_error('DECISION')} cases.update({'multipart-sidecars': completed_multipart_sidecars, 'multipart-integrity': lambda: completed_multipart_sidecars(True), 'multipart-sidecars-applied': lambda: completed_multipart_sidecars(False, True), 'multipart-integrity-applied': lambda: completed_multipart_sidecars(True, True)}) for stage in ('PREACTIVATION', 'PREPARED', 'APPLIED', 'DECISION'): for nodes in (1, 4): diff --git a/scripts/test_helm_templates.sh b/scripts/test_helm_templates.sh index f273430b9..b8d1eeb44 100755 --- a/scripts/test_helm_templates.sh +++ b/scripts/test_helm_templates.sh @@ -180,6 +180,7 @@ scanner_config_output=$(render_chart \ --set config.rustfs.scanner.cache_save_timeout_secs=30 \ --set config.rustfs.scanner.max_concurrent_set_scans=2 \ --set config.rustfs.scanner.max_concurrent_disk_scans=1 \ + --set config.rustfs.scanner.walkers_per_disk=2 \ --set config.rustfs.scanner.yield_every_n_objects=128 \ --set config.rustfs.scanner.alert_excess_versions=100 \ --set config.rustfs.scanner.alert_excess_version_size=1099511627776 \ @@ -197,6 +198,7 @@ grep -q 'RUSTFS_SCANNER_IDLE_MODE: "false"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS: "30"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS: "2"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS: "1"' <<<"$scanner_config_output" +grep -q 'RUSTFS_SCANNER_WALKERS_PER_DISK: "2"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS: "128"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS: "100"' <<<"$scanner_config_output" grep -q 'RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE: "1099511627776"' <<<"$scanner_config_output" @@ -501,6 +503,102 @@ if grep -q "name: data$" <<<"$generic_eight_by_two"; then exit 1 fi +# Issue #8665: an empty storageclass.dataStorageSize means the drives come from +# extraVolumes/extraVolumeMounts, so no data PVC templates, no data mounts and +# no empty `storage:` request may be rendered. The logs PVC is unaffected. +jbod_extra_volumes='[{"name":"disk0","hostPath":{"path":"/mnt/disk0"}},{"name":"disk1","hostPath":{"path":"/mnt/disk1"}}]' +jbod_extra_mounts='[{"name":"disk0","mountPath":"/data/rustfs0"},{"name":"disk1","mountPath":"/data/rustfs1"}]' +jbod_statefulset=$(render_distributed_statefulset \ + --set replicaCount=3 \ + --set drivesPerNode=2 \ + --set storageclass.dataStorageSize= \ + --set-json "extraVolumes=${jbod_extra_volumes}" \ + --set-json "extraVolumeMounts=${jbod_extra_mounts}") +if grep -Eq 'name: data(-rustfs-[0-9]+)?$' <<<"$jbod_statefulset"; then + echo "Empty dataStorageSize must not render data PVC templates or data volume mounts" >&2 + exit 1 +fi +if grep -Eq 'storage:[[:space:]]*$' <<<"$jbod_statefulset"; then + echo "Empty dataStorageSize must not render an empty storage request" >&2 + exit 1 +fi +if grep -Eq 'DRIVES_PER_NODE|mkdir -p /data' <<<"$jbod_statefulset"; then + echo "Empty dataStorageSize must not initialize data directories in the init container" >&2 + exit 1 +fi +if ! grep -Fq 'mountPath: /data/rustfs1' <<<"$jbod_statefulset" || ! grep -Fq 'path: /mnt/disk1' <<<"$jbod_statefulset"; then + echo "Empty dataStorageSize must keep user-supplied extraVolumes and extraVolumeMounts" >&2 + exit 1 +fi +if ! grep -Eq '^[[:space:]]+name: logs$' <<<"$jbod_statefulset"; then + echo "Empty dataStorageSize must not affect the logs PVC" >&2 + exit 1 +fi + +# With neither data nor logs PVCs there is nothing to claim or initialize. +jbod_no_pvc_statefulset=$(render_distributed_statefulset \ + --set replicaCount=3 \ + --set drivesPerNode=2 \ + --set storageclass.dataStorageSize= \ + --set config.rustfs.obs_log_directory=) +for absent_key in volumeClaimTemplates initContainers; do + if grep -Fq "${absent_key}:" <<<"$jbod_no_pvc_statefulset"; then + echo "StatefulSet without data or logs PVCs must not render ${absent_key}" >&2 + exit 1 + fi +done + +# A pool may disable its data PVCs independently of the other pools. +pools_mixed=$(helm template rustfs "$CHART_DIR" \ + --namespace rustfs \ + --set secret.rustfs.access_key=test-access-key \ + --set secret.rustfs.secret_key=test-secret-key \ + --set pools.enabled=true \ + --set-json 'pools.list=[{},{"storageclass":{"dataStorageSize":""}}]') +if grep -Eq 'storage:[[:space:]]*$' <<<"$pools_mixed"; then + echo "A pool with empty dataStorageSize must not render an empty storage request" >&2 + exit 1 +fi +statefulset_named() { + awk -v metadata_name="$1" ' + function flush() { + if (doc ~ /kind: StatefulSet/ && index(doc, "\n name: " metadata_name "\n") > 0) printf "%s", doc + doc = "" + } + /^---$/ { flush(); next } + { doc = doc "\n" $0 } + END { flush() } + ' +} +pool0_statefulset=$(statefulset_named rustfs <<<"$pools_mixed") +pool1_statefulset=$(statefulset_named rustfs-pool1 <<<"$pools_mixed") +if ! grep -Eq 'name: data-rustfs-0$' <<<"$pool0_statefulset"; then + echo "A pool that keeps dataStorageSize must still render its data PVC templates" >&2 + exit 1 +fi +if [[ -z "$pool1_statefulset" ]] || grep -Eq 'name: data(-rustfs-[0-9]+)?$' <<<"$pool1_statefulset"; then + echo "A pool with empty dataStorageSize must not render data PVC templates or mounts" >&2 + exit 1 +fi + +# Standalone mode: no data PVC and no data volume unless an existing claim is named. +standalone_jbod=$(render_chart --set storageclass.dataStorageSize=) +if grep -Eq 'rustfs-data$|storage:[[:space:]]*$' <<<"$standalone_jbod"; then + echo "Standalone mode with empty dataStorageSize must not render a data PVC" >&2 + exit 1 +fi +if grep -Eq '^[[:space:]]+- name: data$|mountPath: /data$' <<<"$standalone_jbod"; then + echo "Standalone mode with empty dataStorageSize must not mount a data volume" >&2 + exit 1 +fi +standalone_jbod_claim=$(render_chart \ + --set storageclass.dataStorageSize= \ + --set mode.standalone.existingClaim.dataClaim=existing-data) +if ! grep -Fq 'claimName: existing-data' <<<"$standalone_jbod_claim"; then + echo "Standalone existingClaim.dataClaim must still mount when dataStorageSize is empty" >&2 + exit 1 +fi + # Issue #5416's exact three-node, one-drive topology must render all three # ordinal hosts while each StatefulSet pod receives its own explicit anchor. three_by_one_configmap=$(render_distributed_configmap --set replicaCount=3 --set drivesPerNode=1) diff --git a/scripts/test_release_build_workflow.py b/scripts/test_release_build_workflow.py new file mode 100644 index 000000000..cce983e90 --- /dev/null +++ b/scripts/test_release_build_workflow.py @@ -0,0 +1,137 @@ +#!/usr/bin/env python3 +"""Exercise server-only release commands and strict acceptance archive readers.""" + +from __future__ import annotations + +import io +import json +import os +import re +import shlex +import subprocess +import sys +import tempfile +import unittest +import zipfile +from pathlib import Path + +from test_security_workflow import named_steps, shell_body, yaml_block + +ROOT = Path(__file__).resolve().parents[1] +SOURCE = (ROOT / '.github/workflows/build.yml').read_text() +STEPS = named_steps(yaml_block(SOURCE.splitlines(), 'build-rustfs', 2)) +MATRIX = json.loads(re.search(r"all='(\{.*?\})'", SOURCE, re.S)[1])['include'] +CATALOG = ('rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256') +EXTRACTORS = ( + ('connect-diagnostic-schedule-acceptance.yml', 'diagnostic-schedule', 'Extract the verified official binary'), + ('connect-health-service-acceptance.yml', 'health-service', 'Extract only the verified official service binary'), + ('connect-profile-memory-acceptance.yml', 'profile-memory', 'Extract only the verified official service binary'), + ('connect-top-disk-acceptance.yml', 'top-disk', 'Extract only the verified official service binary'), +) + + +def render(script, target, package='package.zip'): + context = {f'matrix.{key}': str(value).lower() if isinstance(value, bool) else str(value) for key, value in target.items()} + context['steps.package.outputs.package_file'] = package + return re.sub(r'\$\{\{\s*(.*?)\s*\}\}', lambda match: context[match[1]], script) + + +def archive(path, members, binary): + with zipfile.ZipFile(path, 'w') as package: + for name in members: + entry = zipfile.ZipInfo(name) + entry.external_attr = (0o100755 if name == 'rustfs' else 0o100644) << 16 + package.writestr(entry, binary if name == 'rustfs' else b'fixture') + + +class ReleaseBuildTests(unittest.TestCase): + def test_every_matrix_target_builds_only_server(self): + mock_cargo = r''' +cargo() { + printf '%q ' "$@" >> cargo-call + printf '\n' >> cargo-call + local file="target/$BUILD_TARGET/release/rustfs" + if [[ "$BUILD_TARGET" == *-windows-msvc ]]; then file+='.exe'; fi + mkdir -p "$(dirname "$file")" + cat > "$file" <<'BIN' +#!/usr/bin/env bash +printf 'build target : %s\nbuild os : %s\nrust channel : stable-%s\n' "$BUILD_TARGET" "$BUILD_OS" "$BUILD_TARGET" +BIN + chmod +x "$file" +} +''' + for target in MATRIX: + with self.subTest(target=target['target_id']), tempfile.TemporaryDirectory() as directory: + path = Path(directory) + (path / 'rustfs').mkdir() + (path / 'rustfs/build.rs').touch() + (path / 'scripts/release').mkdir(parents=True) + (path / 'scripts/release/check_glibc.py').write_bytes((ROOT / 'scripts/release/check_glibc.py').read_bytes()) + (path / 'tools').mkdir() + readelf = path / 'tools/readelf' + readelf.write_text("#!/bin/sh\nprintf '%s\\n' 'Version needs section:' 'Name: GLIBC_2.30'\n") + readelf.chmod(0o755) + script = mock_cargo + render(shell_body(STEPS['Build RustFS']), target) + environment = {**os.environ, 'BUILD_TARGET': target['target'], + 'BUILD_OS': target['platform'] + '-' + target['target'].split('-')[0], + 'PATH': str(path / 'tools') + os.pathsep + os.environ['PATH']} + result = subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', script], cwd=path, + env=environment, capture_output=True, text=True) + self.assertEqual(result.returncode, 0, result.stderr) + calls = (path / 'cargo-call').read_text().splitlines() + self.assertEqual(len(calls), 1) + args = shlex.split(calls[0]) + self.assertEqual(args[0], 'zigbuild' if target['cross'] else 'build') + self.assertEqual(args[args.index('--bin') + 1], 'rustfs') + self.assertNotIn('rustfs-cli', args) + self.assertIn('--locked', args) + expected_target = target['target'] + ('.' + target['glibc'] if target['glibc'] else '') + self.assertEqual(args[args.index('--target') + 1], expected_target) + if target['features']: + self.assertEqual(args[args.index('--features') + 1], target['features']) + if target['glibc']: + self.assertEqual(args[args.index('--config') + 1], 'profile.release.strip="none"') + + def test_packaged_linux_server_and_catalog_members(self): + binary = b'#!/bin/sh\necho rustfs-fixture\n' + for libc in ('gnu', 'musl'): + target = next(item for item in MATRIX if item['target_id'] == f'linux-x86_64-{libc}') + expected = ['rustfs', *CATALOG] if libc == 'gnu' else ['rustfs'] + for members, valid in ((expected, True), (expected + ['rustfs-cli'], False)): + with self.subTest(libc=libc, members=members), tempfile.TemporaryDirectory() as directory: + path = Path(directory) + archive(path / 'package.zip', members, binary) + script = render(shell_body(STEPS['Verify packaged Linux binary']), target) + result = subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', script], cwd=path, + capture_output=True, text=True) + self.assertEqual(result.returncode == 0, valid, result.stderr) + + def test_actual_acceptance_extractors_keep_strict_members(self): + source_sha = '0123456789abcdef' * 2 + '01234567' + package_name = f'rustfs-linux-x86_64-gnu-dev-{source_sha[:7]}.zip' + header = bytearray(20) + header[:7] = b'\x7fELF\x02\x01\x01' + header[18:20] = b'\x3e\x00' + valid_members = ['rustfs', *CATALOG] + cases = ((valid_members, True), (['rustfs'], False), (valid_members + ['rustfs-cli'], False), + (valid_members + ['../extra'], False)) + for workflow, job_name, step in EXTRACTORS: + source = (ROOT / '.github/workflows' / workflow).read_text() + job = yaml_block(source.splitlines(), job_name, 2) + body = shell_body(named_steps(job)[step]) + python = body.split("python3 - <<'PY'\n", 1)[1].split('\nPY', 1)[0] + for members, valid in cases: + with self.subTest(workflow=workflow, members=members), tempfile.TemporaryDirectory() as directory: + path = Path(directory) + archive(path / package_name, members, bytes(header)) + with zipfile.ZipFile(path / 'official-artifact.zip', 'w') as outer: + outer.write(path / package_name, package_name) + result = subprocess.run([sys.executable, '-c', python], cwd=path, + env={**os.environ, 'SOURCE_SHA': source_sha}, capture_output=True, text=True) + self.assertEqual(result.returncode == 0, valid, result.stderr) + if valid: + self.assertEqual((path / 'binary/rustfs').read_bytes(), bytes(header)) + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/test_release_glibc.py b/scripts/test_release_glibc.py new file mode 100644 index 000000000..a0359ab11 --- /dev/null +++ b/scripts/test_release_glibc.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +"""Exercise release ABI checks against loader requirements and malformed inputs.""" + +import importlib.util +import subprocess +import unittest +from pathlib import Path +from unittest.mock import patch + +SPEC = importlib.util.spec_from_file_location("check_glibc", Path(__file__).parent / "release/check_glibc.py") +glibc = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(glibc) + + +class GlibcTests(unittest.TestCase): + def inspect(self, needs: str, baseline: str = "2.31") -> str: + output = "Version definition section: Name: GLIBC_9.99\nVersion needs section:\n" + needs + with patch.object(glibc.subprocess, "run", return_value=subprocess.CompletedProcess([], 0, output)): + return glibc.check(Path("rustfs"), baseline) + + def test_uses_loader_needs_and_numeric_order(self): + self.assertEqual(self.inspect("Name: GLIBC_2.9\nName: GLIBC_2.30\nName: GLIBC_2.2.5"), "GLIBC_2.30") + self.assertEqual(self.inspect("Name: GLIBC_2.31.0"), "GLIBC_2.31.0") + + def test_rejects_newer_requirement(self): + with self.assertRaisesRegex(ValueError, "requires GLIBC_2.34"): + self.inspect("Name: GLIBC_2.34") + + def test_rejects_non_numeric_abi_tags(self): + for tag in ("GLIBC_PRIVATE", "GLIBC_ABI_DT_RELR"): + with self.subTest(tag=tag), self.assertRaises(ValueError): + self.inspect(f"Name: {tag}") + + def test_rejects_missing_needs_and_invalid_baselines(self): + with self.assertRaisesRegex(ValueError, "no GLIBC version needs"): + self.inspect("Name: GCC_3.0") + for baseline in ("", "2", "2.31x", "-2.31"): + with self.subTest(baseline=baseline), self.assertRaises(ValueError): + self.inspect("Name: GLIBC_2.2.5", baseline) + + def test_readelf_failure_is_not_a_pass(self): + with patch.object(glibc.subprocess, "run", side_effect=subprocess.CalledProcessError(1, "readelf")): + with self.assertRaises(subprocess.CalledProcessError): + glibc.check(Path("invalid"), "2.31") + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_upgrade_workflow.py b/scripts/test_upgrade_workflow.py index 46a89c121..baa931895 100644 --- a/scripts/test_upgrade_workflow.py +++ b/scripts/test_upgrade_workflow.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Exercise the consolidated upgrade lane without building RustFS.""" +"""Exercise both pinned upgrade sources without building RustFS.""" from __future__ import annotations @@ -27,12 +27,17 @@ class UpgradeWorkflowTests(unittest.TestCase): self.step = self.steps["Run upgrade compatibility tests"] # Explicit bash enables pipefail in Actions, so tee cannot hide a failure. self.assertIn(" shell: bash", self.step) - self.assertNotIn(" strategy:", self.job) + self.assertIn(" fail-fast: false", self.job) + self.assertEqual(re.findall(r"- version: ([^\s]+)", "\n".join(self.job)), ["1.0.0-rc.5", "1.0.1"]) rust = (ROOT / "crates/e2e_test/src/upgrade_compatibility_test.rs").read_text() self.expected = set(re.findall(r'#\[ignore[^\n]*\]\s*async fn (\w+)\(', rust)) self.assertTrue(self.expected) - def run_cases(self, failed_test="", fail_tee=False): + def run_cases(self, version, failed_test="", fail_tee=False): + excluded = ("stable_upgrade_deletes_foreign_reservations_without_losing_live_objects" + if version == "1.0.0-rc.5" else "rc5_rollback_requires_restoring_odm_configuration") + expected = self.expected - {excluded} + self.assertEqual(len(expected), 7) with tempfile.TemporaryDirectory() as directory: root = Path(directory) commands = root / "commands.jsonl" @@ -59,12 +64,12 @@ sys.exit(17 if test == os.environ["FAILED_TEST"] else 0) tee.chmod(0o755) env = dict(os.environ, PATH=f"{root}{os.pathsep}{os.environ['PATH']}", RUNNER_TEMP=str(root), COMMANDS=str(commands), FAILED_TEST=failed_test, - NO_PROXY="127.0.0.1,localhost,::1") + NO_PROXY="127.0.0.1,localhost,::1", UPGRADE_SOURCE_VERSION=version) result = subprocess.run(["bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "-c", shell_body(self.step)], cwd=root, env=env, capture_output=True, text=True) calls = [json.loads(line) for line in commands.read_text().splitlines()] - self.assertEqual({call["test"] for call in calls}, self.expected) - self.assertEqual(len(calls), len(self.expected)) + self.assertEqual({call["test"] for call in calls}, expected) + self.assertEqual(len(calls), len(expected)) for field in ("logs", "evidence"): self.assertEqual(len({call[field] for call in calls}), len(calls)) self.assertTrue(all(Path(call[field]).is_dir() for call in calls)) @@ -74,18 +79,23 @@ sys.exit(17 if test == os.environ["FAILED_TEST"] else 0) return result def test_all_ignored_upgrade_cases_run_once_with_separate_evidence(self): - result = self.run_cases() - self.assertEqual(result.returncode, 0, result.stderr) + for version in ("1.0.0-rc.5", "1.0.1"): + with self.subTest(version=version): + result = self.run_cases(version) + self.assertEqual(result.returncode, 0, result.stderr) def test_early_or_late_failure_does_not_hide_remaining_cases(self): - for name in ("direct_upgrade_from_rc2_preserves_object_contracts", "rc5_baseline_replicates_multipart_layouts"): - with self.subTest(name=name): - result = self.run_cases(failed_test=name) - self.assertNotEqual(result.returncode, 0, result.stdout) - self.assertIn(f"::error::Upgrade compatibility failed: {name}", result.stdout) + for version, last in (("1.0.0-rc.5", "rc5_rollback_requires_restoring_odm_configuration"), + ("1.0.1", "stable_upgrade_deletes_foreign_reservations_without_losing_live_objects")): + for name in ("direct_upgrade_from_previous_release_preserves_object_contracts", last): + with self.subTest(version=version, name=name): + result = self.run_cases(version, failed_test=name) + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn(f"::error::Upgrade compatibility failed: {name}", result.stdout) def test_log_write_failure_fails_the_job(self): - self.assertNotEqual(self.run_cases(fail_tee=True).returncode, 0) + for version in ("1.0.0-rc.5", "1.0.1"): + self.assertNotEqual(self.run_cases(version, fail_tee=True).returncode, 0) def test_reports_are_uploaded_even_after_failure(self): for name, directory in (("Upload scanner/heal G09 evidence", "rustfs-upgrade-g09-evidence"), @@ -93,6 +103,7 @@ sys.exit(17 if test == os.environ["FAILED_TEST"] else 0) step = self.steps[name] self.assertIn(" if: always()", step) self.assertIn(" path: ${{ runner.temp }}/" + directory, step) + self.assertTrue(any("name:" in line and "matrix.source.version" in line for line in step)) if __name__ == "__main__":